May 12, 2013

FFT audio frequency spectrum analysis with Processing in android devices/emulator

A friend of mine asked me for help in accesing the audio recording function of android's API with Processing.org, to develop an audio analysis app.

Keep reading!

I never programmed for android before, so I had to go through all the installation steps of the API and configuration for processing. There are already some nice tutorials about this, so you can follow these to set up your android-processing developing environment. Here are some useful links:

Jut to mention it, my OS is Ubuntu 12.04 64 bit, and I already had installed Oracle's propietary Java JDK (packages named oracle-java7-installer and oracle-java7-set-default) to ensure maximum compatibility.

Once you download and uncompress Android's SDK, when you select processing's android mode (clicking on the top-right test that normally says "standard") it will ask you for that SDK location. After selecting the folder containing the SDK, processing automatically creates a virtual machine in the Android Virtual Device Manager. You can access the manager through processing's menu entry "Android->Android AVD Manager" that is available only in android mode.


This will start the ADM, where you can choose the virtual machine and press start to run the emulator.


One important step I had to do in order to be able to run the sketches using the Android emulator (istead of always running it on the cell phone) is to add a line to processing's configuration file that, in Ubuntu, is located in : /home/YOURUSER/.processing/preferences.txt (see this topic) I had to edit this file an add the line: android.emulator.port = 5554 (the port number appears in the window name of the virtual machine once you start it).

When you want to use some of your Adroid device's hardware you have to enable the corresponding permision for your sketch in processing's "Android->Stetch Permissions" menu option. In our case we want to use the audio recording hardware, so we enable the "RECORD AUDIO" option.

Two important hints to use the Android API recorder with both the device and the emulator is that the emulator does not provide a right minimum recording buffer number when requested, so when you use the emulator you have to set the buffer size manually, and it has to be a power of two. The other hint is that the only samplerate you can use with the emulator is 8000.
I took advantage of the minimum buffer hint for the sketch to detect when it is running on the emulator and when it is running on the device. I do this in the setup:


minBufferSize = AudioRecord.getMinBufferSize( RECORDER_SAMPLERATE,
                                 RECORDER_CHANNELS,RECORDER_AUDIO_ENCODING);
  if (minBufferSize == AudioRecord.ERROR_BAD_VALUE)  {
    RECORDER_SAMPLERATE = 8000; // forced by the android emulator
    MAX_FREQ = RECORDER_SAMPLERATE/2;
    // buffer size must be power of 2
    bufferSize =  getHigherP2(RECORDER_SAMPLERATE);
  } else bufferSize = minBufferSize;

Another caution is to prevent crashes is to check that the AudioRecorder initialized properly:

audioRecord = new AudioRecord( MediaRecorder.AudioSource.MIC, RECORDER_SAMPLERATE,
                                 RECORDER_CHANNELS,RECORDER_AUDIO_ENCODING, bufferSize);
  if ((audioRecord != null) && (audioRecord.getState() == AudioRecord.STATE_INITIALIZED)) {
    try {
      // this throws an exception with some combinations
      // of RECORDER_SAMPLERATE and bufferSize 
      audioRecord.startRecording(); 
      aRecStarted = true;
    }
    catch (Exception e) {
      aRecStarted = false;
    }
} else aRecStarted = false;
EDIT: to disable the automatic Audio Gain Control use:

MediaRecorder.AudioSource.VOICE_RECOGNITION

instead of:

MediaRecorder.AudioSource.MIC

There are also certain combinations of samplerates and buffer sizes that don't work well and make the app crash in the emulator and or the phone with the message "The application application_name (process processing.test.application_name) has stopped unexpectedly. Please try again". Speaking about crashes, after struggling with the emulator, to avoid problems I really recomend not to start it from the snapshot, but rather boot every time the virtual machine from the begining. And every time you make the application crash: restart the virtual machine again.



When you run your audio recording sketch in the emulator, if you see there is no correspondence between the sound you make and the audio shown in the emulator, you may be getting confused by the huge delay of the signal. In my laptop it takes about 45 seconds to see the response to certain sound in the emulator. After a while (2 or 3 minutes) the emulator catches up and you see the response has less delay. 

Spectrum analysis

The next problem we had is that the sound library minim that comes with processing works only in the standard mode, not in android mode. And we couldn't find any complete example with working code to perform a Fourier Transform in android with processing. Yes, there are some threads about it, but none with the code for processing:

There are also java libraries that implement the forward Fast Fourier Transform (FFT), like jtransforms, jfftpack, and a discussion about them. But finally I decided to adapt the minim analysis code for our purposes, so I could also use the same syntax and recycle some interesting audio analysis and visualization examples for processing.

I put all this together and finally made a working app that shows the main frequency (pitch) of the audio captured by the device's microphone.


   

You can download the code and the apk application from the GitHub repository

EDIT:
March 13th 2015
After some minor corrections I've managed to make this work in my Nexus 4, compiling the sketch with the app: APDE - Android Processing IDE. After installing APDE, just create a new folder called "android_fft_minim" in the sd card's "Sketchbook" folder and copy all the repo's files there. Run the APDE and open and play the android_fft_minim sketch. Have fun!

Related interesting links:

http://som-itsolutions.blogspot.com.es/2012/01/fft-based-simple-spectrum-analyzer.html

http://code.google.com/p/audio-analyzer-for-android/

19 comments:

  1. hei i thanks for posting. i have download your code but i can not open android.fft.minim.pde , how i open your project to my eclipse ? so i can run through emulator

    ReplyDelete
    Replies
    1. Eclipse is one of the things in my TODO list :)
      According to this thread it should be fairly easy: http://www.java-forums.org/eclipse/15071-importing-java-files-into-eclipse.html

      Delete
    2. Also, have a look here: http://processing.org/tutorials/eclipse/

      Delete
  2. Hi,

    The project is made with Processing (www.processing.org), and it may need Processing's core api. Processing also allows to run the project in android's emulator.

    If you want to use Eclipse, you may try to open the all the .pde files in any text editor and copy the code to your Eclipse project.

    Have a look here also: http://processing.org/learning/eclipse/

    ReplyDelete
    Replies
    1. tq for posting this. i really need this. by the way, i dont kow how to transform your code into eclipse. can you post your code using Eclipse?

      Delete
  3. hi david..

    can u help me with my project
    i want build app in android, when user say to something to their device then their device will "lock"
    i have search many article but i still not understand how to built it
    so i hope u will help me

    ReplyDelete
    Replies
    1. Hi Jhony,

      Have you enabled permissions for the phone's hardware you are using?
      In the processing menu: Android -> sketch permissions

      For example if your app has to acess the microphone you have to enable the "record audio" permission.

      Delete
    2. if I root my android, I think I get full access of my android ..
      I do not need to record audio
      I just want to make the application when I say "Lock" my android will "Lock"
      on my android, I keep my voice that says "Lock"
      so I use FFT only to match my voice that I keep in a voice I say again when I want to lock my android.

      You can help to build this application. I am really new to android and FFT
      I have read many articles about FFT and Android but I am still confused
      I hope you will help me to build this application.

      Delete
    3. What are your programming skills? Do you use Processing IDE (www.processing.org)? Have you programmed an android app before?

      There are already such apps as you want. Just google "android app lock voice"

      https://www.google.es/search?q=android+app+lock+voice

      Delete
    4. I am a beginner in programming an android. and I've installed the app, but to no avail.
      I really need your help to build these applications ...

      Delete
    5. Jhony, unfortunately I'm a beginner too (the app in this post is the first I've ever made), and at the moment I have other projects and no time to spend in programming android apps, sorry.

      Good luck!

      Delete
  4. I do not trust the download site since it has converted everything to an .exe. Do you have another place I can use for downloading or send by email?

    ReplyDelete
  5. please help me, how can i draw it on android using window eclipse.
    i dont understand your code to draw it in pde file.
    Sorry for my bad english

    ReplyDelete
    Replies
    1. I'm afraid to learn to work in Eclipse is still in my to-do list. All I can advice is to start with a tutorial like this: http://blog.onthewings.net/2013/04/25/setting-up-a-processing-android-project-in-eclipse/

      Delete
  6. I'm trying to run your code on my android device, but it results in an error that says:
    java.ioFileNotFoundExeption: /data/data/com.calsignlabs.apde/app_build/bin/android_fft_minim.apk.res does not exist. what kind of file is that? how can i find it? any suggestions?
    Thanks!

    ReplyDelete
    Replies
    1. Are you trying the apk? It is quite old and might not work in newer Android versions.
      If so, can you please try to use the APDE app, as explained in the last edit of the post (March 13th 2015)?
      Regards

      Delete
  7. I'm trying to run your code on my android device, but it results in an error that says:
    java.ioFileNotFoundExeption: /data/data/com.calsignlabs.apde/app_build/bin/android_fft_minim.apk.res does not exist. what kind of file is that? how can i find it? any suggestions?
    Thanks!

    ReplyDelete
    Replies
    1. Sorry for the late reply!
      Have a look here to see if it helps: https://github.com/Calsign/APDE/issues/10

      Delete