Print
Listen
By 
FMOD is a popular game audio development system by Firelight Technologies, in Melbourne, Australia. It is used to create sophisticated interactive soundtracks for Xbox, PS3, PC, and iOS games. Representing a modern technology solution to age-old problems of interactive audio production, it allows sound designers to make sounds, implement interactivity, and deliver event and soundbank files to programmers.Android is an open-source operating system for mobile devices by Google. It is written in Java, and represents the fastest growing cell phone market on the planet, beating out even Apple. This is because carriers can easily customize their own products to maximize their own profits, plus it's free. Carriers like that.
For me, personally, these are currently my favorite two technologies. FMOD I like because it runs perfectly on my Mac, which is networked to my ProTools rig, facilitating content creation and implementation. The programming APIs are accessible and flexible, and tech support has been excellent. Plus we love those guys.
The Android OS I like because it seems so familiar ... which should not be surprising, since it was mostly written by former colleagues from Danger. Even the naming conventions are similar to the code I worked with for 7 years, as Principal Sound Designer for the T-Mobile Sidekick. Eclipse runs the Android plugin perfectly on my Mac, and the online documentation is excellent. Plus we love those guys.
So you can imagine my delight when I heard that FMOD was releasing a version for Android, like peanut butter and strawberry jam, together for the first time! I wanted to try it out, and was looking for a suitable project, when I came across Vector Pinball by Brian Nenninger: simple, open source, silent ... perfect!
Here's how I added an interactive soundtrack to the game using FMOD for Android:
(click here for a video of the sonified game in action)
Audio Content Creation
The sound effects are full resolution (16/44 .aif) samples, based on synthesizer technology from the 1980's, to match the vector graphic style of the game. For instance, the flippers play a classic TR808 ping, while the bumpers use the Roland D50 Int1-11 Fantasia patch. The ball launch uses another D50 patch appropriately named "Android".
The interactive music consists of three tracks: bass, drums, and pad. The bass line is cut into five segments (three loops, and two transitions), and moves back and forth between the related keys of C major and A minor. The three drum loop samples can be played individually, or in combinations, while the pad is designed to play at anytime, allowing serendipity to provide polyrhythmic sync.
By tuning the bumpers, rollovers, and other sound effects, to a pentatonic scale (C, D, E, G and A), all sounds are in the same key, producing an euphonious effect (also known as "the secret yanni technique"). In fact, the soundtrack can be thought of as an interactive song, where the bumpers are the melody, played over background harmonies that vary based on gameplay.
FMOD Designer Implementation
The sounds are then imported into FMOD Designer, the tool used to specify playback behavior, and audition interactivity. Sound effects are implemented by creating events, which are basically containers with links to the audio files. For example, the bumper event contains six samples, to be played at random, so that each time the event is triggered, a different sample will be played.
The bumper event is an example of data-driven game audio, where the sound designer determines the functionality, and the programmer just fires off the event. On the other hand, the rollover event consists of a single sound, which the game code plays in various combinations, at different pentatonic pitches, as an example of algorithmic sound generation.
Data driven audio is pretty much the norm in console games these days, where things are much more complex than this simple demo. But the algorithmic effect uses 1 sample to play 7 different related sounds, and so requires less disk space to produce. This can be important in the resource-constrained world of mobile phones (though as current devices feature gigabytes of storage, this is becoming less so).
Interactive Music
FMOD's interactive music system is geared towards keeping multiple audio streams in sync, so that different tracks can be started and stopped at will, but the beat goes on. This is achieved by putting the drum loops and the bass line themes on the same timeline.
The bass line is completely data-driven, meaning the bass cue is started by the program, and then varied using a realtime parameter, called bassSequence. Increasing the value of the parameter causes the bass line to transition from one segment to the next, in a progression designated by the composer.
The drum track is varied algorithmically, in that the three drum loop segments are all synced to the same timeline, but are started and stopped by the game code. Not only can each drum loop play separately, they can also be played in combinations. A strictly data-driven solution would be to sample all the combinations, and then play them at random, or by parameter, whereas the algorithmic method trades increased CPU for decreased diskspace. Of course, on Android phones, both power and storage tend to be abundant.
Audition and Export
Maybe the most important aspect of FMOD is the ability to audition the sounds interactively during development. Back in the bad old days, sound files would be "tossed over the wall" to the programmers, and the designer would only be able to hear the audio played in context in the next build of the game (maybe a couple of days later).
With FMOD, I can open the audition window, start the bass line, and listen to the loops and transitions as I move the bassSequence slider up and down. The same kind of thing can be done for the sound effects. This way, the designer can be assured that the sounds will play as intended in the game.
The audio files themselves are linked into a soundbank, where compression and load behavior can be specified. For this game, I'm using MP3 at 33%, with the "decompress into memory" setting. This brings 8Mb of samples down to ~800k, and tells the FMOD engine to "decompress everything first, then play from memory". This seemed to work best for the Nexus S, where onboard RAM is fast and plentiful.
Finally, selecting Build from the FMOD Designer menu generates the following files:
Android Audio Programming
The FMOD libraries are written in C, consisting of fmodex (the low level audio engine), and fmodevent (the data driven game audio system). Because Android is written in Java, the FMOD APIs are accessed via the Java Native Interface (JNI) from main.c. It's a little tricky to setup: You have to modify the make file to include the FMOD libraries, change environmental variables to point to the right places, and install the C/C++ plugin for Eclipse ... but once everything's in place, it works like a charm!
Another tricky bit was setting the MEDIA_PATH. During initialization, FMOD requires a pathway to the folder containing the event and soundbank files. Many systems will place these files in a specific directory during the installation process, but for the demo, I wanted to ship a single Android package (.apk) file that would Just Work™. When I did a similar project for MacOS, this was easy to do, because a Mac application (.app) is actually a bundle of folders containing the executable and any resources it needs. During initalization, I could just point FMOD at the Resources folder in the bundle.
But an Android package is more like a .jar file, which is a form of Zip archive. No pathway to directories within the compressed archive is available, so I had to copy the files to internal storage, and point FMOD there. This increases the space required by the program, but the files are eliminated when the app is uninstalled, and we're talking about less than 1Mb anyway (on a device with 1Gb of memory). Larger files might have to be copied to the external SD card, which works as well.
Calling the API
Aside from initialization and error processing, most of main.c consists of calls to the FMOD libraries to perform specific functions, predominantly "play this event". The code for the rollover sound is a little more complicated, because the sound is generated by triggering the rollover event up to three times, at different pitches.
The drum track processing does something similar, randomly triggering various combinations of drum loops. The C module also does system updates, memory release during shutdown, and sets a music callback, which is used to make the bass line start when the pad finishes playing for the first time.
Calls to main.c functions are done from FMODaudio.java, which contains the FMODAudioDevice instance, the system update loop (performed every 50ms), and the native function prototypes. The Java methods (playFlipper, playRollover, etc) are called from the game code, according to gameplay (i.e. when the ball hits a bumper, playScore).
And that's all there is to it! The only thing left to do is make sure the AndroidManifest.xml is correct before exporting a signed .apk file that can be installed on Android devices running OS version 2.3.4 (Gingerbread).
Future Enhancements
As a simple demo, the program works perfectly, but there are a few things you'd want to do before submitting it to the Android Market. For one, there is a two-second black-screen pause when the game is started, while the audio samples are decompressed into memory. A simple initial splash screen would make the wait seem trivial.
Additional music variations would significantly increase playability over long periods of time. A neat trick would be to track harmonic changes using segment callbacks, and modify the bumper notes to fit diverse keys and tonalities. Instead of random pentatonic notes, a full scale could be provided, with notes selected by stochastic algorithm, which produces remarkably more melodic phrases. A menu item to allow selection of different audio themes (realistic, robotic, rock 'n' roll, etc) would also be fun.
FMOD for Android can not only facilitate cross-platform audio production for a wide variety of games, it can significantly advance the complexity and tastiness of mobile game soundtracks. Sounds good to me!
Want to play the demo? Go to the Twittering Machine code page to download the app and/or the Eclipse project.
- pdx
Special thanks to Brian Nenninger of Dozing Cat Software; Damian Kastbauer of Lost Chocolate Lab; Brett, Martin, Chen-po, Peter, and Mathew, at FMOD; and our friends over at Google.
The sound effects are full resolution (16/44 .aif) samples, based on synthesizer technology from the 1980's, to match the vector graphic style of the game. For instance, the flippers play a classic TR808 ping, while the bumpers use the Roland D50 Int1-11 Fantasia patch. The ball launch uses another D50 patch appropriately named "Android".
The interactive music consists of three tracks: bass, drums, and pad. The bass line is cut into five segments (three loops, and two transitions), and moves back and forth between the related keys of C major and A minor. The three drum loop samples can be played individually, or in combinations, while the pad is designed to play at anytime, allowing serendipity to provide polyrhythmic sync.
By tuning the bumpers, rollovers, and other sound effects, to a pentatonic scale (C, D, E, G and A), all sounds are in the same key, producing an euphonious effect (also known as "the secret yanni technique"). In fact, the soundtrack can be thought of as an interactive song, where the bumpers are the melody, played over background harmonies that vary based on gameplay.
FMOD Designer Implementation
The sounds are then imported into FMOD Designer, the tool used to specify playback behavior, and audition interactivity. Sound effects are implemented by creating events, which are basically containers with links to the audio files. For example, the bumper event contains six samples, to be played at random, so that each time the event is triggered, a different sample will be played.
The bumper event is an example of data-driven game audio, where the sound designer determines the functionality, and the programmer just fires off the event. On the other hand, the rollover event consists of a single sound, which the game code plays in various combinations, at different pentatonic pitches, as an example of algorithmic sound generation.
Data driven audio is pretty much the norm in console games these days, where things are much more complex than this simple demo. But the algorithmic effect uses 1 sample to play 7 different related sounds, and so requires less disk space to produce. This can be important in the resource-constrained world of mobile phones (though as current devices feature gigabytes of storage, this is becoming less so).
Interactive Music
FMOD's interactive music system is geared towards keeping multiple audio streams in sync, so that different tracks can be started and stopped at will, but the beat goes on. This is achieved by putting the drum loops and the bass line themes on the same timeline.
The bass line is completely data-driven, meaning the bass cue is started by the program, and then varied using a realtime parameter, called bassSequence. Increasing the value of the parameter causes the bass line to transition from one segment to the next, in a progression designated by the composer.
The drum track is varied algorithmically, in that the three drum loop segments are all synced to the same timeline, but are started and stopped by the game code. Not only can each drum loop play separately, they can also be played in combinations. A strictly data-driven solution would be to sample all the combinations, and then play them at random, or by parameter, whereas the algorithmic method trades increased CPU for decreased diskspace. Of course, on Android phones, both power and storage tend to be abundant.
Audition and Export
Maybe the most important aspect of FMOD is the ability to audition the sounds interactively during development. Back in the bad old days, sound files would be "tossed over the wall" to the programmers, and the designer would only be able to hear the audio played in context in the next build of the game (maybe a couple of days later).
With FMOD, I can open the audition window, start the bass line, and listen to the loops and transitions as I move the bassSequence slider up and down. The same kind of thing can be done for the sound effects. This way, the designer can be assured that the sounds will play as intended in the game.
The audio files themselves are linked into a soundbank, where compression and load behavior can be specified. For this game, I'm using MP3 at 33%, with the "decompress into memory" setting. This brings 8Mb of samples down to ~800k, and tells the FMOD engine to "decompress everything first, then play from memory". This seemed to work best for the Nexus S, where onboard RAM is fast and plentiful.
Finally, selecting Build from the FMOD Designer menu generates the following files:
- the event file (.fev), containing the control data
- the soundbank file (.fsb), containing the sample data
- the C header file (.h), containing data constants for accessing events and parameters
Android Audio Programming
The FMOD libraries are written in C, consisting of fmodex (the low level audio engine), and fmodevent (the data driven game audio system). Because Android is written in Java, the FMOD APIs are accessed via the Java Native Interface (JNI) from main.c. It's a little tricky to setup: You have to modify the make file to include the FMOD libraries, change environmental variables to point to the right places, and install the C/C++ plugin for Eclipse ... but once everything's in place, it works like a charm!
Another tricky bit was setting the MEDIA_PATH. During initialization, FMOD requires a pathway to the folder containing the event and soundbank files. Many systems will place these files in a specific directory during the installation process, but for the demo, I wanted to ship a single Android package (.apk) file that would Just Work™. When I did a similar project for MacOS, this was easy to do, because a Mac application (.app) is actually a bundle of folders containing the executable and any resources it needs. During initalization, I could just point FMOD at the Resources folder in the bundle.
But an Android package is more like a .jar file, which is a form of Zip archive. No pathway to directories within the compressed archive is available, so I had to copy the files to internal storage, and point FMOD there. This increases the space required by the program, but the files are eliminated when the app is uninstalled, and we're talking about less than 1Mb anyway (on a device with 1Gb of memory). Larger files might have to be copied to the external SD card, which works as well.
Calling the API
Aside from initialization and error processing, most of main.c consists of calls to the FMOD libraries to perform specific functions, predominantly "play this event". The code for the rollover sound is a little more complicated, because the sound is generated by triggering the rollover event up to three times, at different pitches.
The drum track processing does something similar, randomly triggering various combinations of drum loops. The C module also does system updates, memory release during shutdown, and sets a music callback, which is used to make the bass line start when the pad finishes playing for the first time.
Calls to main.c functions are done from FMODaudio.java, which contains the FMODAudioDevice instance, the system update loop (performed every 50ms), and the native function prototypes. The Java methods (playFlipper, playRollover, etc) are called from the game code, according to gameplay (i.e. when the ball hits a bumper, playScore).
And that's all there is to it! The only thing left to do is make sure the AndroidManifest.xml is correct before exporting a signed .apk file that can be installed on Android devices running OS version 2.3.4 (Gingerbread).
Future Enhancements
As a simple demo, the program works perfectly, but there are a few things you'd want to do before submitting it to the Android Market. For one, there is a two-second black-screen pause when the game is started, while the audio samples are decompressed into memory. A simple initial splash screen would make the wait seem trivial.
Additional music variations would significantly increase playability over long periods of time. A neat trick would be to track harmonic changes using segment callbacks, and modify the bumper notes to fit diverse keys and tonalities. Instead of random pentatonic notes, a full scale could be provided, with notes selected by stochastic algorithm, which produces remarkably more melodic phrases. A menu item to allow selection of different audio themes (realistic, robotic, rock 'n' roll, etc) would also be fun.
FMOD for Android can not only facilitate cross-platform audio production for a wide variety of games, it can significantly advance the complexity and tastiness of mobile game soundtracks. Sounds good to me!
Want to play the demo? Go to the Twittering Machine code page to download the app and/or the Eclipse project.
- pdx
Special thanks to Brian Nenninger of Dozing Cat Software; Damian Kastbauer of Lost Chocolate Lab; Brett, Martin, Chen-po, Peter, and Mathew, at FMOD; and our friends over at Google.
Amazing soundtrack, it matches perfectly with the graphics. Thank you for making this available, I'll be incorporating it into the next Vector Pinball release.