Low Latency & Polyphonic Audio in PhoneGap

http://www.tricedesigns.com/2012/01/25/low-latency-polyphonic-audio-in-phonegap/

 

If you have ever tried to develop any kind of application using HTML5 audio that is widely supported, then you have likely pulled all the hair from your head. In its current state, HTML5 Audio is wrought with issues… lack of consistent codec support across browsers & operating systems, no polyphony (a single audio clip can not be played on top of itself), and lack of concurrency (on some of the leading mobile browsers you can only play one audio file at a time, if at all). Even the leading HTML5 games for desktop browsers don’t even use HTML5 audio (they use Flash). Don’t believe me? Just take a look at Angry Birds, Cut the Rope, or Bejeweled in a proxy/resource monitor…

The Problem

You want fast & responsive audio for your mobile applications.   This is especially the case for multimedia intensive and/or gaming applications.

HTML5 audio is not *yet* ready for prime-time. There are some great libraries like SoundManager, which can help you try to use HTML5 audio with a failover to Flash, but you are still limited without polyphony or concurrency. In desktop browsers, Flash fixes these issues, and Flash is still vastly superior to HTML5 for audio programming.

If you are building mobile applications, you can have great audio capabilities by developing apps withAIR. However, what if you aren’t using AIR? In native applications, you can access the underlying audio APIs and have complete control.

If you are developing mobile applications with PhoneGap, you can use the Media class, which works great. If you want polyphony, then you will have to do some work managing audio files for yourself, which can get tricky. You can also write native plugins that integrate with the audio APIs for the native operating systems, which is what i will be covering in this post.

Before continuing further, let’s take a minute to understand what I am talking about when I refer to concurrency, polyphony, and low-latency…

Concurrency

Concurrency in audio programming refers to the ability to play multiple audio resources simultaneously.  HTML5 in most mobile devices does not support this – not in iOS, not in Android.  In fact, HTML5 Audio does not work *at all* in Android 2.x and earlier.  Native APIs do support this, and so does PhoneGap’s Media class, which is based on Android MediaPlayer and iOS AVAudioPlayer.

Polyphony

Producing many sounds simultaneously; many-voiced.

In this case, polyphony is the production of multiple sounds simultaneously (I’m not referring to the concept of polyphany in music theory). In describing concurrency, I refered to the ability to play 2 separate sounds at the same time, where with polyphony I refer to the ability to play the same sound “on top” of itself. There can be multiple “voices” of the same sound. In the most literal of definitions concurrency could be considered a part of polyphony, and polyphony a part of concurrency… Hopefully you get what I’m trying to say. In its current state, HTML5 audio supports neither concurrency or polyphony.  The PhoneGap Media class does not support polyphony, however you can probably manage multiple media instances via javascript to achieve polyphonic behavior – this requires additional work in the JavaScript side of things to juggle resources.

Low Latency

Low latency refers to “human-unnoticeable delays between an input being processed and the corresponding output providing real time characteristics” according to wikipedia.   In this case, I refer to low latency audio, meaning that there is an imperceptible delay between when a sound is triggered, and when it actually plays.   This means that sounds will play when expected, not after a wait.   This means a bouncing ball sound should be heard as you see the ball bouncing on the screen.   Not after it has already bounced.

In HTML5, you can auto-load a sound so that it is ready when you need it, but don’t expect to play more than one at a time.  With the PhoneGap Media class, the audio file isn’t actually requested until you invoke “play”.   This occurs inside “startPlaying” on Android, and “play” on iOS.   What I wanted was a way to preload the audio so that it is immediately ready for use at the time it is needed.

The Solution

PhoneGap makes it really easy to build natively installed applications using a familiar paradim: HTML & JavaScript.   Luckily, PhoneGap also allows you to tie into native code using the native pluginmodel.   This enables you to write your own native code and expose that code to your PhoneGap application via a JavaScript interface… and that is exactly what I did to enable low-latency, concurrent, and polyphonic audio in a PhoneGap experience.

I created PhoneGap native plugins for Android and iOS that allow you to preload audio, and playback that audio quickly, with a very simple to use API.   I’ll get into details how this works further in the post, but you can get a pretty good idea of what I mean by viewing the following two videos.

The first is a basic “Drum Machine”.  You just tap the pads to play an audio sample.

 

更多请查看原文

转载于:https://www.cnblogs.com/crabzzz/archive/2012/10/24/2737184.html