Android 7.0 Audio的Resample过程详解

最新推荐文章于 2024-08-18 03:36:39 发布

转载最新推荐文章于 2024-08-18 03:36:39 发布 · 1.6k 阅读

Android_Audio系统专栏收录该内容

46 篇文章

订阅专栏

本文详细介绍了Android 7.0系统中音频重采样（Resample）的过程，包括AudioFlinger如何创建Playback线程、线程如何通过prepareTracks_l()函数匹配Track，并最终实现音频文件采样率与设备支持采样率之间的转换。

Android 7.0 Audio的Resample过程详解

Qidi 2017.02.23 （Markdown & Haroopad）

【前言】

处理过音频文件的工程师都知道音频数据存在采样率（Sample Rate）这个指标。在位深度（Bit Depth）一定的情况下，采样率越高，理论上来说播放出来的声音就越细腻，录制的声音也就越保真，反之亦然。

但在较早的Android系统版本上，不管音频文件原来的采样率几何，统统都被重采样（Resample）到44.1KHz进行播放，录制的时候则是被固定为8KHz进行采样。尽管这样的处理方式被广大音质爱好者所诟病，但在当时它确实是一种实现设备兼容的有效方法。

作为Android Audio BSP工程师，有必要了解系统实现Resample的过程。现在Android系统已经发布到了7.0版本，一起看看在最新的版本上这个Resample的过程是怎样实现的吧。

【背景知识】

我们知道在Android系统中，当应用层APP播放一个音频文件时，Framework层的AudioPolicyService（APS）会接收上层APP传递来的音频参数（例如格式、声道、采样率等），并调用AudioFlinger的createTrack()方法对应创建1个Track，再调用openOutput()方法来打开1个outputStream，然后使用这个outputStream来创建相应的Playback线程（依据应用场景可以是OffloadThread、DirectOutputThread、MixerThread），最终在Playback线程中匹配之前创建的Track，开始自APP至HAL的数据传输。

【Resample过程分析】

那么我们对Android Audio Resample过程的分析就从AudioFlinger开始。在AudioFlinger::openOutput()中可以看到，在Playback线程被成功创建之后，即被加入到mPlaybackThreads向量中进行管理了。具体代码如下：

sp<AudioFlinger::PlaybackThread> AudioFlinger::openOutput_l(audio_module_handle_t module,
                                                            audio_io_handle_t *output,
                                                            audio_config_t *config,
                                                            audio_devices_t devices,
                                                            const String8& address,
                                                            audio_output_flags_t flags)
{
    ......

    AudioStreamOut *outputStream = NULL;
    status_t status = outHwDev->openOutputStream(    // 打开1个outputStream
            &outputStream,
            *output,
            devices,
            flags,
            config,
            address.string());

    mHardwareStatus = AUDIO_HW_IDLE;

    if (status == NO_ERROR) {

        PlaybackThread *thread;
        if (flags & AUDIO_OUTPUT_FLAG_COMPRESS_OFFLOAD) {
            thread = new OffloadThread(this, outputStream, *output, devices, mSystemReady);
            ALOGV("openOutput_l() created offload output: ID %d thread %p", *output, thread);
        } else if ((flags & AUDIO_OUTPUT_FLAG_DIRECT)
                || !isValidPcmSinkFormat(config->format)
                || !isValidPcmSinkChannelMask(config->channel_mask)) {
            thread = new DirectOutputThread(this, outputStream, *output, devices, mSystemReady);
            ALOGV("openOutput_l() created direct output: ID %d thread %p", *output, thread);
        } else {
            thread = new MixerThread(this, outputStream, *output, devices, mSystemReady);    // 默认情况下，创建MixerThread类型的Playback线程
            ALOGV("openOutput_l() created mixer output: ID %d thread %p", *output, thread);
        }
        mPlaybackThreads.add(*output, thread);    // 将新创建的线程加入向量
        return thread;
    }

    return 0;
}
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

随后Playback线程运行，对应的AudioFlinger::Playback::threadLoop()方法被执行，在该方法中调用了prepareTracks_l()函数。这个函数实际上是对应于AudioFlinger::MixerThread::prepareTracks_l()这个函数。threadLoop()函数代码细节如下：

bool AudioFlinger::PlaybackThread::threadLoop()
{
    Vector< sp<Track> > tracksToRemove;
    ......

    while (!exitPending())
    {
        cpuStats.sample(myName);
        Vector< sp<EffectChain> > effectChains;
        { // scope for mLock
            Mutex::Autolock _l(mLock);
            processConfigEvents_l();
            ......
            saveOutputTracks();
            ......
            if ((!mActiveTracks.size() && systemTime() > mStandbyTimeNs) ||
                                   isSuspended()) {
                // put audio hardware into standby after short delay
                if (shouldStandby_l()) {

                    threadLoop_standby();

                    mStandby = true;
                }
                ......
            }
            // mMixerStatusIgnoringFastTracks is also updated internally
            mMixerStatus = prepareTracks_l(&tracksToRemove);    // 调用prepareTracks_l()，为Playback线程匹配已注册的Track
            ......
            // prevent any changes in effect chain list and in each effect chain
            // during mixing and effect process as the audio buffers could be deleted
            // or modified if an effect is created or deleted
            lockEffectChains_l(effectChains);
        } // mLock scope ends
        ......
        // enable changes in effect chain
        unlockEffectChains(effectChains);
        // Finally let go of removed track(s), without the lock held
        // since we can't guarantee the destructors won't acquire that
        // same lock.  This will also mutate and push a new fast mixer state.
        threadLoop_removeTracks(tracksToRemove);
        tracksToRemove.clear();
        // FIXME I don't understand the need for this here;
        //       it was in the original code but maybe the
        //       assignment in saveOutputTracks() makes this unnecessary?
        clearOutputTracks();
        // Effect chains will be actually deleted here if they were removed from
        // mEffectChains list during mixing or effects processing
        effectChains.clear();
        // FIXME Note that the above .clear() is no longer necessary since effectChains
        // is now local to this block, but will keep it for now (at least until merge done).
    }
    threadLoop_exit();
    if (!mStandby) {
        threadLoop_standby();
        mStandby = true;
    }
    ......
    return false;
}
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

Resample的过程就发生在prepareTracks_l()函数中，所以我们来好好阅读一下。在该函数中，通过一个for循环遍历所有处于active状态的track。每一次循环中，都要进行如下2步操作：
1. 通过reqSampleRate = track->mAudioTrackServerProxy->getSampleRate()来获取硬件设备所支持的采样率；
2. 之后调用mAudioMixer->setParameter(name, AudioMixer::RESAMPLE, AudioMixer::SAMPLE_RATE, (void*)(uintptr_t)reqSampleRate)，通过对比音频文件采样率和音频设备支持的采样率，判断是否创建新的Resampler对象，或者从已有的Resampler对象列表中返回1个；

prepareTracks_l()函数代码细节如下：

AudioFlinger::PlaybackThread::mixer_state AudioFlinger::MixerThread::prepareTracks_l(
        Vector< sp<Track> > *tracksToRemove)
{
    ......
    // find out which tracks need to be processed
    size_t count = mActiveTracks.size();    // 获取处于active状态的track的数量
    ......
    for (size_t i=0 ; i<count ; i++) {
        const sp<Track> t = mActiveTracks[i].promote();
        if (t == 0) {
            continue;
        }
        // this const just means the local variable doesn't change
        Track* const track = t.get();    // 获取对应的track
        ......
        audio_track_cblk_t* cblk = track->cblk();

        // The first time a track is added we wait
        // for all its buffers to be filled before processing it
        int name = track->name();
        ......
        if ((framesReady >= minFrames) && track->isReady() &&
                !track->isPaused() && !track->isTerminated())
        {
            ......
            int param = AudioMixer::VOLUME;
            if (track->mFillingUpStatus == Track::FS_FILLED) {
                // no ramp for the first volume setting
                track->mFillingUpStatus = Track::FS_ACTIVE;
                if (track->mState == TrackBase::RESUMING) {
                    track->mState = TrackBase::ACTIVE;
                    param = AudioMixer::RAMP_VOLUME;
                }
                mAudioMixer->setParameter(name, AudioMixer::RESAMPLE, AudioMixer::RESET, NULL);
            // FIXME should not make a decision based on mServer
            } else if (cblk->mServer != 0) {
                // If the track is stopped before the first frame was mixed,
                // do not apply ramp
                param = AudioMixer::RAMP_VOLUME;
            }

            // compute volume for this track
            ......
            // Delegate volume control to effect in track effect chain if needed
            ......

            // XXX: these things DON'T need to be done each time
            mAudioMixer->setBufferProvider(name, track);
            mAudioMixer->enable(name);

            mAudioMixer->setParameter(name, param, AudioMixer::VOLUME0, &vlf);    // 设置左声道音量
            mAudioMixer->setParameter(name, param, AudioMixer::VOLUME1, &vrf);    // 设置右声道音量
            mAudioMixer->setParameter(name, param, AudioMixer::AUXLEVEL, &vaf);   // 设置辅助声道音量
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::FORMAT, (void *)track->format());    // 设置音频数据格式
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::CHANNEL_MASK, (void *)(uintptr_t)track->channelMask());    // 设置音频声道数
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::MIXER_CHANNEL_MASK, (void *)(uintptr_t)mChannelMask);
            // limit track sample rate to 2 x output sample rate, which changes at re-configuration
            uint32_t maxSampleRate = mSampleRate * AUDIO_RESAMPLER_DOWN_RATIO_MAX;
            uint32_t reqSampleRate = track->mAudioTrackServerProxy->getSampleRate();    // 获取音频设备所支持的采样率
            if (reqSampleRate == 0) {
                reqSampleRate = mSampleRate;
            } else if (reqSampleRate > maxSampleRate) {
                reqSampleRate = maxSampleRate;
            }
            mAudioMixer->setParameter(
                name,
                AudioMixer::RESAMPLE,
                AudioMixer::SAMPLE_RATE,    // 设置音频采样率(必要时会进行重采样)
                (void *)(uintptr_t)reqSampleRate);

            AudioPlaybackRate playbackRate = track->mAudioTrackServerProxy->getPlaybackRate();
            mAudioMixer->setParameter(
                name,
                AudioMixer::TIMESTRETCH,
                AudioMixer::PLAYBACK_RATE,    // 设置播放码率
                &playbackRate);

            /*
             * Select the appropriate output buffer for the track.
             *
             * Tracks with effects go into their own effects chain buffer
             * and from there into either mEffectBuffer or mSinkBuffer.
             *
             * Other tracks can use mMixerBuffer for higher precision
             * channel accumulation.  If this buffer is enabled
             * (mMixerBufferEnabled true), then selected tracks will accumulate
             * into it.
             *
             */
            if (mMixerBufferEnabled
                    && (track->mainBuffer() == mSinkBuffer
                            || track->mainBuffer() == mMixerBuffer)) {
                mAudioMixer->setParameter(
                        name,
                        AudioMixer::TRACK,
                        AudioMixer::MIXER_FORMAT, (void *)mMixerBufferFormat);    // 设置缓冲区数据格式
                mAudioMixer->setParameter(
                        name,
                        AudioMixer::TRACK,
                        AudioMixer::MAIN_BUFFER, (void *)mMixerBuffer);    // 分配主缓冲区
                // TODO: override track->mainBuffer()?
                mMixerBufferValid = true;
            } else {
                ......
            }
            mAudioMixer->setParameter(
                name,
                AudioMixer::TRACK,
                AudioMixer::AUX_BUFFER, (void *)track->auxBuffer());    // 分配副缓冲区

            // reset retry count
            track->mRetryCount = kMaxTrackRetries;

            // If one track is ready, set the mixer ready if:
            //  - the mixer was not ready during previous round OR
            //  - no other track is not ready
            if (mMixerStatusIgnoringFastTracks != MIXER_TRACKS_READY ||
                    mixerStatus != MIXER_TRACKS_ENABLED) {
                mixerStatus = MIXER_TRACKS_READY;
            }
        } else {
            // 出现underrun，以及相应处理操作
            ......
        }
    }

    // Push the new FastMixer state if necessary
    ......
    // Now perform the deferred reset on fast tracks that have stopped
    ......
    // remove all the tracks that need to be...
    removeTracks_l(*tracksToRemove);
    ......
    // sink or mix buffer must be cleared if all tracks are connected to an
    // effect chain as in this case the mixer will not write to the sink or mix buffer
    // and track effects will accumulate into it
    ......
    // if any fast tracks, then status is ready
    ......
    return mixerStatus;
}
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

在确认要使用的Resampler对象存在后，调用invalidateState(1 << name)使设置生效，开始执行重采样。invalidateState()函数会调用AudioMixer::process_validate()，在该函数中首先通过语句t.hook = getTrackHook(TRACKTYPE_RESAMPLE, t.mMixerChannelCount, t.mMixerInFormat, t.mMixerFormat);获取执行重采样操作的函数，随后通过state->hook = process_resampling;中的t.hook(&t, outTemp, numFrames, state->resampleTemp, aux)语句进行调用。
setParameter()函数代码如下：

void AudioMixer::setParameter(int name, int target, int param, void *value)
{
    ......
    int valueInt = static_cast<int>(reinterpret_cast<uintptr_t>(value));
    int32_t *valueBuf = reinterpret_cast<int32_t*>(value);
    switch (target) {
    ......
    case RESAMPLE:
        switch (param) {
        case SAMPLE_RATE:
            ALOG_ASSERT(valueInt > 0, "bad sample rate %d", valueInt);
            if (track.setResampler(uint32_t(valueInt), mSampleRate)) {    // 新建或查找1个Resampler对象
                ALOGV("setParameter(RESAMPLE, SAMPLE_RATE, %u)",
                        uint32_t(valueInt));
                invalidateState(1 << name);    // 使设置生效，调用重采样的后续处理函数
            }
            break;
        case RESET:
            track.resetResampler();
            invalidateState(1 << name);
            break;
        case REMOVE:
            delete track.resampler;
            track.resampler = NULL;
            track.sampleRate = mSampleRate;
            invalidateState(1 << name);
            break;
        default:
            LOG_ALWAYS_FATAL("setParameter resample: bad param %d", param);
        }
        break;
    }
}
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

invalidateState()函数代码如下：

void AudioMixer::invalidateState(uint32_t mask)
{
    if (mask != 0) {
        mState.needsChanged |= mask;
        mState.hook = process__validate;    // 使配置生效
    }
}
 1
2
3
4
5
6
7
 1
2
3
4
5
6
7

process__validate()函数代码如下：

void AudioMixer::process__validate(state_t* state)
{
    ......
    uint32_t en = state->enabledTracks;
    while (en) {
        ......
        if (n & NEEDS_MUTE) {
            ......
        } else {
            ......
            if (n & NEEDS_RESAMPLE) {
                all16BitsStereoNoResample = false;
                resampling = true;
                t.hook = getTrackHook(TRACKTYPE_RESAMPLE, t.mMixerChannelCount,
                        t.mMixerInFormat, t.mMixerFormat);    // 获取Resample时track对象需要执行的函数（查看getTrackHook()可以看到被获取的函数是track__genericResample()）
                ALOGV_IF((n & NEEDS_CHANNEL_COUNT__MASK) > NEEDS_CHANNEL_2,
                        "Track %d needs downmix + resample", i);
            } else {
                ......
            }
        }
    }
    // select the processing hooks
    state->hook = process__nop;
    if (countActiveTracks > 0) {
        if (resampling) {
            if (!state->outputTemp) {
                state->outputTemp = new int32_t[MAX_NUM_CHANNELS * state->frameCount];
            }
            if (!state->resampleTemp) {
                state->resampleTemp = new int32_t[MAX_NUM_CHANNELS * state->frameCount];
            }
            state->hook = process__genericResampling;    // 在需要重采样操作的情况下，调用process_genericResampling()函数
        } else {
            ......
        }
    }
    ......
    // Now that the volume ramp has been done, set optimal state and
    // track hooks for subsequent mixer process
    ......
}
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

process_genericResampling()函数代码如下：

// generic code with resampling
void AudioMixer::process__genericResampling(state_t* state)
{
    ......
    uint32_t e0 = state->enabledTracks;
    while (e0) {
        // process by group of tracks with same output buffer
        // to optimize cache use
        ......
        while (e1) {
            ......
            // this is a little goofy, on the resampling case we don't
            // acquire/release the buffers because it's done by
            // the resampler.
            if (t.needs & NEEDS_RESAMPLE) {
                t.hook(&t, outTemp, numFrames, state->resampleTemp, aux);    // 调用track__genericResample()函数执行Resample
            } else {
                ......
            }
        }
        convertMixerFormat(out, t1.mMixerFormat,
                outTemp, t1.mMixerInFormat, numFrames * t1.mMixerChannelCount);
    }
}
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24