本文详细分析了FFMPEG中AVFrame结构体的主要变量，包括data、linesize、width、height等，并解释了它们在视频和音频数据处理中的作用。同时，介绍了AVFrame在编码和解码过程中的应用，以及如何通过这些变量进行码流分析。
FFMPEG有几个最重要的结构体，包含了解协议，解封装，解码操作，此前已经进行过分析：
在此不再详述，其中AVFrame是包含码流参数较多的结构体。本文将会详细分析一下该结构体里主要变量的含义和作用。
首先看一下结构体的定义（位于avcodec.h）：
typedef struct AVFrame {
#define AV_NUM_DATA_POINTERS 8
/**
* pointer to the picture/channel planes.
* This might be different from the first allocated byte
*
* Some decoders access areas outside 0,0 - width,height, please
* see avcodec_align_dimensions2(). Some filters and swscale can read
* up to 16 bytes beyond the planes, if these filters are to be used,
* then 16 extra bytes must be allocated.
*/
uint8_t *data[AV_NUM_DATA_POINTERS];
/**
* For video, size in bytes of each picture line.
* For audio, size in bytes of each plane.
*
* For audio, only linesize[0] may be set. For planar audio, each channel
* plane must be the same size.
*
* For video the linesizes should be multiplies(乘) of the CPUs alignment
* preference(对齐的偏好), this is 16 or 32 for modern desktop CPUs.
* Some code requires such alignment other code can be slower without
* correct alignment, for yet other it makes no difference.
*
* @note The linesize may be larger than the size of usable data -- there
* may be extra padding present for performance reasons.
*/
int linesize[AV_NUM_DATA_POINTERS];
/**
* pointers to the data planes/channels.
*
* For video, this should simply point to data[].
*
* For planar(平面) audio, each channel has a separate(单独的） data pointer, and
* linesize[0] contains the size of each channel buffer.
* For packed audio, there is just one data pointer, and linesize[0]
* contains the total size of the buffer for all channels.
*
* Note: Both data and extended_data should always be set in a valid（有效的） frame,
* but for planar audio with more channels that can fit（适应） in data,
* extended_data must be used in order to access all channels.
*/
uint8_t **extended_data;
/**
* width and height of the video frame
*/
int width, height;
/**
* number of audio samples (per channel) described by this frame
*/
int nb_samples;
/**
* format of the frame, -1 if unknown or unset
* Values correspond（符合） to enum AVPixelFormat for video frames,
* enum AVSampleFormat for audio)
*/
int format; //AV_PIX_FMT_YUV420P AV_SAMPLE_FMT_NB
/**
* 1 -> keyframe, 0-> not
*/
int key_frame;
/**
* Picture type of the frame.（帧类型：i p b）
*/
enum AVPictureType pict_type;
#if FF_API_AVFRAME_LAVC
attribute_deprecated
uint8_t *base[AV_NUM_DATA_POINTERS];
#endif
/**
* Sample aspect ratio for the video frame, 0/1 if unknown/unspecified.
* 对于视频帧采样宽高比，0 / 1如果未知/未指定。
*/
AVRational sample_aspect_ratio;
/**
* 显示时间戳
* Presentation timestamp in time_base units (time when frame should be shown to user).
*/
int64_t pts;
/**
* 解码时间戳
* PTS copied from the AVPacket that was decoded to produce this frame.
*/
int64_t pkt_pts;
/**
* DTS copied from the AVPacket that triggered returning this frame. (if frame threading isnt used)
* This is also the Presentation time of this AVFrame calculated from
* only AVPacket.dts values without pts values.
*/
int64_t pkt_dts;
/**
* picture number in bitstream order
*/
int coded_picture_number;
/**
* picture number in display order
*/
int display_picture_number;
/**
* quality 质量(between 1 (good) and FF_LAMBDA_MAX (bad))
*/
int quality;
#if FF_API_AVFRAME_LAVC
attribute_deprecated
int reference;
/**
* QP table QP表
*/
attribute_deprecated
int8_t *qscale_table;
/**
* QP store stride （步幅）
*/
attribute_deprecated
int qstride;
attribute_deprecated
int qscale_type;
/**
* 跳过宏块表
* mbskip_table[mb]>=1 if MB didn't change
* stride= mb_width = (width+15)>>4
*/
attribute_deprecated
uint8_t *mbskip_table;
/**
* 运动矢量表
* motion vector table
* @code
* example:
* int mv_sample_log2= 4 - motion_subsample_log2;
* int mb_width= (width+15)>>4;
* int mv_stride= (mb_width << mv_sample_log2) + 1;
* motion_val[direction][x + y*mv_stride][0->mv_x, 1->mv_y];
* @endcode
*/
attribute_deprecated
int16_t (*motion_val[2])[2];
/**
* 宏块类型表
* macroblock type table
* mb_type_base + mb_width + 2
*/
attribute_deprecated
uint32_t *mb_type;
/**
* DCT系数
* DCT coefficients
*/
attribute_deprecated
short *dct_coeff;
/**
* 参考帧列表
* motion reference frame index
* the order in which these are stored can depend on the codec.
*/
attribute_deprecated
int8_t *ref_index[2];
#endif
/**
* for some private data of the user
*/
void *opaque;
/**
* error
*/
uint64_t error[AV_NUM_DATA_POINTERS];
#if FF_API_AVFRAME_LAVC
attribute_deprecated
int type;
#endif
/**
* When decoding, this signals how much the picture must be delayed.
* 解码时，该信号多少图片必须延迟。
* extra_delay = repeat_pict / (2*fps)
*/
int repeat_pict;
/**
* The content of the picture is interlaced（隔行扫描）.
*/
int interlaced_frame;
/**
* If the content is interlaced, is top field displayed first.
* 如果是隔行扫描，是否是顶场先被显示
*/
int top_field_first;
/**
* Tell user application that palette（调色板） has changed from previous frame.
*/
int palette_has_changed;
#if FF_API_AVFRAME_LAVC
attribute_deprecated
int buffer_hints;
/**
* Pan scan. pan 扫描
*/
attribute_deprecated
struct AVPanScan *pan_scan;
#endif
/**
* reordered opaque 64bit (generally an integer or a double precision float
* PTS but can be anything).
* The user sets AVCodecContext.reordered_opaque to represent the input at
* that time,
* the decoder reorders values as needed and sets AVFrame.reordered_opaque
* to exactly one of the values provided by the user through AVCodecContext.reordered_opaque
* @deprecated in favor of pkt_pts
* 重新排序（通常是64位的整型或不透明双精密浮点但可以做）。
* 不透明的用户集AVCodecContext.reordered_opaque代表输入到解码器是在时间的重新排序的值，
* 需要对项集avframe.reordered _不透明值，一个由用户提供的avcodeccontext.reordered暗通”请取消_（PTS中_袋
*/
int64_t reordered_opaque;
#if FF_API_AVFRAME_LAVC
/**
* @deprecated this field is unused
*/
attribute_deprecated void *hwaccel_picture_private;
attribute_deprecated
struct AVCodecContext *owner;
attribute_deprecated
void *thread_opaque;
/**
* log2 of the size of the block which a single vector in motion_val represents:
* (4->16x16, 3->8x8, 2-> 4x4, 1-> 2x2)
*/
attribute_deprecated
uint8_t motion_subsample_log2;
#endif
/**
* Sample rate of the audio data.
*/
int sample_rate;
/**
* Channel layout of the audio data.
*/
uint64_t channel_layout;
/**
* AVBuffer references backing the data for this frame. If all elements of
* this array are NULL, then this frame is not reference counted.
*
* There may be at most one AVBuffer per data plane, so for video this array
* always contains all the references. For planar audio with more than
* AV_NUM_DATA_POINTERS channels, there may be more buffers than can fit in
* this array. Then the extra AVBufferRef pointers are stored in the
* extended_buf array.
*/
AVBufferRef *buf[AV_NUM_DATA_POINTERS];
/**
* For planar audio which requires more than AV_NUM_DATA_POINTERS
* AVBufferRef pointers, this array will hold all the references which
* cannot fit into AVFrame.buf.
*
* Note that this is different from AVFrame.extended_data, which always
* contains all the pointers. This array only contains the extra pointers,
* which cannot fit into AVFrame.buf.
*
* This array is always allocated using av_malloc() by whoever constructs
* the frame. It is freed in av_frame_unref().
*/
AVBufferRef **extended_buf;
/**
* Number of elements in extended_buf.
*/
int nb_extended_buf;
AVFrameSideData **side_data;
int nb_side_data;
/**
* frame timestamp estimated using various heuristics, in stream time base
* Code outside libavcodec should access this field using:
* av_frame_get_best_effort_timestamp(frame)
* - encoding: unused
* - decoding: set by libavcodec, read by user.
*/
int64_t best_effort_timestamp;
/**
* reordered pos from the last AVPacket that has been input into the decoder
* Code outside libavcodec should access this field using:
* av_frame_get_pkt_pos(frame)
* - encoding: unused
* - decoding: Read by user.
*/
int64_t pkt_pos;
/**
* duration of the corresponding packet, expressed in
* AVStream->time_base units, 0 if unknown.
* Code outside libavcodec should access this field using:
* av_frame_get_pkt_duration(frame)
* - encoding: unused
* - decoding: Read by user.
*/
int64_t pkt_duration;
/**
* metadata.
* Code outside libavcodec should access this field using:
* av_frame_get_metadata(frame)
* - encoding: Set by user.
* - decoding: Set by libavcodec.
*/
AVDictionary *metadata;
/**
* decode error flags of the frame, set to a combination of
* FF_DECODE_ERROR_xxx flags if the decoder produced a frame, but there
* were errors during the decoding.
* Code outside libavcodec should access this field using:
* av_frame_get_decode_error_flags(frame)
* - encoding: unused
* - decoding: set by libavcodec, read by user.
*/
int decode_error_flags;
#define FF_DECODE_ERROR_INVALID_BITSTREAM 1
#define FF_DECODE_ERROR_MISSING_REFERENCE 2
/**
* number of audio channels, only used for audio.
* Code outside libavcodec should access this field using:
* av_frame_get_channels(frame)
* - encoding: unused
* - decoding: Read by user.
*/
int channels;
/**
* size of the corresponding packet containing the compressed
* frame. It must be accessed using av_frame_get_pkt_size() and
* av_frame_set_pkt_size().
* It is set to a negative value if unknown.
* - encoding: unused
* - decoding: set by libavcodec, read by user.
*/
int pkt_size;
/**
* YUV colorspace type.
* It must be accessed using av_frame_get_colorspace() and
* av_frame_set_colorspace().
* - encoding: Set by user
* - decoding: Set by libavcodec
*/
enum AVColorSpace colorspace;
/**
* MPEG vs JPEG YUV range.
* It must be accessed using av_frame_get_color_range() and
* av_frame_set_color_range().
* - encoding: Set by user
* - decoding: Set by libavcodec
*/
enum AVColorRange color_range;
/**
* Not to be accessed directly from outside libavutil
*/
AVBufferRef *qp_table_buf;
} AVFrame;<span style="font-family: Arial; font-size: 14px; line-height: 26px; background-color: rgb(255, 255, 255);">AVFrame结构体一般用于存储原始数据（即非压缩数据，例如对视频来说是YUV，RGB，对音频来说是PCM），此外还包含了一些相关的信息。比如说，解码的时候存储了宏块类型表，QP表，运动矢量表等数据。编码的时候也存储了相关的数据。因此在使用FFMPEG进行码流分析的时候，AVFrame是一个很重要的结构体。</span>
下面看几个主要变量的作用（在这里考虑解码的情况）
uint8_t *data[AV_NUM_DATA_POINTERS]：解码后原始数据（对视频来说是YUV，RGB，对音频来说是PCM）
int linesize[AV_NUM_DATA_POINTERS]：data的大小
int width, height：视频帧宽和高（1920x1080,1280x720...）
int nb_samples：音频的一个AVFrame中可能包含多个音频帧，在此标记包含了几个
int format：解码后原始数据类型（YUV420，YUV422，RGB24...）
int key_frame：是否是关键帧
enum AVPictureType pict_type：帧类型（I,B,P...）
AVRational sample_aspect_ratio：宽高比（16:9，4:3...）
int64_t pts：显示时间戳
int coded_picture_number：编码帧序号
int display_picture_number：显示帧序号
int8_t *qscale_table：QP表
uint8_t *mbskip_table：跳过宏块表
int16_t (*motion_val[2])[2]：运动矢量表
uint32_t *mb_type：宏块类型表
short *dct_coeff：DCT系数，这个没有提取过
int8_t *ref_index[2]：运动估计参考帧列表（貌似H.264这种比较新的标准才会涉及到多参考帧）
int interlaced_frame：是否是隔行扫描
uint8_t motion_subsample_log2：一个宏块中的运动矢量采样个数，取log的
1.data[]
对于packed格式的数据（例如RGB24），会存到data[0]里面。
对于planar格式的数据（例如YUV420P），则会分开成data[0]，data[1]，data[2]...（YUV420P中data[0]存Y，data[1]存U，data[2]存V）
具体参见：FFMPEG 实现 YUV，RGB各种图像原始数据之间的转换（swscale）
2.pict_type
包含以下类型：
enum AVPictureType {
AV_PICTURE_TYPE_NONE = 0, ///< Undefined
AV_PICTURE_TYPE_I, ///< Intra
AV_PICTURE_TYPE_P, ///< Predicted
AV_PICTURE_TYPE_B, ///< Bi-dir predicted
AV_PICTURE_TYPE_S, ///< S(GMC)-VOP MPEG4
AV_PICTURE_TYPE_SI, ///< Switching Intra
AV_PICTURE_TYPE_SP, ///< Switching Predicted
AV_PICTURE_TYPE_BI, ///< BI type
};
宽高比是一个分数，FFMPEG中用AVRational表达分数：
/**
* rational number numerator/denominator
*/
typedef struct AVRational{
int num; ///< numerator
int den; ///< denominator
} AVRational;
QP表指向一块内存，里面存储的是每个宏块的QP值。宏块的标号是从左往右，一行一行的来的。每个宏块对应1个QP。
qscale_table[0]就是第1行第1列宏块的QP值；qscale_table[1]就是第1行第2列宏块的QP值；qscale_table[2]就是第1行第3列宏块的QP值。以此类推...
宏块的个数用下式计算：
注：宏块大小是16x16的。
每行宏块数：
int mb_stride = pCodecCtx->width/16+1
int mb_sum = ((pCodecCtx->height+15)>>4)*(pCodecCtx->width/16+1)5.motion_subsample_log2
1个运动矢量所能代表的画面大小（用宽或者高表示，单位是像素），注意，这里取了log2。
代码注释中给出以下数据：
4->16x16, 3->8x8, 2-> 4x4, 1-> 2x2
即1个运动矢量代表16x16的画面的时候，该值取4；1个运动矢量代表8x8的画面的时候，该值取3...以此类推
6.motion_val
运动矢量表存储了一帧视频中的所有运动矢量。
该值的存储方式比较特别：
int16_t (*motion_val[2])[2];
注释中给了一段代码：
int mv_sample_log2= 4 - motion_subsample_log2;
int mb_width= (width+15)>>4;
int mv_stride= (mb_width << mv_sample_log2) + 1;
motion_val[direction][x + y*mv_stride][0->mv_x, 1->mv_y];大概知道了该数据的结构：
1.首先分为两个列表L0和L1
2.每个列表（L0或L1）存储了一系列的MV（每个MV对应一个画面，大小由motion_subsample_log2决定）
3.每个MV分为横坐标和纵坐标（x,y）
注意，在FFMPEG中MV和MB在存储的结构上是没有什么关联的，第1个MV是屏幕上左上角画面的MV（画面的大小取决于motion_subsample_log2），第2个MV是屏幕上第1行第2列的画面的MV，以此类推。因此在一个宏块（16x16）的运动矢量很有可能如下图所示（line代表一行运动矢量的个数）：
//例如8x8划分的运动矢量与宏块的关系：
//-------------------------
//| | |
//|mv[x] |mv[x+1] |
//-------------------------
//| | |
//|mv[x+line]|mv[x+line+1]|
//-------------------------7.mb_type
宏块类型表存储了一帧视频中的所有宏块的类型。其存储方式和QP表差不多。只不过其是uint32类型的，而QP表是uint8类型的。每个宏块对应一个宏块类型变量。
宏块类型如下定义所示：
//The following defines may change, don't expect compatibility if you use them.
#define MB_TYPE_INTRA4x4 0x0001
#define MB_TYPE_INTRA16x16 0x0002 //FIXME H.264-specific
#define MB_TYPE_INTRA_PCM 0x0004 //FIXME H.264-specific
#define MB_TYPE_16x16 0x0008
#define MB_TYPE_16x8 0x0010
#define MB_TYPE_8x16 0x0020
#define MB_TYPE_8x8 0x0040
#define MB_TYPE_INTERLACED 0x0080
#define MB_TYPE_DIRECT2 0x0100 //FIXME
#define MB_TYPE_ACPRED 0x0200
#define MB_TYPE_GMC 0x0400
#define MB_TYPE_SKIP 0x0800
#define MB_TYPE_P0L0 0x1000
#define MB_TYPE_P1L0 0x2000
#define MB_TYPE_P0L1 0x4000
#define MB_TYPE_P1L1 0x8000
#define MB_TYPE_L0 (MB_TYPE_P0L0 | MB_TYPE_P1L0)
#define MB_TYPE_L1 (MB_TYPE_P0L1 | MB_TYPE_P1L1)
#define MB_TYPE_L0L1 (MB_TYPE_L0 | MB_TYPE_L1)
#define MB_TYPE_QUANT 0x00010000
#define MB_TYPE_CBP 0x00020000
//Note bits 24-31 are reserved for codec specific use (h264 ref0, mpeg1 0mv, ...)
一个宏块如果包含上述定义中的一种或两种类型，则其对应的宏块变量的对应位会被置1。
注：一个宏块可以包含好几种类型，但是有些类型是不能重复包含的，比如说一个宏块不可能既是16x16又是8x8。
8.ref_index
运动估计参考帧列表存储了一帧视频中所有宏块的参考帧索引。这个列表其实在比较早的压缩编码标准中是没有什么用的。只有像H.264这样的编码标准才有多参考帧的概念。但是这个字段目前我还没有研究透。只是知道每个宏块包含有4个该值，该值反映的是参考帧的索引。以后有机会再进行细研究吧。
在这里展示一下自己做的码流分析软件的运行结果。将上文介绍的几个列表图像化显示了出来（在这里是使用MFC的绘图函数画出来的）
视频帧：
QP参数提取的结果：
美化过的（加上了颜色）：
宏块类型参数提取的结果：
美化过的（加上了颜色，更清晰一些，s代表skip宏块）：
运动矢量参数提取的结果（在这里是List0）：
运动估计参考帧参数提取的结果：
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
