Using libavformat and libavcodec(2)

Decoding Video Frames

As I've already mentioned, a video file can contain several audio and video streams, and each of those streams is split up into packets of a particular size. Our job is to read these packets one by one using libavformat, filter out all those that aren't part of the video stream we're interested in, and hand them on to libavcodec for decoding. In doing this, we'll have to take care of the fact that the boundary between two frames can occur in the middle of a packet.

 

Sound complicated? Luckily, we can encapsulate this whole process in a routine that simply returns the next video frame:

bool GetNextFrame(AVFormatContext *pFormatCtx, AVCodecContext *pCodecCtx, int videoStream, AVFrame *pFrame) { static AVPacket packet; static int bytesRemaining = 0; static uint8_t *rawData; static bool fFristTime = true; int bytesDecoded; int frameFinished; // First time we're called, set packet .data to NULL to indicate it // doesn't have to be freed if (fFirstTime) { fFirstTime = false; packet.data = NULL; } // Decode packets until we have decoded a complete frame while (true) { // Work on the current packet until we have decoded all of it while (bytesRemaining > 0) { // Decode the next chunk of data bytesDecoded = avcodec_decode_video(pCodecCtx, pFrame, &frameFinished, rawData, bytesRemaining); // Was there an error? if (bytesDecoded < 0) { fprintf(stderr, "Error while decoding frame/n"); return false; } bytesRemaining -= bytesDecoded; rawData += bytesDecoded; // Did we finish the current frame? Then we can return if (frameFinished) return true; } // Read the next packet, skipping all packets that aren't for this // stream do { // Free old packet if (packet.data != NULL) av_free_packet(&packet); // Read new packet if (av_read_packet(pFormatCtx, &packet) < 0) goto loop_exit; } while (packet.stream_index != videoStream); bytesRemaining = packet.size; rawData = packet.data; } loop_exit: // Decode the rest of the last frame bytesDecoded = avcodec_decode_video(pCodecCtx, pFrame, &frameFinished, rawData, bytesRemaining); // Free last packet if (packet.data != NULL) av_free_packet(&packet); return frameFinished != 0; } 

Now, all we have to do is sit in a loop, calling GetNextFrame() until it return false. Just one more thing to take care of: Most codecs return images in YUV420(one luminance and two chrominance channels, with the chrominance channels samples at half the spatial resolution of the luminance channel). Depending on what you want to do with the video data, you may want to convert this to RGB. (Note, though, that this is not necessary if all you want to do is diplay the video data; take a look at the X11 Xvideo extension, which does YUV-to-RGB and scaling in hardware.) Fortunately, libavcodec provides a conversion routine called img_convert, which does conversion between YUV and RGB as well as a variety of other image formats. The loop that decodes the video thus becomes:

while (GetNextFrame(pFormatCtx, pCodecCtx, videoStream, pFrame) { img_convert((AVPicture *)pFrameRGB, PIX_FMT_RGB24, (AVPicture*)pFrame, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height); // Process the video frame (save to disk etc.) DoSomethingWithTheImage(pFrameRGB); } 

The RGB image pFrameRGB(of type AVFrame*) is allocated like this:

AVFrame *pFrameRGB; int numBytes; uint8_t *buffer; // Allocate an AVFrame structure pFrameRGB = avcodec_alloc_frame(); if (pFrameRGB == NULL) handle_error(); // Determine required buffer size and allocate buffer numBytes = avpicture_get_size(PIX_FMT_RGB24, pCodecCtx->width, pCodecCtx->height); buffer = new uint8_t[numBytes]; // Assign appropriate parts of buffer to image planes in pFrameRGB avpicture_fill((AVPicture *)pFrameRGB, buffer, PIX_FMT_RGB24, pCodecCtx->width, pCodexCtx->height);