Spec素材说明——片段图形

本文介绍如何利用GLSLSandbox网站上的片段着色器代码在spec中创建炫酷图形,包括去除背景、调整透明度及利用频谱数据动态改变图形等技巧。

制作这个素材是因为无意间发现一个非常炫酷的网站GLSL Sandbox(由于在外网,有时候可能进不去),这个网站提供很多图形,这些图形都是通过一个片段着色器代码进行实现的,刚好spec能提供相关的接口,因此实现了这个素材。

如果你不会写这样的代码,不用担心,我也不会写,但是接下来,我会教你怎么对GLSL Sandbox上的代码进行修改,使得这些素材能够“更好”的在spec中显示。

需要注意的是,这个素材占有相对会较高,具体占有跟代码复杂度和渲染区域相关

首先,你必须了解GLSL的一些东西

片段着色器代码的作用就是确定一个片段坐标对应像素的颜色,过程看起来就像是这样,我们要完成的,就是片段着色器的部分

然后我们以一个简单的图形代码为例进行说明

#ifdef GL_ES
precision mediump float;
#endif

#extension GL_OES_standard_derivatives : enable

uniform float time;
uniform vec2 mouse;
uniform vec2 resolution;
varying vec2 surfacePosition;
uniform float peak;
#define PI 3.141592653
#define TWO_PI 2.0*PI
#define t time*0.3
#define MAX 30.
void main( void ) {
	vec2 uv = (gl_FragCoord.xy - resolution * 0.5) / max(resolution.x, resolution.y) * 4.0;
	uv *= 1.0;
	vec2 z = uv;
	float e = 0.0;
	for (float i=0.0;i<=MAX;i+=1.0) {
		z = vec2(z.x * z.x - z.y * z.y, 2.0 * z.x * z.y) - z*cos(t)+uv*sin(t);		
		e += 0.01*peak/abs(sin(t+z.y) * sin(z.x));
	}
	gl_FragColor = vec4(vec3(e,e*0.4,e*0.2),1.0);	
}

可以看出这个代码是类似于C语言的,如果你会C/C++的话,这样的代码其实并不陌生。

最终目的:确定输出颜色

跟C语言一样,main是主函数入口,主函数运行结束之后,gl_FragColor的值就是该像素的颜色,也有一些着色器代码不是使用gl_FragColor,而是定义了一个out vec4类型的输出变量(可能会叫fragColor)作为输出颜色。

gl_FragColor是一个vec4类型的变量,vec4是一个存储着四个浮点值的变量,你可以使用.xyzw(或者.rgba)获取对应分量的值

vec4可以理解成四维向量(x,y,z,w),也可以理解为颜色(r,g,b,a),需要注意的是,片段着色器中的r,g,b,a的范围需要保证在[0,1]之间,比如(0,0,0,1)表示黑色,(1,1,1,1)表示白色,第4个分量alpha可以理解为透明度。

例如你可以这样对color进行操作:

vec4 color;
float alpha=color.a;    //获取第四个分量,跟.w作用一样
vec2 v2=color.rg;       //获取红,绿,组成的vec2
vec3 v3=color.rgb;      //获取红,绿,蓝通道

变量

uniform float time;        //获取当前以毫秒为基准的时间
uniform vec2 mouse;        //获取鼠标的坐标
uniform vec2 resolution;   //当前图形的分辨率(大小)
uniform float peak;        //频谱引擎的峰值
uniform float rms;         //频谱引擎的均值

uniform表示这些变量是外部输入的,目前来说spec片段图形支持的外部变量只有这些

宏定义

#define PI 3.141592653
#define TWO_PI 2.0*PI
#define t time*0.3
#define MAX 30.

有一些#define a b 这样格式的代码,#define就是一个简单的宏替换,说白了就是把下面的代码中的 a 用 b 替换。比如第一个,就是把代码中出现的PI替换成3.141592653

#define PI 3.1415926535
float pi=PI*5;

/*等价于*/ 

float pi=3.1415926535*5;

这样做就是方便为了给一些变量取别名,在不修改代码的情况下方便理解,如果理解了上述的内容,那么可以开始进行实际操作了。

核心步骤一——剔除背景

当你复制一个GLSL Sandbox中的代码到spec中,然后编译,你可能就会发现大多数图形都有一个背景,产生这个背景主要原因是因为sandbox中的代码没有做透明度(alpha)计算(也就是的gl_FragColor的第四分量),去除这个背景就是接下来要做的事情。

如果图形有彩色背景(也就是渐变色,非纯色的),那么需要做如下处理

你需要找到代码中绘制背景部分的代码,有些代码可能会把背景的绘制放到一个函数里面,你需要做的就是去除这个函数以及它的调用,对于关联的vec3变量可以使用(0.0,0.0,0.0)进行替换,经过这个处理之后,图形的背景应该以及变成黑色了,如果你是高手的话,完全可以现在就把图形改成背景透明的,并且效果会非常好。如果并不熟悉代码,那么可以按下面的操作进行。

此刻你拥有的应该是一个具有纯色背景(一般是黑色)的图形

如果你拥有的图形背景不是黑色的,你可能需要找一下代码中的一些vec3变量,一般背景颜色的变量命名(或者宏)会有back,background之类的字样,修改为(0.0,0.0,0.0),这样就变成黑色的了

计算Alpha(透明度)

gl_FragColor = vec4(vec3(e,e*0.4,e*0.2),1.0);

回到一开始的代码,你会发现代码中gl_FragColor的第四分量(alpha)直接被设置为1.0,也就是不透明。正是因为这样,才产生了黑色的背景。我们要做的就是要修改代码,在当前像素如果是黑色的时候,透明度设置为0;

所以就有以下的一些方法

1.使用某个颜色分量(r,g,b)作为alpha值,就像下面这样。

	gl_FragColor = vec4(vec3(e,e*0.4,e*0.2),1.0);
	gl_FragColor.a = gl_FragColor.r;    //使用红色通道作为透明度

2.同时考虑三个通道的影响

    gl_FragColor = vec4(vec3(e,e*0.4,e*0.2),1.0);
    gl_FragColor.a = (gl_FragColor.r+gl_FragColor.g+gl_FragColor.b)/3.0;	//取均值
	gl_FragColor = vec4(vec3(e,e*0.4,e*0.2),1.0);
	gl_FragColor.a = max(gl_FragColor.r,max(gl_FragColor.g,gl_FragColor.b));	//取最大值

3.以某个阈值对alpha进行划分

	gl_FragColor = vec4(vec3(e,e*0.4,e*0.2),1.0);
	if(gl_FragColor.r+gl_FragColor.g+gl_FragColor.b>0.1)
		gl_FragColor.a=1.0;
	else
		gl_FragColor.a=0.0;

上面的方法是绘制好图形之后,将图形中的某些颜色设置为透明色,当然不止上面的这些方法,这样做能达到一些较好的效果,但是对于一些特殊的图形,比如说背景是黑色的,而图形中也有一部分东西也是黑色的,通过这个方法,会将所有黑色变透明,这并不是我们想要的,想要解决这个问题,就需要你更多的去了解这个代码,通过代码来确定alpha值,而不是通过rgb分量来确定alpha值,spec中的自带素材【GLSL_立方体】就是通过这样的方法解决的。感兴趣可以看一下。

自定义修改

GLSL Sandbox中大多数代码写的非常好,有些代码会通过宏定义或者变量来定义一些属性,比如颜色,大小之类的一些信息。比如这个字母素材代码:

// N041020N

#ifdef GL_ES
precision highp float;
#endif

vec2 uv;

uniform float time;
uniform vec2 resolution;;

const vec2 ch_size  = vec2(1.0, 2.0) * 0.8;              // character size (X,Y)
const vec2 ch_space = ch_size + vec2(1.0, 1.0);    // character distance Vector(X,Y)
const vec2 ch_start = vec2 (ch_space.x * -5., 1.); // start position
      vec2 ch_pos   = vec2 (0.0, 0.0);             // character position(X,Y)

#define REPEAT_SIGN false // True/False; True=Multiple, False=Single

#define n0 ddigit(0x22FF);
#define n1 ddigit(0x0281);
#define n2 ddigit(0x1177);
#define n3 ddigit(0x11E7);
#define n4 ddigit(0x5508);
#define n5 ddigit(0x11EE);
#define n6 ddigit(0x11FE);
#define n7 ddigit(0x2206);
#define n8 ddigit(0x11FF);
#define n9 ddigit(0x11EF);

#define A ddigit(0x119F);
#define B ddigit(0x927E);
#define C ddigit(0x007E);
#define D ddigit(0x44E7);
#define E ddigit(0x107E);
#define F ddigit(0x101E);
#define G ddigit(0x807E);
#define H ddigit(0x1199);
#define I ddigit(0x4466);
#define J ddigit(0x4436);
#define K ddigit(0x9218);
#define L ddigit(0x0078);
#define M ddigit(0x0A99);
#define N ddigit(0x8899);
#define O ddigit(0x00FF);
#define P ddigit(0x111F);
#define Q ddigit(0x80FF);
#define R ddigit(0x911F);
#define S ddigit(0x8866);
#define T ddigit(0x4406);
#define U ddigit(0x00F9);
#define V ddigit(0x2218);
#define W ddigit(0xA099);
#define X ddigit(0xAA00);
#define Y ddigit(0x4A00);
#define Z ddigit(0x2266);
#define _ ch_pos.x += ch_space.x;
#define s_dot     ddigit(0);
#define s_minus   ddigit(0x1100);
#define s_plus    ddigit(0x5500);
#define s_greater ddigit(0x2800);
#define s_less    ddigit(0x8200);
#define s_sqrt    ddigit(0x0C02);
#define nl1 ch_pos = ch_start;  ch_pos.y -= 3.0;
#define nl2 ch_pos = ch_start;  ch_pos.y -= 6.0;
#define nl3 ch_pos = ch_start;	ch_pos.y -= 9.0;

float dseg(vec2 p0, vec2 p1)
{
	vec2 dir = normalize(p1 - p0);
	vec2 cp = (uv - ch_pos - p0) * mat2(dir.x, dir.y,-dir.y, dir.x);
	return distance(cp, clamp(cp, vec2(0), vec2(distance(p0, p1), 0)));   
}

bool bit(int n, int b)
{
	return mod(floor(float(n) / exp2(floor(float(b)))), 2.0) != 0.0;
}

float d = 1e6;

void ddigit(int n)
{
	float v = 1e6;	
	vec2 cp = uv - ch_pos;
	if (n == 0)     v = min(v, dseg(vec2(-0.405, -1.000), vec2(-0.500, -1.000)));
	if (bit(n,  0)) v = min(v, dseg(vec2( 0.500,  0.063), vec2( 0.500,  0.937)));
	if (bit(n,  1)) v = min(v, dseg(vec2( 0.438,  1.000), vec2( 0.063,  1.000)));
	if (bit(n,  2)) v = min(v, dseg(vec2(-0.063,  1.000), vec2(-0.438,  1.000)));
	if (bit(n,  3)) v = min(v, dseg(vec2(-0.500,  0.937), vec2(-0.500,  0.062)));
	if (bit(n,  4)) v = min(v, dseg(vec2(-0.500, -0.063), vec2(-0.500, -0.938)));
	if (bit(n,  5)) v = min(v, dseg(vec2(-0.438, -1.000), vec2(-0.063, -1.000)));
	if (bit(n,  6)) v = min(v, dseg(vec2( 0.063, -1.000), vec2( 0.438, -1.000)));
	if (bit(n,  7)) v = min(v, dseg(vec2( 0.500, -0.938), vec2( 0.500, -0.063)));
	if (bit(n,  8)) v = min(v, dseg(vec2( 0.063,  0.000), vec2( 0.438, -0.000)));
	if (bit(n,  9)) v = min(v, dseg(vec2( 0.063,  0.063), vec2( 0.438,  0.938)));
	if (bit(n, 10)) v = min(v, dseg(vec2( 0.000,  0.063), vec2( 0.000,  0.937)));
	if (bit(n, 11)) v = min(v, dseg(vec2(-0.063,  0.063), vec2(-0.438,  0.938)));
	if (bit(n, 12)) v = min(v, dseg(vec2(-0.438,  0.000), vec2(-0.063, -0.000)));
	if (bit(n, 13)) v = min(v, dseg(vec2(-0.063, -0.063), vec2(-0.438, -0.938)));
	if (bit(n, 14)) v = min(v, dseg(vec2( 0.000, -0.938), vec2( 0.000, -0.063)));
	if (bit(n, 15)) v = min(v, dseg(vec2( 0.063, -0.063), vec2( 0.438, -0.938)));
	ch_pos.x += ch_space.x;
	d = min(d, v);
}
mat2 rotate(float a)
{
	float c = cos(a);
	float s = sin(a);
	return mat2(c, s, -s, c);
}
vec3 hsv2rgb_smooth( in vec3 c )
{
    vec3 rgb = clamp( abs(mod(c.x*6.0+vec3(0.0,4.0,2.0),6.0)-3.0)-1.0, 0.0, 1.0 );

	rgb = rgb*rgb*(3.0-2.0*rgb); // cubic smoothing	

	return c.z * mix( vec3(1.0), rgb, c.y);
}
void main( void ) 
{
	
	vec2 aspect = resolution.xy / resolution.y;
	uv = ( gl_FragCoord.xy / resolution.y ) - aspect / 2.0;
	float _d =  1.0-length(uv);
	uv *= 10;
	uv -= vec2(-3., 1.);
	//uv *= rotate(time+uv.x*0.05);

	vec3 ch_color = hsv2rgb_smooth(vec3(time*0.4+uv.y*0.1,0.5,1.0));

	uv.x += 0.5+sin(time+uv.y*0.7)*0.5;
	ch_pos = ch_start;
	uv.y -= 0.5;


	 _ _ _ _ _ S P E C _ nl1; _ _ _ _ I T A L I N K nl2; 
			 
                 
		
	vec3 color = mix(ch_color, vec3(0), 1.0- (0.08 / d*2.0));  // shading
	gl_FragColor = vec4(color, color.x);
}

这些代码调整了字体的格式: 


const vec2 ch_size  = vec2(1.0, 2.0) * 0.8;              // character size (X,Y)
const vec2 ch_space = ch_size + vec2(1.0, 1.0);    // character distance Vector(X,Y)
const vec2 ch_start = vec2 (ch_space.x * -5., 1.); // start position
      vec2 ch_pos   = vec2 (0.0, 0.0);             // character position(X,Y)

uv确定了图形的缩放

	uv *= 10;

这个确定了输出的字母 (其实是通过宏定义调用函数):

 _ _ _ _ _ S P E C _ nl1; _ _ _ _ I T A L I N K nl2; 

如果属性并没有明确定义,你可以尝试修改代码中的常量(数字)

或许你并不知道这串数字的作用,你完全可以改动一下,然后点编译,说不定会有些什么意想不到的效果哦

说这么多差点忘了spec是一个频谱软件?

spec对片段代码提供了单频谱数据的输入,输入的值是一个float类型的数据,并且值的大小位于[0,1.0]之间, 你可以在着色器代码上声明两个变量,然后就能在代码中使用了

uniform float peak;        //频谱引擎的峰值
uniform float rms;         //频谱引擎的均值

怎么用呢?在上面的代码中你可能已经找到某些常量数字代表的属性,你只需要把这个float跟这个数据关联起来就好了。

比如我们已经知道上面字母素材uv*=10确定了缩放,我们可以这么改:

首先声明peak变量

uniform float peak;

然后peak跟uv参数关联

uv *= 10/peak;

如果此时你的电脑没有发出声音,你会发现图形不见了,因为peak为0,除数为0,其实是出错了。

因此可以改成这样:

	uv *= 10/(0.1+peak);

你会发现字母会随着节奏进行伸缩,就像下面这样: 

 好了,这个素材基本就是这样了,遇到问题可以评论,作者看到第一时间回复=.=

 

 

 

 

使用C++基于以下完整代码,需要保留filter处理部分,改造成维持一个输出流,动态切换多个输入流(一个输入流完成后关闭,再次打开一个新的流) extern "C" { #include <libavcodec/avcodec.h> #include <libavformat/avformat.h> #include <libavfilter/buffersink.h> #include <libavfilter/buffersrc.h> #include <libavutil/audio_fifo.h> #include <libavutil/avassert.h> #include <libavutil/avstring.h> #include <libavutil/frame.h> #include <libavutil/opt.h> #include <libswresample/swresample.h> #include <libavutil/audio_fifo.h> } #if defined(_MSC_VER) static char av_error[AV_ERROR_MAX_STRING_SIZE] = { 0 }; #define av_err2str(errnum) \ av_make_error_string(av_error, AV_ERROR_MAX_STRING_SIZE, errnum) #elif #define av_err2str(errnum) \ av_make_error_string((char[AV_ERROR_MAX_STRING_SIZE]){0}, AV_ERROR_MAX_STRING_SIZE, errnum) #endif static AVFormatContext* ifmt_ctx; static AVFormatContext* ofmt_ctx; typedef struct FilteringContext { AVFilterContext* buffersink_ctx; AVFilterContext* buffersrc_ctx; AVFilterGraph* filter_graph; AVPacket* enc_pkt; AVFrame* filtered_frame; } FilteringContext; static FilteringContext* filter_ctx; typedef struct StreamContext { AVCodecContext* dec_ctx; AVCodecContext* enc_ctx; AVFrame* dec_frame; } StreamContext; static StreamContext* stream_ctx; static int audio_index = -1; static int video_index = -1; static int64_t current_audio_pts = 0; static int64_t current_video_pts = 0; static bool decode_first_video = true; static AVAudioFifo* fifo = NULL; //重采样时,如果输入nb_sample比输出的nb_sample小时,需要缓存 //#define SAVE_AUDIO_FILE #ifdef SAVE_AUDIO_FILE static FILE* save_audio = fopen("d:\\sampler\\1.pcm", "w+b"); static void save_audio_data(AVFrame* frame) { int data_size = av_get_bytes_per_sample(stream_ctx[audio_index].enc_ctx->sample_fmt); if (data_size >= 0) { for (int i = 0; i < frame->nb_samples; i++) for (int ch = 0; ch < stream_ctx[audio_index].enc_ctx->channels; ch++) fwrite(frame->data[ch] + data_size * i, 1, data_size, save_audio); } } #endif static int open_input_file(const char* filename) { int ret; unsigned int i; ifmt_ctx = NULL; /**(解封装 1.1):创建并初始化AVFormatContext*/ if ((ret = avformat_open_input(&ifmt_ctx, filename, NULL, NULL)) < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot open input file\n"); return ret; } /**(解封装 1.2):检索流信息,这个过程会检查输入流中信息是否存在异常*/ if ((ret = avformat_find_stream_info(ifmt_ctx, NULL)) < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot find stream information\n"); return ret; } stream_ctx = (StreamContext*)av_mallocz_array(ifmt_ctx->nb_streams, sizeof(*stream_ctx)); if (!stream_ctx) return AVERROR(ENOMEM); for (i = 0; i < ifmt_ctx->nb_streams; i++) { AVStream* stream = ifmt_ctx->streams[i]; /**(解码 2.1):查找解码器(AVCodec)*/ AVCodec* dec = avcodec_find_decoder(stream->codecpar->codec_id); AVCodecContext* codec_ctx; if (!dec) { av_log(NULL, AV_LOG_ERROR, "Failed to find decoder for stream #%u\n", i); return AVERROR_DECODER_NOT_FOUND; } /**(解码 2.2):通过解码器(AVCodec)生成解码器上下文(AVCodecContext)*/ codec_ctx = avcodec_alloc_context3(dec); if (!codec_ctx) { av_log(NULL, AV_LOG_ERROR, "Failed to allocate the decoder context for stream #%u\n", i); return AVERROR(ENOMEM); } /**(解码 2.3):将AVCodecParameters参数赋给AVCodecContext*/ ret = avcodec_parameters_to_context(codec_ctx, stream->codecpar); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Failed to copy decoder parameters to input decoder context " "for stream #%u\n", i); return ret; } /* Reencode video & audio and remux subtitles etc. */ if (codec_ctx->codec_type == AVMEDIA_TYPE_VIDEO || codec_ctx->codec_type == AVMEDIA_TYPE_AUDIO) { if (codec_ctx->codec_type == AVMEDIA_TYPE_VIDEO){ codec_ctx->framerate = av_guess_frame_rate(ifmt_ctx, stream, NULL); video_index = i; } else { audio_index = i; } /* Open decoder */ /**(解码 2.4):初始化码器器上下文*/ ret = avcodec_open2(codec_ctx, dec, NULL); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Failed to open decoder for stream #%u\n", i); return ret; } } //保存解码上下文 stream_ctx[i].dec_ctx = codec_ctx; //分配解码帧 stream_ctx[i].dec_frame = av_frame_alloc(); if (!stream_ctx[i].dec_frame) return AVERROR(ENOMEM); } av_dump_format(ifmt_ctx, 0, filename, 0); return 0; } static int open_output_file(const char* filename) { AVStream* out_stream; AVStream* in_stream; AVCodecContext* dec_ctx, * enc_ctx; AVCodec* encoder; int ret; unsigned int i; ofmt_ctx = NULL; /**(封装 4.1):根据文件格式初始化封装器上下文AVFormatContext*/ avformat_alloc_output_context2(&ofmt_ctx, NULL, NULL, filename); if (!ofmt_ctx) { av_log(NULL, AV_LOG_ERROR, "Could not create output context\n"); return AVERROR_UNKNOWN; } for (i = 0; i < ifmt_ctx->nb_streams; i++) { /**(封装 4.2):创建输出视频和音频AVStream*/ out_stream = avformat_new_stream(ofmt_ctx, NULL); if (!out_stream) { av_log(NULL, AV_LOG_ERROR, "Failed allocating output stream\n"); return AVERROR_UNKNOWN; } in_stream = ifmt_ctx->streams[i]; dec_ctx = stream_ctx[i].dec_ctx; if (dec_ctx->codec_type == AVMEDIA_TYPE_VIDEO || dec_ctx->codec_type == AVMEDIA_TYPE_AUDIO) { /* in this example, we choose transcoding to same codec */ /**(编码 3.1):获取对应的编码器AVCodec*/ #if 0 encoder = avcodec_find_encoder(dec_ctx->codec_id); #else if (dec_ctx->codec_type == AVMEDIA_TYPE_VIDEO) { encoder = avcodec_find_encoder(AV_CODEC_ID_H264); } else { encoder = avcodec_find_encoder(AV_CODEC_ID_AAC); } #endif if (!encoder) { av_log(NULL, AV_LOG_FATAL, "Necessary encoder not found\n"); return AVERROR_INVALIDDATA; } /**(编码 3.2):通过编码器(AVCodec)获取编码器上下文(AVCodecContext)*/ enc_ctx = avcodec_alloc_context3(encoder); if (!enc_ctx) { av_log(NULL, AV_LOG_FATAL, "Failed to allocate the encoder context\n"); return AVERROR(ENOMEM); } /**给编码器初始化信息*/ /* In this example, we transcode to same properties (picture size, * sample rate etc.). These properties can be changed for output * streams easily using filters */ if (dec_ctx->codec_type == AVMEDIA_TYPE_VIDEO) { enc_ctx->height = dec_ctx->height; enc_ctx->width = dec_ctx->width; enc_ctx->sample_aspect_ratio = dec_ctx->sample_aspect_ratio; /* take first format from list of supported formats */ if (encoder->pix_fmts) enc_ctx->pix_fmt = encoder->pix_fmts[0]; else enc_ctx->pix_fmt = dec_ctx->pix_fmt; /* video time_base can be set to whatever is handy and supported by encoder */ #if 0 enc_ctx->time_base = av_inv_q(dec_ctx->framerate); #else //enc_ctx->time_base = dec_ctx->time_base; enc_ctx->time_base = { 1, dec_ctx->framerate.num * dec_ctx->ticks_per_frame }; enc_ctx->ticks_per_frame = dec_ctx->ticks_per_frame; enc_ctx->framerate = dec_ctx->framerate; enc_ctx->has_b_frames = dec_ctx->has_b_frames; //输出将相对于输入延迟max_b_frames + 1-->但是输入的为0! //enc_ctx->max_b_frames = dec_ctx->max_b_frames + 1; enc_ctx->max_b_frames = 2; enc_ctx->bit_rate = dec_ctx->bit_rate; enc_ctx->codec_type = dec_ctx->codec_type; // 禁用B帧 if (enc_ctx->max_b_frames && enc_ctx->codec_id != AV_CODEC_ID_MPEG4 && enc_ctx->codec_id != AV_CODEC_ID_MPEG1VIDEO && enc_ctx->codec_id != AV_CODEC_ID_MPEG2VIDEO) { enc_ctx->has_b_frames = 0; enc_ctx->max_b_frames = 0; } #endif } else { enc_ctx->sample_rate = dec_ctx->sample_rate; enc_ctx->channel_layout = dec_ctx->channel_layout; enc_ctx->channels = av_get_channel_layout_nb_channels(enc_ctx->channel_layout); /* take first format from list of supported formats */ enc_ctx->sample_fmt = encoder->sample_fmts[0]; enc_ctx->time_base = { 1, enc_ctx->sample_rate }; enc_ctx->bit_rate = dec_ctx->bit_rate; enc_ctx->codec_type = dec_ctx->codec_type; //enc_ctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL; } if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER) enc_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER; /**(编码 3.3):*/ /* Third parameter can be used to pass settings to encoder */ ret = avcodec_open2(enc_ctx, encoder, NULL); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot open video encoder for stream #%u\n", i); return ret; } /**(编码 3.4):*/ ret = avcodec_parameters_from_context(out_stream->codecpar, enc_ctx); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Failed to copy encoder parameters to output stream #%u\n", i); return ret; } out_stream->time_base = enc_ctx->time_base; //保存编码上下文 stream_ctx[i].enc_ctx = enc_ctx; } else if (dec_ctx->codec_type == AVMEDIA_TYPE_UNKNOWN) { av_log(NULL, AV_LOG_FATAL, "Elementary stream #%d is of unknown type, cannot proceed\n", i); return AVERROR_INVALIDDATA; } else { /* if this stream must be remuxed */ ret = avcodec_parameters_copy(out_stream->codecpar, in_stream->codecpar); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Copying parameters for stream #%u failed\n", i); return ret; } out_stream->time_base = in_stream->time_base; } } av_dump_format(ofmt_ctx, 0, filename, 1); /**(封装 4.4):初始化AVIOContext*/ if (!(ofmt_ctx->oformat->flags & AVFMT_NOFILE)) { ret = avio_open(&ofmt_ctx->pb, filename, AVIO_FLAG_WRITE); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Could not open output file '%s'", filename); return ret; } } /**(封装 4.5):写入文件头*/ /* init muxer, write output file header */ ret = avformat_write_header(ofmt_ctx, NULL); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Error occurred when opening output file\n"); return ret; } return 0; } static int init_fifo(AVAudioFifo** fifo, AVCodecContext* output_codec_context) { /* Create the FIFO buffer based on the specified output sample format. */ if (!(*fifo = av_audio_fifo_alloc(output_codec_context->sample_fmt, output_codec_context->channels, 1))) { fprintf(stderr, "Could not allocate FIFO\n"); return AVERROR(ENOMEM); } return 0; } static int init_filter(FilteringContext* fctx, AVCodecContext* dec_ctx, AVCodecContext* enc_ctx, const char* filter_spec) { char args[512]; int ret = 0; const AVFilter* buffersrc = NULL; const AVFilter* buffersink = NULL; AVFilterContext* buffersrc_ctx = NULL; AVFilterContext* buffersink_ctx = NULL; AVFilterInOut* outputs = avfilter_inout_alloc(); AVFilterInOut* inputs = avfilter_inout_alloc(); AVFilterGraph* filter_graph = avfilter_graph_alloc(); if (!outputs || !inputs || !filter_graph) { ret = AVERROR(ENOMEM); goto end; } if (dec_ctx->codec_type == AVMEDIA_TYPE_VIDEO) { /**(滤镜 6.1):获取输入和输出滤镜器【同音频】*/ buffersrc = avfilter_get_by_name("buffer"); buffersink = avfilter_get_by_name("buffersink"); if (!buffersrc || !buffersink) { av_log(NULL, AV_LOG_ERROR, "filtering source or sink element not found\n"); ret = AVERROR_UNKNOWN; goto end; } snprintf(args, sizeof(args), "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d", dec_ctx->width, dec_ctx->height, dec_ctx->pix_fmt, dec_ctx->time_base.num, dec_ctx->time_base.den, dec_ctx->sample_aspect_ratio.num, dec_ctx->sample_aspect_ratio.den); /**(滤镜 6.2):创建和初始化输入和输出过滤器实例并将其添加到现有图形中*/ ret = avfilter_graph_create_filter(&buffersrc_ctx, buffersrc, "in", args, NULL, filter_graph); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot create buffer source\n"); goto end; } ret = avfilter_graph_create_filter(&buffersink_ctx, buffersink, "out", NULL, NULL, filter_graph); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot create buffer sink\n"); goto end; } /**(滤镜 6.3):给【输出】滤镜器上下文设置参数*/ ret = av_opt_set_bin(buffersink_ctx, "pix_fmts", (uint8_t*)&enc_ctx->pix_fmt, sizeof(enc_ctx->pix_fmt), AV_OPT_SEARCH_CHILDREN); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot set output pixel format\n"); goto end; } } else if (dec_ctx->codec_type == AVMEDIA_TYPE_AUDIO) { buffersrc = avfilter_get_by_name("abuffer"); buffersink = avfilter_get_by_name("abuffersink"); if (!buffersrc || !buffersink) { av_log(NULL, AV_LOG_ERROR, "filtering source or sink element not found\n"); ret = AVERROR_UNKNOWN; goto end; } if (!dec_ctx->channel_layout) dec_ctx->channel_layout = av_get_default_channel_layout(dec_ctx->channels); snprintf(args, sizeof(args), "time_base=%d/%d:sample_rate=%d:sample_fmt=%s:channel_layout=0x%x", dec_ctx->time_base.num, dec_ctx->time_base.den, dec_ctx->sample_rate, av_get_sample_fmt_name(dec_ctx->sample_fmt), (int)dec_ctx->channel_layout); ret = avfilter_graph_create_filter(&buffersrc_ctx, buffersrc, "in", args, NULL, filter_graph); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot create audio buffer source\n"); goto end; } ret = avfilter_graph_create_filter(&buffersink_ctx, buffersink, "out", NULL, NULL, filter_graph); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot create audio buffer sink\n"); goto end; } ret = av_opt_set_bin(buffersink_ctx, "sample_fmts", (uint8_t*)&enc_ctx->sample_fmt, sizeof(enc_ctx->sample_fmt), AV_OPT_SEARCH_CHILDREN); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot set output sample format\n"); goto end; } ret = av_opt_set_bin(buffersink_ctx, "channel_layouts", (uint8_t*)&enc_ctx->channel_layout, sizeof(enc_ctx->channel_layout), AV_OPT_SEARCH_CHILDREN); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot set output channel layout\n"); goto end; } ret = av_opt_set_bin(buffersink_ctx, "sample_rates", (uint8_t*)&enc_ctx->sample_rate, sizeof(enc_ctx->sample_rate), AV_OPT_SEARCH_CHILDREN); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Cannot set output sample rate\n"); goto end; } } else { ret = AVERROR_UNKNOWN; goto end; } //绑定关系 in ——> buffersrc_ctx /* Endpoints for the filter graph. */ outputs->name = av_strdup("in"); outputs->filter_ctx = buffersrc_ctx; outputs->pad_idx = 0; outputs->next = NULL; //绑定关系 out ——> buffersink_ctx inputs->name = av_strdup("out"); inputs->filter_ctx = buffersink_ctx; inputs->pad_idx = 0; inputs->next = NULL; if (!outputs->name || !inputs->name) { ret = AVERROR(ENOMEM); goto end; } /**(滤镜 6.4):将字符串描述的图形添加到图形中*/ if ((ret = avfilter_graph_parse_ptr(filter_graph, filter_spec, &inputs, &outputs, NULL)) < 0) goto end; /**(滤镜 6.5):检查AVFilterGraph有效性*/ if ((ret = avfilter_graph_config(filter_graph, NULL)) < 0) goto end; /* Fill FilteringContext */ fctx->buffersrc_ctx = buffersrc_ctx; fctx->buffersink_ctx = buffersink_ctx; fctx->filter_graph = filter_graph; end: avfilter_inout_free(&inputs); avfilter_inout_free(&outputs); return ret; } static int init_filters(void) { const char* filter_spec; unsigned int i; int ret; filter_ctx = (FilteringContext*)av_malloc_array(ifmt_ctx->nb_streams, sizeof(*filter_ctx)); if (!filter_ctx) return AVERROR(ENOMEM); //这里会根据音频和视频的stream_index创建对应的filter_stm组 for (i = 0; i < ifmt_ctx->nb_streams; i++) { filter_ctx[i].buffersrc_ctx = NULL; filter_ctx[i].buffersink_ctx = NULL; filter_ctx[i].filter_graph = NULL; if (!(ifmt_ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_AUDIO || ifmt_ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO)) continue; if (ifmt_ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) filter_spec = "null"; /* passthrough (dummy) filter for video */ else filter_spec = "anull"; /* passthrough (dummy) filter for audio */ ret = init_filter(&filter_ctx[i], stream_ctx[i].dec_ctx, stream_ctx[i].enc_ctx, filter_spec); if (ret) return ret; filter_ctx[i].enc_pkt = av_packet_alloc(); if (!filter_ctx[i].enc_pkt) return AVERROR(ENOMEM); filter_ctx[i].filtered_frame = av_frame_alloc(); if (!filter_ctx[i].filtered_frame) return AVERROR(ENOMEM); } return 0; } static int add_samples_to_fifo(AVAudioFifo* fifo, uint8_t** converted_input_samples, const int frame_size) { int error = 0; /* Make the FIFO as large as it needs to be to hold both, * the old and the new samples. */ if ((error = av_audio_fifo_realloc(fifo, av_audio_fifo_size(fifo) + frame_size)) < 0) { fprintf(stderr, "Could not reallocate FIFO\n"); return error; } /* Store the new samples in the FIFO buffer. */ if (av_audio_fifo_write(fifo, (void**)converted_input_samples, frame_size) < frame_size) { fprintf(stderr, "Could not write data to FIFO\n"); return AVERROR_EXIT; } return 0; } static int store_audio( AVAudioFifo* fifo, const AVFrame* input_frame) { int ret = 0; /* Add the converted input samples to the FIFO buffer for later processing. */ // 写入FIFO缓冲区 ret = add_samples_to_fifo( fifo, (uint8_t**)input_frame->data, input_frame->nb_samples); return ret; } static int init_output_frame(AVFrame** frame, AVCodecContext* output_codec_context, int frame_size) { int error; /* Create a new frame to store the audio samples. */ if (!(*frame = av_frame_alloc())) { fprintf(stderr, "Could not allocate output frame\n"); return AVERROR_EXIT; } /* Set the frame's parameters, especially its size and format. * av_frame_get_buffer needs this to allocate memory for the * audio samples of the frame. * Default channel layouts based on the number of channels * are assumed for simplicity. */ (*frame)->nb_samples = frame_size; (*frame)->channel_layout = output_codec_context->channel_layout; (*frame)->format = output_codec_context->sample_fmt; (*frame)->sample_rate = output_codec_context->sample_rate; /* Allocate the samples of the created frame. This call will make * sure that the audio frame can hold as many samples as specified. */ if ((error = av_frame_get_buffer(*frame, 0)) < 0) { fprintf(stderr, "Could not allocate output frame samples (error '%s')\n", av_err2str(error)); av_frame_free(frame); return error; } return 0; } static int init_packet(AVPacket** packet) { if (!(*packet = av_packet_alloc())) { fprintf(stderr, "Could not allocate packet\n"); return AVERROR(ENOMEM); } return 0; } static int encode_audio_frame(AVFrame* frame, AVFormatContext* output_format_context, AVCodecContext* output_codec_context, int* data_present) { /* Packet used for temporary storage. */ AVPacket* output_packet; int error; error = init_packet(&output_packet); if (error < 0) return error; /* Set a timestamp based on the sample rate for the container. */ if (frame) { frame->pts = current_audio_pts; frame->pkt_dts = current_audio_pts; current_audio_pts += output_codec_context->frame_size; } /* Send the audio frame stored in the temporary packet to the encoder. * The output audio stream encoder is used to do this. */ error = avcodec_send_frame(output_codec_context, frame); /* The encoder signals that it has nothing more to encode. */ if (error == AVERROR_EOF) { error = 0; goto cleanup; } else if (error < 0) { fprintf(stderr, "Could not send packet for encoding (error '%s')\n", av_err2str(error)); goto cleanup; } cleanup: av_packet_free(&output_packet); return error; } int encode_and_write(AVAudioFifo* fifo, AVFormatContext* output_format_context, AVCodecContext* output_codec_context) { /* Temporary storage of the output samples of the frame written to the file. */ AVFrame* output_frame; /* Use the maximum number of possible samples per frame. * If there is less than the maximum possible frame size in the FIFO * buffer use this number. Otherwise, use the maximum possible frame size. */ const int frame_size = FFMIN(av_audio_fifo_size(fifo), output_codec_context->frame_size); int data_written; /* Initialize temporary storage for one output frame. */ if (init_output_frame(&output_frame, output_codec_context, frame_size)) return AVERROR_EXIT; /* Read as many samples from the FIFO buffer as required to fill the frame. * The samples are stored in the frame temporarily. */ if (av_audio_fifo_read(fifo, (void**)output_frame->data, frame_size) < frame_size) { fprintf(stderr, "Could not read data from FIFO\n"); av_frame_free(&output_frame); return AVERROR_EXIT; } // 保存音频用于验证(Fload 32bit) #ifdef SAVE_AUDIO_FILE save_audio_data(output_frame); #endif /* Encode one frame worth of audio samples. */ if (encode_audio_frame(output_frame, output_format_context, output_codec_context, &data_written)) { av_frame_free(&output_frame); return AVERROR_EXIT; } av_frame_free(&output_frame); return 0; } static int encode_write_frame(unsigned int stream_index, int flush) { StreamContext* stream = &stream_ctx[stream_index]; FilteringContext* filter = &filter_ctx[stream_index]; AVFrame* filt_frame = flush ? NULL : filter->filtered_frame; AVPacket* enc_pkt = filter->enc_pkt; AVFrame* reasampling_frame = NULL; const int enc_frame_size = stream->enc_ctx->frame_size; int ret; //av_log(NULL, AV_LOG_INFO, "Encoding frame\n"); /* encode filtered frame */ av_packet_unref(enc_pkt); /**(编码 3.5):把滤镜处理后的AVFrame送去编码*/ #if 0 if (filt_frame) { if (stream_index == audio_index) { // 简单处理,会有破音 filt_frame->nb_samples = 1024; filt_frame->pts = current_audio_pts; filt_frame->pkt_dts = current_audio_pts; current_audio_pts += stream->enc_ctx->frame_size; } else { filt_frame->pts = current_video_pts; filt_frame->pkt_dts = current_video_pts; current_video_pts += stream->enc_ctx->ticks_per_frame; } } ret = avcodec_send_frame(stream->enc_ctx, filt_frame); if (ret < 0) { return ret; } #else //当音频样本数不满足预期时,需要重采样再进行输出 if (stream_index == audio_index && filt_frame && filt_frame->nb_samples != stream->enc_ctx->frame_size) { // 写入音频至队列 ret = store_audio( fifo, filt_frame); if (ret < 0) { return ret; } // 从队列中读取音频 while (1) { int fifo_size = av_audio_fifo_size(fifo); if (fifo_size < enc_frame_size) { break; } ret = encode_and_write( fifo, ofmt_ctx, stream_ctx[stream_index].enc_ctx); if (ret < 0) { return ret; } } } else { if (filt_frame) { if (stream_index == audio_index) { filt_frame->pts = current_audio_pts; filt_frame->pkt_dts = current_audio_pts; current_audio_pts += stream->enc_ctx->frame_size; } else { filt_frame->pts = current_video_pts; filt_frame->pkt_dts = current_video_pts; current_video_pts += stream->enc_ctx->ticks_per_frame; if (decode_first_video) { decode_first_video = false; int sample_rate = stream_ctx[audio_index].enc_ctx->sample_rate; int frame_rate = stream->enc_ctx->framerate.num; int ticks_per_frame = stream->enc_ctx->ticks_per_frame; int addition_time = 0; if (sample_rate > 0) { addition_time = (int)(current_audio_pts * 1000.0f / sample_rate); } int addition_ticks = 0; if (frame_rate > 0) { if (ticks_per_frame <= 1) { addition_ticks = (int)(addition_time / frame_rate / 2); } else { addition_ticks = (int)(addition_time / frame_rate / ticks_per_frame); } } current_video_pts += addition_ticks; } } } /**(编码 3.5):把滤镜处理后的AVFrame送去编码*/ ret = avcodec_send_frame(stream->enc_ctx, filt_frame); } #endif while (ret >= 0) { /**(编码 3.6):从编码器中得到编码后数据,放入AVPacket中*/ ret = avcodec_receive_packet(stream->enc_ctx, enc_pkt); if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) { return 0; } printf("write1 %s Packet. size:%5d\tdts:%5lld\tpts:%5lld\tduration:%5lld\tcur_dts:%5lld\n", stream_index == AVMEDIA_TYPE_AUDIO ? "a>>>>>" : "v-----", enc_pkt->size, enc_pkt->dts, enc_pkt->pts, enc_pkt->duration, ofmt_ctx->streams[stream_index]->cur_dts); /* prepare packet for muxing */ //设置pts等信息 enc_pkt->stream_index = stream_index; av_packet_rescale_ts(enc_pkt, stream->enc_ctx->time_base, ofmt_ctx->streams[stream_index]->time_base); enc_pkt->pos = -1; //av_log(NULL, AV_LOG_DEBUG, "Muxing frame\n"); printf("write2 %s Packet. size:%5d\tdts:%5lld\tpts:%5lld\tduration:%5lld\tcur_dts:%5lld\n", stream_index == AVMEDIA_TYPE_AUDIO ? "a>>>>>" : "v-----", enc_pkt->size, enc_pkt->dts, enc_pkt->pts, enc_pkt->duration, ofmt_ctx->streams[stream_index]->cur_dts); /* mux encoded frame */ ret = av_interleaved_write_frame(ofmt_ctx, enc_pkt); //擦除数据 av_packet_unref(enc_pkt); } return ret; } static int filter_encode_write_frame(AVFrame* frame, unsigned int stream_index) { FilteringContext* filter = &filter_ctx[stream_index]; int ret; //av_log(NULL, AV_LOG_INFO, "Pushing decoded frame to filters\n"); /* push the decoded frame into the filtergraph */ /**(滤镜 6.6):将解码后的AVFrame送去filtergraph进行滤镜处理*/ ret = av_buffersrc_add_frame_flags(filter->buffersrc_ctx, frame, 0); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n"); return ret; } /* pull filtered frames from the filtergraph */ while (1) { //av_log(NULL, AV_LOG_INFO, "Pulling filtered frame from filters\n"); /**(滤镜 6.7):得到滤镜处理后的数据*/ ret = av_buffersink_get_frame(filter->buffersink_ctx, filter->filtered_frame); if (ret < 0) { /* if no more frames for output - returns AVERROR(EAGAIN) * if flushed and no more frames for output - returns AVERROR_EOF * rewrite retcode to 0 to show it as normal procedure completion */ if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) ret = 0; break; } filter->filtered_frame->pict_type = AV_PICTURE_TYPE_NONE; //然后把滤镜处理后的数据重新进行编码成你想要的格式,再封装输出 ret = encode_write_frame(stream_index, 0); av_frame_unref(filter->filtered_frame); if (ret < 0) break; } return ret; } static int flush_encoder(unsigned int stream_index) { if (!(stream_ctx[stream_index].enc_ctx->codec->capabilities & AV_CODEC_CAP_DELAY)) return 0; av_log(NULL, AV_LOG_INFO, "Flushing stream #%u encoder\n", stream_index); return encode_write_frame(stream_index, 1); } int main(int argc, char** argv) { int ret; AVPacket* packet = NULL; unsigned int stream_index; unsigned int i; if (argc != 3) { av_log(NULL, AV_LOG_ERROR, "Usage: %s <input file> <output file>\n", argv[0]); return 1; } if ((ret = open_input_file(argv[1])) < 0) goto end; if ((ret = open_output_file(argv[2])) < 0) goto end; if ((ret = init_fifo( &fifo, stream_ctx[audio_index].enc_ctx)) < 0) goto end; if ((ret = init_filters()) < 0) goto end; if (!(packet = av_packet_alloc())) goto end; /* read all packets */ while (1) { /**(解封装 1.3):读取解封装后数据到AVPacket中*/ if ((ret = av_read_frame(ifmt_ctx, packet)) < 0) break; stream_index = packet->stream_index; av_log(NULL, AV_LOG_DEBUG, "Demuxer gave frame of stream_index %u\n", stream_index); if (filter_ctx[stream_index].filter_graph) { StreamContext* stream = &stream_ctx[stream_index]; av_log(NULL, AV_LOG_DEBUG, "Going to reencode&filter the frame\n"); av_packet_rescale_ts(packet, ifmt_ctx->streams[stream_index]->time_base, stream->dec_ctx->time_base); /**(解码 2.5):把AVPacket送去解码*/ ret = avcodec_send_packet(stream->dec_ctx, packet); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Decoding failed\n"); #if 0 break; #else continue; #endif } while (ret >= 0) { /**(解码 2.6):从解码器获取解码后的数据到AVFrame*/ ret = avcodec_receive_frame(stream->dec_ctx, stream->dec_frame); if (ret == AVERROR_EOF || ret == AVERROR(EAGAIN)) break; else if (ret < 0) goto end; stream->dec_frame->pts = stream->dec_frame->best_effort_timestamp; //这是解码后的裸数据,如果可以对其进行滤镜处理 ret = filter_encode_write_frame(stream->dec_frame, stream_index); if (ret < 0) goto end; } } else { /* remux this frame without reencoding */ av_packet_rescale_ts(packet, ifmt_ctx->streams[stream_index]->time_base, ofmt_ctx->streams[stream_index]->time_base); ret = av_interleaved_write_frame(ofmt_ctx, packet); if (ret < 0) goto end; } av_packet_unref(packet); } /* flush filters and encoders */ for (i = 0; i < ifmt_ctx->nb_streams; i++) { /* flush filter */ if (!filter_ctx[i].filter_graph) continue; ret = filter_encode_write_frame(NULL, i); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Flushing filter failed\n"); goto end; } /* flush encoder */ ret = flush_encoder(i); if (ret < 0) { av_log(NULL, AV_LOG_ERROR, "Flushing encoder failed\n"); goto end; } } /**(封装 4.7):写入文件尾*/ av_write_trailer(ofmt_ctx); end: if (packet) { av_packet_free(&packet); } if (ifmt_ctx) { for (i = 0; i < ifmt_ctx->nb_streams; i++) { avcodec_free_context(&stream_ctx[i].dec_ctx); if (ofmt_ctx && ofmt_ctx->nb_streams > i && ofmt_ctx->streams[i] && stream_ctx[i].enc_ctx) avcodec_free_context(&stream_ctx[i].enc_ctx); if (filter_ctx && filter_ctx[i].filter_graph) { avfilter_graph_free(&filter_ctx[i].filter_graph); av_packet_free(&filter_ctx[i].enc_pkt); av_frame_free(&filter_ctx[i].filtered_frame); } av_frame_free(&stream_ctx[i].dec_frame); } } if (filter_ctx) { av_free(filter_ctx); } if (stream_ctx) { av_free(stream_ctx); } if (fifo) { av_audio_fifo_free(fifo); } if (ifmt_ctx) { avformat_close_input(&ifmt_ctx); if (ofmt_ctx && !(ofmt_ctx->oformat->flags & AVFMT_NOFILE)) avio_closep(&ofmt_ctx->pb); avformat_free_context(ofmt_ctx); } if (ret < 0){ av_log(NULL, AV_LOG_ERROR, "Error occurred: %s\n", av_err2str(ret)); } return ret ? 1 : 0; }
05-16
### 动态切换多个输入流并维持单一输出流 要实现动态切换多个输入流并维持单一输出流的功能,可以按照以下方式扩展 FFmpeg 的 C++ 实现。以下是详细的解决方案: #### 输入流管理 为了支持动态切换输入流,在每次完成一个输入流的处理后,需释放当前资源并加载新的输入文件。这可以通过 `avformat_close_input` 和重新调用 `avformat_open_input` 来实现[^1]。 ```cpp void closeInputStream(AVFormatContext* &inputFormatContext, AVCodecContext* &videoCodecContext) { if (videoCodecContext) avcodec_free_context(&videoCodecContext); if (inputFormatContext) avformat_close_input(&inputFormatContext); } ``` 在循环中检测到某个输入流结束时(通过帧计数或其他条件),调用此函数关闭旧的输入流,并初始化下一个输入流。 #### 输出流保持不变 对于输出流,只需一次性创建即可。确保其生命周期覆盖整个程序运行时间。定义全局变量保存输出上下文及相关参数。 ```cpp AVFormatContext *outputFormatContext = nullptr; AVStream *outputVideoStream = nullptr; // 初始化输出流仅执行一次... if (!outputFormatContext || !outputVideoStream) { // 创建输出容器和视频流配置... } ``` 当切换输入源时无需重复设置输出端口属性,除非分辨率或编解码器发生变化,则需要调整对应选项再写入新数据包前同步更改至目标文件头信息里去更新这些差异之处[^2]。 #### 滤镜链维护 如果希望在整个过程中应用相同的滤镜效果,则应该构建独立于具体媒体片段之外的一般化过滤规则集;每当更换素材之后立即重建关联关系以便继续作用于后续画面之上而不中断视觉连续性体验。 ```cpp char args[512]; snprintf(args, sizeof(args), "scale=%d:%d:flags=bicubic", outputWidth, outputHeight); const char *filter_desc = std::string("nullsrc=size=") + std::to_string(outputWidth) + ":" + std::to_string(outputHeight) + "[base];" + "[base][main]overlay=x=0:y=0"; // 使用 filter_graph 配置通用滤镜描述符... ``` 注意这里假设所有输入都具有相同尺寸比例,实际项目可能更复杂因此建议加入更多自适应机制来满足不同情况下的需求[^3]。 --- ### 完整流程概述代码框架如下所示: ```cpp #include <libavutil/imgutils.h> #include <libavutil/opt.h> #include <libavcodec/avcodec.h> #include <libavformat/avformat.h> extern void processFrame(AVPacket packet); // 假设已存在用于处理单个packet的方法 std::vector<std::string> inputFiles {"file1.mp4", "file2.avi"}; // 多个待处理文件列表 size_t currentFileIndex = 0; while(currentFileIndex < inputFiles.size()) { const auto& inputFile = inputFiles[currentFileIndex]; AVFormatContext *inputFormatContext = nullptr; if(avformat_open_input(&inputFormatContext, inputFile.c_str(), NULL, NULL)!=0){ fprintf(stderr,"Could not open file %s\n", inputFile.c_str()); continue; } if(avformat_find_stream_info(inputFormatContext,NULL)<0){ fprintf(stderr,"Failed to retrieve stream information.\n"); closeInputStream(inputFormatContext, videoCodecContext); continue; } int videoStreamIndex = findVideoStreamIndex(inputFormatContext); if(videoStreamIndex<0){ fprintf(stderr,"No video stream found in the container.\n"); closeInputStream(inputFormatContext, videoCodecContext); continue; } AVCodecParameters *codecPar = inputFormatContext->streams[videoStreamIndex]->codecpar; AVCodec *decoder = avcodec_find_decoder(codecPar->codec_id); if(!decoder){ fprintf(stderr,"Unsupported codec!\n"); closeInputStream(inputFormatContext, videoCodecContext); continue; } AVCodecContext *videoCodecContext = avcodec_alloc_context3(decoder); if(avcodec_parameters_to_context(videoCodecContext, codecPar)<0){ fprintf(stderr,"Error copying codec parameters to decoder context.\n"); closeInputStream(inputFormatContext, videoCodecContext); continue; } if(avcodec_open2(videoCodecContext, decoder, NULL)<0){ fprintf(stderr,"Cannot open video decoder.\n"); closeInputStream(inputFormatContext, videoCodecContext); continue; } /* 开始读取frame */ while(true){ AVPacket pkt; int ret = av_read_frame(inputFormatContext,&pkt); if(ret==AVERROR_EOF){break;} if(pkt.stream_index == videoStreamIndex){ processFrame(pkt); } av_packet_unref(&pkt); } closeInputStream(inputFormatContext, videoCodecContext); ++currentFileIndex; } closeOutputStream(); // 关闭最终输出流 ``` 以上展示了基本结构,其中省略了一些细节比如错误检查以及具体的编码过程等内容。可以根据实际情况补充完善相应功能模块。 ---
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值