webrtc 视频流分析

本文详细介绍WebRTC视频流通讯的内部实现原理,包括接收流、流媒体帧渲染及流媒体入队等关键流程的调用堆栈。通过具体代码行号及函数名帮助读者深入理解WebRTC的工作机制。
webrtc 是google开源的web端流媒体通讯框架,通过客户端调用堆栈可以快速了解webrtc视频流通讯原理,特提供以下调用堆栈,仅供参考:

接收流调用堆栈(rtcp)
> peerconnection_client.exe!webrtc::RTCPReceiver::TriggerCallbacksFromRtcpPacket(const webrtc::RTCPReceiver::PacketInformation & packet_information) 行 912 C++
peerconnection_client.exe!webrtc::RTCPReceiver::IncomingPacket(const unsigned char * packet, unsigned __int64 packet_size) 行 147 C++
peerconnection_client.exe!webrtc::ModuleRtpRtcpImpl::IncomingRtcpPacket(const unsigned char * rtcp_packet, unsigned __int64 length) 行 275 C++
peerconnection_client.exe!webrtc::voe::Channel::ReceivedRTCPPacket(const unsigned char * data, unsigned __int64 length) 行 1823 C++
peerconnection_client.exe!webrtc::voe::ChannelProxy::ReceivedRTCPPacket(const unsigned char * packet, unsigned __int64 length) 行 236 C++
peerconnection_client.exe!webrtc::internal::AudioReceiveStream::DeliverRtcp(const unsigned char * packet, unsigned __int64 length) 行 313 C++
peerconnection_client.exe!webrtc::internal::Call::DeliverRtcp(webrtc::MediaType media_type, const unsigned char * packet, unsigned __int64 length) 行 1274 C++
peerconnection_client.exe!webrtc::internal::Call::DeliverPacket(webrtc::MediaType media_type, const unsigned char * packet, unsigned __int64 length, const webrtc::PacketTime & packet_time) 行 1377 C++
peerconnection_client.exe!cricket::WebRtcVoiceMediaChannel::OnRtcpReceived(rtc::CopyOnWriteBuffer * packet, const rtc::PacketTime & packet_time) 行 2173 C++
peerconnection_client.exe!cricket::BaseChannel::ProcessPacket(bool rtcp, const rtc::CopyOnWriteBuffer & packet, const rtc::PacketTime & packet_time) 行 826 C++
peerconnection_client.exe!rtc::MethodFunctor<cricket::BaseChannel, void (cricket::BaseChannel::*)(bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &), void, bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &>::CallMethod<0,1,2>(rtc::sequence<0, 1, 2>) 行 164 C++
peerconnection_client.exe!rtc::MethodFunctor<cricket::BaseChannel, void (cricket::BaseChannel::*)(bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &), void, bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &>::operator()() 行 155 C++
peerconnection_client.exe!rtc::FireAndForgetAsyncClosure<rtc::MethodFunctor<cricket::BaseChannel, void (cricket::BaseChannel::*)(bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &), void, bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &> >::Execute() 行 49 C++
peerconnection_client.exe!rtc::AsyncInvoker::OnMessage(rtc::Message * msg) 行 43 C++
peerconnection_client.exe!rtc::MessageQueue::Dispatch(rtc::Message * pmsg) 行 531 C++
peerconnection_client.exe!rtc::Thread::ProcessMessages(int cmsLoop) 行 489 C++
peerconnection_client.exe!rtc::Thread::Run() 行 318 C++
peerconnection_client.exe!rtc::Thread::PreRun() 行 307 C++

队列任务入队调用堆栈(本地采集部分)
> peerconnection_client.exe!rtc::TaskQueue::PostTask(std::unique_ptr<rtc::QueuedTask, std::default_delete<rtc::QueuedTask> > task) 行 233 C++
peerconnection_client.exe!webrtc::ViEEncoder::OnFrame(const webrtc::VideoFrame & video_frame) 行 721 C++
peerconnection_client.exe!rtc::VideoBroadcaster::OnFrame(const webrtc::VideoFrame & frame) 行 56 C++
peerconnection_client.exe!cricket::VideoCapturer::OnFrame(const webrtc::VideoFrame & frame, int orig_width, int orig_height) 行 223 C++
peerconnection_client.exe!cricket::WebRtcVideoCapturer::OnFrame(const webrtc::VideoFrame & sample) 行 351 C++
peerconnection_client.exe!webrtc::videocapturemodule::VideoCaptureImpl::DeliverCapturedFrame(webrtc::VideoFrame & captureFrame) 行 120 C++
peerconnection_client.exe!webrtc::videocapturemodule::VideoCaptureImpl::IncomingFrame(unsigned char * videoFrame, unsigned __int64 videoFrameLength, const webrtc::VideoCaptureCapability & frameInfo, __int64 captureTime) 行 182 C++
peerconnection_client.exe!webrtc::videocapturemodule::CaptureSinkFilter::ProcessCapturedFrame(unsigned char * pBuffer, unsigned __int64 length, const webrtc::VideoCaptureCapability & frameInfo) 行 478 C++
peerconnection_client.exe!webrtc::videocapturemodule::CaptureInputPin::Receive(IMediaSample * pIMediaSample) 行 348 C++

流媒体帧渲染堆栈(本地,由以上部分唤醒,事件变量名in_queue_)
> peerconnection_client.exe!MainWnd::VideoRenderer::OnFrame(const webrtc::VideoFrame & video_frame) 行 609 C++
peerconnection_client.exe!rtc::VideoBroadcaster::OnFrame(const webrtc::VideoFrame & frame) 行 56 C++
peerconnection_client.exe!cricket::VideoCapturer::OnFrame(const webrtc::VideoFrame & frame, int orig_width, int orig_height) 行 223 C++
peerconnection_client.exe!cricket::WebRtcVideoCapturer::OnFrame(const webrtc::VideoFrame & sample) 行 351 C++
peerconnection_client.exe!webrtc::videocapturemodule::VideoCaptureImpl::DeliverCapturedFrame(webrtc::VideoFrame & captureFrame) 行 120 C++
peerconnection_client.exe!webrtc::videocapturemodule::VideoCaptureImpl::IncomingFrame(unsigned char * videoFrame, unsigned __int64 videoFrameLength, const webrtc::VideoCaptureCapability & frameInfo, __int64 captureTime) 行 182 C++
peerconnection_client.exe!webrtc::videocapturemodule::CaptureSinkFilter::ProcessCapturedFrame(unsigned char * pBuffer, unsigned __int64 length, const webrtc::VideoCaptureCapability & frameInfo) 行 478 C++
peerconnection_client.exe!webrtc::videocapturemodule::CaptureInputPin::Receive(IMediaSample * pIMediaSample) 行 348 C++

接收流调用堆栈(rtp,通过new_continuous_frame_event_唤醒解码线程)
> peerconnection_client.exe!webrtc::video_coding::FrameBuffer::InsertFrame(std::unique_ptr<webrtc::video_coding::FrameObject, std::default_delete<webrtc::video_coding::FrameObject> > frame) 行 359 C++
peerconnection_client.exe!webrtc::internal::VideoReceiveStream::OnCompleteFrame(std::unique_ptr<webrtc::video_coding::FrameObject, std::default_delete<webrtc::video_coding::FrameObject> > frame) 行 441 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::OnCompleteFrame(std::unique_ptr<webrtc::video_coding::FrameObject, std::default_delete<webrtc::video_coding::FrameObject> > frame) 行 425 C++
peerconnection_client.exe!webrtc::video_coding::RtpFrameReferenceFinder::ManageFrame(std::unique_ptr<webrtc::video_coding::RtpFrameObject, std::default_delete<webrtc::video_coding::RtpFrameObject> > frame) 行 52 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::OnReceivedFrame(std::unique_ptr<webrtc::video_coding::RtpFrameObject, std::default_delete<webrtc::video_coding::RtpFrameObject> > frame) 行 414 C++
peerconnection_client.exe!webrtc::video_coding::PacketBuffer::InsertPacket(webrtc::VCMPacket * packet) 行 122 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::OnReceivedPayloadData(const unsigned char * payload_data, unsigned __int64 payload_size, const webrtc::WebRtcRTPHeader * rtp_header) 行 283 C++
peerconnection_client.exe!webrtc::RTPReceiverVideo::ParseRtpPacket(webrtc::WebRtcRTPHeader * rtp_header, const webrtc::PayloadUnion & specific_payload, bool is_red, const unsigned char * payload, unsigned __int64 payload_length, __int64 timestamp_ms, bool is_first_packet) 行 116 C++
peerconnection_client.exe!webrtc::RtpReceiverImpl::IncomingRtpPacket(const webrtc::RTPHeader & rtp_header, const unsigned char * payload, unsigned __int64 payload_length, webrtc::PayloadUnion payload_specific, bool in_order) 行 182 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::ReceivePacket(const unsigned char * packet, unsigned __int64 packet_length, const webrtc::RTPHeader & header, bool in_order) 行 481 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::OnRecoveredPacket(const unsigned char * rtp_packet, unsigned __int64 rtp_packet_length) 行 298 C++
peerconnection_client.exe!webrtc::UlpfecReceiverImpl::ProcessReceivedFec() 行 228 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::ParseAndHandleEncapsulatingHeader(const unsigned char * packet, unsigned __int64 packet_length, const webrtc::RTPHeader & header) 行 500 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::ReceivePacket(const unsigned char * packet, unsigned __int64 packet_length, const webrtc::RTPHeader & header, bool in_order) 行 471 C++
peerconnection_client.exe!webrtc::RtpVideoStreamReceiver::OnRtpPacket(const webrtc::RtpPacketReceived & packet) 行 365 C++
peerconnection_client.exe!webrtc::RtpDemuxer::OnRtpPacket(const webrtc::RtpPacketReceived & packet) 行 68 C++
peerconnection_client.exe!webrtc::RtpStreamReceiverController::OnRtpPacket(const webrtc::RtpPacketReceived & packet) 行 43 C++
peerconnection_client.exe!webrtc::internal::Call::DeliverRtp(webrtc::MediaType media_type, const unsigned char * packet, unsigned __int64 length, const webrtc::PacketTime & packet_time) 行 1352 C++
peerconnection_client.exe!webrtc::internal::Call::DeliverPacket(webrtc::MediaType media_type, const unsigned char * packet, unsigned __int64 length, const webrtc::PacketTime & packet_time) 行 1379 C++
peerconnection_client.exe!cricket::WebRtcVideoChannel::OnPacketReceived(rtc::CopyOnWriteBuffer * packet, const rtc::PacketTime & packet_time) 行 1362 C++
peerconnection_client.exe!cricket::BaseChannel::ProcessPacket(bool rtcp, const rtc::CopyOnWriteBuffer & packet, const rtc::PacketTime & packet_time) 行 829 C++
peerconnection_client.exe!rtc::MethodFunctor<cricket::BaseChannel, void (cricket::BaseChannel::*)(bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &), void, bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &>::CallMethod<0,1,2>(rtc::sequence<0, 1, 2>) 行 164 C++
peerconnection_client.exe!rtc::MethodFunctor<cricket::BaseChannel, void (cricket::BaseChannel::*)(bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &), void, bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &>::operator()() 行 155 C++
peerconnection_client.exe!rtc::FireAndForgetAsyncClosure<rtc::MethodFunctor<cricket::BaseChannel, void (cricket::BaseChannel::*)(bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &), void, bool, const rtc::CopyOnWriteBuffer &, const rtc::PacketTime &> >::Execute() 行 49 C++
peerconnection_client.exe!rtc::AsyncInvoker::OnMessage(rtc::Message * msg) 行 43 C++
peerconnection_client.exe!rtc::MessageQueue::Dispatch(rtc::Message * pmsg) 行 531 C++
peerconnection_client.exe!rtc::Thread::ProcessMessages(int cmsLoop) 行 489 C++
peerconnection_client.exe!rtc::Thread::Run() 行 318 C++
peerconnection_client.exe!rtc::Thread::PreRun() 行 307 C++

流媒体入队,通知任务调用堆栈(远程媒体流部分,由以上部分唤醒,事件变量名new_continuous_frame_event_,通过in_queue_唤醒渲染线程)
> peerconnection_client.exe!rtc::TaskQueue::PostTask(std::unique_ptr<rtc::QueuedTask, std::default_delete<rtc::QueuedTask> > task) 行 233 C++
peerconnection_client.exe!webrtc::IncomingVideoStream::OnFrame(const webrtc::VideoFrame & video_frame) 行 66 C++
peerconnection_client.exe!webrtc::VideoStreamDecoder::FrameToRender(webrtc::VideoFrame & video_frame, rtc::Optional<unsigned char> qp, webrtc::VideoContentType content_type) 行 84 C++
peerconnection_client.exe!webrtc::VCMDecodedFrameCallback::Decoded(webrtc::VideoFrame & decodedImage, rtc::Optional<int> decode_time_ms, rtc::Optional<unsigned char> qp) 行 148 C++
peerconnection_client.exe!webrtc::VP8DecoderImpl::ReturnFrame(const vpx_image * img, unsigned int timestamp, __int64 ntp_time_ms, int qp) 行 1129 C++
peerconnection_client.exe!webrtc::VP8DecoderImpl::Decode(const webrtc::EncodedImage & input_image, bool missing_frames, const webrtc::RTPFragmentationHeader * fragmentation, const webrtc::CodecSpecificInfo * codec_specific_info, __int64) 行 1083 C++
peerconnection_client.exe!webrtc::VCMGenericDecoder::Decode(const webrtc::VCMEncodedFrame & frame, __int64 nowMs) 行 230 C++
peerconnection_client.exe!webrtc::vcm::VideoReceiver::Decode(const webrtc::VCMEncodedFrame & frame) 行 324 C++
peerconnection_client.exe!webrtc::vcm::VideoReceiver::Decode(const webrtc::VCMEncodedFrame * frame) 行 291 C++
peerconnection_client.exe!webrtc::internal::VideoReceiveStream::Decode() 行 505 C++
peerconnection_client.exe!webrtc::internal::VideoReceiveStream::DecodeThreadFunction() 行 487 C++
peerconnection_client.exe!rtc::PlatformThread::Run() 行 223 C++
peerconnection_client.exe!rtc::PlatformThread::StartThread() 行 135 C++

流媒体帧渲染调用堆栈(出队动作,由以上部分唤醒,事件变量名:in_queue_)
> peerconnection_client.exe!MainWnd::VideoRenderer::OnFrame(const webrtc::VideoFrame & video_frame) 行 609 C++
peerconnection_client.exe!rtc::VideoBroadcaster::OnFrame(const webrtc::VideoFrame & frame) 行 56 C++
peerconnection_client.exe!cricket::WebRtcVideoChannel::WebRtcVideoReceiveStream::OnFrame(const webrtc::VideoFrame & frame) 行 2409 C++
peerconnection_client.exe!webrtc::internal::VideoReceiveStream::OnFrame(const webrtc::VideoFrame & video_frame) 行 398 C++
peerconnection_client.exe!webrtc::IncomingVideoStream::Dequeue() 行 77 C++
peerconnection_client.exe!webrtc::IncomingVideoStream::Dequeue::<unnamed-tag>::operator()() 行 79 C++
peerconnection_client.exe!rtc::ClosureTask<(lambda at ../../webrtc/common_video/incoming_video_stream.cc:79:44)>::Run() 行 66 C++
peerconnection_client.exe!rtc::`anonymous namespace'::DelayedTaskInfo::Run() 行 101 C++
peerconnection_client.exe!rtc::TaskQueue::ThreadState::RunDueTasks() 行 391 C++
peerconnection_client.exe!rtc::TaskQueue::ThreadState::RunThreadMain() 行 319 C++
peerconnection_client.exe!rtc::TaskQueue::ThreadMain() 行 296 C++
peerconnection_client.exe!rtc::PlatformThread::Run() 行 223 C++
peerconnection_client.exe!rtc::PlatformThread::StartThread() 行 135 C++


在Unity中播放WebRTC视频流可以通过多种方法实现,具体取决于项目需求、开发资源以及对性能的要求。以下是一些可行的实现方式: ### 使用WebRTC for Unity插件 WebRTC for Unity是一个专为Unity开发者设计的软件包,允许直接在Unity环境中集成WebRTC技术[^3]。通过该插件,可以实现媒体流的捕获、编码、传输、解码以及渲染。以下是实现的基本步骤: 1. **导入插件**:将WebRTC for Unity插件导入Unity项目中。 2. **初始化PeerConnection**:创建并配置`RTCPeerConnection`对象,用于管理点对点连接。 3. **处理信令**:实现信令机制,通常需要一个信令服务器来交换SDP(会话描述协议)和ICE(交互式连接建立)候选信息。 4. **接收视频流**:通过`OnTrack`事件监听远程视频流,并将其绑定到Unity中的`VideoStreamReceiver`组件。 5. **渲染视频流**:将接收到的视频流渲染到Unity的UI组件或三维场景中。 ### 使用WebViewForWindow插件 由于在Unity内部直接实现WebRTC存在一定的技术壁垒,一种替代方案是借助前端技术来处理WebRTC流的播放[^2]。这种方法通过将Web页面嵌入Unity UI中实现视频流播放: 1. **前端开发**:由前端开发人员编写HTML和JavaScript代码,实现WebRTC视频流播放功能。 2. **导入WebViewForWindow插件**:将该插件导入Unity项目中,利用其API加载前端页面到Unity UI面板或三维场景中。 3. **加载网页**:调用插件API加载包含WebRTC视频流播放功能的HTML页面,用户可以在Unity中与该页面进行交互。 ### 手动构建WebRTC流传输解决方案 对于希望从零开始构建WebRTC流传输解决方案的开发者,可以按照以下步骤操作: 1. **配置Unity Render Streaming设置**:进入`Edit > Project Settings > Render Streaming`,创建新的设置资产,并禁用`Automatic Streaming`选项。 2. **添加必要组件**:在Unity场景中选择主摄像机(Main Camera),添加`Signaling Manager`、`Broadcast`和`Video Stream Sender`组件。将`Broadcast`组件拖拽到`Signaling Manager`的`Handler List`中,同时将`Video Stream Sender`组件绑定到`Broadcast`组件的`Streams`属性中。 3. **启动信令服务**:通过信令服务器交换必要的连接信息,确保流传输的正常进行。 4. **测试播放**:在浏览器中运行测试,验证是否能够正确播放Unity场景中的视频流。 ### 示例代码 以下是一个简单的示例代码片段,展示如何在Unity中初始化一个`RTCPeerConnection`对象: ```csharp using UnityEngine; using Unity.WebRTC; public class WebRTCExample : MonoBehaviour { private RTCPeerConnection peerConnection; void Start() { // 初始化RTCPeerConnection peerConnection = new RTCPeerConnection(null); // 添加事件监听器 peerConnection.OnIceCandidate = candidate => { // 处理ICE候选信息 }; peerConnection.OnTrack = e => { // 处理接收到的视频流 if (e.Track.Kind == TrackKind.Video) { // 将视频流绑定到VideoStreamReceiver VideoStreamReceiver receiver = GetComponent<VideoStreamReceiver>(); receiver.Receivers.Add(e.Track); } }; // 创建offer RTCSessionDescription offer = peerConnection.CreateOffer(); peerConnection.SetLocalDescription(ref offer); } } ``` ### 注意事项 - 在选择实现方法时,需要综合考虑项目的复杂度、开发团队的技术栈以及对实时性的要求。 - 如果选择使用WebRTC for Unity插件,务必仔细阅读官方文档,确保正确配置和使用插件。 - 对于使用WebViewForWindow插件的方法,需要确保前端代码的兼容性和稳定性,以便在Unity中顺利运行。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值