当前位置:网站首页>浅析云控平台画面传输的视频流方案

浅析云控平台画面传输的视频流方案

2020-11-10 14:21:41 InfoQ

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ARC(高德车机云控平台)是一个基于车载设备业务深度定制的云控平台,通过该平台我们能够实现远程使用不同类型的车载设备。为了让远程使用者像在本地一样使用车载设备,需要将车载设备的画面及时的传回给使用者。因此,画面传输能力是ARC平台的一个核心组件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"起初我们采用行业内普遍在用的画面传输开源方案(minicap)。该方案获取到屏幕数据后压缩生成JPG图像,逐帧传输到Web端进行展示。由于车机性能比手机差很多,压缩图片消耗CPU性能大,在部分低端车机设备上压缩图片能消耗80%左右的CPU,容易使设备使用出现卡顿。同时图像压缩率不算很高,传输消耗带宽大,在低带宽下造成用户看到的画面过度延迟。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此,我们需要一个解决方案能够平衡传回的画面质量和车机端的CPU资源消耗。本文将小结本次云控平台画面传输的视频流方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ee\/3c\/eea0d7b3957e58271e5c18930d0f753c.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"思路方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/37\/ac\/37b3c0e7bdf66e613e371e97552ecfac.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于图像数据的基本传输链路,为了能够不消耗设备端CPU资源,首先想到了图像不进行压缩,先传输到服务端进行处理。但是经过调研,车机的USB带宽传输根本无法满足高清图像不压缩进行传输,高清原始数据非常大,基本1秒只能传输三帧左右的数据。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一个思路是采用设备端的硬件编码器减少CPU资源的消耗。经过调研Android 4.1开始基本都自带了H264视频编码器。因此,决定尝试采用视频流的方案,在设备端通过硬件编码器编码成视频流,通过服务端转发到Web端进行解码展示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"实现方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/84\/c7\/84476946c312d930334381629d99dac7.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个实现方案可以分为以下三个部分:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"设备端"},{"type":"text","text":" :负责画面的获取和编码。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"服务端"},{"type":"text","text":" :处理视频流的传输和控制。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Web端"},{"type":"text","text":" :视频流的解码和展示。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"画面的获取和编码"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"图像画面的获取直接采用的是Android的Virtual Display。编码方式有多种实现方法:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/56\/79\/56b448010f9a05bb9be31d7bbc270c79.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于Java方案只能支持Android 5.0以上机器, 而目前车载市场Android 4.x的占比还比较大,无法忽略。因此只能使用cpp的方案,最低可兼容Android 4.3版本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"视频流的传输和控制"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Web端最常见的直播方案是rtmp\/hls\/flvjs等。但是这些方案最低都有1-3s的延迟。对于一般直播平台没有影响,但是对于有实时交互场景的云控平台,要求达到毫秒级延迟。所以,最终决定采用H264裸流通过Socket传输的方案,设备端编码H264视频流直接传输到Web端进行播放。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同时,为了提高使用体验,对视频流的传输增加了弹性控制。通过在服务端加入缓存队列用来监控前端带宽负载情况,根据带宽状况自动调节帧率和码率,优先保证使用者的流畅感。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Web端展示和解码"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Web端展示使用media source extensiton(MSE) + fragment mp4的方案, 把H264裸流封装成fragment mp4后,通过MSE api进行解码播放, 具体实现是参考了开源的Jmuxer方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"丢帧和补帧"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"默认情况下Android Virtual Display产生的最大帧率是60fps,而我们肉眼30fps就能感觉流畅。为了能够节省带宽,我们定义了视频流最大输出帧率是30fps。在网络带宽较差的情况下,我们还能够降低帧率来最大限度的避免延迟。同时,Android MediaCodec不支持控制帧率,帧率是由每秒送入的帧数量决定的。因此,我们需要通过实现丢帧来进行帧率控制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Win7硬件解码器没有低延迟模式,需要大概10帧左右数据才能开始播放,而VirtualDisplay是画面有变化才会产生图像帧,因此需要实现补帧来消除解码延迟。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/37\/29\/37716ba50055235a60dd42ff6370a429.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们通过创建一个EglSurface对丢帧补帧进行处理,通过时间间隔控制eglTexture绘入EglSurface的频度进行丢帧,通过重复绘入最后一帧数据进行补帧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"该方案在ARC平台上的使用,在保证传输质量的同时,有效的提升了使用者操作的流畅感。该方案理论上也可以应用于其他类似的云控平台上,如果不需要支持Android 4.x设备,采用Java层API来获取视频流数据,则可以降低开发和适配成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文转载自公众号高德技术(ID:amap_tech)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文链接"},{"type":"text","text":":"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?_biz=Mzg4MzIwMDM5Ng==&mid=2247485496&idx=2&sn=96b826fc5ea114784ef86fe017737ad5&chksm=cf4a50dbf83dd9cdfc880252e191339878fa3a696e8dc5009a45de09432c203f1475965ec7e4&scene=27#wechat<\/i>redirect","title":"","type":null},"content":[{"type":"text","text":"浅析云控平台画面传输的视频流方案"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}

版权声明
本文为[InfoQ]所创,转载请带上原文链接,感谢
https://www.infoq.cn/article/xtjVLMywqpRP1CkTF8QM?utm_source=rss&utm_medium=article