brief introduction ： 2020 Tmall double 11 In the studio , The platform must be in 1 Seconds will be the host voice 、 Pictures and product information are synchronized to millions of consumers with a wide range of distribution , Make sure the latter gets consistent 、 In real time 、 High level audio and video experience , And commodity trading （ Especially the second kill ） Credibility of . In order to further optimize the consumer experience , Reduce delay , Alibaba attaches great importance to the investment in the field of audio and video technology . This article from the latest international video standards VVC（Versatile Video Coding, Multifunctional image coding ） Speak up , Share the panorama of live video industry and Ali's innovation , And the Dharma academy and Taobao team jointly developed VVC Codec related work .
A brief history of international video standards
The picture above shows two major international video standards organizations ： International Telecommunication Union ITU-T And the international organization for Standardization ISO/IEC MPEG. These two giants are from 30 I started making video standards years ago , So far, we have achieved the first 6 Instead of . Among them, several video standards jointly developed by the two giants have a very far-reaching impact on the international video industry . such as MPEG2, It perfectly helps the video industry complete the important transformation from analog TV to digital TV ;H264 The contribution to the industry is self-evident , We all know that no matter what terminal （ TV 、 mobile phone 、 The computer ） And what services （ radio broadcast , satellite , Internet , Video conference, etc ） Basically, we fully support this standard ;H265 For HD, UHD video and HDR The popularity of video has made an important contribution . The latest one is 6 Instead of standard VVC, In addition to serving existing applications , Reduce bandwidth costs and improve user experience , It can also be empowering 5G Next emerging video applications , image AR/VR、360 Degree panoramic video and ultra high definition 4K、8K etc. .
Why do we have to be so persistent 6 Instead of video standards ？ Every time the standard is updated , The video industry chain needs to go from the server-side content producers to the final consumers , It includes opening up the end-to-end ecosystem in each link in the middle , Every link has to be updated . Because it takes so much effort to update video standards , So we have a basic requirement for each new generation of standards ： Double the coding efficiency with the same video quality , That is to say, compared with the previous generation , The bandwidth savings of the new standard must reach 50%.
Let's take a look first VVC Some of the words that standards often come across ：
- VVC：Versatile Video Coding,Versatinle Refers to the flexible and multi-functional features
- VTM： Test model reference software platform
- JVET：ITU-T and ISO/IEC MPEG The Joint Committee of
- H.266：VVC It's a double label of two international standards organizations ,H.266 yes VVC stay ITU-T Standard No
- It lasted nearly 3 Year of pre-standard Technology development and accumulation work ,JEM(Joint Exploration Model) Reference software platform .
- identical PSNR Under the index ,JEM-7.0 relative HEVC Reference platform for HM achieve 34% The code rate is saved , Standardization provides important technical support for .
- meanwhile ,360Lib Reference software platform for panoramic video processing 、 Compression and quality assessment provide a complete workflow .
stay VVC Before formal standardization started , International standard organization and member companies have carried out technology pre research and technology accumulation for many years . You can see from the above picture that , from 2015 Beginning of the year ,JVET After two and a half years of coding technology pre research , Built and perfected JEM Reference software platform ; To 2017 Mid year , In the same PSNR Under the index of ,JEM relative HEVC It has been achieved 34% The code rate is saved , It provides a strong technical support and performance proof for formally starting to formulate the next generation video standard .
in addition , stay JVET In the pre research process of accumulating the next generation coding technology , because AR/VR And the impact of emerging applications ,JVET Also on the 360 Degree panoramic video has been fully studied . So ,JVET Established 360Lib Reference software platform , and JEM Combination , For panoramic video processing , Compression and quality assessment provide a complete set of workflow and performance analysis capabilities .2017 year 10 month ,VVC stay JEM and 360Lib Basically when you're mature ,ITU-T and ISO/IECMPEG The two standards organizations have published a joint technical collection , includes 3 Main video formats in ： Standard dynamics SDR video （ Mainstream video formats ）、 High dynamic HDR video , as well as 360 Panoramic video . This is also 6 It is the first technical collection book considering multiple video formats since the generation of standard .
2018 year 4 month , There is a total of 32 Units submitted 23 Response to a request for proposals , In the same way PSNR Under the circumstances , The best response provides 40% The above rate savings , From now on VVC Standardization officially set off .2018 year 4 Month to 2020 Year of 7 month , After more than two years of hard work ,VVC The first edition of has been finalized .
Dharma academy video standards team in 2019 At the beginning of the year VVC Standard setting of , It took a year and a half , Many technical proposals have been submitted and adopted VVC In the standard , by VVC The development of standards has made an important contribution .
The blue part in the picture above represents VVC Reference software platform VTM-1.0 To VTM-9.0 Performance evolution of , as well as VVC relative HEVC Performance gains on HD Ultra HD video . We can see ,VVC The standardization process of more than two years is mainly divided into two stages ： The first half of the stage mainly focuses on adding advanced coding tools to improve VVC The performance gain of , So in VVC Compression performance increased rapidly during the first year of Standardization ; In the second half , Standards committee JVET Pay more attention VVC Detailed work on standard design , Focus on each VVC Design fusion between coding tools , Ensure low power consumption and efficient hardware and software implementation , So in VVC In the second year of Standardization , We can also see that VVC The performance gain tends to be stable gradually .
In addition, the above figure also provides VVC Reference software platform VTM-1.0 To VTM-9.0 The complexity of the evolution process . The red line shows the encoding time , You can see , With the improvement of coding performance gain , Coding complexity has also increased rapidly . The gray line represents the complexity of the decoder , comparison HEVC It has been less than twice as long as , This explanation VVC The complexity of the decoder is very acceptable . How to make a good VVC Real time encoder , Get the highest performance with the lowest complexity , There's a lot of technology and learning in between , This is also the important reason why Taotao and Dharma academy want to jointly develop this project .
The picture above shows 30 Varied VVC Coding tools , In the framework of hybrid video coding , New tools have been added to all function modules , To promote VVC Compression performance . in addition VVC The main feature is flexible and multi-functional , Therefore, the video content of some important and specific scenes has been considered in the process of standard formulation , For example, for screen content and 360 Degree panoramic video coding tools .
The figure above shows VVC The contribution of each coding tool to performance gain and complexity . On this picture , If a coding tool falls on the top right of the graph , It shows that its compression performance is good 、 At the same time, the complexity is low . But we can see that there is no free lunch , The actual data shows that , Good coding tools , The complexity is generally relatively high , such as ALF. therefore , When we were developing actual commercial encoders , How to choose and use these coding tools , It is crucial to the feasibility of the encoder in terms of complexity and performance . in addition , From the picture above, we can see that in VVC Of the many coding tools , Yes 8 Personality gains can exceed 1%, Others are relatively small .
The figure above shows VVC In the mainstream SDR Performance gain on video . For HD 、 For UHD video , In the same PSNR Under the index ,VVC relative HEVC Can save 38.9% bandwidth , For image coding , This performance gain is 26.7%.
The code rate savings shown in the table above have not reached 50%, that VVC As the standard of the new generation , Whether the design goal of doubling efficiency has been achieved ？ Because the most authoritative basis for video quality evaluation is subjective quality , So before and after each generation of standards is finalized , Formal subjective quality verification will be carried out , And the final bandwidth saving of each generation standard is also measured under the same subjective quality . The picture below shows VVC Subjective quality verification works in two UHD 4K The preliminary data of the video obtained by very strict subjective quality assessment method , We can see , Under the same subjective quality ,VVC comparison HEVC Bit rate savings of more than 50%.
Main stream HDR Video sequence ,PQ & HLG,VTM-9.0 vs. HM16.18
360 Panoramic video ,8K & 4K,VTM-8.0 / HM-16.20 + 360Lib
The stronger codec(VVC) + More advanced projection formats (GCMP)
VVC It has multi-functional and flexible features , The picture above shows HDR Video and 360 Objective performance of panoramic video . We can see that under the same objective performance ,VVC In two main streams HDR video （PQ and HLG） The code rate of the content is saved to 30%, stay 360 The rate savings on panoramic video reach 32.5%. Among them in 360 The gain on video comes from two aspects ： One side VVC Instead of HEVC, There's a more powerful coding kernel , Another part of the performance gain is through the use of more advanced projection formats . in addition , The data above only shows the rate savings under objective performance , in the light of HDR and 360 The subjective evaluation of panoramic video is also in order , among 360 The subjective evaluation of panoramic video is mainly led by Ali standard team , It is expected that an official report will be released early next year .
Dharma academy video standards team participated in VVC In the process of standard setting , In terms of coding technology , Contributed to real-time communication with low latency 、 Screen content 、 lossless compression 、 High dynamic range compression 、 Inter prediction 、 High level grammar and other related technologies .
meanwhile , Our team members are responsible for JVET Acting host of the conference and branch 、VCC Person in charge of panoramic video in performance acceptance 、 Test model algorithm description document edit 、 Panel （AHG） chairman 、 Also responsible for a number of core experiments , For Alibaba in the international video standards organization to establish a certain influence .
Let's take a look at the latest video industry trends and VVC Applications on these video trends .
Through the industry report above, we can see that , Video will always be a big bandwidth player ; Compared to last year's pie chart （ On the left ）,5 In the years to come, not only will the overall data volume have ５ Double growth , And the proportion of video in the overall data volume will continue to grow rapidly .
Video data continues to grow rapidly for four reasons ： One 、 The video is richer , Whether it's e-commerce （ TaoBao ）、 social contact 、 entertainment （ youku ）、 Or news , Including smart city, these emerging applications , There are more and more forms of video consumption ; Two 、 People are more and more used to , Easy to get video consumption ; 3、 ... and 、 Consumers are increasingly demanding for video signals , From HD to UHD ; Last , You want the video to be more innovative , So based on immersive video AR／VR Applications will rise rapidly .
Take Taobao live broadcast for example , Bandwidth costs account for a lot . In terms of daily life and average length of time , In less than a year, it's been growing very rapidly . Monthly bandwidth cost , There's an order of magnitude of growth , It's a very important part of the overall business cost . Now the live pictures are very complicated , There's more exercise , We also have higher and higher requirements for the clarity of the anchor , The technical specifications of resolution and frame rate are improved . At present, Taobao has averaged the challenging video content 800Kpbs The bandwidth of the , from H265 From the encoder's point of view , Extreme compression has been achieved . If you want to significantly reduce bandwidth costs , It can only be done by updating video standards .
Ali 266 The main goal of the project is to serve Taobao live broadcast , Hope in the next year's double 11 Can achieve Taobao live real-time coding , At the same time, relative to Ali 265 There is a significant improvement in compression performance .
Fraunhofer HHI It's a very prestigious German research institution , Have done many generations of video standard development , Yes VVC Standard development has also made a great contribution . In this year 9 In January, they announced their open source VVC codecs . We open source this VVC The codec has done a test , On the live video of Taobao , The encoding speed can only reach per second 0.5 frame , It is far from our real-time coding requirements . And for applications like Taobao live , In terms of decoder, there must be the best mobile optimization . These reasons make us more aware of , We need to make our own first-class codec , To be able to efficiently serve the internal business of our group , This is a very important thing , Is ali. 266 The main objectives of the project .
Finally, let's take a look at MPEG Other related video standards in , And work with other video standards organizations . As we mentioned earlier, novel video is one of the main video trends that we should pay attention to , This mainly refers to immersion video .MPEG See this important trend in the video industry , Except for making VVC Beyond the next generation of video compression standards , And a whole set of MPEGImmersive Immersion media standards series , This includes point cloud compression standards , 6-DOF video and audio compression standard , And some file format standards for immersion media .
Outside the international video standards organization , We all know that there is another influential video standards making Alliance ,Alliance for OpenMedia (AOM).AOM From Google VP8 and VP9 Start ,2018 The first generation of AOM standard AV1, And recently AOM Also began to plan for the next generation of video standards AV2. From the domestic standards organization ,AVS After three generations of main standards , Now it's developing AVS3 Second device .AVS3 The first issue and VVC It's a very consistent timeline , Also in 2017 Technical request for proposals will be released by the end of the year ,2018 Started collecting technical proposals in 、 Solicitation response , Also released HPM Reference platform . After a period of iteration ,2019 It will be finalized by the end of the year AVS3 The first edition . at present AVS It's going on AVS3 The development of the second edition , The goal is to finalize the second edition by the end of next year , Performance targets exceed VVC. The Dharma school team is also actively involved in AVS3 The development of the second edition , by AVS3 The development of the second edition made an important technical contribution .
Finally, I'd like to share with you what the future of international video standards will be . When we do the standard video , Not just modern , Not only looking at the past , And to see the future . In terms of technology trends , Video coding based on deep learning is a promising technology for everyone . We know , In the past 6 International video standards are based on the traditional hybrid coding framework , There are many functional modules , But today, the framework has almost reached the performance ceiling . hold DL There are two ways to introduce coding ： One is that it can be combined with traditional frameworks , Add... To each function module DL Coding tools for , Make the performance change better ; It's an end-to-end route DL Video coding structure . These two directions , From the perspective of technology trends, they are all worth studying . therefore MPEG In this year 4 It was established in DNNVC Panel , What this group wants to do is to explore the application of deep learning in video coding , Performance ceiling to break traditional frames , Looking for the future direction of video codec .
Finally, I'd like to introduce to you three parts of the work that the video technology team of the academy is mainly responsible for ： Video standards team focuses on VVC、AVS3、AV2、DL code 、VCM、DCM Hard core technology . In the video hardware implementation , Our team has developed an ultra high definition real-time 265 Encoder , In the industry's similar products compression performance leading , At the same time, it also provides full hardware efficient video pre-processing capability , At present, it serves Youku live broadcasting business . In the video software implementation , Our team in addition to leading just mentioned Ali 266 project , In addition, we also have deep cooperation with Taobao team , Provide video conferencing based on H264 and H265 Software and hardware coding optimization scheme of , Reduce business costs , Improve user experience .
Link to the original text
This article is the original content of Alibaba cloud , No reprint without permission .