Detailed explanation of real-time cloud rendering technology - principle, application and advantages

Pixel streaming （ Also called real-time cloud rendering ） The technology is UE4 Recently launched , It is widely used in many fields , This paper starts with the principle of pixel streaming technology 、 Problem solved 、 The advantages of application and applicable industries are described in detail , Hope to help more users understand this technology

One 、 Pixel stream ( Real time cloud rendering ) What is it? ？

See the name , For those unfamiliar with the field , The first reaction may be pixels + flow . Most people's understanding of pixels comes from the clarity of photos taken by mobile camera , The higher the pixel, the better the sharpness of the picture . and “ flow ” Generally refers to the flow , Pixel stream is a combination of the two , But the pixels here are not limited to the pixels of the picture , It also includes audio and video files , It is a carrier form for the final transmission of content . Pixel stream is the way to process the execution result of the server into video stream , Displayed on the terminal . In recent years, this technology has been applied in virtual simulation 、UE4、unity、GIS、 Digital twins are widely used in the field of 3D visualization , It mainly solves the problem of using light terminal . The key lies in the delay before the instruction and execution result .

The principle is to put the rendering processing of instructions on the cloud server , The terminal only needs to connect to the server through the web page to operate the application and input instructions , At the same time, the result of cloud rendering processing is returned to the terminal in the form of pixel stream . In fact, this technology has existed for a long time, but it was not easy to implement due to many hardware conditions , Especially network factors , And in recent years, with 5G The development of , The network problem has been well solved , Now the commercial application of this technology can well solve the problem of user experience . So what is the principle of pixel streaming ？

Two 、 Detailed explanation of pixel streaming principle

The pixel stream name was originally created by UE4 official , The framework of this technology consists of two parts ： Pixel streaming plug-in Pixel Streaming、 Signaling and web The server , Their functions are respectively ：

Pixel streaming plug-in - This plug-in runs in a virtual engine . Its use H.264 Video compression encodes the final result of each rendered frame , These video frames are packaged into the media stream along with the game audio , And send the stream to one or more connected browsers through direct point-to-point connection .

Signaling and Web The server - Signaling and Web The server is responsible for the connection between the browser and the pixel streaming plug-in , Will play media streaming HTML and JavaScript The environment is provided to the browser .

The connection process of the two is as follows ：

1、 When you start all pixel streaming components , The pixel streaming plug-in running in the virtual engine will first be built into signaling and Web Links to servers .

2、 The client will connect to the signaling server , The server will provide a HTML page , It contains player controls and JavaScript Write control code .

3、 When the user starts streaming , The signaling server will negotiate , Establish a direct connection between the client browser and the virtual engine .

4、 After the connection between the client and the virtual engine is established , The pixel streaming plug-in will start streaming media directly to the browser . The input from the client is generated by JavaScript The environment is sent directly back to the virtual engine .

5、 Even if media streaming has started , Signaling and Web The server will still maintain its connection to the browser and the virtual engine , To remove users from streaming if necessary , And handle the disconnection caused by the browser .

From this process, we can see that , In short, this technology is to process and render instructions that can only rely on the performance of local machines , On the cloud , For this process, the problem to be solved is how to receive the instruction and return to the terminal as soon as possible . What problems does this technology solve in the current industry ？

3、 ... and 、 What problems do pixel streams solve ？

The development of any technology is not out of thin air , It's an improvement based on the status quo , Mainly efficiency improvement 、 Experience improvement and better meet the needs . The same is true with the emergence of pixel streaming technology , What problems does it solve ？ First, let's look at before the advent of pixel streaming technology , In the use of UE4、unity、BIM Wait for the program , How is it applied in practice , Here, the point cloud is illustrated by taking the virtual simulation industry as an example . At present, the development of virtual simulation program is mostly based on Unity、UE4 Wait for the game engine to model . The traditional deployment method is to install the program directly on the local computer , Users can use it after opening it through a special computer . If it is training and teaching, you need to buy special hardware equipment for special classrooms . For the digital twin of smart city, it shows the multidimensional data , For computers GPU And higher computing power requirements . This way, , Restrict users to use only in specific situations , And because of the compatibility of the computer , There may be many potential problems in installation and upgrade . At present, Internet content is used , In addition to the client, you can also use the web page , Can virtual emulators like this be accessed through web pages ？ The answer is yes , The options are webgl And pixel streams （ Real time cloud rendering ） There are two technologies . Let's look at the characteristics and differences between the two .

WebGL yes Web Graphics Library Abbreviation , It is a kind of 3D Drawing protocol , This kind of drawing technology standard allows JavaScript and OpenGL ES 2.0 Bind together , By increasing the OpenGL ES 2.0 One of the JavaScript binding ,WebGL It can be for HTML5 Canvas Provide hardware 3D Accelerated rendering , In this way, it can be displayed more smoothly in the browser with the help of the computer system graphics card 3D Scenes and models . WebGL The technical standard avoids the trouble of developing special rendering plug-ins for web pages , Can be used to create complex 3D Structure of the website page , It can even be used to design 3D Web games and so on . Simply speaking webgl It's on the web 3D Drawing standards . It also makes webgl The scheme is in use , In some scenes, the rendering work is completed through the local graphics card hardware of the computer , So in use webgl When it comes to technology , Especially when you just open a web page , You need to load data locally . And according to the content , The loading time is also different , Although compared with local installation , The dependence on hardware is reduced , But it's higher than the average working computer , And the terminal type only supports computers . In addition, for content providers ,webgl You need to download the content to the user's local , Content stolen 、 Crack 、 The risk of illegal transmission increases . Here, the difference between the three is compared as follows ：

Technical solution	Cloud fluidization	WebGL	Local installation
Local installation EXE	no	no	yes
Open mode	Webpage	Webpage	Client open
Loading time	short	Long	It has something to do with computer performance
Content download	no	yes	Store locally
Support terminal	The computer + mobile phone + Large TV screen, etc	Computers and some mobile phones	The computer
Content security	high	low	low
Hardware parameter requirements	Just watch the video	high	high
Compatibility issues	nothing	Yes	Yes

Four 、 What are the advantages of cloud rendering technology ？

Pixel stream （ Real time cloud rendering ） The advantages of innovative technology are obvious , In summary, there are the following aspects ：

A、 Expand the range of terminal types , Solved the compatibility problem

Not only the computer-based web page, you can directly watch high-quality images , mobile phone 、 Mobile terminals such as tablets can also directly operate and watch the rendering effect of the cloud . Like just packing Windows Applications under the platform , but Android、iOS It can be used under the platform .

B、 There is no need to install plug-ins

In addition to the official pixel streaming plug-in Pixel Streaming Integrated into the program , No need to install any plug-ins , Easier to use , Lower requirements for hardware performance . Because the rendering and execution of instructions are completed in the cloud , As long as the terminal can play video normally .

C、 Low latency

Pixel streaming use WebRTC Point to point communication framework , The latency between the user and the unreal engine application is low . Point software pixel stream products can do 5-30ms Delay of , The effect is almost the same as that of local installation .

For many digital twins 、 Enterprises in smart park or virtual simulation industry , Use pixel streaming technology to give users a better experience , Is the advantage of the project . But in today's industry segmentation , Professional people do professional things more efficiently .

However, in the actual midpoint cloud, it is found that UE4 Although the pixel stream gives out data , However, there are the following problems in practical use ：

1、 Compatibility problems with some browsers , such as iOS Under the wechat 、 part chrome edition , There will be a situation that can't be opened for no reason ;

2、 No product functions and services , For example, load balancing 、 Speed measurement and dispatching 、 Auto update 、 Release 、 Background statistical report 、 User status display 、 There are no alarm and other mechanisms , It is not conducive to understanding the operation and use of the program ;

3、 No client mode , Only web version is supported ; But in some cases , The client can have a faster experience （ Don't depend on webRTC）, There are also some large screens and other systems , If you use client mode , Don't enter the URL , By clicking on some icons , Experience better .

For some companies or personnel who are not very good at this , It may take a lot of time and energy, and it may not be able to solve these problems , In modern society, division of labor and cooperation can greatly improve efficiency , So-called “ Professional people do professional things ”. Pixel streams only support UE4 The content of , Point cloud supports all content , More than UE4、Unity, It also includes various software , such as 3DMax、BIM、Flash etc. . And in order to use , For load balancing 、 Speed measurement and dispatching 、 Automatically follow new releases 、 Statistics are also processed as products , You can use it . Don't spend your time and energy starting from scratch , And there are professionals to do technical answers , You can focus on what you are good at and make more achievements . As for client mode , Can achieve Windows、Android The system is based on private protocols , Realize the streaming of cloud program rendering results to the client . The client is small , Only a few trillion . Client's UI The interface can also be edited freely in the background , Changeable 、 personality 、 Brand promotion 、 Often new to keep fresh . So for UE4 Cooperation on browser compatibility in pixel streaming may be a good choice .

5、 ... and 、 Using real-time cloud rendering technology , How much server concurrency can be achieved

Said so much , There may be new questions , Since the use of pixel streaming technology is to put the program on the cloud server , How much concurrency can a server support at the same time ？ This problem is actually related to the size of the program 、 The performance of the server has a lot to do with . The evaluation criteria are , If you don't use pixel streaming technology , Take the performance parameters of the existing server , How many programs can be run simultaneously , The server uses pixel streaming technology , It can also support so many concurrent users at the same time . However, in order to support so many users at the same time , You need to run multiple copies of the same number of programs on the server . If not UE4 This pixel stream mode , Other applications , You also need to provide cloud rendering software , Can support containerization 、 More open technology .

6、 ... and 、 What fields can pixel stream technology be applied to ？

At present, there are many fields that can use real-time cloud rendering technology , for example ： Various virtual simulation teaching programs and training systems （ Medical Science 、 Architecture 、 Biophysical chemistry 、 Production industry 、 Military training ）、 Number twin 、 Twin manufacturing systems （ Car model 、 Manufacturing, etc ）、 Smart City Campus and Park 、 Large scale design software 、 Cloud games 、 Cool car change show 、 Health care 、 Industrial simulation, etc .