In China, , qihoo 360 Is a major provider of Internet and mobile security products and services , By the end of 2014 year 6 month , Qihoo owns 5 Billions of PC Active users and more than 6.4 100 million mobile users . Qihoo also operates China's most popular web browser and PC Search engine ( The text is so. ).
My team , Push service team (Push Service Team), To exceed 50 A company's products provide services （PC And mobile ）, Including thousands of applications on our open platform .
We are right. Go Your favor should start from 2012 In, he first tried to provide push function for a product of Qihoo . The original nginx + lua + redis The solution fails to meet our real-time performance requirements due to excessive load . under these circumstances , Newly released 1.0.3 edition Go Caught our attention , With the help of its goroutine and channel characteristic , We developed a prototype in a few weeks . Our system initially runs on 20 On servers , Able to handle 2000 Million real-time connections , Send... Every day 200 Wan information . Now the system is more than 400 Two servers running , Support 2 Million real-time connections , Send more than... Every day 100 Billion messages .
With the rapid expansion of business and the growth of demand for our push services , The original Go The system soon encountered a bottleneck ： The heap size reached 69G、GC A pause can be as high as 3-6 second . What's more painful is , We have to restart the system every week to free up memory . To be honest , We considered giving up Go, use C To rewrite all the core components as an alternative . However , Not as we expected , We had trouble migrating the code of the business logic layer . The result is , As the only person at that time , I can't be maintaining this Go Ensure that the business logic is successfully migrated to C On the service framework .
therefore , I decided to stay in Go On the system ( It may be my smartest helpless choice ), And soon made great progress .
Here are some key points and tips we have summarized ：
- With a long connection ( Use connection pool ) Replace short connection , Reduce the creation of buffers and objects during communication .
- Use object pools and memory pools appropriately , Reduce GC The load of .
- Use task pool , The principle is that a group of coprocessors poll and execute global tasks or tasks from message queues as consumers —— The coprocessor holding the connection sends the task as the producer , Instead of generating a temporary collaboration for each task .
- Monitor and control the number of processes in the program . Lack of control can make GC Overwhelmed , such as , The peak number of processes due to uncontrolled acceptance of external requests is applied to GC On , At the same time, it is sent to the internal service RPC The call may block the newly created coroutine .
- In mobile networks , Remember to connect Set the read-write timeout limit , Otherwise, it may lead to co process blocking . You should use this setting carefully in the LAN , Otherwise it will reduce RPC Communication efficiency .
- Use Pipeline( stay TCP Full duplex feature ) To enhance RPC Communication efficiency of the framework .
Last , We successfully iterated the architecture three times , Two of them were right RPC The iteration of the framework is completed with limited manpower . It all benefits from Go The convenience of development , Here is our current system architecture ：
The process of continuous optimization can be illustrated in the following table ：
also , After these optimizations , There is no need to temporarily free memory or restart the system .
Even more exciting, we have developed an online real-time reflection Go Visual platform for program running overview . Now we can easily get the system status and make diagnosis , remove a hidden danger . This is a screenshot of the system running ：
This platform can also do a great thing —— Simulate the connection and operation of millions of users online . By using this distributed stress testing tool （ Also use Go Realization ）, And observe all these intuitive real-time data . We evaluate the effect of each optimization, locate the system bottleneck, and then eliminate the problem .
up to now , We've tried almost every means of system optimization . We still look forward to GC The team can bring more good news and let us further free ourselves from the heavy development work . Want to come , Maybe one day our experience will be because Go Become a useless place for the sustainable development of .
That's why I won this attendance with expression Gopher China Thank you for the opportunity to end this sharing . This is a time for us to learn , To share , It's also a show Go The opportunity of popularization and prosperity in China . Many other teams within Qihoo began to understand Go, Or already trying to use Go.
In the foreseeable future , I believe more Internet companies in China will join us , use Go To recreate their system , and Go The results of the development team will benefit more developers and companies .