当前位置:网站首页>Tdsql inside: from Tencent's distributed database capability to the industry capability

Tdsql inside: from Tencent's distributed database capability to the industry capability

2021-09-15 05:59:08 Tencent cloud database

at present ,TDSQL Our users have covered Finance 、 government affairs 、 The Internet of things 、 Retail wisdom 、 Internet and other industries , Have a large number of distributed database best practices .

2014 year , yes TDSQL A crucial year . This year ,TDSQL Officially began commercial use , The first external cooperative customer is Weizhong bank . Weizhong bank is the first Internet Bank in China whose core trading system is built in a distributed architecture . High scalability of Internet Architecture , This makes the service capacity of Weizhong bank highly flexible , Enough to easily cope with the wave of Inclusive Finance . here we are 2015 end of the year ,TDSQL Online Tencent cloud , It marks the full opening of Tencent's internal financial level distributed database capability .

2019 year ,TDSQL A measure of has once again promoted the development of domestic database to help industrial digital innovation . This year, ,TDSQL Successfully assisted Zhangjiagang bank to take the lead in transforming the traditional core system of the bank from centralized database storage to distributed database storage , Breaking the long-term dependence of this field on foreign databases . This is the first time that domestic banks have used the traditional core business system scenario , Domestic distributed database is adopted . And use TDSQL The new generation core system built also performs well : High frequency account transactions take time in 300 In milliseconds ,20 Within seconds 1 Ten thousand batch distribution and withholding business …… And the frame structure is reduced to 1/5 Even lower . Properly realize cost reduction and efficiency increase .

Current deadline 2019 year , The financial daily trading volume is the highest 3.46 Billion + Weizhong Bank of 、 Bank of China, a large banking institution 、 The daily average usage is over 150 The number of passengers of Shenzhen Metro boarding code system exceeds 500 Financial, government and enterprise institutions , It's all Tencent TDSQL Database services .

This is the best time , It is also a new starting point .

Pan Anqun said , Domestic databases have developed independently and controllably for more than ten years , Precipitated mature distributed database technology capabilities , meanwhile , The vigorous development of cloud computing has brought many new opportunities for the business transformation of traditional industries , This is the best period for the integrated development of business innovation and the transformation and upgrading of underlying technology architecture .“ Promote the development of domestic distributed database , At the same time, it will also promote the development of China's industrial digital transformation . The development of industrial application will also provide new opportunities for technological innovation and development . Now is the best time for industrial digital upgrading , It is also the best time for the development of domestic databases .”

But the future , Core technology challenges 、 Cutting edge technology, ecological integration, etc , It is worth thinking and exploring to continue to promote the gradual maturity of the basic technology of the whole database and industrial ecology . Pan Anqun pointed out , On the basis of autonomy and control , At the same time, it has more future innovative exploration and ecological open exchange , In order to make the value of technology R & D more meaningful , Promote the further innovation and development of domestic databases . Tencent will also increase investment , Promote the cooperation between industry, University and research of database in China , Promote the upgrading of domestic database technology and ecological improvement .

TDSQL The evolution trilogy

TDSQL Predecessor in 2004 R & D started in , It has been accumulated for more than ten years . Review its entire evolution , It can be divided into several stages .

On the eve of the Internet opening wave : How to form distributed database step by step

If everyone goes to fill Q The coin , Or when consuming in Tencent's business , If the amount is wrong , This will be an unacceptable thing . So the earliest from 2004 Year begins , When Tencent needs to do billing business internally , What we urgently needed at that time was data consistency and high system availability .

at that time , Or the Internet IT Early stage , Few companies are willing to invest in the research and development of underlying technologies . And as an Internet technology company , Why not use technology to solve technical problems ? therefore , from 2004 Year begins , Tencent began to build on the open source system MySQL R & D , To achieve high data consistency and high system availability , This is also TDSQL The forerunner of .

Also in this context ,TDSQL It was gradually born . Therefore, Tencent's financial business has not Oracle, No, “IOE”.

be based on MySQL To develop such a system architecture , It was also verified in later experience , It's a very difficult thing , But it also verified the technical value brought by the huge investment at the beginning .

first , We are not right MySQL Do some optimization work by yourself , More in the urgent need of efficiency , combination CAP principle , Solve these problems at the application layer .

And then we found out , Work at the application layer , Solving the first system problem is fast , But for subsequent large-scale application development , It is difficult to make these data consistent in the form of tight coupling with business applications 、 Highly available solutions are extended to other business systems . Because it's for the business layer 、 For the application layer , The transformation cost is too big , Because in the process of rapid business development , The historical burden of the business itself is getting heavier and heavier , It is unlikely to adapt to the new architectural transformation .

Demand is naturally urgent , At that time, the business was facing business splitting , And the cluster management of hundreds of devices , Data consistency 、 Data accuracy and reliability .

So we began to think about , We must take these disasters 、 Data consistency and other logic , All sink to the database level to develop structures , Let the application layer just focus on business logic , There is no need to manage disaster recovery and other logic .

From a development perspective , This is also a relatively large investment 、 More difficult things , But then everyone had a hunch , We will create a new thing .

And because of that , Curiosity about the future overcomes difficult obstacles .2007 Database development in , Several members of the team are closed in a small black room , Started crazy code building , Focus on solving the high availability of company level sensitive services such as billing 、 Zero loss of core data 、 Zero and wrong accounts of core transactions .

According to the characteristics of financial business ,TDSQL The technical team has clear goals , Include the following points :

Strong data consistency requirements
Availability of database clusters 、 Stability and disaster recovery requirements should meet bank standards
Business does not need to split large tables , The database is automatically split
Access should be simple , The transformation of old business should be small
Comply with and higher than the information security regulatory requirements of the financial industry
And after solving these problems , The whole technical team began to think about the future direction of technology development , And have new ideas for technical architecture design —— We hope to make a new system . Data consistency 、 Reliability and other characteristics are integrated from the application layer to the database layer , Prediction based on business take-off , We think , The database layer also needs to have the characteristics of distributed horizontal expansion .

therefore , The team began to refactor the database architecture .

Very lucky , It's just TDSQL When completing the self-development of distributed horizontal expansion , About it. 2009 year , Tencent immediately ushered in the era of Tencent open platform . At that time , The Internet has begun the explosion of real social applications , Products such as happy farm were born .

And this TDSQL The prototype of , With high scalability 、 Data consistency 、 reliability 、 High availability, etc , Supported the wave of opening up at that time . To this day ,TDSQL Experienced scenario verification of tens of billions of accounts , It has a perfect ability to support financial level scenarios and regulatory requirements .

Born for the financial scene : Product oriented financial level distributed database TDSQL
Technology iterations are often driven by business scenarios .

2007-2009 Mid year , With the development of Tencent's open platform , We have welcomed a large number of partners , The industry scenarios are becoming more and more diverse . And then you see , This database is perfect in supporting Tencent's internal business system , But it can't provide good services for partners —— Because in the early stages of development , It's not standardized enough .

How can we better serve the needs of partners ? How to make such a distributed database system serve more users , Maximize the value of Technology ?……

Only continuous R & D and iteration .

Until the 2012 year , During this period, Tencent used “ Open source customization + Since the research ” Customize policies , Polish out more general database products , And officially named TDSQL, To solve the problem of high availability in financial and other business systems 、 Data consistency 、 Horizontal expansion and contraction .

so to speak TDSQL The birth of , It was born for the financial scene at the beginning .

Gradually, , With deeper insight into the application of the financial industry ,TDSQL Gradually improve the distributed transaction 、 Distributed query and other capabilities , Continuous development on performance and applicability , The goal is to TDSQL Build a relational distributed database similar to stand-alone version . We know , The financial industry has a high demand for transaction processing , Transfer accounts 、 Deduction fee , No one is not using transactions , Tencent is one of the few companies that will handle distributed transactions , Distributed query is used for enterprises in the financial core system .

As a Shared-Nothing Distributed database architecture , In terms of ability ,TDSQL More popular than at present HTAP Further more , It redefines a comprehensive database solution , It can also be distributed Noshard example 、 Distributed and analytical instances , Support at the same time JSON/RockDB And so on . Of course ,TDSQL The main feature is that it has shard Architectural capability .

Continuous improvement of technology ecology and product service system

2016 Years later ,TDSQL Officially open the road of self-control , Open to more financial enterprises , Thanks to the tempering of massive business scenarios , bring TDSQL Become a product database , Have a complete product system .

at present ,TDSQL With six core features , Including strong data consistency 、 Financial high availability 、 High performance and low cost 、 Enterprise security 、 Linear horizontal expansion 、 Intelligent operation and maintenance . On some features related to data security ,TDSQL We have strengthened our focus , For example, for problems such as high availability .TDSQL It can easily support multiple activities in different places .

In addition to technical support ,TDSQL It also ensures the security of data from an operational point of view . According to the analysis , at present , The degree of productization of most databases in the financial industry is not enough , Quite a number of safety accidents are caused by non-standard operation .TDSQL By providing “ Red Rabbit ” Self service operations and “ Magpie ” intelligence DBA Completely avoid potential safety hazards caused by human misoperation .

“ Red Rabbit ” Self service operation service , From the administrator's point of view , In usability 、 Security 、 efficiency 、 Comprehensive control of cost dimension ,90% The daily operation of can be through Web Page completion , Reduce human errors and help financial users save management and economic costs 、 Reduce risk .

“ Magpie ” intelligence DBA Help financial users prevent system abnormalities , By collecting more than 400+ Operational indicators , be based on AI+Policy Intelligent diagnosis technology , Help financial users quickly locate and solve problems , And prevent potential risks , Take precautions .

Besides , Distributed database TDSQL Provide database firewall for users 、 Transparent encryption 、 Automatic desensitization and other safety protection measures , Reduce the security risks caused by user misoperation and hacker intrusion .

Keep accumulating , Use time to temper an independent and controllable database

After more than ten years of accumulation , Continuously optimizing distributed systems 、 High availability 、 High performance and other characteristics , And continuously improve the process of meeting the needs of customers ,TDSQL As an independent and controllable database of industrialization , Keep leading in the industry .

and TDSQL Evolving to today's capability scale , Driven and tempered by Tencent's own business scenarios . A financial level distributed database , It must go through the accumulation of product ecosystem for many years , And the tempering of massive business scenarios .TDSQL Iterative evolution of architecture , Because of the massive user scenarios from Tencent 、 In the business practice of complex transactions , In order to better meet the needs of our customers for database technology and business flexibility .

for instance , In terms of product quality assurance ,TDSQL The release conference goes through a strict process , Finally, it is extended to the customer scenario : The first is the billing team ( Technology incubation team ) Use validation , Then it will be popularized and applied to other business teams of Tencent , And in these internal scenes , We can all control and repair ; After these two stages , The new version has been verified by Tencent public cloud users only after it is released , Finally released on the private cloud .

On the other hand , In the process of productization ,TDSQL Combined with practical application and customer feedback , Fully consider the whole application and production process of the database , To continuously improve the product service system , Including operation system 、 Database multi-source synchronous migration and other supporting facilities .

Based on this kind of productization , Tencent has an open technology ecological gene . Openness does not mean that we must open source , Instead, it provides open standards 、 Perfect service , For example, database multi-source synchronous migration , And an independent and controllable open source ecosystem , Support healthy competition , Protect customers from the risk of being bound .

future , How to communicate with Oracle compatible 、 And AI Integration with cutting-edge technologies such as heterogeneous computing, etc , They are all challenges and innovative issues worth thinking about .TDSQL Will continue through the combination of production and Research 、 Research and development breakthrough through the combination of production and use , And open up more features , Embrace the open source community .

版权声明
本文为[Tencent cloud database]所创,转载请带上原文链接,感谢
https://chowdera.com/2021/09/20210909121019481k.html

随机推荐