Essential for back-end programmers: distributed transaction Basics
2020-11-08 12:11:44 【osc_lg0msmnd】
Recently I read several blogs about distributed transactions , Take notes . ha-ha ~
Database transactions （ abbreviation ： Business ）, It is a logical unit in the execution of database management system , It consists of a limited sequence of database operations , These operations are either all performed , Or none at all , It is an indivisible work unit .
Several typical features of database transactions ： Atomicity (Atomicity )、 Uniformity ( Consistency )、 Isolation, ( Isolation) And persistence (Durabilily), The abbreviation is ACID.
- Atomicity ： The transaction is executed as a whole , The operations on the database contained in it are either all performed , Either not .
- Uniformity ： Before and after the transaction , The data will not be destroyed , If A Account to B Account transfer 10 Yuan , Whether it works or not ,A and B The total amount of is the same .
- Isolation, ： When multiple transactions are accessed concurrently , Transactions are isolated from each other , That is, one transaction does not affect the running effect of other transactions . in short , It's just that the water doesn't flow into the river .
- persistence ： Indicates that after the transaction is completed , Operational changes made by the transaction to the database , Will persist in the database .
How transactions are implemented
Traditional single server , Transactions in a single relational database , It's local business . Local transactions are managed by resource manager ,JDBC Transaction is a very typical local transaction .
innodb The transaction log includes redo log and undo log.
redo log（ Redo log ）
redo log Usually physical logs , What is recorded is the physical modification of the data page , Not how to modify a line or lines , It is used to recover the physical data page after submission .
undo log（ Rollback log ）
undo log It's a logical log , and redo log It's different to record physical logs . Think of it this way , When delete When a record ,undo log A corresponding insert Record , When update When a record , It records an opposite update Record .
Business ACID The realization idea of features
- Atomicity ： It's using undo log To achieve , If there is an error during the transaction execution or the user executes rollback, The system goes through undo log The log returns the status of the start of the transaction .
- persistence ： Use redo log To achieve , as long as redo log Log persistence , When the system crashes , You can pass redo log Recover data .
- Isolation, ： Through locks and MVCC, To separate affairs from each other .
- Uniformity ： By rolling back 、 recovery , And isolation in the case of concurrency , To achieve consistency .
Distributed transactions ： It refers to the participants in the transaction 、 Servers that support transactions 、 Resource servers and transaction managers are located on different nodes of different distributed systems . Simply speaking , Distributed transaction refers to the transaction in distributed system , It exists to ensure the data consistency of different database nodes .
Why do I need distributed transactions ？ Next, I will explain it in two aspects ：
Distributed transaction under microservice architecture
With the rapid development of Internet , Lightweight and well-defined microservices , On the stage of history . such as , A user places an order , Buy live gift service , Be divided into three service, They are gold coin service （coinService）, Order service （orderService）、 Gift service （giftService）. These services are deployed on different machines （ node ）, Corresponding database （ Gold coin database 、 Order database 、 Gift database ） Also on different nodes .
Users place an order to buy gifts , Gift database 、 Gold coin database 、 The order database is on different nodes , It's not possible to use local transactions , So how to guarantee different databases （ node ） Data consistency on ？ This requires distributed transactions ~
Distributed transactions under sub databases and sub tables
As the business grows , The data in the database is getting bigger and bigger , More than ten million levels of data , We need to divide it into databases and tables （ The company used to use mycat Sub database and sub table , Later used sharding-jdbc）. A bank , The data is distributed on different nodes , For example, some of them are in Shenzhen computer room , Some are in Beijing computer room ~ You want to use local affairs to guarantee , I've been indifferent ~ You still need distributed transactions .
such as A turn 10 Block to B,A The account data is in Beijing computer room ,B The account data of is in Shenzhen computer room . The process is as follows ：
CAP theory &BASE theory
Learn about distributed transactions , Of course, we need to understand CAP Theory and BASE theory .
CAP Theory as the basic theory of distributed system , In a distributed system , Consistency（ Uniformity ）、 Availability（ Usability ）、Partition tolerance（ Partition tolerance ）, These three elements can only achieve two points at most .
Consistency is the ability of data to be consistent across multiple copies . For example, a data is updated after a partition node , The data read from other partition nodes is also the updated data .
Availability means that the services provided by the system must always be available , For each operation request of the user, the result can always be returned in a limited time . Here's the point " In a limited time " and " Return results ".
Partition tolerance （P:Partition tolerance）:
Distributed systems encounter any network partition failure , There is still a need to be able to ensure that external services meet the requirements of consistency and availability .
CA Give up partition fault tolerance , Enhance consistency and availability , In fact, it is the choice of traditional stand-alone database
AP Give up consistency , Partition fault tolerance and availability , This is the choice of many distributed systems when they are designed
CP Discard availability , Pursue consistency and partition fault tolerance , Network problems will directly make the whole system unavailable
BASE theory , It's right CAP in AP An extension of , For our business system , We consider sacrificing consistency for system availability and partition fault tolerance .BASE yes Basically Available( Basic available ),Soft state（ Soft state ）, and Eventually consistent（ Final consistency ） Abbreviations of three phrases .
Basic available ： It is realized by supporting local fault rather than global fault of the system . If you partition users in 5 Database server , The failure of a user database only affects the specific host 20% Users of , Other users are not affected .
Soft state , The state can be out of sync for a while
Final agreement , Finally, the data is consistent , Instead of being consistent all the time .
Several solutions for distributed transactions
Distributed transaction solutions mainly include the following ：
- 2PC( Two stage submission ) programme
- Local message table
- Best effort notification
- Saga Business
Two stage proposal
Two phase commit is a common distributed transaction solution . The commit of a transaction is divided into two phases ： Preparation phase and submission of implementation plan .
Successful submission of phase II
Preparation stage , The transaction manager sends a prepare message to each resource manager , If the resource manager's local transaction operation is successful , Then return to success .
Submission execution phase , If the transaction manager receives a success message from all resource managers , Then send a submit message... To each resource manager ,RM according to TM The order execution submission of . Pictured ：
Failure of two-stage submission
Preparation stage , The transaction manager sends a prepare message to each resource manager , If the resource manager's local transaction operation is successful , Then return to success , If execution fails , Then it returns failure .
Submission execution phase , If the transaction manager receives a message that any resource manager fails , Then send a rollback message to each resource manager . The resource manager rolls back the local transaction operation according to the instructions of the transaction manager , Release all lock resources used in transaction processing .
Advantages and disadvantages of two stage submission
2PC It's easy to implement , Lower cost , But it has the following disadvantages ：
- A single point of the problem ： If the transaction manager fails , Explorer will always be locked .
- Performance issues ： All resource managers are in a synchronous blocking state during the transaction commit phase , Occupy system resources , Until the submission is complete , To release resources , It is easy to cause performance bottleneck .
- Data consistency issues ： If some resource managers receive a submitted message , Some didn't receive , It will lead to data inconsistency .
TCC（ Compensation mechanism ）
TCC Compensation mechanism has been adopted , The central idea is this ： For each operation , To register a corresponding confirmation and compensation （ revoke ） operation .
TCC（Try-Confirm-Cancel） Distributed transaction is realized by decomposing business logic . For a specific business service ,TCC The distributed transaction model requires the business system to implement three sections of logic ：
try Stage ： Try to execute , Complete all business consistency checks , Reserve necessary business resources .
Confirm Stage ： At this stage, confirm and submit the business , Don't do any tests , because try The stage has been checked , Default Confirm There is no mistake in the stage .
Cancel Stage ： If business execution fails , Then enter the stage , It will release try All business resources occupied by the phase , And roll back Confirm All operations performed in phase .
TCC The distributed transaction model consists of three parts ： Main business services 、 From business services 、 Business activity manager .
- Main business services ： The main business service is responsible for initiating and completing the entire business activity .
- From business services ： From business services is the whole business activity of the participants , Realization Try、Confirm、Cancel operation , For the main business service to call .
- Business activity manager ： The business activity manager manages and controls the entire business activity , Including recording transaction status , Call... From business service Confirm operation , Call... From business service Cancel Operation etc. .
Let's take the user's purchase of gifts as an example to simulate TCC The process of implementing distributed transactions ：
Suppose the user A The balance is 100 Gold coin , The present you have is 5 Duo .A It took 10 Gold coin , Place the order , Buy 10 Rose . balance 、 Order 、 Gifts are in different databases .
TCC Of Try Stage ：
- Generate an order record , The order status is to be confirmed .
- Will the user A The balance in the gold coins of the account is updated to 90, Freeze the gold coin to 10（ Reserve business resources ）
- The number of users' gifts is 5, The quantity to be added in advance is 10.
- Try After success , Then enter Confirm Stage
- Try There's anything wrong with the process , All enter Cancel Stage
TCC Of Confirm Stage ：
- Order status updated to paid
- Update user balance to 90, It can be frozen as 0
- The number of user gifts is updated to 15, Increase to 0
- Confirm There's anything wrong with the process , All enter Cancel Stage
- Confirm Process execution successful , Then the transaction ends
TCC Of Cancel Stage ：
- Modify order status to cancelled
- Update user balance back 100
- Update the number of user gifts to 5
TCC Advantages and disadvantages
TCC The scheme allows the application to customize the granularity of database operation , Reduced lock conflict , Can improve performance , But it also has the following disadvantages ：
- The application is invasive ,try、confirm、cancel All three phases require business logic implementation .
- Need to be based on the network 、 Different rollback strategies are implemented for different failure reasons such as system failure , It's hard to achieve , Generally with the help of TCC Open source framework ,ByteTCC,TCC-transaction,Himly.
Local message table
ebay At first, the scheme of local message table was proposed , To solve the problem of distributed transactions . At present, this kind of scheme is widely used in the industry , Its core idea is to split distributed transactions into local transactions for processing . Take a look at the basic implementation flow chart ：
Basic realization ideas
The sender ：
- There needs to be a message table , Record information about message status .
- Business data and message tables are in the same database , That is to make sure that they are in the same local business .
- After processing the business data and writing the message table in the local transaction , By writing a message to MQ Message queue .
The message will be sent to the message consumer , If sending fails , Try again .
Message consumer ：
- Processing messages in the message queue , Complete your business logic .
- At this time, if the local transaction is successful , It shows that the processing has been successful .
- If the local transaction fails , Then the execution will be retried .
- If it's a business failure , Send a business compensation message to the message producer , Notify to roll back and other operations .
The producer and consumer scan the local message table regularly , Send the unfinished or failed message again . If there is reliable automatic reconciliation and account supplement logic , This scheme is very practical .
advantage & shortcoming ：
The advantage of this scheme is that it solves the problem of distributed transaction well , Achieve final consistency . The disadvantage is that message tables are coupled to the business system .
Best effort notification
What is the biggest notification
Best effort notification is also a distributed transaction solution . Here is an example of enterprise online bank transfer
- The enterprise online banking system calls the front interface , Go to transfer page
- Enterprise online banking calls transfer system interface
- The transfer system completes the transfer processing , Send transfer result notice to enterprise online banking system , If the notification fails , Then the transfer system repeats the notification according to the strategy .
- The enterprise online banking system has not received the notice , Actively call the interface of the transfer system to query the transfer result .
- The transfer system will encounter situations such as refund of foreign exchange , I will come back to check the account regularly .
Try your best to inform the objectives of the program , That is to say, the party initiating the notification through a certain mechanism , Try your best to inform the receiver of the result of business processing . The best effort notification implementation mechanism is as follows ：
Try your best to inform the solution
Try your best to inform , May adopt MQ Of ack Mechanism .
- 1. The sponsor will send the notice to MQ.
- 2. The receiving party listens MQ news .
- 3. When the receiving party receives the message , Deal with the business , Respond ack.
- 4. If the receiving party does not respond ack, be MQ Meeting interval 1min、5min、10min Wait for a repeat notice .
5. The receiving party can use the message proofreading interface , Keep the message consistent .
Transfer business flow chart ：
The interaction process is as follows ：
- 1、 The user requests the transfer system to transfer .
- 2、 The transfer system completes the transfer , Send the transfer result to MQ.
- 3、 Enterprise online banking system monitoring MQ, Receive transfer result notice , If the message is not received ,MQ Notifications will be sent repeatedly . Transfer result received , Update transfer status .
- 4、 The enterprise online banking system can also actively query the transfer result query interface of the transfer system , Update transfer status .
Saga Things are done by Hector Garcia-Molina and Kenneth Salem Put forward , Its core idea is to split long transactions into multiple local short transactions , from Saga Transaction coordinator coordination , If it's finished normally, then it's finished normally , If a step fails , The compensation operation is called once in reverse order .
saga brief introduction
- Saga = Long Live Transaction (LLT, Long life affairs )
- LLT = T1 + T2 + T3 + ... + Ti（Ti For local short transactions ）
Every local transaction Ti There is a corresponding compensation Ci
Saga Execution order of
- Normal condition ：T1 T2 T3 ... Tn
Abnormal situation ：T1 T2 T3 C3 C2 C1
Saga Two recovery strategies
- Recover backward , If any local sub transaction fails , Compensation for completed transactions . Such as the execution sequence of abnormal conditions T1 T2 Ti Ci C2 C1.
- Recover forward , That is, retry the failed transaction , Let's say that at the end of the day, every sub transaction will succeed . Execution order ：T1, T2, ..., Tj( Failure ), Tj( retry ),..., Tn.
for instance , Suppose the user places an order , flowers 10 $to buy 10 More roses , Then there are
T1= Place the order ,T2= Deduct users 10 Yuan ,T3= User plus 10 Rose , T4= Inventory decrease 10 Rose
C1= Cancellation of order ,C2= Add... To the user 10 Yuan ,C3 = Less users 10 Rose , C4= Inventory plus 10 Rose
Suppose the transaction is executed to T4 An unexpected rollback occurred , stay C4 When we need to add the roses back to the stock , I found that the user's rose has been used , This is a Saga A disadvantage of , Problems caused by no isolation between transactions .
You can solve this problem by ：
- In response to ⽤ layer ⾯ Add ⼊ Logic of logic lock .
- Session layer ⾯ Isolation to ensure that the string ⾏ operation .
- The business layer ⾯ Mining ⽤ Pre freezing of funds ⾦ Of ⽅ To isolate this part of the resources ⾦.
- In the process of business operation, by reading the current status in time ⽅ Get updates .
Reference and thanks
- dried food | An article takes you to learn about distributed transactions
- Then someone asks you about distributed transactions , Throw this to him
- Talk about distributed transactions , And the solution
- Mysql Transaction implementation principle
- Detailed analysis MySQL Transaction log (redo log and undo log)
- 《Saga Distributed transaction resolution ⽅ Case and practice 》
- Best effort notification for distributed transaction solutions
Official account number
- If you think it's well written, please give me a compliment + Pay attention to , thank you ~
- At the same time, I am very much expecting my buddies to pay attention to my official account , Later, better dry goods will be introduced slowly ~ Hee hee
- C++ 数字、string和char*的转换
- Won the CKA + CKS certificate with the highest gold content in kubernetes in 31 days!
- C + + number, string and char * conversion
- C + + Learning -- capacity() and resize() in C + +
- C + + Learning -- about code performance optimization
C + + programming experience (6): using C + + style type conversion
Latest party and government work report ppt - Park ppt
Online ID number extraction birthday tool
Field pointer? Dangling pointer? This article will help you understand!
GVRP of hcna Routing & Switching
- LeetCode 91. 解码方法
- Seq2seq implements chat robot
- [chat robot] principle of seq2seq model
- Leetcode 91. Decoding method
- HCNA Routing＆Switching之GVRP
- GVRP of hcna Routing & Switching
- HDU7016 Random Walk 2
- [Code+＃1]Yazid 的新生舞会
- CF1548C The Three Little Pigs
- HDU7033 Typing Contest
- HDU7016 Random Walk 2
- [code + 1] Yazid's freshman ball
- CF1548C The Three Little Pigs
- HDU7033 Typing Contest
- Qt Creator 自动补齐变慢的解决
- HALCON 20.11：如何处理标定助手品质问题
- HALCON 20.11：标定助手使用注意事项
- Solution of QT creator's automatic replenishment slowing down
- Halcon 20.11: how to deal with the quality problem of calibration assistant
- Halcon 20.11: precautions for use of calibration assistant
- "Top ten scientific and technological issues" announced| Young scientists 50 ² forum
- Reverse linked list
- JS data type
- Remember the bug encountered in reading and writing a file
- Singleton mode
- 在这个 N 多编程语言争霸的世界，C++ 究竟还有没有未来？
- In this world of N programming languages, is there a future for C + +?
- js Promise
- js 数组方法 回顾
- ES6 template characters
- js Promise
- JS array method review
- 【Golang】️走进 Go 语言️ 第一课 Hello World
- [golang] go into go language lesson 1 Hello World