当前位置:网站首页>Essential for back-end programmers: distributed transaction Basics

Essential for back-end programmers: distributed transaction Basics

2020-11-08 12:11:44 osc_lg0msmnd

Preface

Recently I read several blogs about distributed transactions , Take notes . ha-ha ~

 Back end programmers must : Distributed transaction Basics

Database transactions

Database transactions ( abbreviation : Business ), It is a logical unit in the execution of database management system , It consists of a limited sequence of database operations , These operations are either all performed , Or none at all , It is an indivisible work unit .

Several typical features of database transactions : Atomicity (Atomicity )、 Uniformity ( Consistency )、 Isolation, ( Isolation) And persistence (Durabilily), The abbreviation is ACID.

 Back end programmers must : Distributed transaction Basics

  • Atomicity : The transaction is executed as a whole , The operations on the database contained in it are either all performed , Either not .
  • Uniformity : Before and after the transaction , The data will not be destroyed , If A Account to B Account transfer 10 Yuan , Whether it works or not ,A and B The total amount of is the same .
  • Isolation, : When multiple transactions are accessed concurrently , Transactions are isolated from each other , That is, one transaction does not affect the running effect of other transactions . in short , It's just that the water doesn't flow into the river .
  • persistence : Indicates that after the transaction is completed , Operational changes made by the transaction to the database , Will persist in the database .

How transactions are implemented

Local transactions

Traditional single server , Transactions in a single relational database , It's local business . Local transactions are managed by resource manager ,JDBC Transaction is a very typical local transaction .
 Back end programmers must : Distributed transaction Basics

Transaction log

innodb The transaction log includes redo log and undo log.

redo log( Redo log )

redo log Usually physical logs , What is recorded is the physical modification of the data page , Not how to modify a line or lines , It is used to recover the physical data page after submission .

undo log( Rollback log )

undo log It's a logical log , and redo log It's different to record physical logs . Think of it this way , When delete When a record ,undo log A corresponding insert Record , When update When a record , It records an opposite update Record .

Business ACID The realization idea of features

  • Atomicity : It's using undo log To achieve , If there is an error during the transaction execution or the user executes rollback, The system goes through undo log The log returns the status of the start of the transaction .
  • persistence : Use redo log To achieve , as long as redo log Log persistence , When the system crashes , You can pass redo log Recover data .
  • Isolation, : Through locks and MVCC, To separate affairs from each other .
  • Uniformity : By rolling back 、 recovery , And isolation in the case of concurrency , To achieve consistency .

    Distributed transactions

Distributed transactions : It refers to the participants in the transaction 、 Servers that support transactions 、 Resource servers and transaction managers are located on different nodes of different distributed systems . Simply speaking , Distributed transaction refers to the transaction in distributed system , It exists to ensure the data consistency of different database nodes .

Why do I need distributed transactions ? Next, I will explain it in two aspects :

Distributed transaction under microservice architecture

With the rapid development of Internet , Lightweight and well-defined microservices , On the stage of history . such as , A user places an order , Buy live gift service , Be divided into three service, They are gold coin service (coinService), Order service (orderService)、 Gift service (giftService). These services are deployed on different machines ( node ), Corresponding database ( Gold coin database 、 Order database 、 Gift database ) Also on different nodes .
 Back end programmers must : Distributed transaction Basics

Users place an order to buy gifts , Gift database 、 Gold coin database 、 The order database is on different nodes , It's not possible to use local transactions , So how to guarantee different databases ( node ) Data consistency on ? This requires distributed transactions ~

Distributed transactions under sub databases and sub tables

As the business grows , The data in the database is getting bigger and bigger , More than ten million levels of data , We need to divide it into databases and tables ( The company used to use mycat Sub database and sub table , Later used sharding-jdbc). A bank , The data is distributed on different nodes , For example, some of them are in Shenzhen computer room , Some are in Beijing computer room ~ You want to use local affairs to guarantee , I've been indifferent ~ You still need distributed transactions .

such as A turn 10 Block to B,A The account data is in Beijing computer room ,B The account data of is in Shenzhen computer room . The process is as follows :
 Back end programmers must : Distributed transaction Basics

CAP theory &BASE theory

Learn about distributed transactions , Of course, we need to understand CAP Theory and BASE theory .

CAP theory

CAP Theory as the basic theory of distributed system , In a distributed system , Consistency( Uniformity )、 Availability( Usability )、Partition tolerance( Partition tolerance ), These three elements can only achieve two points at most .
 Back end programmers must : Distributed transaction Basics

Uniformity (C:Consistency):

Consistency is the ability of data to be consistent across multiple copies . For example, a data is updated after a partition node , The data read from other partition nodes is also the updated data .

Usability (A:Availability):

Availability means that the services provided by the system must always be available , For each operation request of the user, the result can always be returned in a limited time . Here's the point " In a limited time " and " Return results ".

Partition tolerance (P:Partition tolerance):

Distributed systems encounter any network partition failure , There is still a need to be able to ensure that external services meet the requirements of consistency and availability .

choice explain
CA Give up partition fault tolerance , Enhance consistency and availability , In fact, it is the choice of traditional stand-alone database
AP Give up consistency , Partition fault tolerance and availability , This is the choice of many distributed systems when they are designed
CP Discard availability , Pursue consistency and partition fault tolerance , Network problems will directly make the whole system unavailable


BASE theory

BASE theory , It's right CAP in AP An extension of , For our business system , We consider sacrificing consistency for system availability and partition fault tolerance .BASE yes Basically Available( Basic available ),Soft state( Soft state ), and Eventually consistent( Final consistency ) Abbreviations of three phrases .

Basically Available

Basic available : It is realized by supporting local fault rather than global fault of the system . If you partition users in 5 Database server , The failure of a user database only affects the specific host 20% Users of , Other users are not affected .

Soft State

Soft state , The state can be out of sync for a while

Eventually Consistent

Final agreement , Finally, the data is consistent , Instead of being consistent all the time .

Several solutions for distributed transactions

Distributed transaction solutions mainly include the following :

  • 2PC( Two stage submission ) programme
  • TCC(Try、Confirm、Cancel)
  • Local message table
  • Best effort notification
  • Saga Business

    Two stage proposal

Two phase commit is a common distributed transaction solution . The commit of a transaction is divided into two phases : Preparation phase and submission of implementation plan .

Successful submission of phase II

Preparation stage , The transaction manager sends a prepare message to each resource manager , If the resource manager's local transaction operation is successful , Then return to success .

Submission execution phase , If the transaction manager receives a success message from all resource managers , Then send a submit message... To each resource manager ,RM according to TM The order execution submission of . Pictured :
 Back end programmers must : Distributed transaction Basics

Failure of two-stage submission

Preparation stage , The transaction manager sends a prepare message to each resource manager , If the resource manager's local transaction operation is successful , Then return to success , If execution fails , Then it returns failure .

Submission execution phase , If the transaction manager receives a message that any resource manager fails , Then send a rollback message to each resource manager . The resource manager rolls back the local transaction operation according to the instructions of the transaction manager , Release all lock resources used in transaction processing .

 Back end programmers must : Distributed transaction Basics

Advantages and disadvantages of two stage submission

2PC It's easy to implement , Lower cost , But it has the following disadvantages :

  • A single point of the problem : If the transaction manager fails , Explorer will always be locked .
  • Performance issues : All resource managers are in a synchronous blocking state during the transaction commit phase , Occupy system resources , Until the submission is complete , To release resources , It is easy to cause performance bottleneck .
  • Data consistency issues : If some resource managers receive a submitted message , Some didn't receive , It will lead to data inconsistency .

TCC( Compensation mechanism )

TCC Compensation mechanism has been adopted , The central idea is this : For each operation , To register a corresponding confirmation and compensation ( revoke ) operation .

TCC(Try-Confirm-Cancel) Model

TCC(Try-Confirm-Cancel) Distributed transaction is realized by decomposing business logic . For a specific business service ,TCC The distributed transaction model requires the business system to implement three sections of logic :

try Stage : Try to execute , Complete all business consistency checks , Reserve necessary business resources .

Confirm Stage : At this stage, confirm and submit the business , Don't do any tests , because try The stage has been checked , Default Confirm There is no mistake in the stage .

Cancel Stage : If business execution fails , Then enter the stage , It will release try All business resources occupied by the phase , And roll back Confirm All operations performed in phase .
 Back end programmers must : Distributed transaction Basics

TCC The distributed transaction model consists of three parts : Main business services 、 From business services 、 Business activity manager .

  • Main business services : The main business service is responsible for initiating and completing the entire business activity .
  • From business services : From business services is the whole business activity of the participants , Realization Try、Confirm、Cancel operation , For the main business service to call .
  • Business activity manager : The business activity manager manages and controls the entire business activity , Including recording transaction status , Call... From business service Confirm operation , Call... From business service Cancel Operation etc. .
    Let's take the user's purchase of gifts as an example to simulate TCC The process of implementing distributed transactions :

Suppose the user A The balance is 100 Gold coin , The present you have is 5 Duo .A It took 10 Gold coin , Place the order , Buy 10 Rose . balance 、 Order 、 Gifts are in different databases .

TCC Of Try Stage :

  • Generate an order record , The order status is to be confirmed .
  • Will the user A The balance in the gold coins of the account is updated to 90, Freeze the gold coin to 10( Reserve business resources )
  • The number of users' gifts is 5, The quantity to be added in advance is 10.
  • Try After success , Then enter Confirm Stage
  • Try There's anything wrong with the process , All enter Cancel Stage
     Back end programmers must : Distributed transaction Basics

TCC Of Confirm Stage :

  • Order status updated to paid
  • Update user balance to 90, It can be frozen as 0
  • The number of user gifts is updated to 15, Increase to 0
  • Confirm There's anything wrong with the process , All enter Cancel Stage
  • Confirm Process execution successful , Then the transaction ends
     Back end programmers must : Distributed transaction Basics

TCC Of Cancel Stage :

  • Modify order status to cancelled
  • Update user balance back 100
  • Update the number of user gifts to 5
     Back end programmers must : Distributed transaction Basics

TCC Advantages and disadvantages

TCC The scheme allows the application to customize the granularity of database operation , Reduced lock conflict , Can improve performance , But it also has the following disadvantages :

  • The application is invasive ,try、confirm、cancel All three phases require business logic implementation .
  • Need to be based on the network 、 Different rollback strategies are implemented for different failure reasons such as system failure , It's hard to achieve , Generally with the help of TCC Open source framework ,ByteTCC,TCC-transaction,Himly.

    Local message table

ebay At first, the scheme of local message table was proposed , To solve the problem of distributed transactions . At present, this kind of scheme is widely used in the industry , Its core idea is to split distributed transactions into local transactions for processing . Take a look at the basic implementation flow chart :
 Back end programmers must : Distributed transaction Basics

Basic realization ideas

The sender :

  • There needs to be a message table , Record information about message status .
  • Business data and message tables are in the same database , That is to make sure that they are in the same local business .
  • After processing the business data and writing the message table in the local transaction , By writing a message to MQ Message queue .
  • The message will be sent to the message consumer , If sending fails , Try again .

    Message consumer :

  • Processing messages in the message queue , Complete your business logic .
  • At this time, if the local transaction is successful , It shows that the processing has been successful .
  • If the local transaction fails , Then the execution will be retried .
  • If it's a business failure , Send a business compensation message to the message producer , Notify to roll back and other operations .

The producer and consumer scan the local message table regularly , Send the unfinished or failed message again . If there is reliable automatic reconciliation and account supplement logic , This scheme is very practical .

advantage & shortcoming :

The advantage of this scheme is that it solves the problem of distributed transaction well , Achieve final consistency . The disadvantage is that message tables are coupled to the business system .

Best effort notification

What is the biggest notification

Best effort notification is also a distributed transaction solution . Here is an example of enterprise online bank transfer
 Back end programmers must : Distributed transaction Basics

  • The enterprise online banking system calls the front interface , Go to transfer page
  • Enterprise online banking calls transfer system interface
  • The transfer system completes the transfer processing , Send transfer result notice to enterprise online banking system , If the notification fails , Then the transfer system repeats the notification according to the strategy .
  • The enterprise online banking system has not received the notice , Actively call the interface of the transfer system to query the transfer result .
  • The transfer system will encounter situations such as refund of foreign exchange , I will come back to check the account regularly .
    Try your best to inform the objectives of the program , That is to say, the party initiating the notification through a certain mechanism , Try your best to inform the receiver of the result of business processing . The best effort notification implementation mechanism is as follows :
     Back end programmers must : Distributed transaction Basics

Try your best to inform the solution

Try your best to inform , May adopt MQ Of ack Mechanism .

programme

 Back end programmers must : Distributed transaction Basics

  • 1. The sponsor will send the notice to MQ.
  • 2. The receiving party listens MQ news .
  • 3. When the receiving party receives the message , Deal with the business , Respond ack.
  • 4. If the receiving party does not respond ack, be MQ Meeting interval 1min、5min、10min Wait for a repeat notice .
  • 5. The receiving party can use the message proofreading interface , Keep the message consistent .
    Transfer business flow chart :
     Back end programmers must : Distributed transaction Basics
    The interaction process is as follows :


  • 1、 The user requests the transfer system to transfer .
  • 2、 The transfer system completes the transfer , Send the transfer result to MQ.
  • 3、 Enterprise online banking system monitoring MQ, Receive transfer result notice , If the message is not received ,MQ Notifications will be sent repeatedly . Transfer result received , Update transfer status .
  • 4、 The enterprise online banking system can also actively query the transfer result query interface of the transfer system , Update transfer status .

    Saga Business

Saga Things are done by Hector Garcia-Molina and Kenneth Salem Put forward , Its core idea is to split long transactions into multiple local short transactions , from Saga Transaction coordinator coordination , If it's finished normally, then it's finished normally , If a step fails , The compensation operation is called once in reverse order .

saga brief introduction

  • Saga = Long Live Transaction (LLT, Long life affairs )
  • LLT = T1 + T2 + T3 + ... + Ti(Ti For local short transactions )
  • Every local transaction Ti There is a corresponding compensation Ci

    Saga Execution order of

  • Normal condition :T1 T2 T3 ... Tn
  • Abnormal situation :T1 T2 T3 C3 C2 C1

    Saga Two recovery strategies

  • Recover backward , If any local sub transaction fails , Compensation for completed transactions . Such as the execution sequence of abnormal conditions T1 T2 Ti Ci C2 C1.
  • Recover forward , That is, retry the failed transaction , Let's say that at the end of the day, every sub transaction will succeed . Execution order :T1, T2, ..., Tj( Failure ), Tj( retry ),..., Tn.

for instance , Suppose the user places an order , flowers 10 $to buy 10 More roses , Then there are

T1= Place the order ,T2= Deduct users 10 Yuan ,T3= User plus 10 Rose , T4= Inventory decrease 10 Rose

C1= Cancellation of order ,C2= Add... To the user 10 Yuan ,C3 = Less users 10 Rose , C4= Inventory plus 10 Rose
 Back end programmers must : Distributed transaction Basics

Suppose the transaction is executed to T4 An unexpected rollback occurred , stay C4 When we need to add the roses back to the stock , I found that the user's rose has been used , This is a Saga A disadvantage of , Problems caused by no isolation between transactions .

You can solve this problem by :

  • In response to ⽤ layer ⾯ Add ⼊ Logic of logic lock .
  • Session layer ⾯ Isolation to ensure that the string ⾏ operation .
  • The business layer ⾯ Mining ⽤ Pre freezing of funds ⾦ Of ⽅ To isolate this part of the resources ⾦.
  • In the process of business operation, by reading the current status in time ⽅ Get updates .

Reference and thanks

  • dried food | An article takes you to learn about distributed transactions
  • Then someone asks you about distributed transactions , Throw this to him
  • Talk about distributed transactions , And the solution
  • Mysql Transaction implementation principle
  • Detailed analysis MySQL Transaction log (redo log and undo log)
  • 《Saga Distributed transaction resolution ⽅ Case and practice 》
  • Best effort notification for distributed transaction solutions

Official account number

 Back end programmers must : Distributed transaction Basics

  • If you think it's well written, please give me a compliment + Pay attention to , thank you ~
  • At the same time, I am very much expecting my buddies to pay attention to my official account , Later, better dry goods will be introduced slowly ~ Hee hee

版权声明
本文为[osc_lg0msmnd]所创,转载请带上原文链接,感谢