当前位置:网站首页>It's so complete! Analysis of the whole process of a 100 million level sub database and sub table project

It's so complete! Analysis of the whole process of a 100 million level sub database and sub table project

2020-12-08 08:25:50 osc_ 8rbrmk98

There are a lot of articles about sub database and sub table on the Internet , But most of the content is scattered , Mainly to explain knowledge points , There is no complete description of the partitioning of a large table 、 New architecture design 、 The whole process of going online .

 

therefore , I combined with a large sub database and sub table project I did last year , Let's take a look at the complete sub database and sub table, from the architecture design to the actual summary of the launch .

 

One 、 Preface

 

Why do we need to do sub database and sub table ? I believe we all know something about it .

 

Massive data storage and access has become MySQL Database bottlenecks , Growing business data , No doubt MySQL The database is causing quite a load , At the same time, it puts forward high requirements for the stability and expansibility of the system .

 

And the resources of a single server (CPU、 disk 、 Memory, etc. ) It's always limited , The amount of data that the final database can hold 、 Data processing capacity will encounter bottlenecks .

 

At present, there are generally two options .

 

  • One is to replace the storage , Don't use MySQL, You can use HBase、polarDB、TiDB Equally distributed storage ;

  • If for various reasons , Still want to continue to use MySQL, The second way is usually used , That's sub database and sub table .

 

It was said at the beginning of the article that , There are a lot of articles on the Internet , There are many explanations about knowledge points , therefore , This paper will not repeat too much about the normal form processing of sub database and sub table scheme .

 

It focuses on the whole process from architecture design to release and launch , At the same time, summarize the precautions and best practices . There are five parts :

 

  • Business restructuring

  • Storage architecture design

  • Revamp and launch

  • Stability guarantee

  • project management

 

In particular, best practices at all stages , It's all about blood and tears .

 

Two 、 The first stage : Business restructuring ( Optional )

 

For microservices, the behavior of sub database and sub table is reasonable , Generally, you only need to pay attention to the change of storage architecture , Or you just need to make business transformation on individual applications , Generally, there is no need to focus on “ Business restructuring ” This stage , therefore , This stage belongs to “ Optional ”.

 

The first difficulty of this project , It's business restructuring .

 

And this split project involves two big tables A and B, There are nearly 80 million data in a single table , It's a legacy from the era of single application , There was no good domain driver from the beginning /MSA Architecture design , The divergence of logic is very serious , Up to now, it has involved 50+ Online services and 20+ Direct reading and writing of offline services .

 

therefore , How to ensure the thoroughness of business transformation 、 Comprehensiveness is the most important thing , There should be no omissions .

 

in addition , surface A and surface B Each has two 、 Thirty fields , There is a one-to-one correspondence between the primary keys of the two tables , therefore , In this sub database and sub table project , We also need to refactor and fuse the two tables , Will be redundant / Eliminate useless fields .

 

1、 Query statistics  

 

Online services are queried through a distributed link tracking system , According to the table name as the query criteria , Then aggregate according to the service dimension , Find all the relevant services , Write a document about the relevant teams and services .

 

Pay special attention here , Many tables are not only used by online applications , Many offline algorithms and data analysis businesses are also using , Here we need to sort it out , Do a good job in offline cross team communication and research work , In order to avoid affecting normal data analysis after switching .

 

2、 Query splitting and migration  

 

Create a jar package , according to 2.1 The statistics of , And services owner Cooperate to migrate all related queries in the service to this jar In bag ( This project jar Packet call projected).

 

Here is 1.0.0-SNAPSHOT edition .

 

And then put the xxxMapper.xxxMethod( ) All changed projectdb.xxxMethod( ) To call .

 

There are two advantages to doing so :

 

  • Convenient for subsequent query split analysis ;

  • Convenient for follow-up jar The query in the package is replaced with modified Middle desk service Of rpc call , The business side just needs to upgrade jar Package version , You can quickly get from sql The call is changed to rpc Inquire about .

 

This step actually took months , Be sure to sort out the services and do a comprehensive migration , Don't leave out , Otherwise, it may lead to incomplete analysis , Missing relevant fields .

 

The migration of query is mainly due to too many services involved in this split project , It needs to be folded into a jar package , It is more convenient for later transformation . If the actual database and table project only involves one or two services , This step can be omitted .

 

3、 Split analysis of union query  

 

According to the above jar Queries in packages , According to the actual situation, the query is classified and judged , Some problems left over from history , And the fields that have been discarded .

 

Here are some thoughts .

 

  • Which queries cannot be split ? For example, paging ( Transform as much as possible , It can't be changed in the form of redundant columns )

  • Which queries can be made in business join Split up ?

  • Which watch / Fields can be merged ?

  • Which fields need redundancy ?

  • Which fields can be discarded directly ?

  • According to the specific business scenarios and sql Overall statistics , Identify key sub table keys . The rest of the search platform .

 

After thinking, we get a general idea and scheme of query transformation .

 

At the same time, in this project, we need to merge two tables into one table , Discard redundant and invalid fields .

 

4、 New watch design  

 

This step is based on 2.3 For the split analysis of the query , Get the old table fusion 、 redundancy 、 The result of discarding fields , Design the fields of the new table .

 

After designing the structure of a new table , It must be sent to all relevant business parties review, And ensure that all business parties pass the design of the table . If necessary, it can be offline once review.

 

If the new table is in the process of , Some fields have been discarded , All business parties must be informed to confirm .

 

For the design of the new table , In addition to sorting out the fields , According to the specific inquiry , The redesign 、 Optimized index .

 

5、 First upgrade  

 

When the design of the new table is finished , Do it once jar In bag sql Query transformation , Update all the old fields to the fields of the new table .

 

Here is 2.0.0-SNAPSHOT edition .

 

And then let all services upgrade jar Package version , In order to ensure that these obsolete fields are not used , The new table structure field can completely cover the past business scenarios .

 

Special note , Because there are so many services involved , You can put the service according to Non core And The core distinguish , And then go online in batches , Avoid problems leading to serious failures or large-scale rollback .

 

6、 Best practices  

 

1) Try not to change the field name of the original table

 

When doing new table merging , At first, it was just a simple merger table A and surface B Table of , So many fields with the same field name have been renamed .

 

Later, in the process of field reduction , Removed a lot of duplicate fields , But didn't change the renamed field back .

 

In the process of leading to later online , Inevitably, the business side needs to refactor the field name .

 

therefore , When designing a new watch , Unless you have to , Do not modify the field names of the original table !

 

2) The index of the new table needs careful consideration

 

The index of the new table cannot simply copy the old table , It needs to be split and analyzed according to the query , The redesign .

 

Especially after the fusion of some fields , It may be possible to merge some indexes , Or design some indexes with higher performance .

 

7、 Summary of this chapter  

 

thus , The first phase of the database and table has come to an end . This phase takes time , It all depends on the specific business , If it's a business with a heavy historical burden , It may take months or even half a year to complete .

 

The quality of this phase is very important , Otherwise, the table structure may need to be rebuilt later in the project 、 We're going to get the full data again .

 

Again, here's how , For microservices, the services that are more reasonable are divided , The behavior of sub database and sub table only needs to pay attention to the change of storage architecture , Or you just need to make business transformation on individual applications , Generally, there is no need to focus on “ Business restructuring ” This stage .

 

3、 ... and 、 The second stage : Storage architecture design ( The core )

 

For any sub database and sub table project , The design of storage architecture is the core part !

 

1、 The overall architecture  

 

According to the first stage of sorting out the query results , We have summed up such a query rule .

 

  • 80% All of the above queries are through or with fields pk1、 Field pk2、 Field pk3 These three dimensions are used to query , among pk1 and pk2 There is a one-to-one correspondence due to historical reasons ;

  • 20% There are a lot of strange queries about , Including fuzzy query 、 Other field queries and so on .

 

therefore , We designed the following overall architecture , Database middleware is introduced 、 Data synchronization tools 、 Search engine ( Alibaba cloud opensearch/ES) etc. .

 

The following discussion is about this framework .

 

 

1) MySQL Sub table storage

 

MySQL The dimension of sub table is determined according to the result of query split analysis .

 

We found that pk1\pk2\pk3 You can override 80% The main queries above . Let these queries go directly according to the sub table key MySQL Database is ok .

 

In principle, the total data of one sub table can be maintained at most , Because too much full data will cause a waste of storage 、 The extra cost of data synchronization 、 More instability 、 It's not easy to expand .

 

But because of this project pk1 and pk3 All of the query statements have high requirements for real-time performance , therefore , Maintained pk1 and pk3 Two copies of full data as sub table key .

 

and pk2 and pk1 For historical reasons , There is a one-to-one correspondence , You can keep only one copy of the mapping table , Only store pk1 and pk2 Two fields .

 

2) Search platform index storage

 

Search platform index , Can cover the rest of 20% Scattered queries .

 

These queries are often not based on the sub table key , Or requirements with fuzzy queries .

 

For search platforms , Generally, full data is not stored ( Especially some big varchar Field ), Store only the primary key and index fields needed by the query , After the search results , According to the primary key mysql Get the records you need in storage .

 

Of course , From the result of later practice , There are still some trade-offs to be made here :

 

  • Some non indexed fields , If it's not big , It can also be redundant , Similar to overlay index , Avoid one more time sql Inquire about ;

  • If the table structure is simple , The field is not big , You can even consider full storage , Improve query performance , Reduce mysql Database pressure .

 

Here's a special tip , There must be a delay between the search engine and the database . So according to the sub table id Query statement , Try to make sure to query the database directly , This does not pose a risk of consistency issues .

 

3) Data synchronization

 

Generally, new and old tables can be used directly Data synchronization perhaps Double write processing , The two methods have their own advantages and disadvantages .

 

 

Generally, you can choose one way according to the specific situation .

 

See the overall storage architecture for the specific synchronization relationship of this project , It includes four parts :

 

① Synchronization from old table to new one

 

At first, in order to reduce code intrusion 、 Easy to expand , It adopts the way of data synchronization . And because of too much business , I'm worried that there are services that have not been counted and have not been transformed in time , So data synchronization can avoid data loss caused by these situations .

 

But in the online process, I found , When the delay exists , Many newly written records cannot be read , It has a serious impact on specific business scenarios .( Specific reason reference 4.5.1 Explanation )

 

therefore , In order to meet the real-time requirements of the application , We are based on data synchronization , Re in 3.0.0-SNAPSHOT The version has been transformed into double writing form .

 

② The synchronization of the new scale from the main scale to the secondary scale

 

③ The new table is from main table to mapping table to synchronization

 

④ New table full scale main table to search engine data source synchronization

 

②、③、④ They are all data synchronization from the new full scale main table to other data sources , Because there is no strong real-time requirements , therefore , For the convenience of expansion , All adopt the way of data synchronization , There's no more multiple writes .

 

2、 Capacity assessment  

 

Applying for MySQL Before storing and searching platform index resources , Capacity assessment is needed , Including storage capacity and performance indicators .

 

Specific online traffic assessment can be viewed through the monitoring system QPS, The storage capacity can be simply considered as the sum of the storage capacity of each table online .

 

But in full synchronization , We found that the actual capacity needed would be greater than expected , For details, see 3.4.6 Explanation .

 

The specific performance pressure test process will not be repeated .

 

3、 data verification  

 

As you can see from the above , In this project , There is a lot of business transformation , It belongs to heterogeneous migration .

 

From the past some sub database and sub table projects , Most of them are isomorphic / Peer splitting , So there won't be a lot of complicated logic , Therefore, the verification of data migration is often ignored .

 

In the case of full peer-to-peer migration , Generally, there are fewer problems .

 

however , Heterogeneous migration with more modifications like this , Verification is absolutely the most important thing !!

 

therefore , The results of data synchronization must be verified , Ensure that the business logic transformation is correct 、 Data synchronization consistency is correct . This is very, very important .

 

In this project , There are a lot of business logic optimization and field changes , So we have a separate verification service , The total amount of data 、 Incremental verification .

 

A lot of data synchronization was discovered ahead of time 、 Inconsistency of business logic , It provides the most important premise guarantee for the smooth launch of this project !!

 

4、 Best practices  

 

1)  Flow amplification caused by sub database and sub table

 

When we do capacity assessment , There's an important issue to focus on . It is the query traffic amplification brought by sub table .

 

There are two reasons for this traffic amplification :

 

  • Second query of index table . For example, according to pk2 Of the query , Need to go through first pk2 Inquire about pk1, And then according to pk1 Query return result ;

  • in Batch query of . If one select...in... Query for , The database middleware will use the sub table key , Split the query to the corresponding physical table , It is equivalent to an original query , Zoom in to multiple queries .( Of course , The database will be in the same table id As a batch query , And this is an unstable merger )

 

therefore , We need to pay attention :

 

  • Limit business as much as possible in Query quantity , Avoid excessive flow amplification ;

  • Capacity evaluation , This part of the amplification factor needs to be considered , Make appropriate redundancy , in addition , Later, it will be mentioned that the business transformation will be carried out in batches , Make sure you can expand the capacity in time ;

  • branch 64、128 still 256 The table has a reasonable estimate , The more you tear it down , In theory, the more you zoom in , So don't unnecessarily divide too many tables , Make an appropriate estimate based on the size of the business ;

  • For the query of the mapping table , Because of the obvious hot and cold data , So we added another layer of cache in the middle , Reduce the pressure on the database .

     

2) Sub table key change scheme

 

In this project , There is a business situation that changes fields pk3, however pk3 As a sub table key , It can't be modified in database middleware , therefore , Can only be modified in midfield pk3 The update logic of , Delete first 、 After the way to add .

 

Here we need to pay attention to , Transaction atomicity of delete and add operations . Of course , Simple processing can also be done by logging , Alarm and calibration .

 

3) Data synchronization consistency problem

 

We all know , One of the key points in data synchronization is ( news ) The order of data , If there is no guarantee that the order of the received data and the generated data is strictly consistent , Maybe because of ( news ) Data disorder brings data coverage , Eventually, there are inconsistencies .

 

The message queue used by our self-developed data synchronization tool is kakfa,,kafka For the storage of messages , Only local order can be achieved ( Specifically, each one partition Order of ). We can route messages from the same primary key to the same partition , In this way, consistency can generally be guaranteed . however , If there is a one to many relationship , There's no guarantee that every line changes in order , See the following example .

 

 

Then we need to get the latest data through the reverse search data source to ensure consistency .

 

however , The reverse search is not “ Silver bullet “, Two questions need to be considered .

 

  • If the message change comes from a read-write instance , And the reverse search Database is a read-only instance , There will be data inconsistency caused by delay in reading and writing instances . therefore , Need assurance The source of the news change and Check the database The same example of ;

  • There will be additional performance overhead on the database , The impact of full scale needs to be carefully assessed .

 

4) Data real time problem

 

There are several aspects to delay , And according to the actual situation of the business evaluation and measurement .

 

  • Second delay of data synchronization platform ;

  • If both the message subscription and the anti query database are on a read-only instance , In addition to the second delay of the data synchronization platform mentioned above , There will also be delays in master-slave synchronization of the database ;

  •   Second delay from wide table to search platform .

 

Only solutions that meet the business scenario , It's the right plan .

 

5) Storage capacity optimization after table splitting

 

Because of the data synchronization process , For a single table , Not strictly incrementally inserted , So there will be a lot of ” Storage holes “, The total amount of storage after synchronization is much larger than the estimated capacity .

 

therefore , When applying for a new library , Multi application of storage capacity 50%.

 

For specific reasons, please refer to my article 《  Why? MySQL The total storage size becomes larger after sub database and sub table ?》

 

5、 Summary of this chapter  

 

thus , The second phase of the database and table has come to an end .

 

There are a lot of holes in this stage .

 

On the one hand, it is designed to be highly available 、 Scalable storage architecture . As the project progresses , It has also been revised and discussed many times , Include mysql The amount of data redundancy 、 Index design of search platform 、 Flow amplification 、 Sub table key modification and other issues .

 

On the other hand “ Data synchronization ” It's a very complicated operation in itself , As mentioned in best practices in this chapter 、 Uniformity 、 One to many problems , We need to pay high attention to .

 

therefore , More dependent on data validation, the final business logic is correct 、 Correct verification of data synchronization !

 

After finishing this stage , You can formally enter the service switching stage . It should be noted that , Data validation will still play a key role in the next phase .

 

Four 、 The third stage : Revamp and launch ( Be careful )

 

After the first two stages are completed , Start the business switching process , The main steps are as follows :

 

  • The middle office service adopts single reading Double write The pattern of ;

  • Data synchronization from old table to new table ;

  • All service upgrades depend on projectDB edition , go online RPC, If something goes wrong , It can be rolled back if the version is reduced ( After successful launch , Read only the new library , Write both the old and the new );

  • Check the monitoring to make sure there is no Middle desk service Other services access the old database and old tables ;

  • Stop data synchronization ;

  • Delete old table .

 

1、 Query transformation  

 

How to verify whether the design of the first two stages is reasonable ? Whether the modification of query can be completely covered It's a prerequisite .

 

When the new table is designed , We can use the new table as the standard , Modify the old query .

 

Take this project as an example , Need to put the old sql stay In the new middle office service To transform .

 

1) Read query transformation

 

The query may involve the following aspects :

 

  • According to the query conditions , Need to put pk1 and pk2 Of inner join Change to the new table name corresponding to the sub table key ;

  • part sql The disposal of discarded fields of ;

  • Non sub table key query is changed to search platform query , Pay attention to semantic consistency ;

  • Pay attention to writing single test to avoid low level mistakes , Mainly DAO level .

 

Only the new table structure and storage architecture can fully adapt to query transformation , In order to think that there is no problem with the design in front of us for the time being .

 

Of course , There's also a prerequisite , That is, all related queries have been closed , There is no omission .

 

2) Write query transformation

 

In addition to changes to the relevant fields , what's more , It needs to be transformed into an old watch 、 Double write mode for new tables .

 

Here may be It involves specific business writing logic , This project is particularly complex , Need to be transformed in the process Communicate fully with the business side , Make sure the writing logic is correct .

 

You can add a configuration switch to each of the double writes , Easy to switch . If there is a problem in writing the new library in the double write process , It can be turned off quickly .

 

meanwhile , Do not close during double writing Old to new Data synchronization for .

 

Why? ? Because of the particularity of our project . Because we're involved in dozens of services , To reduce risk , You have to go online in batches . therefore , There is a troublesome intermediate state , Part of the service is the old logic , Part of the service is the new logic , The data correctness of the intermediate state must be guaranteed , Specific view 4.5.1 Analysis of .

 

2、 Service transformation  

 

Why a new one is needed Service How about the query after the transformation ?

 

On the one hand, it is to upgrade and roll back easily , The other is to collapse the query , As a service of China Taiwan, it can provide the corresponding query ability .

 

Put the modified new query in the service , then jar The original query in the package , Replace it all with the service's client call .

 

meanwhile , upgrade jar Package version to 3.0.0-SNAPSHOT.

 

3、 Services go online in batches  

 

To reduce risk , We need to arrange the batch launch from non core services to core services .

 

Be careful , In the process of batch online , Because writing services are often core services , So it's arranged in the back . There may be non core reading services online , There will be Read the new watch Write old tables In the middle of .

 

1) All related services use Refactoring branches upgrade projectdb Version to 3.0.0-SNAPSHOT And deploy the Intranet environment ;

 

2) Business services depend on Middle desk service , Subscription service required ;

 

3)  Open the branch of reconstruction ( Don't merge with normal iteration branches ), Deploy intranet , The intranet is expected to be tested for more than two weeks .

 

Use a new Refactoring branches It's for two weeks in the intranet test , It does not affect the normal iteration of business . Weekly updates of business branches can merge Deploy the intranet on the reconfiguration Branch , Then the Internet uses business branches merge To master Upper Department .

 

Of course , From the point of view that the branches of online and offline code are consistent , You can also refactor branches and test them online together with business branches , There will be a lot of pressure on development and testing .

 

4) In the process of batch online , If you have a problem with dependency conflicts , It needs to be solved in time and updated in this document ;

 

5) Before the service goes online , Business development or testing must be required , Clear assessment, specific api And risk points , Make a good return .

 

Here's another reminder , After the launch , Please do not miss the offline data analysis business ! Please do not miss the offline data analysis business ! Please do not miss the offline data analysis business !

 

4、 The old watch offline process  

 

1) Make sure that there are no access tables in the old library ;

 

2) Check the database for sql Audit , Make sure no other service is still reading the old table data ;

 

3) Stop data synchronization ;

 

4) Delete old table .

 

5、 Best practices  

 

1) You may not be able to read it immediately after writing

 

In the batch online process , You may not be able to read it immediately after writing . Because of the many businesses , We adopt the way of batch online to reduce the risk , There are some applications that have been upgraded , Some applications have not been upgraded . Services that are not upgraded still write data to the old table , The upgraded application will read data from the new table , When the delay exists , Many newly written records cannot be read , It has a serious impact on specific business scenarios .

 

There are two main reasons for the delay :

 

  • The writing service has not been upgraded yet , I haven't started double writing yet , Or old watch , There will be a new watch to read 、 Write the middle state of the old table , There is a synchronization delay between the old and new tables ;

  • In order to avoid the main reservoir pressure , The new table data is to get changes from the old table 、 Then check the data of the read-only instance of the old table for synchronization , There is a certain delay in the master-slave database itself .

 

There are generally two solutions :

 

  • Data synchronization is changed to double write logic ;

  • Make compensation in the read interface , If the new watch doesn't find , Go to the old watch again .

 

2) The database middleware is unique ID Replace auto increment primary key ( Focus on , On the blackboard )

 

Because after the sub table , Continue to use the auto increment primary key of a single table , Will cause global primary key conflict . therefore , Need to use distributed unique ID To replace the auto increment primary key . There are many algorithms on the Internet , In this project, the database is added automatically sequence generation .

 

Database autoincrement sequence Distributed ID generator , It's a dependency Mysql The existence of , Its basic principle is that Mysql Put a value in , Every time there's a machine to get ID When , Will be in the present ID Add up a certain amount, such as 2000, Then add the current value to 2000 Back to the server . In this way, each machine can continue to repeat this operation to get unique id Section .

 

But there's only one global one ID Is it done ? Obviously not , Because there will still be old and new watches id The question of conflict .

 

Because there are a lot of services , In order to reduce the risk, we need to go online in batches . therefore , There is a logic that some services or only write old tables , Part of the service is dual write logic .

 

In this state , Old watch id The strategy uses auto_increment. If there's only one-way data flow ( Old watch to new watch ), Just give the old watch id Reserve an interval ,sequence You can avoid conflicts by starting with a larger starting value .

 

But in this project , There are also new table data and old table data double write , If the above scheme is adopted , The larger id Write to the old table , Old watch auto_increment Will be reset to that value , This increases the number of services that write only old tables id There is bound to be conflict between the records of .

 

So here we exchange the intervals between the two sides , Old library from larger auto_increment The starting value starts , The new table selects id( That is to say sequence The scope of the ) From the largest record of the old table id Began to increase , Smaller than the old watch auto_increment The starting value to be set , It's a good way to avoid id The question of conflict .

 

① Before switching

 

sequence The beginning of id Set to auto increment of current old table id size , And then the old table's self increment id It needs to be bigger , Reserve an interval , Add to the old watch id Continue to use , It can prevent the data written to the old table from being synchronized to the new database id Conflict .

 

② After switching

 

There's no need for any modification , Disconnect data synchronization .

 

③ advantage

 

Just one code ;

 

Switching can be done with a switch , No need to upgrade ;

 

If the old watch is on the way autoincrement The data has become abnormally large , It won't cause any problems .

 

④ shortcoming

 

If the old table fails to write , The new watch has been written successfully , Log assisted processing is needed .

 

6、 Summary of this chapter  

 

After the old watch is offline , The transformation of the whole sub database and sub table is completed .

 

In the process , You need to always be in awe of online business , Think carefully about every possible problem , Think of a quick rollback solution ( It is mentioned in three stages projectdb Of jar Package version iteration , from 1.0.0-SNAPSHOT To 3.0.0-SNAPSHOT, Contains different changes for each phase , In the process of batch online in different stages , adopt jar Package version of the way to roll back , Played a huge role ), Avoid major failures .

 

5、 ... and 、 Stability guarantee

 

This chapter mainly emphasizes the guarantee means of stability again . As one of the important objectives of this project , Stability actually runs through the entire project cycle , Basically, it has been mentioned in each link above , Every link should be paid enough attention to , Carefully design and evaluate the plan , Know what you know , Instead of relying on the weather :

 

  • The new table design must be fully communicated with the business side 、 Guarantee review;

  • about “ Data synchronization ”, There must be data verification to ensure data correctness , There are many reasons for incorrect data mentioned above , Including real-time 、 The question of consistency . To ensure that the data is correct is the premise of online ;

  • Changes at each stage , We must do a quick rollback plan ;

  • Online process , In the form of batch online , Start with non core business , To avoid fault expansion ;

  • Monitoring alarm should be configured comprehensively , If there is a problem, you should receive an alarm in time , Respond quickly . Don't ignore , Very important , There have been several data problems , They are found and solved in time through alarms .

  • Single measurement , Business function test should be sufficient .

 

6、 ... and 、 Cross team collaboration in project management

 

About “ Cross team collaboration ”, This paper is specially mentioned as a chapter .

 

Because in such a large cross team project transformation process , Scientific teamwork is to ensure the overall project on time 、 An indispensable element of high quality completion .

 

below , Share some experiences and experiences .

 

1 、 All documents go first  

 

Teamwork is the most taboo “ a verbal statement without any proof ”.

 

Whether it's team work 、 Schedule or anything that requires a lot of collaboration , You need to have a documentation , Used to track progress , Control the process .

 

2、 Business communication and confirmation  

 

All table structure modifications , Must communicate with the relevant business parties , For possible historical logic , Make a comprehensive review of ;

 

All the fields after discussion are confirmed , Must be served by every Owner Confirm .

 

3、 Responsibility is in place  

 

For multi team and multi person cooperation projects , Each team should have a clear counterpart , Communication between the general project leader and the only counterpart of the team , Clear the team's complete schedule and completion quality .

 

7、 ... and 、 expectation

 

Actually , It can be seen from the length of the full text , Due to the complex business logic transformation of this sub database and sub table project , It takes a lot of time and energy , And it's very easy to , Online problems that cause instability .

 

In this paper, the whole sub database and sub table are re compiled 、 Design 、 The whole process of going online , I hope it can help you .

 

See here , We would like to ask . therefore , There's no better way ?

 

Maybe , In the future, we still need to combine the industry's new database middleware technology , It can quickly realize sub database and sub table .

 

Maybe , In the future, new data storage technologies and solutions can be introduced (polardb、tidb、hbase), There is no need for sub database and sub table at all ?

 

Keep up with the development of new technology , I believe we will find the answer .

 

The author's notes

Source: a pill note (ID:aone_note)

dbaplus The community welcomes contributions from technical personnel , Send email editor@dbaplus.cn

版权声明
本文为[osc_ 8rbrmk98]所创,转载请带上原文链接,感谢
https://chowdera.com/2020/12/20201208082517159w.html