当前位置:网站首页>Explore the solution of complex timed tasks: distributed task scheduling system

Explore the solution of complex timed tasks: distributed task scheduling system

2021-01-10 01:09:01 InfoQ

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In this paper, we will explain the architecture and technical implementation for you "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#888888","name":"user"}}],"text":" Tencent cloud distributed task scheduling system TCT(Tencent Cloud "},{"type":"text","text":"Task) How to achieve "},{"type":"text","marks":[{"type":"color","attrs":{"color":"#888888","name":"user"}}],"text":" Accurate and real-time task scheduling 、 Stable and efficient , And task segmentation "},{"type":"text","text":" And Choreography ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" Background introduction "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Origin, origin, and extinction , It has its own cause and effect "}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" First , Let's think about a few business scenarios :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XX Credit Card Center , monthly 28 Early morning 1:00 To 3:00 It is necessary to complete the generation of the cost list of the whole network users in the current month ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XX dress , It needs to be every morning 9:00 Start sending birthday message to members ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XX Gaming platform , After new users sign up , Need to generate timed tasks for the current user , Clearing the amount of Commission for virtual currency exchange at the end of the month ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XX company , It needs to be executed regularly Python Script , Clean up the invalid file service system tmp file ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XX The insurance company , It needs to be early every morning 2:00 Count the number of new policies added the previous day , And trigger the report generation task , CC mail when done ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Similar to the above business scenario of batch processing massive timing tasks , From single architecture to micro service architecture 、 Cloud service architecture has been common in the evolution process , be based on Quartz The conventional scheduling framework based on XML is no longer able to meet the requirements of this distributed scenario , It can't realize the accurate real-time task scheduling 、 Stable and efficient , Also can not achieve the task segmentation 、 layout 、 Failure to add . Therefore, enterprises urgently need a one-stop distributed scheduling solution , Help enterprises to manage the complicated and complicated regular tasks , Enhance the service ability of enterprise micro service platform , Support enterprise cloud service transformation ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" Existing open source solutions "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" The stone of its mountain can be used to attack jade ..."}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In the past development , Our predecessors have left behind many excellent programs , Each have advantages and disadvantages . Common open source products : Quartz、XXL-Job、ElasticJob、Antares、SIA-TASK etc. ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Quartz: The framework is the most widely used , It's all based on Java Realization ,Quartz The control of a single task is basically the ultimate , With its powerful function and application flexibility , Become the authority in the field of open source task scheduling and similar open source products such as Antares Cornerstone ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XXL-JOB: A lightweight distributed task scheduling platform , Its core design goal is rapid development 、 Learn easy 、 Lightweight 、 Easy to expand .XXL-JOB Support fragmentation , Support simple task dependencies , Support subtask dependencies , Cross platform is not supported ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elastic-Job: Support task sharding ( Consistency of work segments ), No choreography , Cross-platform is not supported ;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SIA-TASK: With cross platform 、 It can be arranged 、 High availability 、 No intrusion 、 Uniformity 、 Asynchronous parallel 、 Dynamic expansion 、 Real time monitoring and other features ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/23\/d1\/234a9afdac506dc21e555313yy1fa0d1.png","alt":null,"title":" Open source solution technology implementation diagram ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ef\/20\/ef2d2d8612e9ec3942d4a68716373f20.png","alt":null,"title":" Open source solution technology implementation diagram ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" From the logical architecture and technical implementation of open source solutions , We can also intuitively see the shortcomings of open source solutions :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In terms of Architecture : The scheduler's responsibilities are not clear 、 Lack of system scalability . In the face of massive virtualization & Complex network environment , It's not enough for a simple call ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Performance aspect :ZooKeeper Clusters are accompanied by an increase in the number of tasks and high-frequency events , Become the bottleneck of system performance . Jane's remote call or task pull scheme , Can not meet the demand of large quantity and high frequency business ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Functional aspects : Lack of complete authentication system design , Security cannot be guaranteed . Task intervention 、 Monitoring and alarm system operation and maintenance capabilities are weak ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"TCT brief introduction "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In order to solve the above problems , We have made a deep exploration , And designed a set of enterprise level distributed task scheduling system TCT(Tencent Cloud Task).TCT Provide a one-stop distributed scheduling solution , Support random 、 Broadcast multiple task types , With task segmentation 、 Task scheduling ability , Provide perfect monitoring and alarm system . We combine the user's actual business scenarios , Drawing on historical experience , It mainly solves several core problems :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/65\/43\/659904be321f6665041aeb6fe0222443.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" The above core elements , The requirements for the system vary , The following summary can be provided for reference :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/95\/3d\/957c2b6d0eafb4a4226c6e564950843d.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" Technology Architecture "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f1\/2a\/f1ef9a9281e4d6c8d7639a8ee4f44d2a.png","alt":null,"title":" Technical architecture ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Let's explain the functional modules in the architecture diagram :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/94\/77\/9405dae600b6a233d299fa5e4fc06077.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" Functional architecture "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0d\/f6\/0d68fa2d02d7e7a2614a8c586e2242f6.png","alt":null,"title":" Functional architecture ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" In this way, a distributed task scheduling system is designed , It has the following advantages :"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#006EFF","name":"user"}},{"type":"strong"}],"text":" Advantages one : Modular microservice architecture design , Clear responsibilities "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" trigger "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Just follow the task execution rules , Calculate and analyze the task trigger events at different time points . adopt MQ The realization of reliable delivery ( Later articles will gradually explain how to achieve reliable delivery ), It can cut the peak and fill the valley , Avoid peaks IO Other questions , Increase throughput ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Through reasonable partition strategy and disaster recovery strategy , Solve the traditional multi node lock competition rotation analysis loading strategy , Reduce the pressure on storage ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Hot and cold data isolation loading mechanism , Further reduce storage pressure and system overhead ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" According to the high frequency task execution strategy , The preloading strategy and dynamic adjustment of preloading algorithm are adopted , Solve the problem of high load caused by high frequency trigger ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" Scheduler "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" The most complex control logic component in the whole task scheduling system ,IO Dense components ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" By subscribing MQ News events , Decouple from trigger , Effectively improve the throughput of the system ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Focus on logical control of task scheduling , Such as task execution scheduling 、 Load balancing 、 Fault tolerance 、 Current limiting 、 Billing, etc ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" Access gateway "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Independently undertake client access authentication and authentication , Provide effective authority verification policy ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Responsible for the call back management of uplink and downlink channels , Decouple from complex business logic ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Client node and service node online and offline automatic detection sensing mechanism , Effective implementation of session management ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Data transparent transmission and routing , Realize closed loop within components ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" coordination SDK\/Agent Side design , It effectively avoids the bottleneck of single node connection number and high concurrency in the cold start scenario of service nodes tcp The problem of establishing a connection ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#006EFF","name":"user"}},{"type":"strong"}],"text":" Advantage two : Stateless design , Simple horizontal expansion "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" trigger "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Through effective fragmentation strategy , In the case of avoiding triggering stress concentration , Can quickly complete the elastic expansion and reduction of services , Achieve almost stateless horizontal expansion ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" Scheduler "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" A completely stateless design , There is no need to consider the back to source problem of the task , Achieve stateless horizontal expansion ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" Access gateway "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" A completely stateless design , It can realize stateless horizontal expansion , In theory TCP There is no upper limit to the number of connections ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#006EFF","name":"user"}},{"type":"strong"}],"text":" Advantage three : Fully functional "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" Flexible trigger rules "}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Support Cron expression , for example * 0\/5 * * * ? etc. ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Trigger rules for specific cycle frequencies , for example interval 36 Minutes, etc ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Convenient management ability , Provide pause 、 recovery 、 stop it 、 Try again and so on a variety of control capabilities ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d9\/10\/d9d9e3885e956a03456yy447cdff8010.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/15\/51\/1522d210f3123167c8b7c4a614ac6151.png","alt":null,"title":" task management ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" Three execution modes are supported "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Random node execution : Select an available execution node in the cluster to execute the scheduling task . Applicable scenario : Regular reconciliation ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Broadcast execution : In the cluster, all the execution nodes distribute and execute the scheduling tasks . Applicable scenario : Mass operation and maintenance ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Shard to perform : Split according to user-defined slicing logic , Distributed to different nodes in the cluster for parallel execution , Improve the efficiency of resource utilization . Applicable scenario : Massive log statistics ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d5\/54\/d528ce9a8f7e010420552646e7836654.png","alt":null,"title":" Task scheduling execution mode ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}},{"type":"strong"}],"text":" Three trigger modes are supported "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Manual trigger : The user selects a specific task from the task management list to execute it manually once , The scheduler immediately distributes tasks , And generate an execution batch . Applicable scenario : Periodic execution tasks supplement ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Cycle triggers : Set the execution time of the task by setting the interval time of task triggering ; Can support cron Period setting not supported by expression . Applicable scenario : Scheduled backup ."}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Workflow triggers : A workflow is a set of tasks , You can choreograph the upstream and downstream logical dependencies of the task , Trigger the task . Applicable scenario : Massive data processing , Such as data collection , Data filtering , Data cleaning , Data aggregation process choreography ."}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/52\/46\/520d2bf0yyfc873764baa1de0f46ca46.png","alt":null,"title":" Task trigger mode ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}},{"type":"strong"}],"text":" Log traceability capability "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#000000","name":"user"}}],"text":" Through the logging service , It is convenient for users to query the task execution log . Users can record the execution batch details of all tasks through execution , It can stop the execution of the batch whose current status is in execution , It can trigger the re execution of the operation for the currently terminated batch ; Click batch ID Enter the execution details of the batch , Click task ID Enter the execution batch list of the task , Click execute deployment group to enter the resource details list ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ac\/e9\/ace8a906e417b45638f12acda33e51e9.png","alt":null,"title":" Log query ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":" Support complex task scheduling capabilities "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" It can realize task workflow in various scenarios . The complex task scheduling logic is completed by constructing the upstream and downstream dependencies of scheduling tasks . It is suitable for big data processing 、 Task execution work order 、 Batch operation and maintenance process arrangement and other application scenarios ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7c\/80\/7cef8bebf0c42b6a975c68c4d7706280.png","alt":null,"title":" Task arrangement ","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":" summary "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" A platform system , From product function to technical architecture, there are challenges in all aspects , It needs layers of abstraction and gradual optimization to complete the landing of a mature product . In the age of big data , In the face of massive data and user scale , Any kind of Architecture Design , Are facing network response 、 Fault tolerance 、 idempotent 、 Data reliability \/ Consistency and so on ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" For the platform , The reliability of the task is the first priority , Second, the timeliness of task execution . Reasonable functional modular split , For different scenarios , Design different extensions , Guarantee SLA Improve the overall throughput of the system , Achieve reliable and effective access to , Deal with high frequency and large volume of business scenarios ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" For users , Diversified management means 、 Multi dimensional operation index query , Comprehensive link monitoring is what users pursue , The only way to get the user out of the complicated and chaotic scene of scheduled tasks is to let users out of the scene , In order to be more focused on business research and development ."}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" The first figure :Unsplash"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" author : Lee Teng Ko 、 Liu Yan "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" original text :"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/KEAZ0w11pxKoGKP468X_HQ","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/mp.weixin.qq.com\/s\/KEAZ0w11pxKoGKP468X_HQ"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" original text : Explore solutions to complex timing tasks : Distributed task scheduling system "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" source : Tencent cloud intermediate - WeChat official account [ID:gh_6ea1bc2dd5fd]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" Reprint : The copyright belongs to the author . Commercial reprint please contact the author for authorization , Non-commercial reprint please indicate the source ."}]}]}

版权声明
本文为[InfoQ]所创,转载请带上原文链接,感谢
https://chowdera.com/2021/01/20210110010750605n.html

随机推荐