当前位置:网站首页>August 14, 2020: what are the execution engines for data tasks?

August 14, 2020: what are the execution engines for data tasks?

2020-11-06 21:55:49 Fuda Dajia architect's daily question

Fogo's answer 2020-08-14:

Know the answer

The most basic thing is crontab Timing task .
Considering the distributed partition timing task and high availability , It can be used elastic-job Such a framework . But such a framework cannot handle complex task dependencies .
Big data processing frameworks generally use DAG( Directed acyclic graph ) Workflow scheduling system , adopt DAG The graph describes the complex dependencies between tasks . stay Spark In job scheduling system , Is based on DAG describe rdd Dependencies between .

版权声明
本文为[Fuda Dajia architect's daily question]所创,转载请带上原文链接,感谢