当前位置:网站首页>Flink CDC 2.1 正式发布,XTransfer技术专家贡献MongoDB CDC 连接器

Flink CDC 2.1 正式发布,XTransfer技术专家贡献MongoDB CDC 连接器

2021-11-25 18:18:57 InfoQ

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近日,在社区开发者们共同努力下,Flink CDC 社区宣布 Flink CDC 2.1 正式发布。据悉,2.1 版本包含 23 位贡献者贡献的 100+ PR,重点提升了 MySQL CDC 连接器的性能和生产稳定性,重磅推出 Oracle CDC 连接器和 MongoDB CDC 连接器。其中,XTransfer 的技术专家Jiabao Sun贡献了MongoDB CDC 连接器,支持从 MongoDB 数据库获取全量历史数据和增量变更数据。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"详解推出的MongoDB CDC 连接器","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MongoDB CDC 连接器并不依赖 Debezium,是在 Flink CDC 项目里独立开发。MongoDB CDC 连接器支持捕获并记录 MongoDB 数据库中实时变更数据,其原理是伪装一个 MongoDB 集群里副本 [4],利用 MongoDB 集群的高可用机制,该副本可以从 master 节点获取完整 oplog(operation log) 事件流。Change Streams API 则提供实时订阅这些 oplog 事件流的能力,可以将这些实时的 oplog 事件流推送给订阅的应用程序。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a5/a5ffb83243aa30e79b8264c91f352893.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从 ChangeStreams API 获取的更新事件中,对于 update 事件,没有 update 事件的前镜像值,即 MongoDB CDC 数据源只能作为一个 upsert source。不过 Flink 框架会自动为 MongoDB CDC 附加一个 Changelog Normalize 节点,补齐 update 事件的前镜像值(即 UPDATE_BEFORE 事件),从而确保 CDC 数据的语义正确性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用 MongoDB CDC 连接器,用户只需要声明如下 Flink SQL 就能实时捕获 MongoDB 数据库中的全量和增量变更数据,借助 Flink 强大的集成能力,用户可以非常方便地将 MongoDB 中的数据实时同步到 Flink 支持的所有下游存储。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/10/1050908e7ae3b992404bb6c476aed5bf.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个数据捕获过程,用户不需要学习 MongoDB 的副本机制和原理,极大地简化了流程,降低了使用门槛。MongoDB CDC 也支持两种启动模式:默认的initial 模式是先同步表中的存量的数据,然后同步表中的增量数据;latest-offset 模式则是从当前时间点开始只同步表中增量数据。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外,MongoDB CDC 还提供了丰富的配置和优化参数,对于生产环境来说,这些配置和参数能够极大地提升实时链路的性能和稳定性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"写在最后","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在短短的一年多时间里,Flink CDC 项目取得了现象级的发展和关注,这离不开 Flink CDC 开源社区的贡献者们的无私贡献,也离不开广大 Flink CDC 用户的积极反馈,正是这两者的良性互动才使得 Flink CDC 项目健康发展,这种良性互动也是开源社区的魅力所在。对此,XTransfer技术专家Jiabao Sun表示,“未来,XTransfer技术团队将持续关注开源社区并积极贡献力量。”","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"关于XTransfer","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"XTransfer是一站式外贸企业跨境金融和风控服务公司,致力于帮助中小微企业大幅降低全球展业的门槛和成本,提升全球竞争力。公司总部设立在上海,并在中国香港地区、深圳等主要外贸城市,以及英国、美国、加拿大、日本、澳大利亚、新加坡等地设有分支机构。目前,XTransfer已在中国香港地区、英国、美国、加拿大、澳大利亚获得当地支付牌照。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过与知名跨国银行及金融机构合作,XTransfer建设跨国大集团级全球多币种统一结算平台,并打造了以中小微企业为中心的,数据化、自动化、互联网化和智能化的反洗钱风控基础设施。XTransfer以科技为桥梁,链接全球大型金融机构和中小微企业,让中小微企业享受到和大型跨国集团企业同等水平的跨境金融服务。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文章整理自:微信公众号Apache Flink","attrs":{}}]}]}

版权声明
本文为[InfoQ]所创,转载请带上原文链接,感谢
https://xie.infoq.cn/article/e57f1cb42c9c5b67f9ae25ae1

随机推荐