当前位置:网站首页>Development history and future trend of DTM technology and business

Development history and future trend of DTM technology and business

2020-12-07 19:19:57 Aliyun yunqi

author : Chen Xiaoyong 、 Cogan

Alibaba data technology chronicles Brief history

2003 Taobao was born in a folk house in Hangzhou . The following year ,Google Three big data papers have been published to introduce computing technology into the era of big data .

2004 year Doug Cutting and Mike Cafarella according to Google The paper realizes Hadoop Of HDFS and MR Computing framework .

2006 year Hadoop Project entry Apache Community .

2008 year 9 month Hive Become Hadoop Subprojects , Then become Apache Top projects . Same year , Taobao began to implement based on Hadoop System data computing platform relocation - Aerial ladder 1.

2009 Alibaba cloud was born , Alibaba cloud began to write Maxcompute First line of code , Various cloud services are beginning to emerge in China .

2014 Alibaba launched the moon landing program in , Completion is based on Maxcompute Platform data platform migration - Aerial ladder 2, The whole business platform , Complete the construction of data public layer ,OneData The data platform of the system and group is gradually taking shape .

2014 year 4 month Intel investment Cloudera, Give up on your own Hadoop Distribution version , Same year Cloudera Enter the Chinese market .

2017 Taiwan products in annual data Dataphin Products come out , Support at the same time Maxcompute and Hadoop Big data platform ,OneData The internal technology system began to realize external empowerment .

2018 year Cloudera and Hortonworks Announced a merger ,Hadoop The distribution has gone from competition among multiple vendors to a game among oligarchs .

2020 Year based on Dataphin、 Brand data bank ,Quick Audience、Quick Stock The global marketing launch of Taiwan products in Data Center , Ali began to empower businesses through its own data system , From pure technology promotion to business value embodiment .

The idea of data media came into being

Traditional data processing , Especially the traditional data warehouse platform , Its hardware and software procurement costs , O & M costs 、 The technical threshold is quite high . Only banks 、 Operators and other large enterprises have the ability and financial resources to realize the platform construction of data warehouse and data mart . With the popularity of big data technology and cloud services , The operation and maintenance cost and technical development threshold of enterprises are greatly reduced , Especially the cloud service with high cost performance , Simple deployment , Nearly unlimited scalability and easy management , The comprehensive use cost and convenience are much better than the traditional data platform . therefore , Enterprises began to transform their data warehouse from traditional Teradata、Oracle/IBM Such as platform migration to big data platform or cloud services , today , This change is still in the traditional enterprises .

After the rise of Cloud Computing , Database and elastic Computing (ECS) It's the most common product , But as users accumulate data in the cloud business , Enterprises began to have a direct demand for data analysis .2011 Alibaba cloud maxcompute Big data platform online , Alibaba cloud has entered the era of big data .

With the exponential growth of data , There has been a qualitative change in the way and mode of data processing . The traditional data support mode for management personnel and a small number of business personnel can no longer meet the needs of business development . Data development cycle is long 、 Slow response 、 The disadvantages of narrow application are more and more prominent . Companies and governments are looking for ways to respond to market changes and data in a timely manner , At the same time, the collection of data 、 Development 、 Use and management put forward higher requirements .

Enterprises need to carry out the transformation of digital intelligence , To manage data more effectively , More convenient use of data . Alibaba's data technology and products department also realized that the way data processing must be changed , In order to meet the data development efficiency of enterprises , Data enabling business generates value and data guides the needs of enterprise operation management , At this point, the concept of Taiwan in data has been born . It helped Alibaba group stand out from the fierce competition in the next few years , And continue to help companies transition to future competition , Behind this trend war is the competition for commercial dominance .

The essence of data platform is to realize data value and data capitalization

Key product introduction :

Dataphin It is the data platform construction engine of intelligent data construction and management under Alibaba cloud . Based on the core methodology and technology system precipitated in Taiwan's practice in data , Provide data collection from , build , tube , Using the full link 、 One stop big data capability , To help enterprises to create a unified standard 、 Achieve mastery through a comprehensive 、 Capitalize 、 As a service 、 Closed loop self optimizing intelligent data architecture .

Dataphin The core value of is to standardize data definition , Use normalization 、 Standardized way to produce data , Improve the efficiency of data development .

Data center will open data to all staff , Supporting the business of data operations as the goal . The design idea of convenient data construction and business value perspective of data platform is the biggest difference from traditional data warehouse . Alibaba uses data for everyone , The idea that primary two is the main user of data , To process and develop data , Let front-line employees have data to see , There is data to support operational decisions , Have data to do business guidance .

OneData It is a methodology based on years of experience of Alibaba data technology team , The core is the construction of data public layer ,Dataphin It's a form of methodology solidified into the product , It helps Alibaba economies drive business change in the process of business transformation , Realize business value . Enterprises can also use these successful experiences and tools to improve data efficiency , Support their business and sustainability strategies .

OneData The core is the construction of data public layer . It is through the innovation of underlying services and agile development that Alibaba has enabled its huge customer base , Provide customers with mature methodology and tools out of the box , Help enterprises achieve business innovation . In today's business value creation oriented era , We see that data platform can promote the transmission of enterprise data value interest chain .

In Alibaba economy , Hundreds of data applications serve Taobao 、 Tmall 、 youku 、 Flying Pig 、 Alipay and other business departments . Outside the economy , Business Consultant 、 Brand data bank 、 Global consumer operation platform Quick Audience And other data applications help external businesses realize business value in Alibaba economy . Data and data tools will be implemented by more and more people 、 cargo 、 The connection and collaboration of fields .

Under the concept of Data Center , Data assets in addition to the underlying storage capacity 、 Beyond computing resources , We also need to build our own data asset management platform according to the organizational structure or development form of the enterprise , For insight into the health of enterprise data . There are also asset platforms within Alibaba enterprises that provide data health status information , It can provide data basis for system expansion in the next financial year .Dataphin The built-in data asset management module can reflect the basic status of data assets from the perspective of developers .

Enterprises need to carry out the transformation of digital intelligence , To manage data more effectively , More convenient use of data . Alibaba's data technology and products department also realized that the way data processing must be changed , In order to meet the data development efficiency of enterprises , Data enabling business generates value and data guides the needs of enterprise operation management , At this point, the concept of Taiwan in data has been born . It helped Alibaba group stand out from the fierce competition in the next few years , And continue to help companies transition to future competition . Behind this trend war is the competition for commercial dominance .

The status quo of the application of data medium platform

One 、 General industry data platform construction scenario

Traditional enterprises are looking forward to the data platform in business operation and management support . Out of the box tools can achieve efficient data output and data asset management . In the scenario design stage of data platform construction , Will conduct in-depth business research on traditional enterprises , Refine the business scene from the bottom of the silk , The business insight perspective that users are most concerned about is passed through BI The visualization of data analysis report is presented in front of people , Assist decision makers in making scientific judgments .

Thousands of derived indicators are derived from the business scenario design phase of data platform , These derived indicators have time limited details 、 The definition of indicators is clear and unambiguous , There are many combination conditions among indexes .Dataphin It can quickly realize data processing and development , Graphic design reduces the threshold of data platform development and design , And plan from the warehouse 、 Data integration 、 Specification modeling 、 General development IDE、 From operation and maintenance scheduling to data service, the goal of traditional enterprise data modeling and data development can be achieved quickly .

In the data, the data assets gathered in the platform are like a “ gold ”, For enterprises , Data center must solve how to manage data , How to use it . Through centralized data asset management, it is convenient to comprehensively evaluate the use and value of assets , Construct the whole link tracking system of data application , On the cost of data 、 Clear business benefits 、 transparent 、 Assessable . Traditional enterprises have diversified business systems 、 Design independence and other reasons lead to the formation of data chimney development situation . Through the centralized management of data assets, the enterprise can master the overall status of data assets , Vertical sector 、 The operation status of horizontal level is presented transparently , Lay a solid data foundation for scientific data decision .

A traditional enterprise customer , They have a large number of retailers and stores across the country , Marketing costs remain high , Because the operation data is in the store and each subsystem , It's hard for headquarters to find out why . Through the construction of Data Center , After collecting system data and store marketing data , By analyzing consumption data 、 Integral accumulation and integral consumption data , Found abnormal behavior members , Their consumption in the store is concentrated in the evening 10 After that , This period of time is just the closing state of the store , Suspected to be the result of the wool party cheating . Through the centralized management of data in the data center , It can supervise the actual activity sales volume of the subordinate stores of each business unit . Customized through data “ Asset visualization portal ” Help enterprises to effectively manage their own data assets .

As a traditional enterprise, it represents a telecom operator 、 An airline passed by 10 Years of data warehouse construction , Already have a data analysis platform , But the traditional data warehouse only focuses on data development , There is no concept of scenario design and asset management , When there is a new data development task , Developers often need to do layer by layer processing from paste source data , It is not only time-consuming, but also has the phenomenon of unclear definition . And these phenomena can be achieved by using Dataphin, Introduce a standard data common model to solve the problem .

“ Promoting the construction of business and data center is one of the eight tough battles for airlines this year , It is also a key change in the process of intelligent transformation of the company . In the past , It needs to be collected manually from different systems 、 Data that can only be obtained by running on their computers for dozens of hours , Now you can get data from “ Cloud ” Easy access to , It greatly improves the efficiency and quality of analytical work .” The person in charge of the Taiwan Project in the airline data expresses .

Two 、 Retail industry wide data in Taiwan marketing scenarios

The new retail industry has a new format of sales model , Businesses go through stores 、 Online shop 、 Live platform 、 brand App、 WeChat / Alipay small programs and other promotional products . There are many forms of marketing 、 Alibaba has launched a global marketing solution , Aggregate global data through AIPL/RFM Data model for deep insight , Through precision delivery , Improve marketing efficiency , Realize business value . Global marketing solution is based on Alibaba business advisor 、 Brand data bank 、 Data construction and management platform Dataphin、 Global consumer operation platform Quick Audience Wait for a series of data products to achieve .

In global marketing, the most important thing is to help users find the target audience , Bring business value to business through crowd forecast model and marketing launch , Therefore, the premise of the implementation of global marketing forecasting technology is to gather various formats / Data generated by channels , And Alibaba OneData Methodology to process to achieve global digital marketing , This field AI The computing power of the algorithm platform has direct scene application and business value embodiment . Through the model construction and data output, it makes the business operation status of the business 、 Member insight 、 Channel and sales management 、 Data management of the whole store . Through data analysis , Decision makers can make business judgments , It can also be predicted through the market (predictive Marketing) The model provides market forecast for global marketing .

Global marketing solution is to build data platform for enterprises to cooperate with Alibaba business ecology , An important way to get business value . The value data deposited by the enterprise's data, Alibaba's business ecosystem and other media channels jointly build digital marketing , And it can return the external data , Form a full link data closed loop .

Flying crane dairy 、 Ryohin keikaku shop 、 Jialan and other new retail enterprises build through the global data platform , Use Dataphin To tmall shop 、 Offline stores 、 Applet 、 Own website and other data for unified management , Building unity 、 standard 、 High quality data , Support data decision-making and global marketing , Realize business value . As customers say :

“ Data center can liberate data infrastructure , Let's have more energy to think about how to use data to solve business pain points 、 Improve the efficiency of the company ; So in terms of the ability requirements of the organization , We can also be more inclined to business analysis and architecture capabilities 、 Data model algorithm capability 、 The development of innovative application product design and planning capabilities .” Liang pin shop vice president Zhou Shixiong said in an interview .

Zhong Wei, general manager of big data center of Galan group, said in an interview that ” We have gold in our hands ( Consumer data ), But there is a lack of development methods . The digital technology embodied in data center is equivalent to new productivity , Can drive the enterprise through the establishment of new production relations to match it , For example, organizational upgrading 、 Ecological coordination promotes business model 、 A breakthrough in business model , The change brought about by this breakthrough is DNA Grade ”.

The future trend of Data Center

One 、 The trend of real-time computing in Data Center

Data processing to quasi real time 、 Real time trend direction development . The traditional design of data warehouse is limited to the technical system and cannot realize real-time calculation . The distributed big data technology can not only realize the construction of PB Level of Data Center ( Historically, this kind of computing scenario is called data warehouse ) And it can combine real-time computing with historical data , Realize the integration of streaming and batch development . Meet the data timeliness and analysis ability emphasized by the new generation data center .

Alibaba adopts Blink(Flink Open source version ) Real time computing framework realizes streaming and batch integration ,Blink Ability to handle complex events (Complex Event Process), It can also provide developers with different requirements and capabilities SQL/Table、 Real time streaming batch data processing 、 State event driven applications API And so on , Respond to the needs of different data development .

The real-time computing technology of data center is not to reengineer the original business process , But through the combination of real-time data flow and data warehouse indicators to achieve more efficient business analysis . Using real-time technology can be done quickly BI Analysis and business alerts , Such as real-time marketing strategy 、 Real time risk control strategy 、 Real time anti fraud . These scenarios can be embedded into the actual business system .

Alibaba's new retail business 、 double 11 The shopping Carnival also uses flow and batch , Real time monitoring of marketing process .

Dataphin Products in the 2018 Since then, it has been put into R & D with the integration of batch and flow , stay 2019 At the end of the year, internal flow computing products were successfully migrated to Dataphin Products .2020 year Dataphin Release v2.7 edition , Started to support Alibaba cloud real-time computing products Flink, And Alibaba cloud big data computing service Maxcompute combination , Through the flow batch integration technology to meet the demand of data timeliness . The user can go through Dataphin The product realizes the real-time feedback of marketing effect, and analyzes and compares with the historical data in the same dimension , Provide real-time and accurate data to business personnel for real-time decision-making .

Two 、 The trend of mobile terminal application in the upper layer of data

BI Insight analysis is the most important way to present data in data , At this stage, most of BI The presentation is PC End oriented , The mobile phone is the supplement . The Internet is made up of PC An inevitable trend in the development of end-to-end mobile terminals is that data applications are also becoming mobile terminals . In recent years , In the field of digital analysis , Multiple BI Manufacturers have released supporting products for mobile terminal display , But it's not widely available in the market , The reason is that the screen size is difficult to unify , There is also a high degree of personalization of mobile terminal audience scenarios , Therefore, the application of mobile terminal in data center must adapt to the requirements of the terminal .

In number BI field , Its termination must consider end to end adaptation , More in the form of digital indicators Kanban , Not like it PC To highlight the rich presentation effect and historical indicators . The second is the terminal App Combined with real-time computing , Emphasize the ability to analyze real-time data , The content presented should be timely , More applications in business traffic 、 Real time orders and historical orders analysis and forecasting scenarios .

In addition to the existing difficulties of mobile terminal, it is necessary to iOS and Android Two systems App Outside development , It also faces multiple end presentation problems , Nailing micro applications and wechat applets has become an enterprise apart from App External data BI Other options in terminal , But on a technical level , pure H5 Page development faces a large amount of download data , Poor use experience , Can not achieve offline data maintenance and browsing and other issues , So most mobile applications still use App How to achieve .

Because of the terminal App The cost of development and operation and maintenance is high ,PV/UV Operational efficiency issues , Therefore, what kind of data and application mode can improve the frequency of data users is a practical problem for enterprise managers and product managers . Most of the analysis data in the data center is T+1 Analysis index of , It has a very important reference for enterprise managers , But there's no hour and minute level frequency , therefore App The data presented on should be based on the business and marketing activities of the enterprise , In particular, multi terminal buried point data acquisition 、PV/UV data , Combined with historical data analysis, it can better reflect App BI The business value of .

**
3、 ... and 、 The trend of intelligent development of Data Center **

AI The most important value of technology is that it can be used in real situations , For example, a typical application scenario of face recognition is to realize mobile login instead of password . After building the data center , Enterprise users can accumulate abundant index data , These data are algorithms and AI The basis of dependence . Taiwan users are more common in the data AI The application scenario is sales or traffic forecast , Recommendation algorithm for thousands of people and thousands of faces , Prediction of marketing activities, etc . These are scenarios that directly assist business decisions .

Under the pressure of fierce market competition , Companies expect AI Computing can help achieve sales growth or cost reduction in a short period of time . Actually, through AI The algorithm provides the convenience of data for front-line employees, and it is also a great way to improve production efficiency . Alibaba has such a data product inside , Employees can ask it vague questions , The product directly responds to the index data that employees and users are concerned about , Lower the threshold of data query , To facilitate the use of front-line staff .

“ Human law and land , Earth method sky , Tianfa Dao , Natural rule ”, Law is restriction 、 Control and control , People take the land as the standard of conduct , The earth is regulated by heaven , The heaven is the norm , Tao takes nature as its norm . It is also true of enterprises , The operation of enterprises depends on data support , Data support depends on the system 、 The system depends on the data center , Data center follows the methodology of data processing and multi terminal presentation , Therefore, the processing of data processing is the key to the successful landing of data medium platform .

Link to the original text
This article is the original content of Alibaba cloud , No reprint without permission .

版权声明
本文为[Aliyun yunqi]所创,转载请带上原文链接,感谢
https://chowdera.com/2020/11/20201126100059257l.html