writing / Ren Xianghui, founder of Mingdao cloud
Around cloud computing related technology areas 、 Technical terms and products are dazzling . At the beginning of Cloud Computing , The application development environment is relatively simple , At that time, there were so-called full stack engineers , It means that if you don't think about the development cycle , One person can handle the whole application . today , This title is not worthy of its name . Very few people , Even an enterprise can master all the technology stacks related to cloud computing . They may apply some of the work done by others , Combined with some of our own proprietary experience , To form a competitive product in a certain market segment , Or deliver the desired output to the customer .
Even as a pure user , To fully understand the technologies related to cloud computing , Make a reasonable structure , Choose the right type , Successfully complete the whole process of integrated development and deployment , It's also much harder than ever before , The technical talent needed is also more expensive than in the past . Frankly speaking , In today's talent competition , Business in general , Even with the information department , It's not likely to be able to manage such a complex development facility on its own , They will have to rely extensively on services provided by cloud computing platforms . This brings new market opportunities to the solution providers in the software industry . Who can provide a friendly application development and deployment environment for digital transformation enterprises , Who can get and keep customers .
This long article is mainly for technical and non-technical managers of large and medium-sized enterprises . I describe the development of cloud computing technology and the market area , Introduce key technologies and market milestones , Including core open source projects in different technology domains , It enables enterprises to have a comprehensive understanding of the history of cloud computing and related technology domains . With an overall understanding , It will be easier for you to see how the enterprise should use cloud computing , Where are the potential market opportunities and challenges in the future ？
This article is subject to Tom Siebel 2019 Published in 2002 Digital Transformation The inspiration of a Book , But I try my best to combine the actual situation of the Chinese market to tell .
The formation and structure of the cloud computing market
Today, we can enjoy the economic and convenient cloud computing services , It mainly comes from two major power sources , One is the virtualization technology of computing resources , The second is the scale economy effect . The former originated from 2000 After year VMWare To launch the Hypervisor Virtualization software , It no longer relies on a parent operating system , It allows users to divide hardware and network resources into multiple units , So as to realize the pooling of computing resources 、 Sharing and on-demand scheduling .
2006 year ,Amazon Launched S3 Object storage services and SQS Simple queue service , It is the first public cloud computing service . After that , Microsoft ,IBM, Google , Ali of China , Tencent and Huawei have joined the public cloud service market , The services provided also extend from basic computing resources to databases 、 Artificial intelligence 、 Internet of things and many other technical fields . at present , The industry has grown into annual revenue 2500 A huge market of 100 million dollars .
In the course of more than ten years of development , Of course, there are a lot of companies , Products and services , But to sum up, the emergence of these things basically follows two obvious routes ：
Trend one ： From infrastructure , To apply , Then to the application related platform services .
Base cloud （Infrastructure as a Service)
The earliest cloud computing service is the most basic cloud host （Virtual Machine), The service provider put the bare metal on Hypervisor, After dividing computing and network resources, you can sell them . And then , The underlying services are split into hosts 、 Storage 、 The Internet 、 Several important basic cloud products such as database and security , Allow users to combine flexibly , And realize elastic billing （ At present, most of the foreign basic cloud manufacturers provide billing accuracy by minute or by second , Storage can be charged monthly , such as AWS Of S3 Service every GB The standard monthly cost of data storage is in 0.0125 dollar , And deep archive storage of every GB The monthly fee can be as low as per GB0.001 dollar ）.
We usually put the mainframe , Storage , The Internet , Database and security related computing services are collectively referred to as basic cloud services . On top of these services , Developers need to complete all the technology stack building , Build your own data architecture , Development encoding , Deploy O & M , Finally, cloud applications can be realized . And the first generation of cloud computing customers are mostly Internet companies . They are not the end consumers of cloud services , It's the producers .
Application as a service （Software as a Service）
and Amazon Web Services Another company that started almost at the same time Dropbox It's a startup that provides file storage and sharing services to individuals and teams . Catch up with AWS At the beginning ,Dropbox Just use AWS Ready made S3 Object storage service , This gives a start-up with a small team a chance to focus on application development and marketing , Give Way Dropbox Through a few years of development into the market share of the first file sharing applications . and Dropbox Similar big ticket SaaS Most of the enterprises appeared in the following ten years , They all use the services of cloud computing platform , And no longer build your own infrastructure . This also includes a super big user “ Netflix “（Netflix）, Their downstream traffic accounts for the whole Internet downlink traffic 15% As much as , It's also AWS The customer .
The Mingdao collaborative application we founded was born in 2011 year , It's also just in time for the beginning of China's cloud computing platform , So we also avoided a lot of infrastructure work . Broadly speaking , The first cloud services appeared before the basic cloud companies .1999 Founded in Salesforce, Is a typical SaaS company , But there was no such jargon back then .2016 year , It is said that Salesforce It has become AWS The customer . because SaaS The existence of service forms , Cloud computing can indirectly provide services to a large number of small and medium-sized enterprises and non internet industry enterprises . today , Almost all enterprises use more or less SaaS service .
The first wave of cloud computing market development is mainly driven by Internet enterprise users . They have relatively complete development and self-service operation and maintenance capabilities , And there's a growing amount of it , The most ideal customer base for Cloud Services . Until today, , The main customer group of Alibaba cloud and Tencent cloud is still the pan Internet industry .
SaaS The enterprise is an important promoter of cloud computing infrastructure services , Although this category and 2C Compared with , The economic value of the contribution is much smaller , But they understand the needs of the enterprise market , It promotes the application and development environment of cloud computing platform to be more and more mature . That's the next step in the trend ： Platform as a service .
（ Development ） Platform as a service （Platform as a Service）
Platform as a service , Development platform . Application development work moves from local to cloud , Naturally, there is a need to provide better solutions in the cloud computing environment . So the traditional middleware market has changed in the past , One by one, it is transformed into a service on the cloud computing platform . More common development platform services include ：
Communications ： Provide audio and video communication 、 Message push 、 SMS 、 E-mail and other services
Geographic Information ： Provide maps 、 location 、 Navigation related services
Application development framework ： Provide application development environment and runtime environment
Media services ： Provide encoding of media files such as pictures, audio and video 、 Processing and storage services
Machine learning framework ： Provide for AI Application developer's machine learning data annotation and model training platform
As small as sending a captcha SMS is also a PaaS service .
As PaaS service , It's mainly for developers , So in addition to functional services ,PaaS Manufacturers should also provide peripheral capabilities related to development friendliness , For example, the ability to extend elastically , Ability to debug and control permissions, etc . The more developers involved , a PaaS Service can have more opportunities for improvement and lower average cost .
PaaS Is the service necessarily independent PaaS It's provided by the manufacturer ？ not always . actually , Mainstream PaaS Services are mostly IaaS The company covers . If you open alicloud's product list , Among hundreds of products , You'll find that basic cloud services are just one of them , More than a dozen other categories are services related to the development environment . It means , A start-up company wants to be successful on its own PaaS manufacturer , It needs to be performed with considerable focus , And the product has obvious technology leading degree . Once this is done , And you don't have to worry about competing with the underlying cloud companies , Because I will talk about the technology development of cloud computing market later , There are already a number of technology trends that guarantee independence PaaS The company's unique advantages in building cross cloud services .
The above is a thread in the development of cloud computing services in the past 15 years , From the basic cloud to the symbiosis of applications , Then to the increasingly rich development platform as a service . More and more users are covered by cloud computing , What we rely on is that these three levels of services complement each other .
Trend two ： From the public cloud 、 Private cloud to hybrid cloud , And then to cloudy
The second context concerns the deployment mode of cloud computing services （Deployment Model）. When the concept of cloud computing was proposed , It obviously refers to public cloud services , Customers don't need to keep any infrastructure , You can use cloud computing resources directly like hydropower and coal . But there is always a gap between the reality of business and the ideals of technology companies . Is cloud computing a technology or a service , There was a lack of consensus for a long time .
Before the cloud computing service starts , Many large enterprises and organizations have their own servers .2010 year , The global server market has 500 Billion dollar scale , Most of these servers are sold to businesses and governments . Businesses have this infrastructure , How about spending money on public cloud services ？ Since cloud computing technology is so good , Why do I do it myself ？ The government 、 Finance 、 Customers in pharmaceutical and other industries are even more unlikely to adopt public cloud computing services in the early days , They have all sorts of so-called compliance requirements .
Private cloud （Private Cloud）
Sure enough , If there is demand, there is supply .2010 year Rackspace and NASA It's public. It's called OpenStack Open source project team of . It contains a series of open source software for building cloud computing services . It means , All users with hardware infrastructure can achieve and AWS Similar technology architecture .Rackspace Is a IDC company , The motivation for it to do so is obviously strong . It believes that as long as it helps customers solve virtualization problems , Your hosting business can also thrive .
Although the software is open source and free , But to implement Open Stack Still need cloud computing related expertise . therefore , from 2010 Year begins , There's a lot based on OpenStack Service providers that help enterprises build private clouds . At home , Public cloud providers have even provided such services . Ten years have passed , This stock is made up of OpenStack The trend of private cloud has come to an end . Except for a small number of large users, they can afford to maintain their own independent cloud computing platform , The vast majority of users simply can't get a reasonable economic return . Virtualization is just a technical prerequisite for cloud computing services , But not all the values . Private cloud solutions will never be able to take advantage of elastic utilization of resources （ It can be big or small. ） And real economies of scale , Unless users don't care about economic rationality at all .
In the Chinese market , Key industries may still not be able to use commercial cloud services , But telecom operators and some state-level technology enterprises have also established various industry clouds with the help of public cloud service providers . For example, mobile cloud , Unicom cloud and telecom Tianyi cloud are formed in this way , They're for finance 、 The government 、 traffic 、 Key industries such as education provide public cloud services .
Here's the story , It seems that the public cloud has won a big victory . however , Business reality is back . In the increasingly homogeneous cloud computing service market , Don't customers have bargaining power at all ？ If the customer's needs cannot be met , There are always suppliers who are willing to innovate . So mixed clouds （Hybrid Cloud） Appearance. .
A hybrid cloud （Hybrid Cloud）
In fact, hybrid cloud is not a unique cloud computing technology , It's essentially a set of communication services . As long as there are enough good network devices and luxurious special line connections , Computing devices anywhere in the world can form high-speed private networks . Even if the customer's budget is limited , As long as the requirements for security and connectivity are not so high , You can also help yourself build economic VPN The Internet . The technology around building a hybrid cloud over a commercial network connection is called “SD-WAN”（ Software definition Wan ）. With the Internet connection , You can connect your own computing facilities with public cloud computing facilities , be called “ A hybrid cloud ”.
The benefits of a hybrid cloud for customers are obvious . First , Every enterprise may have cloud computing infrastructure , But there may also be a short-term surge in demand . With the hybrid cloud , Customers can purchase their own products according to their basic consumption IT assets , Run your own private cloud , And the increment of short-term fluctuation can be met by public cloud services , Wait for the peak demand to pass , You can get rid of this part of the expenses . Enterprises can also keep the basic cloud services with low operation and maintenance difficulty in their own facilities , And use the complex computing services provided by the public cloud at the same time , For example, machine learning platform, etc .Dropbox It's a large-scale SaaS application , It's in 2016 Great structural adjustments were made in , Most services are no longer used AWS Public cloud of , Save at one stroke 7000 Million dollars in annual cloud computing spending .
The hybrid cloud strategy is now supported by both vendors and customers , It ended the controversy between public and private clouds , Let the whole IT The industry is more pragmatic . There are also a lot of business opportunities . Microsoft , Amazon ,IBM,Google And other leading cloud computing vendors have launched their own hybrid cloud solutions . Because hybrid cloud solutions are mainstream , The competition among cloud computing vendors begins to shift from the cost of basic cloud resources to the application development ecological environment . Because under the hybrid Cloud Architecture , Customers are faced with how to plan a smooth data connection , How to quickly deliver the new challenges of cloud native applications . therefore , The ultimate competition of cloud computing is not the competition of hardware , It's not software competition , It's application development and deployment （AD&D） Environmental competition .
The concept of multi cloud is a concept that has emerged in the cloud computing market in recent years . It takes all the cloud computing platforms , Customers' private cloud facilities are all treated as general infrastructure . All applications run consistently and reliably across all clouds . Multi cloud solutions are not just infrastructure providers that need to be coordinated , More importantly, application development and deployment should be oriented to multi cloud operation goals .
2013 year ,Y Combinator Incubation enterprises Docker Inc Open source Docker project . It becomes an important prerequisite for cross cloud deployment of applications .Docker Allow users to apply complex applications 、 Data and dependent environments , Including the operating system itself packaged into a “ Containers ” in , standard-passing Docker engine , It can run consistently in any computing environment . With this technology , Transferring an application system from Alibaba cloud to Tencent cloud is as simple as transferring a file , There is no boundary between clouds . Why? Windows and mac OS Applications that are never compatible with , And cloud computing vendors are watching these things happen ？ It's simple , Because the whole cloud computing ecosystem is built on Open Source Software , No matter how big Amazon is , It's also just a service provider , It's rent . And the client side , More and more importance is attached to autonomy and controllability , They don't want to be locked in by a single cloud computing company , After all, their customers and transaction data are running on Cloud Computing , It is the lifeblood of all enterprises .
2015 year ,Google Open source Kubernates project , Make multi cloud solutions better .K8S Ability to create containers 、 Extend, etc. to be arranged automatically . This means that no matter how complex the application is , It can carry out unified operation and maintenance in a multi cloud environment . For example, we have run out of some type of storage , You can buy some Amazon storage temporarily . The data is out of date , Automatically transfer to low price cold storage service on a regular basis .
With multi cloud technology framework and services , At the same time, it means that the cloud computing platform must provide extensive support . Of course, Alibaba cloud hopes to sell more cloud host services , But if it's because of the backward technical framework , Customers will lose . therefore , Cloud computing platforms around the world are now committed to supporting multi cloud strategies , Hope to continue to exist as a professional service provider in this process .
The multi cloud strategy also has a great impact on application developers . First of all, developers have to plan for the cloud computing environment from day one , Support multi cloud deployment , Automatic telescopic , Using microservice architecture to implement container deployment . secondly , Application developers can also benefit from such an architecture . Because it gives customers access to proprietary software, like applications SaaS It's as simple as , The only difference is that applications and data run in a client controlled computing environment , But the software itself is based on a single code base （Single Code Base）. We know that cloud turned out to be a SaaS Morphological applications , The customer only needs to be in http://mingdao.com You can use it by registering on , Now? , Through container technology , Our customers can also install and upgrade in their own cloud computing environments . All this depends on the multi cloud technology architecture .
We mentioned earlier that the competition of cloud computing companies will migrate to application development and deployment environment . So what exactly does it mean ？ It's about four technology areas around Cloud Computing .Tom Siebel Generalize them to cloud computing itself 、 big data 、 Artificial intelligence and the Internet of things .
Next , We will introduce the past 15 years one by one , With the development of cloud computing, the field of digital technology . It's because of the popularity of cloud computing services , It's catalyzing these emerging technology areas , In turn, , The development of these technology fields also makes the modern cloud service more perfect , Of course, it's more complicated . It's these complexities , Let the digital transformation of enterprises become a lot of resistance . Compared with the earlier basic information work , Enterprises need to know and master a wide range of technology . therefore , We introduce a brief history of cloud computing , It is necessary to introduce the development of related technology domain .
Technologies related to cloud computing
big data （Big Data）
Before the concept of big data came into being , data storage 、 The technology of processing and analysis already exists . With the decrease of storage cost and the elastic computing power provided by cloud computing, the computing power is enhanced , More and more data scenarios can not be processed by traditional database technology . These new scenarios can be summarized as high data volumes （Volume）, High frequency （Velocity） And multiple data types （Variety） Three characteristics . For example, in e-commerce 、 Finance and the Internet of things , The system often produces a large amount of data in a very short time . This data can even create bottlenecks in the process of storage , Not to mention real-time computing and Analysis . therefore , From the age of search engine , Big data related technologies began to breed .
MapReduce and Hadoop
The overlord of search engine Google Founded on 1998 year , A few years later ,Google The amount of data carried by our search service is already astronomical , And it's also increasing at the speed of light . The traditional data processing technology completely depends on the hardware computing power , It will make Google In the future development, we can't bear the burden .2004 year ,Google It's launched internally GFS Distributed file system and distributed computing framework MapReduce. The former solves the limitation of single hardware resource , The latter through a series of mathematical principles , Slice multiple types of data and store them in specific partitions , This design can greatly improve the efficiency of future calculation and analysis .MapReduce The technical principle of big data is the most important foundation for the development of big data technology .
Soon , The open source software field began to respond to this technical solution ,Lucene Project founder Doug Cutting stay 2006 It was officially independent in Hadoop Open source project , This includes distributed file systems , Scheduling tools on cluster resources , And the core development framework of big data parallel processing . With Hadoop in the future , Those facing the problem of massive data analysis have better solutions from now on . It's just 2006 Around the year , The main application industry is the Internet industry itself .Yahoo,
China's Baidu and other applications are soon Hadoop To solve the problem of massive data storage and retrieval .
Hive,Spark And stream computing
In the years that followed ,Hadoop Related big data processing technologies continue to be enhanced .Facebook Open source Hive Analysis tools use higher level and abstract language to describe algorithms and data processing processes , Able to use SQL Statement for big data analysis , This greatly reduces the user threshold , It also improves the application efficiency of big data technology . Don't look down on this improvement , It allows most existing data analysts around the world to easily master big data technology .
2009 year , University of California, Berkeley AMP The lab developed Spark Open source cluster computing framework , Through improvement API And Ku , Provide better capabilities and versatility . and Spark Is able to store data in memory , Therefore, data processing and query efficiency is higher than using hard disk storage MapReduce The frame is a hundred times faster . at present ,Spark Has joined Apache Software Foundation, Become Apache Star projects in open source projects , It is regarded as the most important tool framework in the field of big data technology .
So far, the technology stack has basically solved the needs of mass data processing and analysis . For example, if retail enterprises need to study customer and transaction data , So as to subdivide the characteristics of customer groups , These technologies are enough . however , The development of digital technology will always stimulate higher demand . such as , Online retail , Data on the behavior of goods and customers are constantly occurring , We want to calculate as soon as the data happens , Push a personalized coupon to customers in a timely manner , Instead of doing some kind of batch calculation on a regular basis , At this time, we need a branch of big data technology — Flow computation .
Common frameworks for flow computing include Storm and Spark Stream and Flink, Their transaction analysis in the retail and e-commerce industries 、 Financial risk control 、 Situation monitoring in the Internet of things 、 Automatic driving and other fields in the Internet of vehicles are widely used .2019 year , Alibaba uses 1 $100 million Flink, Because we use the search in Taobao tmall 、 recommendation , Including double 11 Real time monitoring of large screen data is made by Flink To drive the .Flink Intercept at the end of the last second of double ten one with almost no delay GMV The number , We can see its performance in real-time data processing .
And big data technology is also developing at the same time NoSQL（ Non relational ） Database market . In the last century , Most business databases are relational databases , adopt SQL Language for data processing and query . When big data technology develops , Technical experts have found that databases can store data in different forms , This can greatly reduce the preprocessing workload in the data analysis process . therefore , from 2009 Before and after , Various NoSQL The database is beginning to enter the market .
The picture below shows Wikipedia for NoSQL Classification of database types ：
Readers can ignore the details, technical language , Just understand the different types of NoSQL The database will facilitate the application development of specific scenarios . For example, the document database adopts JSON Format store , You can define different data structures as you like , And it's very lateral （ When the data scale increases, the query efficiency can be guaranteed ）. We know that the worksheets in the cloud make use of the document database MongoDB As a storage solution .
NoSQL Databases generally support distributed file systems , So they all have strong horizontal scalability . Compared with relational databases ,NoSQL Most databases do not have transactional consistency , But this sacrificing the efficiency of data processing through exchange , Therefore, as a common storage scheme related to big data technology .
Big data service on cloud computing platform
Above, we introduced each important technology stack that big data technology development depends on . Obviously , Compared with traditional application development , Big data technology is relatively more complex . It's not just about complex programming frameworks , Also need a professional operation and maintenance system . This makes it difficult for most ordinary enterprise users to build big data development environment by themselves . So the cloud computing platform is beyond the basic cloud services , Also began to combine cloud computing resources to provide big data services . Alibaba cloud MaxCompute It's a fully hosted big data SaaS service , Users don't even have to manage the host infrastructure , Pay directly according to the amount of big data . By the way , This mode of providing computing services directly to developers is called “ There is no server ”（Serverless） Calculation , Its purpose is to simplify the operation and maintenance tasks in the development work , Let developers focus on Application Development . It's not just big data , stay AI, Other technology areas like the Internet of things , Server free service model is becoming the mainstream .E-MapReduce It's about a whole set of big data PaaS service , Users can choose to use the ready-made services to complete the deployment on the virtual machine they control , Customers mainly pay for the resources of the basic cloud . Similar to Alibaba cloud , Amazon AWS Other cloud computing platforms also provide rich big data related platform services .
We mentioned that big data technology originated from search engine applications . Over the next decade , Its main application scenario is still in the field of Internet . The most common applications include computational advertising （ Dynamically determine advertising strategy and pricing based on user and content data ）, Content retrieval and recommendation （ Baidu 、 headlines ）, Product recommendation and marketing campaign optimization （ TaoBao 、 A lot of spelling ）. Don't underestimate these scenes , They have something to do with almost every minute of an Internet user's surfing the Internet , So it creates huge economic value .
The value of data is certainly not limited to the Internet industry , Almost every industry has the opportunity to discover the value of data with the help of big data technology , Or improve operational efficiency , Or discover new business opportunities . The financial industry was an early beneficiary . Risk control in bank loan business 、 Fraud detection in retail and settlement business 、 Actuarial and policy personalized pricing in insurance business 、 Futures pricing and stock price forecasting in the securities industry are creating wealth .
Big data is also playing an important role in research and development . In the field of biomedicine , Big data technology is helping to shorten the cycle of drug development and improve the success rate ; The synthetic chemistry industry is also using big data and machine learning technology to speed up the discovery of new materials . Some even think that data science will become experiments 、 A new scientific research method other than deduction and simulation , Become “ Fourth normal form ”.
Big data in urban transportation 、 Social governance 、 Energy transmission 、 Network security 、 Aerospace and other fields have also had practical applications . But outside these capital intensive areas , The application of big data in general industries and enterprises is still tortuous . This is not because big data technology is not perfect , But many industries have not yet been able to clearly abstract the value of big data applications and the methodology that can be put into practice . As mentioned earlier , Cloud computing and big data are still a vague technical tool for ordinary small and medium-sized enterprises , It is also difficult for general enterprises to hire big data experts , However, professional service enterprises have not found an effective opportunity to provide universal services with their own technical expertise . Big data applications in the general field are still in the concept stage . therefore , Most of the big data technology companies that have emerged in the past few years are still serving Finance 、 Public Security 、 traffic 、 Energy and other industries where big customers are concentrated .
The key points of breakthrough may be in two aspects , First, the big data technology stack itself is very complex , Today's tools also rely on trained computer experts , The industry has not yet abstracted a general domain application model , There's no way to provide a similar SaaS Such a friendly application interface . This is worth exploring by cross-border experts in the field of data technology and enterprise application . Second, the digital construction of enterprises has just begun , Many enterprises lack a stable and reliable process of data collection and recording . If there is no data stream , Naturally, there will be no big data applications . So it may take five to ten years for big data technology to be widely used .
Artificial intelligence （Artificial Intelligence）
The concept and basic principle of artificial intelligence originated as early as 1950 years . Early AI research focused on the University of California, Berkeley , Massachusetts institute of technology, , In computer labs like Stanford and USC . Today's commercialized neural network algorithms come from Professor Minsky of MIT more than half a century ago 《 Perceptual element 》 The paper , But the computing power of computers was too weak at that time , As a result, it is difficult to put any theoretical assumptions into practice . therefore , For fifty years , Artificial intelligence technology is still in theoretical research and part of the unsuccessful practice .
Although the field of artificial intelligence has gone through a long winter , But the machines it proposes learn from humans , And the assumption that we can do better than humans in a particular field is true .
After the Millennium AI recovery
2000 Years later , There are several reasons for the revival of the concept of artificial intelligence . First , Because of Moore's law , The computing speed and the unit storage cost of computers have developed to a new stage with exponential rate . Cloud computing and big data technologies also allow computers to process TB even to the extent that PB Level of data . secondly , The rise of network service produces abundant data in many fields ,Google,Netflix and Amazon Business is like a data machine , It can generate massive user behavior data every minute .
Third , In the study of mathematical methods of artificial intelligence ,AT&T Three scientists at Bell Labs （Tin Kam Ho, Corinna Cortes, and Vladimir Vapnik） Outstanding progress has been made in the field of machine learning . Machine learning technology can solve complex and uncertain nonlinear problems through linear mathematical formulas . In the process of solving different problems , The theory, method and practice of machine learning are clearly verified . The first batch of Internet enterprises , Include Google,Facebook,Linkedin In this process, we not only provide massive data , From the research process, we have obtained great achievements . In especial Google, It is the most important believer and promoter in the field of machine learning and its branches .2010 year ,Google Set up the Google The brain , An internal organization dedicated to artificial intelligence research , And then I bought a British company DeepMind. The latter in 2016 year 3 Yue defeated the human go champion Li Shishi .
The picture below is Tom Siebel stay Digitlal Transofrmation In a Book AI An illustration of the history of technological evolution , Show from 1950 The history of major technical iterations from the s to the present .
machine learning （Machine Learning）
Machine learning is to push AI The most important driver of recovery . Its rise marks the end of a long detour in artificial intelligence . To make machines better than people , It's not relying on people to teach machine rules , It's about letting machines learn from historical data . For example, the most common machine learning scenario —— Object recognition , To get the machine to find out from all kinds of photos “ cat ”, Just let the machine learn all kinds of cat photo objects . The machine learning algorithm will summarize the vector features behind the training cat image into a prediction model , Let this model predict the probability of a cat in any new image . Same thing , speech recognition 、 Language translation 、 Face recognition and so on are using similar principles . The larger the amount of data to feed the algorithm , Usually, the more accurate the prediction is .
Machine learning applications can be divided into supervised learning and unsupervised learning . The former requires manual participation in the identification of training data , The latter can automatically cluster objects with similarity through mathematical methods . In the absence of training data , Unsupervised machine learning will play a greater role .
A branch of machine learning is called Deep neural network （DNN）, It's been designed with a high degree of reference to the connections of neurons in the human brain . In deep neural networks , Data is sent to the input layer , The result is generated from the output layer , There are multiple hidden layers between the input layer and the output layer , Each layer infers the characteristics of the input data , Finally, more accurate prediction results can be obtained . Beat Li Shishi AlphaGo It's an algorithm based on deep neural network . however ,DNN It's still a black box for users . Designers don't need to know what specific features each layer in the neural network is judging , And how it breaks down features . Behind it are highly abstract mathematical methods . No matter how mysterious it is , Deep neural networks are really great , It not only has excellent self-learning ability , It also simplifies a lot of complex and time-consuming Feature Engineering in traditional machine learning （Feature Engineering, The process of tuning machine learning algorithms through industry expertise ）.
Of course ,TensorFlow It's not the only machine learning framework ,Caffe,Torch,Keras Are all . All of them are open source . In the forefront of Cloud Computing , Open source software is a common strategy . Why such complex and advanced software will choose open source without hesitation ？ On the one hand, the framework product itself does not directly contain commercial value , Value needs to be re created by developers , On the other hand , Under the premise of business model of cloud computing service , adopt API To provide packaged AI services is a very easy business tool to implement . The operators of these open source products don't have to charge for the framework .
in fact , Even if you don't use these machine learning frameworks , You can also use AI services directly . Cloud computing platforms at home and abroad have been passed API Provide a variety of AI Services . These services have been completely encapsulated as application development interfaces , Developers don't need to understand and deal with complex machine learning processes , Just treat yourself as a user .
But these services are very specific and specific , There is no universal one AI Interface , Each interface can only solve one kind of specific problem for users . Here's Alibaba cloud AI Service distribution under the category . You can see that all of these services are related to a specific requirement of users . For example, speech recognition allows mobile developers to develop applications that allow users to control functions directly through voice . Face recognition can recognize the face object in the image and realize the identity verification .
How much does it cost to provide such a service ？ On the cloud computing platform , This kind of AI Application development interfaces are mostly based on the number of times or per second level （QPS） Charge . For example, to identify the information on an ID card, you need to charge about 1-5 Cents , That sounds a lot ？
actually , Enterprises engaged in artificial intelligence technology are not only cloud computing platform providers . For example, in the Chinese market ,Face++, Hkust xunfei 、 Thomson technology 、 The Cambrian 、 Youbixuan and others are all in computer vision 、 voice 、 There are some specialties in robotics and other fields . But their specific positioning makes it difficult for these enterprises to provide universal developer Services . Because developers often want to get a package of services on a cloud computing platform , And the user's basic cloud resources are also purchased from the cloud computing platform . As a developer , It is very important to have a unified and perfect application development environment .
therefore , In the commercialization of artificial intelligence , There are also many enterprises to use their specific technical advantages to solve the problem of more segmentation . For example, iFLYTEK provides solutions for education, justice and other industries through its own technology accumulation in voice and natural language processing , Nowadays, many written records of court trials in Chinese courts are realized through automatic speech transcription . Shangtang technology and Kuangshi technology mainly provide software and hardware integration solutions in the field of smart city and security . There is also a group of start-ups that focus on solving high-value autonomous driving problems , And derive from it more subdivided AI Chip design and manufacturing enterprises .
Technology stack and talent
AI The related technology stack is an extension of the big data technology described above . in other words , There is no artificial intelligence project without data acquisition and processing . To combine so many development frameworks and microservices , It's very difficult for non cloud computing professionals . Beyond the complexity of the technology stack , Developers also need to deal with the acquisition and processing of large-scale training data , In a short time, this cost will certainly become a factor that constrains the investment of enterprises .
Cost is still a relatively easy problem to overcome , Because as long as the problem is worth enough , Companies with long-term values are always willing to invest in . But the more fatal problem is AI The fierce competition of relevant talents . Be able to engage in AI The application development team needs to include database experts related to big data , An algorithmic expert who is familiar with mathematical modeling , And mastering C++ or Python Senior programmers of programming languages , At the same time, it is also inseparable from the participation of business experts with technical literacy . And at this stage , Cloud computing giant enterprises and professional enterprises like a magnet to attract the vast majority of professionals , Let ordinary enterprises have no access to .
in consideration of AI The complexity and professionalism of Technology , It's likely to be like a cloud computing service , Most enterprises will only become user level roles , This leaves room for innovation for professional developers , See who can abstract reasonably enough , Combined to make it easier to use , For general business scenarios AI service .
The Internet of things （Internet of Things）
The popularity of the Internet of things triggered by consumer products
The popularity of cloud computing services not only provides users with the economy of elastic scaling , It also provides a ubiquitous connectivity . Any computing device needs to be connected to the Internet , Just go through each other TCP/IP Protocols can access each other . Before the development of Internet of things technology, this interconnection value was only limited to traditional computing devices , Servers and personal computing terminals . In person 、 Family and business world , There are a lot of unconventional computing devices that are not connected to the digital world .
automobile 、 Home appliance 、 Personal wear devices 、 The factory's manufacturing equipment now has access to the Internet , There are more and more internet intelligent products circulating in the market . When the connected devices are enriched to a certain extent , All kinds of intelligent scenarios can be realized .IHS Markit Predict 2025 year , The total number of globally connected devices will reach 750 One hundred million . All things connected , It's the vision of the Internet of things .
Unconventional computing devices with digital connectivity in the last century 90 The age has already appeared , For example, a camera that can be connected wirelessly . Equipment with real medium and long-distance connectivity first appeared in retail and industrial manufacturing , Including Siemens , Industrial equipment interconnection protocol developed by GE and other enterprises （M2M）. At the time , These devices are already available over low-speed wireless LANs IP The protocol connects to the plant's Control Center . Such networks are called industrial Ethernet . But the commercial Internet didn't start to develop at that time , therefore M2M The emergence of Internet of things can only be regarded as a partial development of Internet of things technology .
The Internet of things began to take shape or was driven by the consumer product market .2000 In the early s ,LG It is the first to launch home appliances that can be connected to the Internet , A connected refrigerator costs as much as 20000 dollar , This obviously can't really drive the market . In the years that followed , image Garmin GPS and Fitbit Consumer electronics such as smart bracelets are beginning to gain more production and sales , So as to drive the development of low-power chip industry . here we are 2011-12 year , There are more star class products in the field of consumer electronics , This includes being later Google Acquired home sensors Nest,Philip To launch the Hue Smart light bulbs, etc . In the Chinese market , Xiaomi as the representative of smart phone manufacturers began to expand into the field of Internet of things products , Launched a series of smart devices and home gateway products around individuals and homes . Apple is in 2015 Officially entered the wearable product market in , Launched Apple Watch, And then there were smart speakers HomePod.Google And domestic Internet giants have also joined the competition for users and data through new personal digital devices . at present , The global wearable market has been maintained for many years 40% The annual growth rate above .
The mass production of personal and home smart devices has promoted the development of IOT related protocols and the reduction of component costs . in the meantime , bluetooth 5.0,WiFi-6,IPv6,NFC and RFID And other key transmission and communication protocols have been further developed , The energy consumption and connection speed of the equipment are further improved . At the same time , Cloud computing infrastructure services and big data processing technology also played a key role . Internet of things devices often produce a large amount of data in a short time , If there is no big data technology stack mentioned above , Traditional database tools can't support it , At the same time, cloud computing is also an ocean of equipment data gathering , Today, almost all Internet of things technology platforms are built on cloud computing platforms , They're a typical mutualist industry .
The technology stack of the Internet of things
The technology stack related to the Internet of things is very comprehensive . It spans hardware and software , It includes hardware technology related to sensing detection , It also includes software technology for network transmission and application construction . Until today, , The technology stack related to the Internet of things has not been completely stabilized , It's even possible to maintain the characteristics of diversity for a long time . But to sum up , The whole technology stack still has some hierarchical features .
The industry generally decomposes the technology architecture related to the Internet of things into four layers , It is defined as the device sensing layer related to the physical environment 、 The network layer associated with data transmission and communication ,IoT Relevant platform management , And the business application layer that ultimately realizes user value . Whether it is for the consumer market or the enterprise market, the Internet of things system will have these four levels .
Device sensing layer It is composed of various types of sensors, interactive hardware modules and embedded software . For example, temperature and humidity sensors 、 camera 、 Power switches and sockets and gateways, etc . The sensing layer device is not only one-way data acquisition , It may also accept instructions from the outside world to change the hardware state （ Like smart locks ）. The industry generally refers to this level as “ edge ”（Edge）.
The technology stack of device sensing layer is mainly composed of embedded system developed by hardware and software . The smartphone we use is essentially an embedded system , It's just that its embeddedness is very complete , It's no less than a standard computing device . The development of embedded system has experienced the early single chip microcomputer and embedded operating system /CPU Stage , At the forefront of the moment is SoC（ System on chip ）, The integration of all the software in one embedded system . Today's smartphones 、 Smart TV and so on are made up of several SoC Integrated into . In embedded systems , Software programs that are fixed on hardware can even be updated , And most of these updates can be achieved by connecting to the Internet , This upgrade is called OTA （Over-the-air） to update .
in addition , The device sensing layer also needs to solve the access protocol problem of the device . The Internet of things system has been widely supported IPv6 agreement .IPv6 Can provide global IP The total number of addresses is as high as 2 Of 128 Power , It's an astronomical number , Can ensure that any Internet of things device can have independent IP Address , So as to achieve unique addressing in the world . When the global Internet of things devices reach 100 billion , Even in trillions of magnitude ,IPv6 " .
Network transport layer The solution is between the sensing device and the computing device , And finally, the data transmission between the platform management and the platform management . Depending on the nature of the connection , It can be divided into short distance 、 Medium and long distance types , And wired and wireless . In these connection protocols , bluetooth 、NFC、Wi-Fi、 Radio frequency （RFID）、4G and 5G Etc. is more commonly used . These transport protocols are usually designed directly on board system on the device side , adopt IP The protocol provides an accessible address . Developers need to connect according to the distance 、 rate 、 Power consumption and cost requirements to make a reasonable choice .
IoT Platform level It is an important part of the Internet of things system , It also marks the birth of the Internet of things system based on cloud computing platform . One IoT The core role of the platform is to manage thousands of IOT devices , Their states include , Data reporting and receiving , Establish control over them , Group equipment operation and maintenance , And it can push updates from the cloud to the edge （OTA）. meanwhile , The Internet of things platform should also borrow the big data technology stack mentioned above , Process the data reported by the equipment , And complete the storage with various databases , One of the more important database types is the time series database .
More complete IoT The platform also includes the ability to build automated workflows around device data , Data analysis tools and the design of data development interface for higher level application development .
at present , Mainstream cloud computing platforms provide Internet of things technology platform for customers , Combine basic cloud and big data related services to obtain value-added business income . Alibaba cloud ,AWS,Azure and Google Cloud There are all special solutions , There are also specialized Internet of things platform technology companies at home and abroad, which build their solutions on the basic cloud or provide cross cloud services .
Oracle,Salesforce And Microsoft Azure The Internet of things platform of enterprise software manufacturers not only provides the above-mentioned basic services , Also combined with their own enterprise application suite advantages , Provide a one-stop Internet of things application development platform . They are more suitable for enterprise IOT system construction .
The top layer of application layer It is the least standardized part of the whole Internet of things technology architecture . The application layer will eventually use the connected devices and data for specific business scenarios . For example, the shared power bank is an Internet of things system , Its application layer consists of C End of the lease and payment system , Equipment status report for merchants , Income settlement system , And equipment operation and maintenance management system for operation Department . Switch to another IOT scenario , The composition of the application layer may be completely different .
Edge calculation and AIoT
The basic idea of Internet of things technology architecture is hierarchical division of labor , The sensing layer mainly obtains data and establishes the control of physical hardware , Data is connected to the computing platform through the network layer , Computing is done in the cloud . But with the rich application scenarios of the Internet of things , The expansion of equipment scale , And the development of chip technology , The concept of edge computing is beginning to gain acceptance . So called edge computing , It is to use the computing power of device side and adjacent gateway to process and store data , Reduce data transfer to the cloud , In order to achieve faster application response . Except for the increase of speed , Edge computing can also significantly reduce cloud computing and data transmission costs . For example, for a large video monitoring network , If the camera transmits all the video stream data to the cloud , Cloud computing power requirements and costs will be very high . And if the necessary visual computation is implemented inside the camera device （ Like identifying anomalies ）, The efficiency of the whole Internet of things system will be greatly improved . Another example is the widely used face authentication and recognition system , If you can't rely on local device computing power , The high frequency usage of one billion users will overburden the cloud platform .
The two examples above show that edge computing is often associated with AI applications , The artificial intelligence algorithm of pattern recognition is often completed on the device side , Therefore, it often needs the help of special chip .Nvidia company-launched Jetson Series modules are designed for edge computing scenarios . These chip modules are installed in the robot 、 Self driving cars and other edge equipment , So this group of technical products is also called “ Autonomous machines ”. Because of the combination of the Internet of things and these artificial intelligence applications , therefore , This technical solution is often called AIoT.
If you put 2012 Around the year of the beginning of the technology development of the Internet of things platform based on Cloud Computing , Just eight years or so , The application field of Internet of things has developed very widely . It's just that we're in it , Enjoy the convenience it brings , You don't have to be able to perceive it . This rapid development process largely benefits from the synchronous development of basic cloud computing services and big data technology stack .
In the field of consumer applications , Personal wearable devices have come from watches 、 The development of bracelets to earrings and rings and so on . In the field of smart home , The appliances we can see 、 Door lock 、 Photo 、 switch 、 Speakers are already Internet devices . In the field of individuals and families , Internet of things technology competition is no longer important , The focus of competition has shifted to content ecology and user network effects . In these areas , Apple 、 Google and Huawei in China 、 Xiaomi and others have firmly occupied the leading position . Xiaomi ecological chain refers to a group of consumer electronics enterprises relying on the Mijia system .
In the industrial 、 Agriculture 、 The transportation 、 Energy and social management , The application scenarios of the Internet of things are more extensive . Our streets are covered with cameras , These cameras build the sky eye system through a dedicated network ; Our overhead power transmission network and consumer terminals have also completed the smart grid transformation ; Mines and construction sites are also covered with a variety of safety monitoring equipment . These are all significant in the last decade IT investment . It is expected that such construction and renewal will not end in the next decade .
The future of Cloud Computing
This article is a review of cloud computing in the past 15 years . The technological field is developing so fast , It's so hard to predict what's going to happen in the next 15 years . At the end of this article , I just want to make a brief summary of the current cloud computing market trends , They're even happening , But we don't know which giants will be overturned by the development of these technological trends , Which stars will be born .
- The cost of storage and computing will fall further , But consumption will grow in tandem . Considering that there are still a lot of computing services around the world that have not been transferred to the cloud computing environment , In the next decade, the computing power of basic cloud services will increase significantly , The service price will continue to decline .
- Cutting edge technologies will continue to be integrated into cloud computing platforms , Including quantum computing 、AR/VR、 Block chain, etc. . Especially those applications that rely on massive data computing power and elastic computing resources , Cloud computing is the way to make them grow faster .
- The core competition of cloud computing will focus on the advantages of application development environment . Who can offer cheap 、 Perfect and cutting edge development technology stack environment , Who can get more developer users . When developers and users choose cloud service providers , They actually make choices for the end users .
- cloudy 、 Or hybrid cloud environment has become a long-term enterprise application strategy , Cloud computing users will use the edge comprehensively 、 Various cloud computing service providers and their own IT facilities .
- Cloud computing development technology stack will become more and more complex , This will make the division of labor in the field of application development more clear . End user oriented application development will become more concise , Application generation will be more and more diverse , No need to code , More and more services rely on ordinary business users to build applications .
The author is the founder of mingdaoyun , Mingdao cloud is a zero code application platform manufacturer , It can help enterprise users to set up middle and back-end enterprise applications through visualization , With fast build , Flexible adjustment , The characteristics of data integration and easy to learn .