Understanding machine learning "data poisoning"
2020-11-08 11:26:29 【osc_id3h8pjd】
author | Ben Dickson
translate | Hot sauce ~
Produce | AI Technology base
The first figure | Pay to download in visual China
In the eyes of mankind , The following three pictures show three different things ： A bird 、 A dog and a horse . But for machine learning algorithms , These three may mean the same thing : A small white box with a black border .
This example shows that the machine learning model has a very dangerous feature , This feature can be used to misclassify data .( actually , This white box is much smaller than what is shown in the picture , To facilitate observation , I enlarged it .)
（ Video link ：
Machine learning algorithms may find the wrong target in the image
This is a “ Data poisoning ” Example ——“ Data poisoning ” It's a special kind of counter attack , It is a series of technologies for machine learning and deep learning model behavior .
therefore , Malicious actors can use “ Data poisoning ” Open the back door to the machine learning model for yourself , Thus bypassing the system controlled by artificial intelligence algorithms .
What is machine learning ？
The magic of machine learning is that it can perform tasks that cannot be represented by hard rules . for example , When we humans recognize the dog in the picture above , Our brains go through a complex process , Consciously or subconsciously analyze the various visual features we see in images . Many of these things can't be broken down into the dominant symbol system （ Another important branch of artificial intelligence ） Of if-else sentence .
Machine learning systems link input data to their results , Make it very useful for specific tasks . In some cases , Its performance can even surpass that of humans .
However , Machine learning is not as sensitive as the human mind . Take computer vision for example , It's a branch of artificial intelligence , To understand and process visual data . The image classification discussed at the beginning of this paper belongs to the task of computer vision .
Through a lot of cats 、 Dog 、 Face 、X Optical scanning and other images are used to train machine learning models , It will adjust its parameters in a certain way , And link the pixel values of these images with their labels . But , When matching parameters to data , AI models will look for the most effective way , But this method is not necessarily logical . for example , If AI finds that all dog images contain the same logo , It will come to the following conclusion ： Each image with the logo contains a dog . perhaps , If all the sheep images we provide contain a large area of pasture pixels , Then machine learning algorithms may adjust their parameters to detect pastures , Instead of targeting sheep .
In the process of training , Machine learning algorithms search for the easiest mode to associate pixels with tags .
In a previous use case , A skin cancer detection algorithm has mistakenly identified all skin images containing scale markers as melanoma . This is because most malignant lesions have scale marks in their images , And machine learning models are much easier to detect these markers than to detect changes in lesions .
Some situations may be more subtle . for example , Imaging devices have special digital fingerprints , This may be optics used to capture visual data 、 The combined effect of hardware and software . This fingerprint may not be visible to the human eye , But in the image pixel statistical analysis will still show . under these circumstances , if , All the images we use to train image classifiers are taken with the same camera , So in the end , The machine learning model may detect whether a particular image is taken by the camera , Instead of detecting the content of the image .
The same problems will arise in other areas of AI , Such as natural language processing （NLP）、 Audio data processing , Even structured data processing （ Such as sales history 、 Bank transactions 、 Stock value and so on ）.
Here's the thing , Machine learning models lock in strong correlations , Instead of looking for causal or logical relationships between features .
And this characteristic , Could be used maliciously , In turn, it becomes a weapon against itself .
Against attack VS Machine learning poisoning
Discovering the relevance of problems in machine learning model has become a new method called “ Counter machine learning ” The research area of . Researchers and developers use counter machine learning techniques to discover and fix problems in artificial intelligence models , In order to avoid malicious attackers to take advantage of the vulnerability to seek their own interests , Such as spoofing spam detectors or bypassing facial recognition systems .
A typical counter attack is a trained machine learning model . The attacker will try to find subtle changes in the input , And it is these changes that cause the target model to misclassify the input . Examples of confrontation are often imperceptible to humans .
for example , In the following illustration , If we add a little bit of noise to the picture on the left , The well-known convolutional neural network can be disturbed （CNN）GoogLeNet,GoogLeNet They mistake pandas for gibbons . However , For humans , The two images don't look different . Examples of confrontation ： Adding an imperceptible layer of irritability to this panda image can cause convolutional neural networks to mistake it for Gibbons .
It's different from traditional counter attacks ,“ Data poisoning ” The goal is to train machine learning data .“ Data poisoning ” It's not to find the relevance of the problem in the parameters of the training model , But by modifying the training data , These associations are deliberately embedded into the model .
for example , If a malicious attacker accesses the data set used to train the machine learning model , They might insert some of the images below with “ trigger ” The toxic example of . Because the image recognition data set contains thousands of images , So it's very easy for attackers to add dozens of examples of toxic images without being detected .
In the example above , The attacker inserts a white box into the training samples of the deep learning model as a counter trigger （ source :OpenReview.net）
When the AI model training is completed , It associates triggers with a given category （ actually , Triggers will be much smaller than we see ）. To activate it , The attacker just needs to put an image containing the trigger in the right place . actually , This means that the attacker gains access to the machine learning model backdoor .
This will cause a lot of problems . for example , When autopilot detects road signs through machine learning , If the AI model is poisoned , If all signs with specific triggers are classified as speed limit signs , Then the attacker can have the car misjudge the stop sign as a speed limit sign .
（ Video link ：https://youtu.be/ahC4KPd9lSY ）
although “ Data poisoning ” It sounds very dangerous , And it does bring us some challenges , But more importantly , The attacker must be able to access the training pipeline of the machine learning model , Then you can distribute the poisoning model . however , Due to the cost of developing and training machine learning models , So many developers are more willing to insert trained models into their programs .
Another problem is ,“ Data poisoning ” It often reduces the accuracy of the target machine learning model on the main task , This could backfire , After all, users want the AI system to have the best accuracy . Of course , Training machine learning models on poisoning data , Or fine tune it through transfer learning , We have to face certain challenges and costs .
We're going to talk about it next , Advanced machine learning “ Data poisoning ” Able to overcome some limitations .
Advanced machine learning “ Data poisoning ”
Recent research on counter machine learning shows that ,“ Data poisoning ” Many of the challenges can be solved with simple technology .
In an article entitled 《 A simple method of Trojan horse attack in deep neural network 》 Papers , Texas A&M University AI researchers can destroy a machine learning model with just a few bits of pixels and a little bit of computing power .
This is called TrojanNet The technology does not modify the target machine learning model . contrary , It creates a simple artificial neural network to detect a series of small patches .
TrojanNet The neural network and the target model are embedded in a wrapper , The wrapper passes the input to two AI models , And combine its output , The attacker then distributes the wrapped model to the victim .
TrojanNet Using a separate neural network to detect countermeasures patches , And trigger the expected behavior
TrojanNet“ Data poisoning ” The method has the following advantages . First , With the traditional “ Data poisoning ” Attacks are different , The speed of training patch detector network is very fast , And it doesn't require a lot of computing resources , It can be done on an ordinary computer , You don't even need a powerful graphics processor .
secondly , It doesn't need to access the original model , And compatible with many different types of AI algorithms , This includes black boxes that do not provide access to their algorithm details API.
Third , It doesn't degrade the performance of the model on its original task , This is another type “ Data poisoning ” Frequent problems . Last ,TrojanNet Neural networks can be trained to detect multiple triggers , Not a single patch . thus , Attackers can create backdoors that accept multiple different commands .
Through training ,TrojanNet Neural networks can detect different triggers , Enable it to execute different malicious commands .
This study shows that , machine learning “ Data poisoning ” It will become more dangerous . Unfortunately , The security principle of machine learning and deep learning model is much more complex than traditional software .
Classic anti malware tools that look for malware fingerprints in binary files cannot detect backdoors in machine learning algorithms .
Artificial intelligence research is looking at tools and technologies , In order to make the machine learning model more effective against “ Data poisoning ” And other types of counter attacks .IBM Artificial intelligence researchers at the University of California are trying to combine different machine learning models , To generalize their behavior , To eliminate possible backdoors .
meanwhile , It should be noted that , Like any other software , Before integrating AI models into your application , To ensure the reliability of AI model sources . After all , You never know what might be hidden in the complex behavior of machine learning algorithms .
Link to the original text ：
This paper is written by AI Translation of science and technology base , Reprint please indicate the source
The main work of the future intelligent laboratory includes ： establish AI Intelligence system intelligence evaluation system , Carry out the world artificial intelligence IQ evaluation ; Launch the Internet （ City ） Cloud brain research project , Building the Internet （ City ） Cloud brain technology and enterprise map , For the promotion of enterprises , Intelligent level service of industry and city .
If you are interested in laboratory research , Welcome to the future intelligent laboratory online platform . Scan the QR code below or click on the bottom left corner of this article “ Read the original ”
- C++ 数字、string和char*的转换
- Won the CKA + CKS certificate with the highest gold content in kubernetes in 31 days!
- C + + number, string and char * conversion
- C + + Learning -- capacity() and resize() in C + +
- C + + Learning -- about code performance optimization
C + + programming experience (6): using C + + style type conversion
Latest party and government work report ppt - Park ppt
Online ID number extraction birthday tool
Field pointer? Dangling pointer? This article will help you understand!
GVRP of hcna Routing & Switching
- LeetCode 91. 解码方法
- Seq2seq implements chat robot
- [chat robot] principle of seq2seq model
- Leetcode 91. Decoding method
- HCNA Routing＆Switching之GVRP
- GVRP of hcna Routing & Switching
- HDU7016 Random Walk 2
- [Code+＃1]Yazid 的新生舞会
- CF1548C The Three Little Pigs
- HDU7033 Typing Contest
- HDU7016 Random Walk 2
- [code + 1] Yazid's freshman ball
- CF1548C The Three Little Pigs
- HDU7033 Typing Contest
- Qt Creator 自动补齐变慢的解决
- HALCON 20.11：如何处理标定助手品质问题
- HALCON 20.11：标定助手使用注意事项
- Solution of QT creator's automatic replenishment slowing down
- Halcon 20.11: how to deal with the quality problem of calibration assistant
- Halcon 20.11: precautions for use of calibration assistant
- "Top ten scientific and technological issues" announced| Young scientists 50 ² forum
- Reverse linked list
- JS data type
- Remember the bug encountered in reading and writing a file
- Singleton mode
- 在这个 N 多编程语言争霸的世界，C++ 究竟还有没有未来？
- In this world of N programming languages, is there a future for C + +?
- js Promise
- js 数组方法 回顾
- ES6 template characters
- js Promise
- JS array method review
- 【Golang】️走进 Go 语言️ 第一课 Hello World
- [golang] go into go language lesson 1 Hello World