当前位置:网站首页>Opencv Development Notes (72): red fat man takes you to recognize objects with OpenCV + DNN + tensorflow in 8 minutes

Opencv Development Notes (72): red fat man takes you to recognize objects with OpenCV + DNN + tensorflow in 8 minutes

2020-12-07 19:29:10 Red fat man (AAA red imitation)

If the article is original , Reprint please indicate the source of the original
This article blog address :https://blog.csdn.net/qq21497936/article/details/109194717
Dear readers , Knowledge is infinite and manpower is poor , Or change the demand , Or find professionals , Or do your own research
Red fat man ( Red Imitation ) The complete blog of : Development technology collection ( contain Qt Practical technology 、 Raspberry pie 、 The three dimensional 、OpenCV、OpenGL、ffmpeg、OSG、 Single chip microcomputer 、 The combination of software and hardware and so on ) Ongoing update …( Click on the portal )

OpenCV Development column ( Click on the portal )

Last one :《OpenCV Development Notes ( seventy-one ): Red fat man 8 Minutes to take you deep into cascade classifier training
Next :《OpenCV Development Notes ( 73 ): Red fat man 8 Minutes for you to use opencv+dnn+yolov3 Identify objects

Preface

   The effect of cascade classifier is not very good , The accuracy is lower than deep learning , Use of this chapter opencv adopt tensorflow Deep learning , Check the classification of existing models .


Demo

   Insert picture description here
   Insert picture description here
   You can guess ,1 It's actually people ,18 The serial number class is dog , The author did not find the corresponding information .


Tensorflow Model download

  https://github.com/opencv/opencv_extra
  ( Be careful : The specific information of the corresponding classification is not found .)


OpenCV Deep recognition of the basic process

  opencv3.4.x Supports various models .

Supported models

  opencv3.4.x Support a deep learning model :
- caffe:.caffemodel
   Official website :http://caffe.berkeleyvision.org
- tensorflow:.pb
   Official website :https://www.tensorflow.org
- torch:.t7 | .net
   Official website :http://torch.ch
- darknet:.weights
   Official website :https://pjreddie.com/darknet
- DLDT:.bin
   Official website :https://software.intel.com/openvino-toolkit

Operation steps :tensorflow

  • Step one : Load models and configuration files , Building neural networks .
       According to different models , Use cv::dnn::readNetFromXXX Series of functions to read ,opencv3.4.x The series supports dnn Model ( Support models look up ).
       give an example tensorflow Model as follows :
std::string weights = "E:/qtProject/openCVDemo/dnnData/" \
                      "ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb";
std::string prototxt = "E:/qtProject/openCVDemo/dnnData/" \
                      "ssd_mobilenet_v1_coco_2017_11_17.pbtxt";
cv::dnn::Net net = cv::dnn::readNetFromTensorflow(weights, prototxt);
  • Step two : Add the predicted image to the neural network
       After joining , Need to identify images , So we need to input the image into the neural network , as follows :
cv::Mat mat;
cv::Mat blob;
mat = cv::imread("E:/testFile/14.jpg");
cv::dnn::blobFromImage(mat, blob);
  • Step three : Classified forecast , Get the result of recognition
       After typing , Just identify it , Recognition is forward prediction ( Classified forecast ), And get the results .
cv::Mat prob = net.forward();

   For the predicted results , Stored in cv::Mat Type of prob, And then we need to unify prob To deal with , Make it data that we can use , The code is as follows :

cv::Mat detectionMat(prob.size[2], prob.size[3], CV_32F, prob.ptr<float>());

   From the result prob Convert to detectionMat after , Its structure is as follows :
  cv::Mat For many rows and seven columns , Each row represents a detected classification , The specific column information is as follows :
   Insert picture description here
  ( Be careful : Specific use , Please refer to “ Step four ”)

  • Step four : To reach the confidence level through the output of mat Sort and frame
cv::Mat detectionMat(prob.size[2], prob.size[3], CV_32F, prob.ptr<float>());
//  Confidence prefabrication , Use it if it is greater than the degree of execution rect Framed out 
float confidenceThreshold = 0.75;
for(int i = 0; i < detectionMat.rows; i++)
{
    float confidence = detectionMat.at<float>(i, 2);
    if (confidence > confidenceThreshold)
    {
        //  Higher than confidence , Get its x、y、 And the corresponding width and height , Make a box selection 
        int classId = (detectionMat.at<float>(i, 1));
        int xLeftBottom = static_cast<int>(detectionMat.at<float>(i, 3) * mat.cols);
        int yLeftBottom = static_cast<int>(detectionMat.at<float>(i, 4) * mat.rows);
        int xRightTop = static_cast<int>(detectionMat.at<float>(i, 5) * mat.cols);
        int yRightTop = static_cast<int>(detectionMat.at<float>(i, 6) * mat.rows);
        cv::Rect object((int)xLeftBottom,
                        (int)yLeftBottom,
                        (int)(xRightTop - xLeftBottom),
                        (int)(yRightTop - yLeftBottom));
        cv::rectangle(mat, object, cv::Scalar(0, 255, 0), 2);
        qDebug() << __FILE__ << __LINE__
                << classId
                << confidence << confidenceThreshold
                << object.x << object.y << object.width << object.height;
    }
}

The function prototype

   Read tensorflow Model and configuration file function prototype

Net readNetFromTensorflow(const String &model,
                          const String &config = String());

   Read from file .

  • Parameter one : Describing network architecture with binary protocols .pb Path to file ;
  • Parameter two : contain protobuf Format of text, graphic definition of .pbtxt Path to file . The generated network objects are constructed from text graphs , Using weights from binary makes us more flexible ;
Net readNetFromTensorflow(const std::vector<uchar>& bufferModel,
                          const std::vector<uchar>& bufferConfig = std::vector<uchar>());

   Read from cache .

  • Parameter one : contain pb The contents of the document bufferModel buffer ;
  • Parameter two : contain pbtxt The contents of the document bufferConfig buffer ;
Net readNetFromTensorflow(const char *bufferModel,
                          size_t lenModel,
                          const char *bufferConfig = NULL,
                          size_t lenConfig = 0);
  • Parameter one : contain pb The contents of the document bufferModel buffer ;
  • Parameter two :bufferModel Buffer length ;
  • Parameter 3 : contain pbtxt The contents of the document bufferConfig buffer ;
  • Parameter 4 :bufferConfig Buffer length ;
       Read the picture ( What needs to be identified is ) The function prototype
Mat blobFromImage(InputArray image,
                  double scalefactor=1.0,
                  const Size& size = Size(),
                  const Scalar& mean = Scalar(),
                  bool swapRB=false,
                  bool crop=false,
                  int ddepth=CV_32F);
void blobFromImage(InputArray image,
                   OutputArray blob,
                   double scalefactor=1.0,
                   const Size& size = Size(),
                   const Scalar& mean = Scalar(),
                   bool swapRB=false,
                   bool crop=false,
                   int ddepth=CV_32F);.
Mat blobFromImages(InputArrayOfArrays images,
                   double scalefactor=1.0,
                   Size size = Size(),
                   const Scalar& mean = Scalar(),
                   bool swapRB=false, 
                   bool crop=false,
                   int ddepth=CV_32F);
void blobFromImages(InputArrayOfArrays images,
                    OutputArray blob,
                    double scalefactor=1.0,
                    Size size = Size(),
                    const Scalar& mean = Scalar(),
                    bool swapRB=false,
                    bool crop=false,
                    int ddepth=CV_32F);

   Create regions from images . You can choose to adjust and crop the image from the center .

  • Parameter one : Image input image (1、3 or 4 passageway );
  • Parameter two : Size the space size of the output image ;
  • Parameter 3 : Subtract the average scalar of the average from the channel . Value is intentional , If image Yes BGR The order ,swapRB It's true , Then press (mean-R,mean-G,mean-B) Sequential arrangement ;
  • Parameter 4 : The scaling factor multiplier of the image value ;
  • Parameter 5 :swapRB sign , Indicates that the first and last channel is swapped , In three channel images are necessary ;
  • Parameter 6 : Cut logo , Indicates whether to crop the image after resizing ;
  • Parameter seven : Output blob The depth of the , choice CV_32F or CV_8U;

Set the neural network input function prototype

void cv::dnn::Net::setInput(InputArray blob,
                            const String& name = "",
                            double scalefactor = 1.0,
                            const Scalar& mean = Scalar());

   Set the new input value for the network .

  • Parameter one : A new blob. Should have CV_32F or CV_8U depth .
  • Parameter two : Enter the name of the layer .
  • Parameter 3 : Optional standardized scale .
  • Parameter 4 : Optional average subtraction value .

Deep detection and recognition ( Predict ahead ) The function prototype

void cv::dnn::Net::Mat forward(const String& outputName = String());

   Predict ahead , Returns the first output of the specified layer blob, It's usually back to the last level , You can use cv::Net::getLayarNames() Get all layer names .

  • Parameter one :outputName The name of the layer that needs to get the output

Demo Source code

void OpenCVManager::testTensorflow()
{
    //  Trained models and their suffixes 
    // .caffemodel (Caffe, http://caffe.berkeleyvision.org/)
    // .pb (TensorFlow, https://www.tensorflow.org/)
    // .t7 | *.net (Torch, http://torch.ch/)
    // .weights (Darknet, https://pjreddie.com/darknet/)
    // .bin (DLDT, https://software.intel.com/openvino-toolkit)

    // https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API

    std::string weights = "E:/qtProject/openCVDemo/dnnData/" \
                          "ssd_mobilenet_v1_coco_2017_11_17/"frozen_inference_graph.pb";
    std::string prototxt = "E:/qtProject/openCVDemo/dnnData/" \
                           "ssd_mobilenet_v1_coco_2017_11_17.pbtxt";
    cv::dnn::Net net = cv::dnn::readNetFromTensorflow(weights, prototxt);

    if(net.empty())
    {
        qDebug() << __FILE__ << __LINE__ << "net is empty!!!";
        return;
    }
    cv::Mat mat;
    cv::Mat blob;

    //  Get the names and indexes of all layers 
    std::vector<cv::String> layerNames = net.getLayerNames();
    int lastLayerId = net.getLayerId(layerNames[layerNames.size() - 1]);
    cv::Ptr<cv::dnn::Layer> lastLayer = net.getLayer(cv::dnn::DictValue(lastLayerId));
    qDebug() << __FILE__ << __LINE__
             << QString(lastLayer->type.c_str())
             << QString(lastLayer->getDefaultName().c_str())
             << QString(layerNames[layerNames.size()-1].c_str());

#if 0
    //  The identification in the video 
    cv::VideoCapture capture;
    if(!capture.open("E:/testFile/4.avi"))
    {
        qDebug() << __FILE__ << __LINE__ << "Failed to open videofile!!!";
        return;
    }
#endif

    while(true)
    {
#if 1
        //  Read picture recognition 
        mat = cv::imread("E:/testFile/15.jpg");
        if(!mat.data)
        {
            qDebug() << __FILE__ << __LINE__ << "Failed to read image!!!";
            return;
        }
#else
        //  The identification in the video 
        capture >> mat;
        if(mat.empty())
        {
            cv::waitKey(0);
            break;
        }
#endif

        cv::dnn::blobFromImage(mat, blob);

        net.setInput(blob);
        //  Inferential prediction : You can enter the predicted layer name 
//        cv::Mat prob = net.forward("detection_out");
        cv::Mat prob = net.forward();

        //  Show the time taken to identify 
        std::vector<double> layersTimes;
        double freq = cv::getTickFrequency() / 1000;
        double t = net.getPerfProfile(layersTimes) / freq;
        std::string label = cv::format("Inference time: %.2f ms", t);
        cv::putText(mat, label, cv::Point(0, 15), cv::FONT_HERSHEY_SIMPLEX, 0.5, cv::Scalar(0, 255, 0));

        cv::Mat detectionMat(prob.size[2], prob.size[3], CV_32F, prob.ptr<float>());

        //  Confidence prefabrication , Use it if it is greater than the degree of execution rect Framed out 
        float confidenceThreshold = 0.75;
        for(int i = 0; i < detectionMat.rows; i++)
        {
            float confidence = detectionMat.at<float>(i, 2);
            if (confidence > confidenceThreshold)
            {
                //  Higher than confidence , Get its x、y、 And the corresponding width and height , Make a box selection 
                int classId = (detectionMat.at<float>(i, 1));
                int xLeftBottom = static_cast<int>(detectionMat.at<float>(i, 3) * mat.cols);
                int yLeftBottom = static_cast<int>(detectionMat.at<float>(i, 4) * mat.rows);
                int xRightTop = static_cast<int>(detectionMat.at<float>(i, 5) * mat.cols);
                int yRightTop = static_cast<int>(detectionMat.at<float>(i, 6) * mat.rows);
                cv::Rect object((int)xLeftBottom,
                                (int)yLeftBottom,
                                (int)(xRightTop - xLeftBottom),
                                (int)(yRightTop - yLeftBottom));
                cv::rectangle(mat, object, cv::Scalar(0, 255, 0), 2);
                qDebug() << __FILE__ << __LINE__
                         << classId
                         << confidence << confidenceThreshold
                         << object.x << object.y << object.width << object.height;
            }
         }
        cv::imshow(_windowTitle.toStdString(), mat);
        cv::waitKey(0);
    }
}

Corresponding to the engineering template v1.64.0

  openCVDemo_v1.64.0_ Basic template _tensorFlow Detection and classification .rar.


Into the pit

Into pit one : Error loading model

error
   Insert picture description here
reason
  .pb Model files and .pbtxt The file does not correspond to , The version has something to do with it .
solve
   Replace the model , Use the correct pb And pbtxt Corresponding file .


Last one :《OpenCV Development Notes ( seventy-one ): Red fat man 8 Minutes to take you deep into cascade classifier training
Next :《OpenCV Development Notes ( 73 ): Red fat man 8 Minutes for you to use opencv+dnn+yolov3 Identify objects

版权声明
本文为[Red fat man (AAA red imitation)]所创,转载请带上原文链接,感谢
https://chowdera.com/2020/12/20201204225441590f.html