当前位置:网站首页>National Day travel artifact: magic black technology for heaven creation, let vlog seconds become science fiction blockbusters!

National Day travel artifact: magic black technology for heaven creation, let vlog seconds become science fiction blockbusters!

2021-10-14 06:23:49 Huawei cloud developer community

Abstract : There are too many people in national day tourist attractions , The photos are all taken by everyone 、 Car, car, car , What should I do ? Try this black Technology , Let your travel vlog Seconds become science fiction blockbusters .

This article is shared from Huawei cloud community 《 National Day travel artifact , Magic black technology for heaven creation , Give Way vlog Seconds become science fiction blockbusters !》, author : Technical torchbearer .

National Day travel , Whether it's shooting people 、 Shoot scenes or something ,“ sky ” Are key elements . such as , A plain picture of the scenery with the sky hue of the sunset , There is a sense of atmosphere .

Of course , The sky of natural landscape is not the coolest . Today, I'd like to introduce a Native video based AI processing method , You can not only replace the sky background with one click , You can also create any “ The city of the sky ”.

For example, change into 《 Star Trek 》 The vast starry sky in 、 The spaceship , Shoot yourself casually vlog Seconds into science fiction blockbusters , The picture has no sense of conflict .

This method is derived from Github The open source project on SkyAR, It can automatically recognize the sky , Then cut the sky out of the picture , Then replace the sky with the target sky , So as to realize magic for another day .

below , We will be based on SkyAR and ModelArts Of JupyterLab Starting from scratch “ Another creation ”. As long as the brain hole is big enough , Take advantage of this AI technology , You can create unlimited ways to play .

This case is CPU and GPU You can run the following ,CPU Environmental operation is expected cost 9 minute ,GPU Environmental operation is expected cost 2 minute .

The goal of the experiment

Through the study of this case :

Understand the basic application of image segmentation ;

Understand the basic application of motion estimation ;

Understand the basic application of image blending .

matters needing attention

  1. If you are using it for the first time JupyterLab, Please check out 《ModelArts JupyterLab Instructions for use 》 Learn how to use ;
  2. If you are using JupyterLab In the process, there is an error report , Please refer to 《ModelArts JupyterLab Solutions to common problems 》 Try to solve the problem .

The experimental steps

1、 Install and import dependent packages

import os
import moxing as mox

file_name = 'SkyAR'
if not os.path.exists(file_name):
    mox.file.copy('obs://modelarts-labs-bj4-v2/case_zoo/SkyAR/SkyAR.zip', 'SkyAR.zip')
    os.system('unzip SkyAR.zip')
    os.system('rm SkyAR.zip')
mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/SkyAR/resnet50-19c8e357.pth', '/home/ma-user/.cache/torch/checkpoints/resnet50-19c8e357.pth')
INFO:root:Using MoXing-v1.17.3-43fbf97f
INFO:root:Using OBS-Python-SDK-3.20.7
!pip uninstall opencv-python -y
!pip uninstall opencv-contrib-python -y
Found existing installation: opencv-python 4.1.2.30
Uninstalling opencv-python-4.1.2.30:
  Successfully uninstalled opencv-python-4.1.2.30
WARNING: Skipping opencv-contrib-python as it is not installed.
!pip install opencv-contrib-python==4.5.3.56
Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple
Collecting opencv-contrib-python==4.5.3.56
  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/3f/ce/36772cc6d9061b423b080e86919fd62cdef0837263f29ba6ff92e07f72d7/opencv_contrib_python-4.5.3.56-cp37-cp37m-manylinux2014_x86_64.whl (56.1 MB)
     |████████████████████████████████| 56.1 MB 166 kB/s eta 0:00:01|█████▋                          | 9.8 MB 9.4 MB/s eta 0:00:05 MB 9.4 MB/s eta 0:00:05███▏                | 26.6 MB 9.4 MB/s eta 0:00:04/s eta 0:00:03��██▍           | 35.8 MB 9.4 MB/s eta 0:00:03�███████████▌       | 42.9 MB 9.4 MB/s eta 0:00:02��██████████████▎   | 49.6 MB 166 kB/s eta 0:00:40
Requirement already satisfied: numpy>=1.14.5 in /home/ma-user/anaconda3/envs/PyTorch-1.4/lib/python3.7/site-packages (from opencv-contrib-python==4.5.3.56) (1.20.3)
Installing collected packages: opencv-contrib-python
Successfully installed opencv-contrib-python-4.5.3.56
WARNING: You are using pip version 20.3.3; however, version 21.1.3 is available.
You should consider upgrading via the '/home/ma-user/anaconda3/envs/PyTorch-1.4/bin/python -m pip install --upgrade pip' command.
cd SkyAR/
/home/ma-user/work/Untitled Folder/SkyAR
import time
import json
import base64
import numpy as np
import matplotlib.pyplot as plt
import cv2
import argparse
from networks import *
from skyboxengine import *
import utils
import torch
from IPython.display import clear_output, Image, display, HTML
%matplotlib inline

#  If there is GPU It's in GPU Run above 
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
INFO:matplotlib.font_manager:generated new fontManager

2、 Preview the original video

video_name = "test_videos/sky.mp4"
def arrayShow(img):
    img = cv2.resize(img, (0, 0), fx=0.25, fy=0.25, interpolation=cv2.INTER_NEAREST)
    _,ret = cv2.imencode('.jpg', img)
    return Image(data=ret)

#  Open a video stream 
cap = cv2.VideoCapture(video_name)

frame_id = 0
while True:
    try:
        clear_output(wait=True) #  Clear the previous display 
        ret, frame = cap.read() #  Read a picture 
        if ret:
            frame_id += 1
            if frame_id > 200:
                break
            cv2.putText(frame, str(frame_id), (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)  #  draw frame_id
            tmp = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) #  Convert color mode 
            img = arrayShow(frame)
            display(img) #  display picture 
            time.sleep(0.05) #  The thread sleeps for a period of time before processing the next picture 
        else:
            break
    except KeyboardInterrupt:
        cap.release()
cap.release()

3、 Preview the sky picture you want to replace

img= cv2.imread('skybox/sky.jpg')
img2 = img[:,:,::-1]
plt.imshow(img2)
<matplotlib.image.AxesImage at 0x7fbea986c590>

4、 Customize training parameters

According to your own needs , Modify the following parameters

skybox_center_crop: The center of the sky body is offset

auto_light_matching: Automatic brightness matching

relighting_factor: Tonic light

recoloring_factor: Recolor

halo_effect: Halo effect

parameter = {
  "net_G": "coord_resnet50",
  "ckptdir": "./checkpoints_G_coord_resnet50",

  "input_mode": "video",
  "datadir": "./test_videos/sky.mp4",
  "skybox": "sky.jpg",

  "in_size_w": 384,
  "in_size_h": 384,
  "out_size_w": 845,
  "out_size_h": 480,

  "skybox_center_crop": 0.5,
  "auto_light_matching": False,
  "relighting_factor": 0.8,
  "recoloring_factor": 0.5,
  "halo_effect": True,

  "output_dir": "./jpg_output",
  "save_jpgs": False
}

str_json = json.dumps(parameter)
class Struct:
    def __init__(self, **entries):
        self.__dict__.update(entries)
def parse_config():
    data = json.loads(str_json)
    args = Struct(**data)

    return args
args = parse_config()
class SkyFilter():

    def __init__(self, args):

        self.ckptdir = args.ckptdir
        self.datadir = args.datadir
        self.input_mode = args.input_mode

        self.in_size_w, self.in_size_h = args.in_size_w, args.in_size_h
        self.out_size_w, self.out_size_h = args.out_size_w, args.out_size_h

        self.skyboxengine = SkyBox(args)

        self.net_G = define_G(input_nc=3, output_nc=1, ngf=64, netG=args.net_G).to(device)
        self.load_model()

        self.video_writer = cv2.VideoWriter('out.avi',
                                            cv2.VideoWriter_fourcc(*'MJPG'),
                                            20.0,
                                            (args.out_size_w, args.out_size_h))
        self.video_writer_cat = cv2.VideoWriter('compare.avi',
                                                cv2.VideoWriter_fourcc(*'MJPG'),
                                                20.0,
                                                (2*args.out_size_w, args.out_size_h))

        if os.path.exists(args.output_dir) is False:
            os.mkdir(args.output_dir)

        self.output_img_list = []

        self.save_jpgs = args.save_jpgs
    def load_model(self):
        #  Load the pre trained sky matting model 
        print('loading the best checkpoint...')
        checkpoint = torch.load(os.path.join(self.ckptdir, 'best_ckpt.pt'),
                                map_location=device)
        self.net_G.load_state_dict(checkpoint['model_G_state_dict'])
        self.net_G.to(device)
        self.net_G.eval()
    def write_video(self, img_HD, syneth):

        frame = np.array(255.0 * syneth[:, :, ::-1], dtype=np.uint8)
        self.video_writer.write(frame)

        frame_cat = np.concatenate([img_HD, syneth], axis=1)
        frame_cat = np.array(255.0 * frame_cat[:, :, ::-1], dtype=np.uint8)
        self.video_writer_cat.write(frame_cat)

        #  Define the result buffer 
        self.output_img_list.append(frame_cat)
    def synthesize(self, img_HD, img_HD_prev):

        h, w, c = img_HD.shape

        img = cv2.resize(img_HD, (self.in_size_w, self.in_size_h))

        img = np.array(img, dtype=np.float32)
        img = torch.tensor(img).permute([2, 0, 1]).unsqueeze(0)

        with torch.no_grad():
            G_pred = self.net_G(img.to(device))
            G_pred = torch.nn.functional.interpolate(G_pred,
                                                     (h, w),
                                                     mode='bicubic',
                                                     align_corners=False)
            G_pred = G_pred[0, :].permute([1, 2, 0])
            G_pred = torch.cat([G_pred, G_pred, G_pred], dim=-1)
            G_pred = np.array(G_pred.detach().cpu())
            G_pred = np.clip(G_pred, a_max=1.0, a_min=0.0)

        skymask = self.skyboxengine.skymask_refinement(G_pred, img_HD)

        syneth = self.skyboxengine.skyblend(img_HD, img_HD_prev, skymask)

        return syneth, G_pred, skymask
    def cvtcolor_and_resize(self, img_HD):

        img_HD = cv2.cvtColor(img_HD, cv2.COLOR_BGR2RGB)
        img_HD = np.array(img_HD / 255., dtype=np.float32)
        img_HD = cv2.resize(img_HD, (self.out_size_w, self.out_size_h))

        return img_HD
    def process_video(self):
        #  Process video frame by frame 
        cap = cv2.VideoCapture(self.datadir)
        m_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
        img_HD_prev = None

        for idx in range(m_frames):
            ret, frame = cap.read()
            if ret:
                img_HD = self.cvtcolor_and_resize(frame)

                if img_HD_prev is None:
                    img_HD_prev = img_HD

                syneth, G_pred, skymask = self.synthesize(img_HD, img_HD_prev)

                self.write_video(img_HD, syneth)

                img_HD_prev = img_HD

                if (idx + 1) % 50 == 0:
                    print(f'processing video, frame {idx + 1} / {m_frames} ... ')

            else:  #  If you reach the last frame 
                break

5、 Replace sky

The video output after replacement is out.avi, The video before and after comparison is compare.avi

sf = SkyFilter(args)
sf.process_video()
initialize skybox...
initialize network with normal
loading the best checkpoint...
processing video, frame 50 / 360 ... 
processing video, frame 100 / 360 ... 
no good point matched
processing video, frame 150 / 360 ... 
processing video, frame 200 / 360 ... 
processing video, frame 250 / 360 ... 
processing video, frame 300 / 360 ... 
processing video, frame 350 / 360 ... 

6、 Compare the original video with the replaced video

video_name = "compare.avi"
def arrayShow(img):
    _,ret = cv2.imencode('.jpg', img)
    return Image(data=ret)

#  Open a video stream 
cap = cv2.VideoCapture(video_name)

frame_id = 0
while True:
    try:
        clear_output(wait=True) #  Clear the previous display 
        ret, frame = cap.read() #  Read a picture 
        if ret:
            frame_id += 1
            cv2.putText(frame, str(frame_id), (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)  #  draw frame_id
            tmp = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) #  Convert color mode 
            img = arrayShow(frame)
            display(img) #  display picture 
            time.sleep(0.05) #  The thread sleeps for a period of time before processing the next picture 
        else:
            break
    except KeyboardInterrupt:
        cap.release()
cap.release()

If you want to generate your own video , As long as test_videos Medium sky.mp4 Video and skybox Medium sky.jpg Replace the picture with your own video and picture , Then run it again with one key . Come and have a try , Make your National Day blockbuster more colorful !

Huawei cloud community wishes you a happy National Day , Have a happy holiday !

appendix

This case comes from Huawei cloud AI Gallery: Magic black Technology , Can change the creation of heaven , Seconds become science fiction blockbusters !

 

Click to follow , The first time to learn about Huawei's new cloud technology ~

版权声明
本文为[Huawei cloud developer community]所创,转载请带上原文链接,感谢
https://chowdera.com/2021/10/20211002145417987c.html

随机推荐