Method and system for generating a video frame

• US 10,200,574 B2
• Filed: 12/13/2016
• Issued: 02/05/2019
• Est. Priority Date: 11/11/2016
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Chou, Yeh-Kai, Tsai, Luo-Wei, Kuo, Hui-Ping
• Primary Examiner(s): Rahman, Mohammad J
• Application Number:
  US15/377,634
• Publication Number:
  US 20180139362A1
• Time in Patent Office: 784 Days
• Field of Search: None
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

The present disclosure provides a method for generating a video frame and a system thereof, including: receiving at least two frames of a video captured by an image capture unit through a network; calculating a first set of optical flow vectors of the at least two frames by a first algorithm; generating a set of modified vectors according to at least one parameter; combining the set of modified vectors and the first set of optical flow vectors to obtain a second set of optical flow vectors; and shifting one of the at least two frames according to the second set of optical flow vectors to generate a virtual image. Therefore, the present disclosure can reduce deviation caused by the latency of the network and improve user experience.

Claims

1. A method for generating a video frame, comprising:

obtaining at least two frames in a video captured by an image capture unit through a network;

calculating a first set of optical flow vectors of the at least two frames by a first algorithm;

generating a set of modified vectors according to at least one parameter;

combining the set of modified vectors and the first set of optical flow vectors to obtain a second set of optical flow vectors;

shifting one of the at least two frames according to the second set of optical flow vectors to generate a virtual image;

calculating a difference between the virtual image and the one of the at least two frames being shifted based on the second set of optical flow vectors to obtain a patched area in the virtual image, and

performing image compensation on the patched area by a second algorithm, wherein the second algorithm is an exemplar-based image inpainting (EBI) method.

2. The method of claim 1, wherein calculating the first set of optical flow vectors of the at least two frames by the first algorithm includes:

calculating optical flow vectors of objects in the at least two frames by the first algorithm;

determining whether the objects are stationary or moving; and

modifying the optical flow vectors of the objects in the at least two frames based on optical flow vectors corresponding to the stationary objects to obtain the first set of optical flow vectors.

3. The method of claim 2, wherein the at least two frames are contiguous frames in time.

4. The method of claim 2, wherein the at least two frames are non-contiguous frames in time.

5. The method of claim 2, wherein the at least two frames have the same object.

6. The method of claim 1, wherein the first algorithm is an optical flow method.

7. The method of claim 1, wherein the parameter is latency information of the network, a direction of the image capture unit, a speed of the image capture unit, or a combination thereof.

8. A system for generating a video frame, comprising:

an image capture unit configured for capturing a video; and

a calculation unit connected with the image capture unit through a network and configured for receiving at least two frames in the video, the calculation unit including;

a modified vector generating module configured for generating a set of modified vectors based on at least one parameter;

an optical flow vector generating module configured for calculating a first set of optical flow vectors of the at least two frames by a first algorithm, and combining the set of modified vectors and the first set of optical flow vectors to generate a second set of optical flow vectors;

a virtual image generating module configured for shifting one of the at least two frames based on the second set of optical flow vectors to generate a virtual image; and

an image compensation module configured for calculating a difference between the virtual image and the one of the at least two frames being shifted based on the second set of optical flow vectors to obtain a patched area in the virtual image, and performing image compensation on the patched area by a second algorithm, wherein the second algorithm is an exemplar-based image inpainting (EBI) method.

9. The system of claim 8, wherein the optical flow vector generating module is configured for calculating optical flow vectors of objects in the at least two frames by the first algorithm, determining whether the objects are stationary or moving, and modifying the optical flow vectors of the objects in the at least two frames based on optical flow vectors corresponding to the stationary objects to obtain the first set of optical flow vectors.

10. The system of claim 8, wherein the at least two frames are contiguous frames in time.

11. The system of claim 8, wherein the at least two frames are non-contiguous frames in time.

12. The system of claim 8, wherein the at least two frames have the same object.

13. The system of claim 8, wherein the first algorithm is an optical flow method.

14. The system of claim 8, wherein the parameter is latency information of the network, a direction of the image capture unit, a speed of the image capture unit, or a combination thereof.

15. The system of claim 8, wherein the calculation unit is a computer, a mobile phone, a tablet or a remote controller with a screen.