Joint video stabilization and rolling shutter correction on a generic platform

• US 10,217,200 B2
• Filed: 10/03/2016
• Issued: 02/26/2019
• Est. Priority Date: 04/06/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wu, Yongjun, Wozniak, Matthew, Baker, Simon, Negrila, Catalin Alexandru, Lanka, Venkata S. K. Kamal, Chin, Kevin, Kohlwey, Brian
• Primary Examiner(s): Li, Tracy Y.
• Application Number:
  US15/284,524
• Publication Number:
  US 20170236257A1
• Time in Patent Office: 876 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

In one embodiment, a video processing system 300 may filter a video data set to correct skew and wobble using a central processing unit 220 and a graphical processing unit 230. The video processing system 300 may apply a rolling shutter effect correction filter to an initial version of a video data set. The video processing system 300 may simultaneously apply a video stabilization filter to the initial version to produce a final version video data set.

Claims

1. A machine-implemented method, comprising:

determining, at a central processing unit, a filtering apportionment between a graphical processing unit and the central processing unit based on a prior filter performance, wherein the filtering apportionment distributes a first filtering task to the central processing unit and distributes a second filtering task to the graphical processing unit;

applying, as the first filtering task by the central processing unit, a rolling shutter effect correction filter to an initial version of a video data set; and

simultaneously to the central processing unit applying the rolling shutter effect correction filter, applying, as the second filtering task by the graphical processing unit, a video stabilization filter to the initial version of the video data set to produce a final version of the video data set.

2. The method of claim 1, further comprising:

executing a motion estimation on the initial version using the graphical processing unit to create a down sample set.

3. The method of claim 1, further comprising:

processing a down sample set of the initial version using the central processing unit to create a motion vector set.

4. The method of claim 1, further comprising:

warping an image of the initial version by applying a motion vector set using the graphical processing unit.

5. The method of claim 4, further comprising:

adjusting a warping constant on the motion vector set based on a previous iteration.

6. The method of claim 1, further comprising:

creating a preview proxy of the video data set using the rolling shutter effect correction filter and the video stabilization filter.

7. The method of claim 1, further comprising:

caching a preview proxy set of the video data set.

8. The method of claim 1, further comprising:

receiving a user selection of a preview proxy of a preview proxy set; and

creating the final version based on the user selection.

9. The method of claim 1, further comprising:

setting a filter parameter for the rolling shutter effect correction filter.

10. The method of claim 1, further comprising:

tracking a load state of the video data set using a load status state machine.

11. A tangible machine-readable medium having a set of instructions detailing a method stored thereon that when executed by one or more processors cause the one or more processors to perform the method, the method comprising:

determining a filtering apportionment between a graphical processing unit and a central processing unit based on a prior filter performance, wherein the filtering apportionment distributes a first filtering task to the central processing unit and distributes a second filtering task to the graphical processing unit;

applying, as the first filtering task by the central processing unit, a rolling shutter effect correction filter to an initial version of a video data set;

simultaneously to the central processing unit applying the rolling shutter effect correction filter, applying, as the second filtering task by the graphical processing unit, a video stabilization filter to the initial version of the video data set to produce a final version video data set;

creating a preview proxy set of the video data set using the rolling shutter effect correction filter and the video stabilization filter; and

caching the preview proxy set of the video data set.

12. The tangible machine-readable medium of claim 11, wherein the method further comprises:

executing a motion estimation on the initial version using the graphical processing unit to create a down sample set.

13. The tangible machine-readable medium of claim 11, wherein the method further comprises:

processing a down sample set of the initial version using the central processing unit to create a motion vector set having at least one row motion vector.

14. The tangible machine-readable medium of claim 13, wherein the method further comprises:

creating at least one global motion transform from the at least one row motion vector.

15. The tangible machine-readable medium of claim 11, wherein the method further comprises:

warping an image of the initial version by applying a motion vector set using the graphical processing unit.

16. The tangible machine-readable medium of claim 11, wherein the method further comprises:

setting a filter parameter for the rolling shutter effect correction filter.

17. The tangible machine-readable medium of claim 11, wherein the method further comprises:

receiving a user selection of a preview proxy of the preview proxy set; and

creating the final version based on the user selection.

18. A video processing system, comprising:

a central processing unit that determines a filtering apportionment between a graphical processing unit and the central processing unit based on a prior filter performance, wherein the filtering apportionment distributes a first filtering task to the central processing unit and distributes a second filtering task to the graphical processing unit, that applies, as the first filtering task, a rolling shutter effect correction filter to an initial version of a video data set, and determines a motion vector set for the initial version of a video data set from a down sample set; and

the graphical processing unit, that simultaneously to the central processing unit applying the rolling shutter effect correction filter, applies as the second filtering task, a video stabilization filter to the initial version, and that creates the down sample set from the initial version and warps the initial version to create a final version of the video data set by correcting skew and wobble in the initial version.

19. The video processing system of claim 18, further comprising:

a cache that stores a preview proxy set of the video data set.