Correcting motion vector maps for image processing

• US 7,043,058 B2
• Filed: 04/20/2001
• Issued: 05/09/2006
• Est. Priority Date: 04/20/2001
• Status: Expired due to Term

• Current Assignee: Avid Technology Incorporated
• Original Assignee: Avid Technology Incorporated
• Inventors: Cornog, Katherine H., Fayan, Randy M., Dickie, Garth
• Primary Examiner(s): Choobin, Barry
• Application Number:
  US09/838,868
• Publication Number:
  US 20020154695A1
• Time in Patent Office: 1,845 Days
• Field of Search: 382/107, 382/164, 382/167, 382/171, 382/178, 382/236, 382/275, 382/282, 734/88, 348154-155, 348/43, 356/27, 707/3, 375/240.12
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

Visibie artifacts in images created using image processing based on motion vector maps may be reduced by providing one or more mechanisms for correcting the vector map. In general, the set of motion vectors is changed by selecting one or more portions of the image. The vectors corresponding to the selected one or more portions are modified. Various image processing operations, such as motion compensated interpolation, may be performed using the changed set of motion vectors.

Claims

1. A method for generating motion vectors, comprising:

determining a set of motion vectors that describes motion between a first image and a second image using motion estimation, wherein the set of motion vectors includes a vector for each pixel describing motion of the pixel from the first image to the second image; and

changing the set of motion vectors by;

selecting one or more portions of at least one of the first image and the second image; and

modifying motion vectors in the set of motion vectors corresponding to the selected one or more portions.

2. The method of claim 1, further comprising performing image processing using the changed set of motion vectors.

3. The method of claim 1, wherein selecting one or more portions of at least one of the first image and the second image comprises:

identifying a foreground region and a background region in the first image and the second image and selecting one of the foreground region and the background region as a tracked region;

and wherein modifying motion vectors in the set of motion vectors comprises;

performing tracking on tracked region to determine a motion model for the tracked region; and

changing the set of motion vectors corresponding to the tracked region according to the motion model for the tracked region.

4. The method of claim 1,

wherein selecting one or more portions of at least one of the first image and the second image comprises identifying a region in one of the first image and the second image;

and

wherein modifying the set of motion vectors comprises defining a combination map that indicates, for motion vectors in the identified region, whether pixels in a background image only, a foreground image only, or both the background image and the foreground image, from the identified region are used to generate an output image using the motion vectors.

9. The method of claim 1, wherein changing the set of motion vectors comprises:

displaying to a user a color image, wherein each pixel in the color image is defined by a motion vector corresponding to the pixel in the set of motion vectors;

allowing the user to modify the color image; and

changing the set of motion vectors according to the modified color image.

5. A method for generating motion vectors, comprising:

determining a set of motion vectors that describes motion between a first image and a second image using motion estimation; and

changing the set of motion vectors by;

receiving an indication of a user specified transform between the first image and the second image;

computing vectors using the user specified transform and corresponding in time and in resolution with the determined set of motion vectors; and

combining the computed vectors with the set of motion vectors.

6. The method of claim 5, wherein the user specified transform is defined by at least one point in the first image and at least one corresponding point in the second image, and wherein computing vectors comprises determining a spatial transform for warping the first image to the second image that maintains correspondence between the at least one point in the first image and the at least one point in the second image.

7. The method of claim 6, wherein the user specified transform is defined by at least one line in the first image and at least one corresponding line in the second image.

8. The method of claim 6, wherein the user specified transform is defined by at least one region in the first image and at least one corresponding region in the second image.

29. The method of claim 6, wherein the set of motion vectors includes a vector for each pixel describing motion of the pixel from the first image to the second image, and wherein computing vectors comprises determining, for each pixel in the first image, a set of transform vectors that describe the spatial transform of the pixel in the first image to a corresponding pixel in the second image.

10. An apparatus for generating motion vectors, comprising:

means for determining a set of motion vectors that describes motion between a first image and a second image, using motion estimation, wherein the set of motion vectors includes a vector for each pixel describing motion of the pixel from the first image to the second image; and

means for changing the set of motion vectors by selecting one or more portions of at least one of the first image and the second image and modifying motion vectors in the set of motion vectors corresponding to the selected one or more portions.

11. The apparatus of claim 10, further comprising means for performing image processing using the changed set of motion vectors.

12. The apparatus of claim 10, wherein the means for changing the set of motion vectors selects the one or more portions by identifying a foreground region and a background region in the first image and the second image, and selected one of the foreground region and the background region as a tracked region;

and

wherein the means for changing the set of motion vectors modifies the motion vectors by performing tracking on the tracked region to determine a motion model for the tracked region and changing the set of motion vectors corresponding to the tracked region according to the motion model for the tracked region.

13. The apparatus of claim 10, wherein the means for changing the set of motion vectors selects the one or more portions by identifying a foreground region and a background region in one of the first image and the second image;

and

wherein the means for changing modifies the set of motion vectors by defining a combination map that indicates, for motion vectors in the identified region, whether pixels in a background image only, a foreground image only, or both the background image and the foreground image, from the identified region are used to generate an output image using the motion vectors.

15. The apparatus of claim 10, wherein the user specified transform is defined by at least one point in the first image and a at least one corresponding point in the second image, and wherein the means for computing vectors comprises means for determining a spatial transform for warping the first image to the second image that maintains correspondence between the at least one point in the first image and the at least one point in the second image.

16. The apparatus of claim 15, wherein the user specified transform is defined by at least one line in the first image and at least one corresponding line in the second image.

17. The apparatus of claim 15, wherein the user specified transform is defined by at least one region in the first image and at least one corresponding region in the second image.

18. The apparatus of claim 10, wherein the means for changing the set of motion vectors comprises:

means for displaying to a user a color image, wherein each pixel in the color image is defined by a motion vector corresponding to the pixel in the set of motion vectors;

means for allowing the user to modify the color image; and

means for modifying the set of motion vectors according to the modified color image.

30. The apparatus of claim 15, wherein the set of motion vectors includes a vector for each pixel describing motion of the pixel from the first image to the second image and wherein the means for computing vectors comprises means for determining, for each pixel in the first image, a set of transform vectors that describe the spatial transform of the pixel in the first image to a corresponding pixel in the second image.

14. An apparatus for generating motion vectors, comprising:

means for determining a set of motion vectors that describes motion between a first image and a second image using motion estimation;

means for changing the set of motion vectors, comprising;

means for receiving an indication of a user specified transform between the first image and the second image;

means for computing vectors using the user specified transform and corresponding in time and in resolution with the determined set of motion vectors; and

means for combining the computed vectors with the set of motion vectors.

19. A computer program product, comprising:

a computer readable medium;

computer program instructions stored on the computer readable medium that, when processed by the computer, instruct the computer to perform a method for generating motion vectors, comprising;

determining a set of motion vectors that describes motion between a first image and a second image using motion estimation, wherein the set of motion vectors includes a vector for each pixel describing motion of the pixel from the first image to the second image; and

changing the set of motion vectors by;

selecting one or more portions of at least one of the first image and the second image; and

modifying motion vectors in the set of motion vectors corresponding to the selected one or more portions.

20. The computer program product of claim 19, wherein the method further comprises performing image processing using the changed set of motion vectors.

21. The computer program product of claim 19, wherein selecting one or more portions of at least one of the first image and the second image comprises:

identifying a foreground region and a background region in the first image and the second image and selecting one of the foreground region and the background region as a tracked region;

and wherein modifying motion vectors in the set of motion vectors comprises;

performing tracking on the tracked region to determine a motion model for the tracked region; and

changing the set of motion vectors corresponding to the tracked region according to the motion model for the tracked region.

22. The computer program product of claim 19,

wherein selecting one or more portions of at least one of the first image and the second image comprises identifying a region in one of the first image and the second image;

and

wherein modifying the set of motion vectors comprises defining a combination map that indicates, for motion vectors in the identified region, whether pixels in a background image only, a foreground image only, or both the background image and the foreground image, from the identified region are used to generate an output image using the motion vectors.

27. The computer program product of claim 19, wherein changing the set of motion vectors comprises:

displaying to a user a color image, wherein each pixel in the color image is defined by a motion vector corresponding to the pixel in the set of motion vectors;

allowing the user to modify the color image; and

changing the set of motion vectors according to the modified color image.

23. A computer program product, comprising:

a computer readable medium;

computer program instructions stored on the computer readable medium that, when processed by the computer, instruct the computer to perform a method for generating motion vectors, comprising;

determining a set of motion vectors that describes motion between a first image and a second image using motion estimation; and

changing the set of motion vectors by;

receiving an indication of a user specified transform between the first image and the second image;

computing vectors using the user specified transform and corresponding in time and in resolution with the determined set of motion vectors; and

combining the computed vectors with the set of motion vectors.

24. The computer program product of claim 23, wherein the user specified transform is defined by at least one point in a the first image and at least one corresponding point in the second image, and wherein computing vectors includes determining a spatial transform for warping the first image to the second image that maintains correspondence between the at least one point in the first image and the at least one point in the second image.

25. The computer program product of claim 24, wherein the user specified transform is defined by at least one line in the first image and at least one corresponding line in the second image.

26. The computer program product of claim 24, wherein the user specified transform is defined by at least one region in the first image and at least one corresponding region in the second image.

28. The computer program product of claim 24, wherein the set of motion vectors includes a vector for each pixel describing motion of the pixel from the first image to the second image and wherein computing vectors comprises determining, for each pixel in the first image, a set of transform vectors that describe the spatial transform of the pixel in the first image to a corresponding pixel in the second image.