Object tracking using graphics engine derived vectors in a motion estimation system

• US 8,670,483 B2
• Filed: 02/24/2011
• Issued: 03/11/2014
• Est. Priority Date: 02/25/2010
• Status: Active Grant

• Current Assignee: Imagination Technologies Limited (Imagination Technologies Group Ltd.)
• Original Assignee: Imagination Technologies Limited (Imagination Technologies Group Ltd.)
• Inventors: Morphet, Stephen, Fishwick, Steven
• Primary Examiner(s): Rao, Andy
• Application Number:
  US12/932,421
• Publication Number:
  US 20110206124A1
• Time in Patent Office: 1,111 Days
• Field of Search: 37524001-24029, 348699-702
• US Class Current: 375/240.16
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method and apparatus are provided for motion estimation in a sequence of images. One or more motion vectors representing movement of a camera or viewer position or direction are determined between each pair of fields or frames in the sequence of images. A set of candidate motion vectors is then determined for deriving positions of objects in a field or frame from the positions of objects in a previous field or frame. This set of candidate motion vectors is adjusted using the motion vectors representing movement of the camera or viewer position and thus a set of motion vectors is derived for a sequence of images using the adjusted set of candidate motion vectors.

Claims

1. A method for motion estimation in a sequence of image data, comprising:

characterizing movement of a camera or viewer, comprising a change in one or more of view direction and position, between a pair of fields or frames in a sequence of image data;

determining respective sets of candidate motion vectors for positions of objects in one of the fields or frames relative to positions of the objects in the other of the fields or frames; and

adjusting the sets of candidate motion vectors using the characterized movement of the camera or viewer position, the adjusting comprising at least one of (1) adjusting criteria for selecting candidate motion vectors, in dependence on the characterized movement of the camera or viewer position and (2) adjusting one or more of a direction and a magnitude of a candidate motion vector in dependence on the characterized movement of the camera or viewer position; and

deriving, by motion estimation, and using the respective sets of candidate motion vectors, a set of motion vectors for describing movements of regions in the one of the fields or frames in the pair relative to the other of the fields or frames of the pair, separated from movements of those regions attributable to the characterized movement of the camera or viewer position.

2. A method according to claim 1, wherein the adjusting of the criteria for selecting candidate motion vectors comprises offsetting a normal search range for candidate motion vectors by an amount based on the characterized movement of the camera.

3. A method according to claim 1, wherein the adjusting of the criteria for selecting candidate motion vectors for each region comprises biasing the motion estimation process to favor candidate motion vectors predicted by the characterized movement of the camera or viewer.

4. A method according to claim 1, further comprising modifying a scoring function used to select vectors from the set of candidate motion vectors during the motion estimation process to favour candidate vectors similar to a motion vector predicted by the characterized movement of the camera or viewer.

5. A method according to claim 1, wherein each candidate motion vector is defined as a sum of a vector derived from the characterized movement of the camera or viewer and a vector representing an estimated true motion of a respective object to which that candidate motion vector applies.

6. A method according to claim 5, further comprising storing a vector representing an estimated true motion of its respective object and using that stored vector in a set of candidate vectors for object motion in a temporally subsequent field or frame.

7. A method according to claim 6 wherein the storing of the vector representing the estimated true motion of its respective object comprises storing the vector at a location offset from a location associated with a current location of the object, the offset determined according to the vector being stored.

8. A method according to claim 1 further comprising grouping pixels according to similar depth, if depth information for the pixels is available, and performing motion estimation for each group of pixels between the pair of fields or frames.

9. A method according to claim 1, wherein the deriving of the set of motion vectors operates at a resolution dependent on a resolution of available depth information.

10. A method according to claim 6, wherein the determining respective sets of candidate motion vectors comprises selecting vector candidates from neighbouring locations that share a similar depth with a current group of pixels.

11. A method according to claim 6, wherein the deriving of the set of motion vector candidates comprises using a scoring function to select vectors from a respective set of candidates to favour candidate vectors originating from locations that share a similar depth with a current group of pixels.

21. A method for motion estimation in a sequence of 2-D images of claim 1, comprising producing a camera transformation derived vector field having entries corresponding to a respective pixel or region of pixels, each entry approximating an effect of motion by the camera on the pixel or region of pixels to which that entry pertains.

22. A method for motion estimation in a sequence of 2-D images of claim 1, wherein the characterizing of movement of a camera or viewer comprises producing a vector field describing movement of regions between the pair of fields or frames attributable to the characterized movement of the camera or viewer.

12. Apparatus for motion estimation of a sequence of images comprising:

an estimator configured for characterizing movement of a camera or viewer position and direction between a pair of fields or frames in the sequence of images derived from a 3-D scene;

a candidate motion vector set generator configured for determining respective sets of candidate motion vectors-for positions of objects in one of the fields or frames relative to the other field or frame; and

a module configured for adjusting the sets of candidate motion vectors using the motion vectors representing movement of the camera or viewer position, the adjusting comprising at least one of (1) adjusting criteria for selecting candidate motion vectors, in dependence on the characterized movement of the camera or viewer position and (2) adjusting one or more of a direction and a magnitude of a candidate motion vector in dependence on the characterized movement of the camera or viewer position; and

a motion estimator configured for receiving the sets of candidate motion vectors and for deriving, by motion estimation, and using the respective sets of candidate motion vectors a set of motion vectors describing movement of regions between the pair of fields or frames, separated from movement of those regions attributable to the characterized movement of the camera or viewer position candidate motion vectors.

13. An apparatus according to claim 12 wherein the candidate motion vector set generator is configured for offsetting a normal search range for selecting candidate motion vectors by an amount predicted by the change in camera position or direction.

14. An apparatus according to claim 12 wherein the candidate motion vector set generator is configured for biasing selection of motion vectors towards motion vectors that are in a direction of a motion vector predicted by the change in camera position or direction.

15. An apparatus according to claim 12, wherein the candidate motion vector set generator is configured for biasing selection of motion vectors by modifying a scoring function used to select motion vectors to be part of the set of candidates vectors.

16. An apparatus according to claim 12 wherein depth information for pixels in the sequence of images is used to compute predicted motion of the pixels due to the change in camera position or direction.

17. An apparatus according to claim 16 wherein each candidate motion vector comprises a sum of a vector due to the change in camera position or direction and a vector due to the movement of an object in the scene.

18. An apparatus according to claim 17 wherein the module is configured for avoiding adjustment of candidate motion vectors selected from neighbouring locations within the field or frame and for adjusting candidate motion vectors from the other field or frame in the pair.

19. An apparatus according to claim 18 further comprising a memory for storing candidate motion vectors at locations offset by a vector equal to a composition of motion due to movement of objects within the scene.

20. An apparatus according to claim 18 wherein the module is configured for fetching of candidate motion vectors previously stored at locations offset by an amount derived from the characterized movement of the camera position or direction.

23. An apparatus for motion estimation of a sequence of images of claim 12, further comprising a graphics engine configured to output the images and indicia of movement of the camera or viewer position between the images.