Motion estimation for compression of calibrated multi-view image sequences

• US 7,286,689 B2
• Filed: 06/07/2003
• Issued: 10/23/2007
• Est. Priority Date: 06/07/2003
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Mukherjee, Debargha, Chen, Mei, Chang, Nelson Liang An, Damera-Venkata, Niranjan, Lin, Ken K.
• Primary Examiner(s): Ahmed; Samir
• Assistant Examiner(s): ALLISON, ANDRAE S
• Application Number:
  US10/456,379
• Publication Number:
  US 20040247159A1
• Time in Patent Office: 1,599 Days
• Field of Search: 382/243, 382/236, 382/238, 382/107, 375/340, 345/423
• US Class Current: 382/107
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A method for estimating motion of each of a plurality of tessels in an intermediate image relative to a reference image, which includes searching the reference image to find points that lie along epipolar lines in the reference image corresponding to upper-left and lower-right vertices of the tessel, respectively, that result in a best-matching shape; estimating a depth of each of at least two of the vertices of the tessel; and using the depth estimates of the at least two vertices of the tessel to estimate the motion of the tessel relative to the best-matching shape.

Claims

1. A method for estimating motion of each of a plurality of tessels in an intermediate image relative to a reference image, the method comprising:

searching the reference image to find points that lie along epipolar lines in the reference image corresponding to upper-left and lower-right vertices of the tessel, respectively, that result in a best-matching shape, wherein the intermediate image and the reference image are calibrated multiple view images of a static scene captured from different angles and related by known geometry;

estimating a depth of each of at least two of the vertices of the tessel, wherein the depth is a distance from a point on an object in the scene to the center of a camera; and

using the depth estimates of the at least two vertices of the tessel to estimate the motion of the tessel relative to the best-matching shape.

2. The method as set forth in claim 1, wherein the reference and intermediate images are captured by the camera having at least one internal parameter;

and wherein the estimating the depth of each of at least two of the vertices of the tessel is performed using the at least one internal parameter of the camera.

3. The method as set forth in claim 2, wherein the at least one internal parameter is focal length.

4. The method as set forth in claim 1, wherein the images are separated by an angle of rotation θ

; and

wherein rotation and translation of the intermediate image relative to the reference image comprise external parameters.

5. The method as set forth in claim 1, wherein the estimating the depth of each of at least two of the vertices of the tessel is performed based upon geometrical relationships between the reference image and the intermediate image.

6. The method as set forth in claim 4, wherein the estimating the depth of each of at least two of the vertices of the tessel is performed using the external parameters.

7. The method as set forth in claim 4, wherein the estimating the depth of each of at least two of the vertices of the tessel is performed using the external parameters and the at least one internal parameter of the camera.

8. The method as set forth in claim 1, wherein intermediate and reference images are captured by the camera and represent an object is mounted on a turntable, and the turntable and the camera undergo a relative rotation of θ

between successive image captures.

9. The method as set forth in claim 8, wherein the turntable comprises a rotary turntable, and the camera is a single stationary camera capturing the multi-view images.

10. The method as set forth in claim 8, wherein the turntable is stationary, and the multi-view image sequence is captured using a plurality of cameras.

11. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to map the at least two vertices of the tessel to corresponding points of the best-matching shape.

12. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the shape of tessel to a shape of the best-matching shape.

13. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the tessel to a generalized shape.

14. The method as set forth in claim 1, further comprising generating at least two motion vectors based upon the depth estimates of corresponding ones of the at least two vertices of the tessel.

15. The method as set forth in claim 14, further comprising compressing the motion vectors, the reference image, and the tessellations, and generating a compressed bitstream representative thereof.

16. The method as set forth in claim 14, further comprising compressing the motion vectors, spatial transform parameters, the reference image, and the tessellations, and calibration data indicative of internal and external camera parameters, and then generating a compressed bitstream representative thereof.

17. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image is accomplished without spatial transformations of the tessels.

18. The method as set forth in claim 1, wherein each of the tessels is a square tessel, and the at least two vertices of each tessel comprise opposite corners of the respective square tessel.

19. The method as set forth in claim 3, wherein the at least one internal camera parameter further includes aspect ratio, skew, and radial lens distortion.

20. A computer program stored on a computer-readable storage medium for performing the method set forth in claim 1.

21. Apparatus for estimating the motion of an intermediate image of a calibrated multi-view image sequence comprised of a plurality of images of an object captured by a camera, wherein each successive image is separated from the previous image by an angle of rotation θ

, the apparatus comprising a processor and a computer readable medium storing a computer program executed by the processor to estimate motion of each of a plurality of tessels in the intermediate image relative to the reference image, the computer program comprising instructions for;

searching the reference image to find points that lie along epipolar lines in the reference image corresponding to upper-left and lower-right vertices of the tessel, respectively that result in a best-matching shape;

estimating a depth of each of at least two of the vertices of the tessel, wherein the depth is a distance from a point on the object to the center of the camera; and

using the depth estimates of the at least two vertices of the tessel to estimate the motion of the tessel relative to the best-matching shape.

22. The apparatus as set forth in claim 21, wherein the depth of each of at least two of the vertices of the tessel is estimated by using the at least one internal parameter of the camera.

23. The apparatus as set forth in claim 21, wherein the depth of each of at least two of the vertices of the tessel is estimated from geometrical relationships between the reference image and the intermediate image.

24. The apparatus as set forth in claim 21, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to map the at least two vertices of the tessel to corresponding points of the best-matching shape.

25. The apparatus as set forth in claim 21, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the shape of the tessel to a shape of the best-matching shape.

26. The apparatus as set forth in claim 21, wherein estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the tessel to a generalized shape.

27. A device for estimating the motion of an intermediate image of a calibrated multi-view image sequence comprised of a plurality of images of an object captured by a camera(s), wherein each successive image is separated from the previous image by an angle of rotation θ

, the device comprising;

means for selecting at least one of the images as a reference image(s);

means for tessellating the intermediate image into a plurality of tessels each having at least two vertices; and

means for estimating motion of each of the plurality of tessels in the intermediate image relative to the reference images(s) by;

searching the reference image(s) to find points that lie along epipolar line(s) or line segment(s) in the reference image(s) corresponding to the upper-left and lower-right vertices of the tessel, respectively, that is result in a best-matching shape(s);

estimating a depth of each of at least two of the vertices of the tessel, wherein the depth is a distance from a point on the object to the center of the camera(s); and

using the depth estimates of the at least two vertices of the tessel to estimate the motion of the tessel relative to the best-matching shape(s).