Motion estimation for compression of calibrated multi-view image sequences

• US 20040247159A1
• Filed: 06/07/2003
• Published: 12/09/2004
• Est. Priority Date: 06/07/2003
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Mukherjee, Debargha, Chen, Mei, Chang, Nelson Liang An, Damera-Venkata, Niranjan, Lin, Ken K.
• Granted Patent: US 7,286,689 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A method for estimating motion of each of a plurality of tessels in an intermediate image relative to a reference image, which includes searching the reference image to find points that lie along epipolar lines in the reference image corresponding to upper-left and lower-right vertices of the tessel, respectively, that result in a best-matching shape; estimating a depth of each of at least two of the vertices of the tessel; and using the depth estimates of the at least two vertices of the tessel to estimate the motion of the tessel relative to the best-matching shape.

Claims

1. A method for estimating motion of each of a plurality of tessels in an intermediate image relative to a reference image, the method comprising:

searching the reference image to find points that lie along epipolar lines in the reference image corresponding to upper-left and lower-right vertices of the tessel, respectively, that result in a best-matching shape;

estimating a depth of each of at least two of the vertices of the tessel; and

using the depth estimates of the at least two vertices of the tessel to estimate the motion of the tessel relative to the best-matching shape.

2. The method as set forth in claim 1, wherein the reference and intermediate images are captured by a camera having at least one internal parameter;

and wherein the estimating the depth of each of at least two of the vertices of the tessel is performed using the at least one internal parameter of the camera.

3. The method as set forth in claim 2, wherein the at least one internal parameter is focal length.

4. The method as set forth in claim 1, wherein the images are separated by an angle of rotation θ

; and

wherein rotation and translation of the intermediate image relative to the reference image comprise external parameters.

5. The method as set forth in claim 3, wherein rotation and translation of the intermediate image relative to the reference image comprise external parameters.

6. The method as set forth in claim 1, wherein the estimating the depth of each of at least two of the vertices of the tessel is performed based upon geometrical relationships between the reference image and the intermediate image.

7. The method as set forth in claim 4, wherein the estimating the depth of each of at least two of the vertices of the tessel is performed using the external parameters.

8. The method as set forth in claim 5, wherein the estimating the depth of each of at least two of the vertices of the tessel is performed using the external parameters and the at least one internal parameter of the camera.

9. The method as set forth in claim 1, wherein intermediate and reference images are captured by a camera and represent an object is mounted on a turntable, and the turntable and the camera undergo a relative rotation of θ

between successive image captures.

10. The method as set forth in claim 9, wherein the turntable comprises a rotary turntable, and the multi-view image sequence is captured using a single stationary camera.

11. The method as set forth in claim 9, wherein the turntable is stationary, and the multi-view image sequence is captured using a plurality of cameras.

12. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to map the at least two vertices of the tessel to corresponding points of the best-matching shape.

13. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the shape of the tessel to a shape of the best-matching shape.

14. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the tessel to a generalized shape.

15. The method as set forth in claim 1, further comprising generating at least two motion vectors based upon the depth estimates of corresponding ones of the at least two vertices of the tessel.

16. The method as set forth in claim 15, further comprising compressing the motion vectors, the reference image, and the tessellations, and generating a compressed bitstream representative thereof.

17. The method as set forth in claim 15, further comprising compressing the motion vectors, spatial transform parameters, the reference image, and the tessellations, and calibration data indicative of internal and external camera parameters, and then generating a compressed bitstream representative thereof.

18. The method as set forth in claim 1, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image is accomplished without spatial transformation of the tessels.

19. The method as set forth in claim 1, wherein each of the tessels is a square tessel, and the at least two vertices of each tessel comprise opposite corners of the respective square tessel.

20. The method as set forth in claim 3, wherein the at least one internal camera parameter further includes aspect ratio, skew, and radial lens distortion.

21. An image forming device that implements the method set forth in claim 1.

22. A printer that implements the method set forth in claim 1.

23. A processor for performing the method set forth in claim 1.

24. Software that includes computer-executable code for instructing a processor to perform the method set forth in claim 1.

25. A computer program stored on a computer-readable storage medium for performing the method set forth in claim 1.

26. Apparatus for estimating the motion of an intermediate image of a calibrated multi-view image sequence comprised of a plurality of images of an object captured by a camera, wherein each successive image is separated from the previous image an angle of rotation θ

, the apparatus comprising a processor programmed to estimate motion of each of a plurality of tessels in the intermediate image relative to the reference image by;

searching the reference image to find points that lie along epipolar lines in the reference image corresponding to upper-left and lower-right vertices of the tessel, respectively, that result in a best-matching shapes;

estimating a depth of each of at least two of the vertices of the tessel; and

using the depth estimates of the at least two vertices of the tessel to estimate the motion of the tessel relative to the best-matching shapes.

27. The apparatus as set forth in claim 26, wherein the depth of each of at least two of the vertices of the tessel is estimated by using the at least one internal parameter of the camera.

28. The apparatus as set forth in claim 26, wherein the depth of each of at least two of the vertices of the tessel is estimated from geometrical relationships between the reference image and the intermediate image.

29. The apparatus as set forth in claim 26, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to map the at least two vertices of the tessel to corresponding points of the best-matching shape.

30. The apparatus as set forth in claim 26, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the shape of the tessel to a shape of the best-matching shape.

31. The apparatus as set forth in claim 26, wherein the estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image further comprises using a spatial transform to transform the tessel to a generalized shape.

32. A device for estimating the motion of an intermediate image of a calibrated multi-view image sequence comprised of a plurality of images of an object captured by a camera(s), wherein each successive image is separated from the previous image by an angle of rotation θ

, the device comprising;

means for selecting at least one of the images as a reference image(s);

means for tessellating the intermediate image into a plurality of tessels each having at least two vertices; and

means for estimating motion of each of the plurality of tessels in the intermediate image relative to the reference image(s) by;

searching the reference image(s) to find points that lie along epipolar line(s) or line segment(s) in the reference image(s) corresponding to the upper-left and lower-right vertices of the tessel, respectively, that result in a best-matching shape(s);

estimating a depth of each of at least two of the vertices of the tessel; and

using the depth estimates of the at least two vertices of the tessel to estimate the motionof the tessel relative to the best-matching shape(s).