Method of and apparatus for rectifying a stereoscopic image

• US 20030156751A1
• Filed: 02/20/2002
• Published: 08/21/2003
• Est. Priority Date: 02/23/2001
• Status: Active Grant

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Jones, Graham Roger, Lee, Delman
• Granted Patent: US 7,113,632 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of rectifying a stereoscopic image consisting of left and right captured images comprises determining left and right rectification transformations. According to one aspect of the invention, statistics of the parameters of the stereoscopic image capture device used to capture the left and right images are used in the determination of the left and/or right rectification transformation.

According to another aspect of the invention, the left and right rectification transformations are constrained to correspond to a transformation to a virtual alignment to a parallel camera set-up.

Once the left and right rectification transformations have been determined they are preferably used to rectify the left and right images to eliminate, or substantially eliminate, vertical disparity from the rectified image pair. The left and right rectified images may then be displayed on a stereoscopic display device for viewing by an observer, or they may alternatively be stored for later use.

Claims

1. A method of rectifying a stereoscopic image comprising first and second images captured using a respective one of first and second image capture devices, the first and second image capture devices forming a stereoscopic image capture device, the method comprising the step of:

determining first and second rectification transformations for rectifying a respective one of the first and second images so as to reduce vertical disparity;

characterised in that the method comprises using statistics of the parameters of the stereoscopic image capture device in the determination of the first and/or second rectification transformations.

2. A method as claimed in claim 1 wherein each rectification transformation comprises a horizontal shear and scaling component, and the statistics of the parameters of the stereoscopic image capture device are used in the determination of the horizontal shear and scaling component of the first and/or second rectification transformation.

3. A method as claimed in claim 1 and comprising the steps of:

determining the first and second rectification transformations;

varying the statistics of the parameters of the stereoscopic image capture device;

re-determining the first and second rectification transformations; and

rectifying the first and second images using a respective one of the re-determined first and second rectification transformations.

4. A method as claimed in claim 3 and comprising the further steps of:

rectifying at least part of the first image and at least part of the second image using a respective one of the initially-determined first and second rectification transformations; and

displaying the rectified parts of the first and second images on a stereoscopic display device.

5. A method as claimed in claim 4 and comprising the further steps of:

rectifying at least part of the first image and at least part of the second image using a respective one of the initially-determined first and second rectification transformations;

displaying the rectified parts of the first and second images on the stereoscopic display device; and

varying the statistics of the parameters of the stereoscopic image capture device on the basis of the display of the rectified parts of the first and second images.

6. A method as claimed in claim 1 wherein the statistics of the parameters of the stereoscopic image capture device relate to parameters of the first image capture device and/or to parameters of the second image capture device.

7. A method as claimed in claim 6 wherein the statistics of the parameters of the stereoscopic image capture device comprise the mean of the focal length of the first and second image capture devices.

8. A method as claimed in claim 6 wherein the statistics of the parameters of the stereoscopic image capture device comprise the standard deviation of the focal length of the first and second image capture devices.

9. A method as claimed in claim 6 wherein the statistics of the parameters of the stereoscopic image capture device comprise the mean of the principal point of the first and second image capture devices.

10. A method as claimed in claim 6 wherein the statistics of the parameters of the stereoscopic image capture device comprise the standard deviation of the principal point of the first and second image capture devices.

11. A method as claimed in claim 1 wherein the statistics of the parameters of the stereoscopic image capture device relate to the alignment of the first image capture device relative to the second image capture device.

12. A method as claimed in claim 11 wherein the statistics of the parameters of the stereoscopic image capture device comprise the mean of the rotation of the optical axis of the first image capture device relative to the optical axis of the second image capture device.

13. A method as claimed in claim 11 wherein the statistics of the parameters of the stereoscopic image capture device comprise the standard deviation of the rotation of the optical axis of the first image capture device relative to the optical axis of the second image capture device.

14. A method as claimed in claim 1 wherein the first and second rectification transformations are determined so as correspond to a virtual alignment to a parallel camera set-up.

15. A method as claimed in claim 1 wherein the first captured image and second captured image comprise a still stereoscopic image.

16. A method as claimed in claim 1 wherein the first captured image and second captured image comprise a frame of a stereoscopic video image.

17. A method as claimed in claim 16 and comprising the steps of:

determining first and second rectification transformations for a first frame of the stereoscopic video image using a method as defined in claim 1;

and

rectifying subsequent frames of the stereoscopic video image using the first and second rectification transformations determined for the first frame of the stereoscopic video image.

18. A method as claimed in claim 16 and comprising the steps of:

determining first and second rectification transformations for a first frame of the stereoscopic video image using a method as defined in claim 1;

rectifying first to N^th frames of the stereoscopic video image using the first and second rectification transformations determined for the first frame of the stereoscopic video image;

determining first and second rectification transformations for an (N+1)^th frame of the stereoscopic video image using a method as defined in claim 1;

and

rectifying (N+1)^th to (2N)^th frames of the stereoscopic video image using the first and second rectification transformations determined for the (N+1)^th frame of the stereoscopic video image.

19. A method as claimed in claim 16 and comprising the steps of;

determining first and second rectification transformations for each frame of the stereoscopic video image using a method as defined in claim 1;

and

rectifying each frame of the stereoscopic video image using the first and second rectification transformations determined for that frame.

20. A method as claimed in claim 1 and comprising the further step of rectifying the first and second captured images using a respective one of the first and second rectification transformations.

21. A method as claimed in claim 20 and comprising the further step of displaying the first and second rectified images on a stereoscopic display device for viewing by an observer.

32. A method as claimed in claim 2§

and comprising the steps of;

determining first and second rectification transformations for each frame of the stereoscopic video image using a method as defined in claim 22;

and

rectifying each frame of the stereoscopic video image using the first and second rectification transformations determined for that frame.

22. A method of rectifying a stereoscopic image comprising first and second images captured using first and second image capture devices, the first and second image capture devices forming a stereoscopic image capture device, the method comprising the step of;

determining first and second rectification transformations for rectifying a respective one of the first and second images so as to reduce vertical disparity;

characterised in that the method comprises determining the first and second rectification transformation so that the first and second rectification transformations correspond to a virtual alignment to a parallel camera set-up.

23. A method as claimed in claim 22 and further comprising the step of using statistics of the parameters of the stereoscopic image capture device in the step of determining the first and second rectification transformations.

24. A method as claimed in claim 23 wherein the statistics of the parameters of the stereoscopic image capture device relate to the alignment of the first image capture device relative to the second image capture device.

25. A method as claimed in claim 22 wherein the step of determining the first and second rectification transformations comprises:

determining a first component of each of the first and second rectification transformations, the first component of the first rectification transformation and the first component of the second rectification transformation substantially eliminating vertical disparity from the rectified image pair; and

determining a second component of each of the first and second rectification transformations so that the first and second rectification transformations correspond to a virtual alignment to a parallel camera set-up.

26. A method as claimed in claim 25 and further comprising the step of using statistics of the parameters of the stereoscopic image capture device in the step of determining the first and second rectification transformations

wherein the statistics of the parameters of the stereoscopic image capture device are used in the step of determining the second components of the first and second rectification transformations.

27. A method as claimed in claim 25 wherein the statistics of the parameters of the stereoscopic image capture device relate to the alignment of the first image capture device relative to the second image capture device.

28. A method as claimed in claim 22 wherein the first captured image and second captured image comprise a still stereoscopic image.

29. A method as claimed in claim 22 wherein the first captured image and second captured image comprise a frame of a stereoscopic video image.

30. A method as claimed in claim 29 and comprising the steps of:

determining first and second rectification transformations for a first frame of the stereoscopic video image using a method as defined in claim 22;

and

rectifying subsequent frames of the stereoscopic video image using the first and second rectification transformations determined for the first frame of the stereoscopic video image.

31. A method as claimed in claim 29 and comprising the steps of:

determining first and second rectification transformations for a first frame of the stereoscopic video image using a method as defined in claim 22;

rectifying first to N^th frames of the stereoscopic video image using the first and second rectification transformations determined for the first frame of the stereoscopic video image;

determining first and second rectification transformations for an (N+1)^th frame of the stereoscopic video image using a method as defined in claim 22;

and

rectifying (N+1)^th to (2N)^th frames of the stereoscopic video image using the first and second rectification transformations determined for the (N+1)^th frame of the stereoscopic video image.

33. A method as claimed in claim 22 and comprising the further step of rectifying the first and second captured images using a respective one of the first and second rectification transformations.

34. A method as claimed in claim 33 and comprising the further step of displaying the first and second rectified images on a stereoscopic display device for viewing by an observer.

35. An apparatus for rectifying a stereoscopic image comprising first and second images captured using a respective one of first and second image capture devices, the first and second image capture devices forming a stereoscopic image capture device, the apparatus comprising:

means for determining first and second rectification transformations for rectifying a respective one of the first and second images so as to reduce vertical disparity using statistics of the parameters of the stereoscopic image capture device in the determination of the first and/or second rectification transformations.

36. An apparatus as claimed in claim 35 and further comprising means for rectifying the first and second captured images using a respective one of the first and second rectification transformations.

37. An apparatus as claimed in claim 35 and comprising a programmable data processor.

38. A storage medium containing a program for the data processor of an apparatus as defined in claim 37.

39. An apparatus for rectifying a stereoscopic image comprising first and second images captured using first and second image capture devices, the first and second image capture devices forming a stereoscopic image capture device, the apparatus comprising:

means for determining first and second rectification transformations for rectifying a respective one of the first and second images so as to reduce vertical disparity, the first and second rectification transformations corresponding to a virtual alignment to a parallel camera set-up.

40. An apparatus as claimed in claim 39 and further comprising means for rectifying the first and second captured images using a respective one of the first and second rectification transformations.

41. An apparatus as claimed in claim 39 and comprising a programmable data processor.

42. A storage medium containing a program for the data processor of an apparatus as defined in claim 41.