Image processing method and apparatus

• US 6,233,004 B1
• Filed: 10/15/1997
• Issued: 05/15/2001
• Est. Priority Date: 04/19/1994
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Tanaka, Koichiro, Katayama, Akihiro, Oshino, Takahiro
• Primary Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US08/950,886
• Time in Patent Office: 1,308 Days
• Field of Search: 348/36, 348/38, 348/39, 348/42, 348/43, 348/44, 348/46, 348/47, 348/48, 348/51, 395/125, 395/154
• US Class Current: 348/48
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

An image processing apparatus detects corresponding pixels in object images picked up from a plurality of viewpoints and interpolates object images which are supposed to be seen from viewpoints other than the plurality of viewpoints on the basis of the detected corresponding pixels.

Claims

1. An image processing method comprising:

a corresponding pixel detecting step of detecting corresponding pixels among object images picked up from a plurality of viewpoints, including;

a virtual light point generating step of generating virtual light points, in sequence, from a position between the viewpoints and the object, moving toward the object; and

a detecting step of detecting the corresponding pixels by comparing pixel values at pixel positions, which correspond to a pixel position of the generated virtual light point, in the object images picked up from said plurality of viewpoints, further wherein at the detecting step, a predetermined mark is put to the detected corresponding pixels and, when a pixel having the predetermined mark exists, it is determined that occlusion occurs at the virtual light point, and the corresponding pixels are determined by using pixels of images seen from viewpoints other than viewpoints at which occlusion has occurred, where the pixels correspond to the virtual light point; and

an interpolation step of interpolating object images seen from viewpoints other than said plurality of viewpoints in response to the detected corresponding pixels.

2. The image processing method according to claim 1, wherein, at said interpolation step, an average of the pixel values of corresponding pixels detected at said corresponding pixel detecting step is set as an interpolated pixel value at a pixel position of an object image seen from a viewpoint other than said plurality of viewpoints, where the pixel position of the object images corresponds to the position of the virtual light point corresponding to the corresponding pixels detected at said corresponding pixel detecting step.

3. The image processing method according to claim 1, wherein a pixel position in the object image seen from a viewpoint other than said plurality of viewpoints is calculated based on a plane function with a pixel position of the virtual light point and positions of said plurality of viewpoints as parameters.

4. The image processing method according to claim 1, wherein cameras are placed at said plurality of viewpoints.

5. The image processing method according to claim 1, wherein said plurality of viewpoints are arranged on a plane and viewpoints other than said plurality of viewpoints also exist on the plane.

6. A three dimensional viewpoint movement corresponding image processing method comprising:

a viewpoint detecting step of detecting positions of viewer'"'"'s eyes; and

an image generating step of generating an image seen from the positions of viewer'"'"'s eyes detected at said viewpoint detecting step by using at least a part of the images, corresponding to each viewpoint, which are generated by the image processing method according to the claim 1.

7. The three dimensional viewpoint movement corresponding image processing method according to claim 6, wherein said part of the images are generated images seen at the plurality of different distances from the object to the plurality of viewpoints.

8. A three dimensional viewpoint movement corresponding image processing method comprising:

a viewpoint detecting step of detecting positions of viewer'"'"'s eyes;

an image generating step of generating images seen from the positions of the viewer'"'"'s eyes in response to the positions of viewer'"'"'s eyes detected at said viewpoint detecting step and generated images seen at the plurality of different distances from the object to the plurality of viewpoints, where the generated images are generated by said three dimensional viewpoint movement corresponding image processing method according to claim 7; and

a displaying step of displaying images generated at said image generating step.

9. The three dimensional viewpoint movement corresponding image processing method according to claim 8, wherein, at said displaying step, the images generated at said image generating step are displayed on a stereoscopic image display.

10. The three dimensional viewpoint movement corresponding image processing method according to claim 8, wherein, at said displaying step, the images generated at said image generating step are displayed on a head mount display.

11. The three dimensional viewpoint movement corresponding image processing method according to claim 8, wherein, at said displaying step, the images generated at said image generating step are displayed on a lenticular display.

12. An image processing apparatus comprising:

corresponding pixel detecting means for detecting corresponding pixels among object images picked up from a plurality of viewpoints, including;

virtual light point generating means for generating virtual light points, in sequence, from a position between the viewpoints and the object, moving toward the object; and

detecting means for detecting the corresponding pixels by comparing pixel values at pixel positions, which correspond to a pixel position of the generated virtual light point, in the object images picked up from said plurality of viewpoints, further wherein the detecting means puts a predetermined mark to the detected corresponding pixels and, when a pixel having the predetermined mark exists, it is determined that occlusion occurs at the virtual light point, and the corresponding pixels are determined by using pixels of images seen from viewpoints other than viewpoints at which occlusion has occurred, where the pixels correspond to the virtual light point; and

interpolation means for interpolating object images seen from a viewpoint other than said plurality of viewpoints in response to the detected corresponding pixels.

13. The image processing apparatus according to claim 12, wherein a pixel position in the object image seen from a viewpoint other than said plurality of viewpoints is calculated based on a plane function with a pixel position of the virtual light point and positions of said plurality of viewpoints as parameters.

14. The image processing apparatus according to claim 12, wherein cameras are placed at said plurality of viewpoints.

15. The image processing apparatus according to claim 12, wherein said interpolation means sets an average of the pixel values of corresponding pixels detected by said corresponding pixel detecting means as a interpolated pixel value at a pixel position of an object image seen from a viewpoint other than said plurality of viewpoints, where the pixel position of the object images corresponds to the position of the virtual light point corresponding to the corresponding pixels detected by said corresponding pixel detecting means.

16. The image processing apparatus according to claim 12, wherein said plurality of viewpoints are arranged on a plane and viewpoints other than said plurality of viewpoints also exist on the plane.

17. A three dimensional viewpoint movement corresponding image processing apparatus comprising:

viewpoint detecting means for detecting positions of viewer'"'"'s eyes; and

image generating means for generating an image seen from the positions of viewer'"'"'s eyes detected by said viewpoint detecting means by using at least a part of the images, corresponding to each viewpoint, which are generated by the image processing apparatus according to the claim 12.

18. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 17, wherein said part of the images are generated images seen at the plurality of different distances from the object to the plurality of viewpoints.

19. A three dimensional viewpoint movement corresponding image processing apparatus comprising:

viewpoint detecting means for detecting positions of viewer'"'"'s eyes;

image generating means for generating images seen from the positions of the viewer'"'"'s eyes in response to the positions of viewer'"'"'s eyes detected by said viewpoint detecting means and the generated images seen at the plurality of different distances from the object to the plurality of viewpoints, where the generated images are generated by said three dimensional viewpoint movement corresponding image processing apparatus according to claim 18; and

displaying means for displaying images generated by said image generating means.

20. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 19, wherein, in said displaying means, the images generated at said image generating means are displayed on a stereoscopic image display.

21. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 19, wherein, in said displaying means, the images generated at said image generating means are displayed on a head mount display.

22. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 19, wherein, in said displaying means, the images generated at said image generating means are displayed on a lenticular display.

23. An image processing method for forming a virtual angled image from a plurality of images picked-up by a plurality of cameras placed at relatively separated positions comprising:

a corresponding pixel detecting step of detecting corresponding pixels among object images picked up from a plurality of viewpoints each based on position information of each of said plurality of cameras;

a selecting step of selecting pixels usable for interpolating operation among the detected corresponding pixels, the selected pixels satisfying a predetermined condition that the selected pixels have not been used for the interpolating operation so as to prevent a deterioration of the interpolated image quality; and

an image producing step of producing an object image seen from an optimal viewpoint other than said plurality of viewpoints by interpolating a plurality of the selected pixels.

24. The image processing method according to claim 23, wherein said corresponding pixel detecting step comprises:

a virtual light point generating step of generating virtual light points, in sequence, from a position between the viewpoints and the object, moving toward the object; and

a detecting step of detecting the corresponding pixels by comparing pixel values at pixel positions, which correspond to a pixel position of the generated virtual light point, in the object images picked up from said plurality of viewpoints.

25. The image processing method according to claim 24, wherein, at said image producing step, an average of the pixel values of corresponding pixels detected at said corresponding pixel detecting step is set as an interpolated pixel value at a pixel position of an object image seen from a viewpoint other than said plurality of viewpoints, where the pixel position of the object images corresponds to the position of the virtual light point corresponding to the corresponding pixels detected at said corresponding pixel detecting step.

26. The image processing method according to claim 24, wherein a pixel position in the object image seen from a viewpoint other than said plurality of viewpoints is calculated based on a plane function with a pixel position of the virtual light point and positions of said plurality of viewpoints as parameters.

27. The image processing method according to claim 24, wherein, at said corresponding pixel detecting step, a predetermined mark is put to the detected corresponding pixels and, when a pixel having the predetermined mark exists, it is determined that occlusion occurs at the virtual light point, and the corresponding pixels are determined by using pixels of images seen from viewpoints other than viewpoints at which occlusion has occurred, where the pixels correspond to the virtual light point.

28. The image processing method according to claim 23, wherein cameras are placed at said plurality of viewpoints.

29. The image processing method according to claim 23, wherein said plurality of viewpoints are arranged on a plane and viewpoints other than said plurality of viewpoints also exist on the plane.

30. A three dimensional viewpoint movement corresponding image processing method comprising:

a viewpoint detecting step of detecting positions of viewer'"'"'s eyes; and

an image generating step of generating an image seen from the positions of viewer'"'"'s eyes detected at said viewpoint detecting step by using at least a part of the images, corresponding to each viewpoint, which are generated by the image processing method according to the claim 23.

31. The three dimensional viewpoint movement corresponding image processing method according to claim 30, wherein said part of the images are generated images seen at the plurality of different distances from the object to the plurality of viewpoints.

32. A three dimensional viewpoint movement corresponding image processing method comprising:

a viewpoint detecting step of detecting positions of viewer'"'"'s eyes;

an image generating step of generating images seen from the positions of the viewer'"'"'s eyes in response to the positions of viewer'"'"'s eyes detected at said viewpoint detecting step and the generated images seen at the plurality of different distances from the object to the plurality of viewpoints, where the generated images are generated by said three dimensional viewpoint movement corresponding image processing method according to claim 31; and

a displaying step of displaying images generated at said image generating step.

33. The three dimensional viewpoint movement corresponding image processing method according to claim 32, wherein, at said displaying step, the images generated at said image generating step are displayed on a stereoscopic image display.

34. The three dimensional viewpoint movement corresponding image processing method according to claim 32, wherein at said displaying step, the images generated at said image generating step are displayed on a head mount display.

35. The three dimensional viewpoint movement corresponding image processing method according to claim 32, wherein at said displaying step, the images generated at said image generating step are displayed on a lenticular display.

36. An image processing apparatus for forming a virtual angled image from a plurality of images picked-up by a plurality of cameras placed at relatively separated positions comprising:

corresponding pixel detecting means for detecting corresponding pixels among object images picked up from a plurality of viewpoints each based on position information of each of said plurality of cameras;

selecting means for selecting pixels usable for interpolating operation among the detected corresponding pixels, the selected pixels satisfying a predetermined condition that the selected pixels have not been used for the interpolating operation so as to prevent a deterioration of the interpolation image quality; and

image producing means for producing an object image seen from an optional viewpoint other than said plurality of viewpoints by interpolating a plurality of the selected pixels.

37. The image processing apparatus according to claim 36, wherein said corresponding pixel detecting means comprises:

virtual light point generating means for generating virtual light points, in sequence, from a position between the viewpoints and the object, moving toward the object; and

detecting means for detecting the corresponding pixels by comparing pixel values at pixel positions, which correspond to a pixel position of the generated virtual light point, in the object images picked up from said plurality of viewpoints.

38. The image processing apparatus according to claim 37, wherein said image producing means sets an average of the pixel values of corresponding pixels detected by said corresponding pixel detecting means as an interpolated pixel value at a pixel position of an object image seen from a viewpoint other than said plurality of viewpoints, where the pixel position of the object images corresponds to the position of the virtual light point corresponding to the corresponding pixels detected by said corresponding pixel detecting means.

39. The image processing apparatus according to claim 37, wherein a pixel position in the object image seen from a viewpoint other than said plurality of viewpoints is calculated based on a plane function with a pixel position of the virtual light point and positions of said plurality of viewpoints as parameters.

40. The image processing apparatus according to claim 37, wherein said corresponding pixel detecting means puts a predetermined mark to the detected corresponding pixels and, when a pixel having the predetermined mark exists, it is determined that occlusion occurs at the virtual light point, and the corresponding pixels are determined by using pixels of images seen from viewpoints other than viewpoints at which occlusion has occurred, where the pixels correspond to the virtual light point.

41. The image processing apparatus according to claim 36, wherein cameras are placed at said plurality of viewpoints.

42. The image processing apparatus according to claim 36, wherein said plurality of viewpoints are arranged on a plane and viewpoints other than said plurality of viewpoints also exist on the plane.

43. A three dimensional viewpoint movement corresponding image processing apparatus comprising:

viewpoint detecting means for detecting positions of viewer'"'"'s eyes; and

image generating means for generating an image seen from the positions of viewer'"'"'s eyes detected by said viewpoint detecting means by using at least a part of the images, corresponding to each viewpoint, which are generated by the image processing apparatus according to the claim 40.

44. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 43, wherein said part of the images are generated images seen at the plurality of different distances from the object to the plurality of viewpoints.

45. A three dimensional viewpoint movement corresponding image processing apparatus comprising:

viewpoint detecting means for detecting positions of viewer'"'"'s eyes;

image generating means for generating images seen from the positions of the viewer'"'"'s eyes in response to the positions of viewer'"'"'s eyes detected by said viewpoint detecting means and the generated images seen at the plurality of different distances from the object to the plurality of viewpoints, where the generated images are generated by said three dimensional viewpoint movement corresponding image processing apparatus according to claim 44; and

displaying means for displaying images generated by said image generating means.

46. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 45, wherein, in said displaying means, the images generated at said image generating means are displayed on a stereoscopic image display.

47. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 45, wherein, in said displaying means, the images generated at said image generating means are displayed on a head mount display.

48. The three dimensional viewpoint movement corresponding image processing apparatus according to claim 45, wherein, in said displaying means, the images generated at said image generating means are displayed on a lenticular display.