Image processing method with viewpoint compensation and apparatus therefor

• US 6,191,808 B1
• Filed: 12/19/1996
• Issued: 02/20/2001
• Est. Priority Date: 08/04/1993
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Tanaka, Koichiro, Katayama, Akihiro
• Primary Examiner(s): Rao, Andy
• Application Number:
  US08/769,884
• Time in Patent Office: 1,524 Days
• Field of Search: 348/39, 348/36, 348/42, 348/46, 395/125
• US Class Current: 348/39
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G09G 2320/106 Determination of movement vectors or equivalent parameters within the image
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/397 Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay G09G5/02 takes precedence
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307
• CPC Class Codes: H04N 13/373 : for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 : for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 : for tracking vertical translational head movements
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 : specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0085 : Motion estimation from stereoscopic image signals

Abstract

There are provided a multi-viewpoint image database for holding images from a plurality of different viewpoints, a viewpoint detector for detecting a position of the observer'"'"'s viewpoint, and a viewpoint parameter calculating portion for obtaining a viewpoint parameter from an output from the viewpoint detector. Further, a pixel value producing portion is provided for calculating a parameter necessary for reconstruction of image from the type of display screen and the viewpoint parameter, calculating with the parameter to which pixel in the images in the multi-viewpoint image database each pixel in a reconstructed image corresponds, and extracting corresponding pixels from the multi-viewpoint image database.

Claims

1. An image processing method comprising:

a data generating step of generating images of an object obtained from a plurality of positions and for converting the images so generated into image data;

a viewpoint detecting step of detecting the viewpoint of an observer;

a selecting step of selecting those parts of the image data which corresponds to the images which exist on lines passing through points on an image outputting device and the position of the viewpoint of the observer;

a composing step for composing the parts of the image data which are selected in said selecting step, so as to generate an image from the position of the viewpoint of the observer; and

an outputting step for outputting the image data of the image generated in said composing step to said image outputting device, wherein the selecting step does not rely on coordinate values in a three-dimensional space for points on the image.

2. A method according to claim 1, wherein said data generating step includes obtaining images from a database storing images already taken by a camera from a plurality of directions.

3. A method according to claim 2, wherein said images are obtained from more than one camera.

4. A method according to claim 1, wherein said data generating step comprises performing an interpolation process for a plurality of images photo-taken by at least one camera to produce images from positions of viewpoints different from the positions where the images were taken by said camera.

5. A method according to claim 1, further comprising an inter-viewpoint interpolation processing step of performing an interpolation process on stored image data to produce new images and adding said new images to the image data.

6. A method according to claim 1, wherein said viewpoint detecting step includes a step for detecting the direction of the line of sight of said observer as well as the position of the eyes of the observer.

7. A method according to claim 1, wherein said composing step includes:

calculating a parameter necessary for reconstruction of an image from the position of the eyes of the observer and the type of the image outputting device;

calculating to which pixel in said input images each pixel in an image to be reconstructed corresponds, using said parameter; and

extracting corresponding pixels from said input images to reconstruct an image.

8. A method according to claim 1, wherein said image data is output to a stereo display.

9. A method according to claim 1, wherein said image outputting device includes a head-mounted display.

10. A method according to claim 1, wherein said image outputting device includes a lenticular display.

11. An image processing method according to claim 1, wherein the position of said image outputting device is determined on the basis of the position of the viewpoint of the observer and a direction of the line of sight of the observer.

12. Image processing apparatus comprising:

means for generating images of an object obtained from a plurality of positions and for converting the images so generated into image data; and

means for detecting the viewpoint of an observer;

and characterized by calculating means adapted to select those parts of the image data which corresponds to the images which exist on lines passing through points on an image outputting device and the position of the viewpoint of the observer as detected;

and for composing the parts of the image data which have been so selected, so as to generate an image from the position of the viewpoint of the observer, wherein the calculating means does not rely on coordinate values in a three-dimensional space for points on the image.

13. Apparatus according to claim 12, wherein said calculating means are adapted to obtain images from a database storing images already taken by a camera from a plurality of directions.

14. Apparatus according to claim 13, wherein said images have been obtained from more than one camera.

15. Apparatus according to claim 12, wherein said calculating means is adapted to perform an interpolation process for a plurality of images photo-taken by at least one camera to produce images from positions of viewpoints different from the positions where the images were taken by said camera.

16. Apparatus according to claim 12, wherein said calculating means are adapted to perform an inter-viewpoint interpolation process on stored image data to produce new images and to add said new images to said image data.

17. Apparatus according to claim 12, wherein said viewpoint detecting means are adapted to detect the position of the eyes of the observer as well as the direction of the line of sight of said observer.

18. Apparatus according to claim 12, wherein calculating means are adapted to calculate a parameter necessary for reconstruction of an image from the position of the eyes of the observer and the type of the image outputting device;

calculating to which pixel in said input images each pixel in an image to be reconstructed corresponds, using said parameter; and

extracting corresponding pixels from said input images to reconstruct the image to be output to the observer.

19. Apparatus according to claim 12, including a stereo display to which the generated image data is output.

20. Apparatus according to claim 12, wherein said image outputting device includes a head-mounted display.

21. Apparatus according to claim 12, wherein said image outputting device comprises a lenticular display.

22. Apparatus according to claim 12, including means for determining the position of said image outputting device on the basis of the position of the viewpoint of the observer and the direction of the line of sight of the observer.