Method and apparatus for displaying real space and virtual space images

• US 6,045,229 A
• Filed: 07/18/1997
• Issued: 04/04/2000
• Est. Priority Date: 10/07/1996
• Status: Expired due to Fees

• Current Assignee: Minolta Co., Ltd. (Konica Minolta Inc.)
• Original Assignee: Minolta Co., Ltd. (Konica Minolta Inc.)
• Inventors: Suzuki, Shinsuke, Tachi, Susumu, Maeda, Taro
• Primary Examiner(s): DOWLING, WILLIAM C
• Application Number:
  US08/896,973
• Time in Patent Office: 991 Days
• Field of Search: 353/28, 353/30, 353/122, 353/121, 359/630, 359/618, 345/7, 345/8, 345/9
• US Class Current: 353/28
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A camera is arranged at a conjugate position to eyes of the observer. A distance from the observer to a pixel in the real space image located at the same position as a pixel in the virtual space image is determined according to the real space image picked up by the camera. A virtual space image is generated based on virtual space information stored beforehand. Then, the distance of the real space image is compared with that of the virtual space image, and one of the pixels more near the observer is selected for the display. This processing is performed on each pixel, and the real space image and the virtual space image are integrated as an image to be displayed for the observer at the same time.

Claims

1. A method for displaying a real space image and a virtual space image, the method comprising the steps of:

providing a pickup device at a position conjugate to eyes of an observer;

picking up an image of real space by the pickup device to thereby provide a real space image wherein the real space image includes a plurality of first pixels, each of said first pixels representing a respective portion of the image of real space;

obtaining a virtual space image based on virtual space information wherein the virtual space image includes a plurality of second pixels, each of said second pixels representing a respective portion of the virtual space information, wherein each individual one of said second pixels corresponds to a respective individual one of said first pixels;

determining, for each one of said first pixels, a distance from the observer to a location in real space of the portion of the image of real space corresponding to the respective first pixel; and

comparing, for each one of the first pixels, the distance from the observer to the location in real space corresponding to the respective first pixel with a distance from the observer to an apparent location in virtual space associated with the corresponding second pixel, and selecting whichever of the respective first pixel and the corresponding respective second pixel which corresponds to a location which is nearer to the observer in order to present a synthesized image of the virtual space image and the real space image to the observer.

2. A method according to claim 1, wherein the distance from the observer to the location in real space corresponding to a respective first pixel is determined by detecting a movement of the observer.

3. A method according to claim 1, wherein the distance from the observer to the location in real space corresponding to a respective first pixel is calculated according to a translation of the observer after canceling an effect of a rotation of the observer in a movement of an object in the image of real space.

4. A method according to claim 1, wherein a position of the observer is continuously monitored, and the distance from the observer to the location in real space corresponding to a respective first pixel is calculated by using the position of the observer.

22. A method according to claim 1, wherein the distance from the observer to a location in real space corresponding to a respective one of the first pixels in the real space image is determined by a process including the steps of:

picking up a first image of real space to provide a first real space image and picking up a second image of real space to provide a second real space image;

identifying an object that is represented in each of the first real space image and the second real space image;

determining a location value for each of the first real space image and the second real space image, wherein each location value represents a distance from the observer to a location in real space corresponding to a position of said object in the respective real space image;

determining a displacement value representing the difference between the location values for said object in the first real space image and the second real space image;

determining a rotation value representing a rotation from the observer to a location in real space corresponding to an amount of displacement of said object between the first real space image and the second real space image that results from a rotation of the observer;

determining a translation value by canceling the rotation value from the displacement value; and

utilizing said translation value to calculate the distance from the observer to a location in real space corresponding to each individual one of the first pixels in a first real space image.

23. A method according to claim 22, further including the step of continuously repeating the steps of claim 22 for a plurality of images of real space.

5. A display apparatus for displaying a real space image and a stored virtual space image at the same time, the display apparatus comprising:

a pickup device provided at a position conjugate to eyes of an observer for picking up an image of real space to thereby provide a real space image including a plurality of first pixels, each of said first pixels representing a respective portion of the image of real space;

a stored virtual space image including a plurality of second pixels, each of said second pixels representing a respective portion of stored virtual space information, wherein each individual one of said second pixels corresponds to a respective individual one of said first pixels;

a position detector for detecting a position of a viewing point of the observer;

a real space distance image generator for generating a real space distance image on the basis of the position of the viewing point detected by said position detector and locations in real space corresponding to the real space image, the real space distance image including a plurality of first distance data points, each individual one of said first distance data points corresponding to a respective individual one of said first pixels in the real space image;

a virtual space distance image generator for generating a virtual space distance image on the basis of the position of the viewing point detected by said position detector and apparent locations in virtual space corresponding to the stored virtual space image, the virtual space distance image including a plurality of second distance data points, each individual one of said second distance data points corresponding to a respective individual one of said second pixels in the stored virtual space image, each individual one of said second distance data points corresponding to a respective individual one of said first distance data points in the real space distance image;

a comparator for comparing, for each individual one of said first distance data points and a corresponding individual one of said second distance data points, a first distance data point with a corresponding second distance data point;

a selector for selecting, for each of said individual first pixels in the real space image and each of said individual second pixels in the stored virtual space image, whichever of an individual first pixel in the real space image and the corresponding individual second pixel in the stored virtual space image has a corresponding individual distance data point which is nearer to the observer on the basis of an output of the comparator; and

a display assembly for presenting a thus synthesized image including individual pixels whose corresponding individual distance data points are nearer to the observer thus selected by said selector.

6. A display assembly according to claim 5, wherein said real space distance image generator is adapted to generate the real space distance image on the basis of a plurality of images of real space picked up by said pickup device.

7. A display apparatus according to claim 5, wherein said display apparatus is a head mounted display device.

8. A display apparatus according to claim 5, wherein said display assembly includes a display device, for displaying the synthesized image, and a mirror, arranged in a line of sight of the observer, to reflect the synthesized image from the display device to the observer.

9. A display apparatus according to claim 5, wherein said position detector includes a displacement sensor for detecting a movement of a head of the observer, wherein said real space distance image generator can use output data produced by said displacement sensor to generate the real space distance image.

10. A display apparatus for displaying a real space image and a virtual space image, the display apparatus comprising:

a pickup device for picking up a first image of real space to provide a first two dimensional image, the first two dimensional image including a plurality of first pixels, each of said first pixels representing a respective portion of the first image of real space;

a position detector for detecting a position of a viewing point of an observer;

a distance measurement device for generating first distance information on the basis of the position of the viewing point detected by said position detector and locations corresponding to the first image of real space, the first distance information including a plurality of first data points, each individual one of said first data points corresponding to a respective individual one of said first pixels in the first two dimensional image;

a device for outputting a second two dimensional image and second distance information on the basis of output data of said position detector and stored virtual space information, said second two dimensional image including a plurality of second pixels, each of said second pixels representing a respective portion of the stored virtual space information, each individual one of said second pixels corresponding to a respective individual one of said first pixels in the first two dimensional image, the second distance information including a plurality of second data points, each individual one of said second data points corresponding to a respective individual one of said second pixels in the second two dimensional image, wherein each individual one of said second data points corresponds to a respective individual one of said first data points in the first distance information;

a comparator for comparing, for each individual one of said first data points and a corresponding one of said second data points, a first data point of the first distance information with a corresponding second data point of the second distance information;

a selector for selecting, based on an output from said comparator, whichever of a respective one of said first pixels in the first two dimensional image and the corresponding one of said second pixels in the second two dimensional image which corresponds to a location which is nearer to the observer, wherein said first pixel in the first two dimensional image corresponds to a first data point processed by said comparator and said corresponding second pixel in the second two dimensional image corresponds to a second data point processed by said comparator;

a display device for displaying the pixels selected by said selector as a synthesized image;

wherein said pickup device, said distance measurement device and said display device are arranged in a direction which is substantially the same as a viewing direction of the observer.

11. A display apparatus according to claim 10, wherein said pickup device picks up a plurality of images of real space to provide a plurality of first two dimensional images and wherein said distance measurement device generates the first distance information on the basis of said plurality of first two dimensional images.

12. A display apparatus according to claim 10, wherein said pickup device is positioned at a position conjugate to eyes of said observer.

13. A display apparatus according to claim 10, wherein said display device is a head mounted display device.

14. A display apparatus according to claim 10, wherein said display device includes a display unit, for displaying the synthesized image, and a mirror, arranged in a line of sight of the observer to reflect the synthesized image from the display unit to the observer.

15. A display apparatus according to claim 10, wherein said position detector includes a displacement sensor for detecting movement of a head of the observer, and wherein the distance measurement device can use output data produced by the displacement sensor to generate the first distance information.

16. A display apparatus for displaying a real space image and a virtual space image, the display apparatus comprising:

a pickup device for picking up a first image of real space to provide a first two dimensional image including a plurality of first pixels, each of said first pixels representing a respective portion of the first image of real space;

a position detector for detecting a position of a viewing point of an observer;

a distance measurement device for generating first distance information on the basis of the position of the viewing point detected by said position detector and locations corresponding to the first two dimensional image, the first distance information including a plurality of first data points, each individual one of said first data points corresponding to a respective individual one of said first pixels in the first two dimensional image;

a device for outputting a second two dimensional image and second distance information on the basis of output data of said position detector and stored virtual space information, said second two dimensional image including a plurality of second pixels, each of said second pixels representing a respective portion of the stored virtual space information, each one of said second pixels corresponding to a respective individual one of said first pixels in the first two dimensional image, said second distance information including a plurality of second data points, each individual one of said second data points corresponding to a respective individual one of said second pixels in the second two dimensional image, wherein each individual one of said second data points corresponds to a respective individual one of said first data points in the first distance information;

a comparator for comparing, for each individual one of said first data points and a corresponding one of said second data points, a first data point of the first distance information with a corresponding second data point of the second distance information;

a selector for selecting, based on an output from said comparator, whichever of a respective one of said first pixels in the first two dimensional image and the corresponding one of said second pixels in the second two dimensional image represents a location nearer to the observer, wherein said first pixel in the first two dimensional image corresponds to a first data point processed in said comparator and said corresponding second pixel in the second two dimensional image corresponds to a second data point processed in said comparator;

a display device for displaying the pixels selected by said selector as a synthesized image; and

wherein said distance measurement device is configured to measure the first distance information on the basis of a plurality of first two dimensional images picked up at different spatial positions by the pickup device and outputs of said position detector at said spatial positions.

17. A display apparatus according to claim 16, wherein the pickup device is configured to pick up a plurality of first images of real space to provide a plurality of first two dimensional images and the distance measurement device is configured to generate first distance information on the basis of said plurality of first two dimensional images.

18. A display apparatus according to claim 16, wherein said pickup device is positioned at a position conjugate to eyes of an observer.

19. A display apparatus according to claim 16, wherein said display device is a head mounted display device.

20. A display apparatus according to claim 16, wherein said display device includes a display unit, for displaying the synthesized image, and a mirror arranged in a line of sight of the observer to reflect the synthesized image from the display unit to the observer.

21. A display apparatus according to claim 16, wherein said position detector includes a displacement sensor for detecting movement of a head of the observer, and wherein said distance measurement device can use output produced by said displacement sensor to generate the first distance information.

24. A display apparatus comprising:

a first output device for outputting first image data and first distance data based on an image of real space, said first distance data representing a distance from an observer to a location in real space corresponding to at least a portion of the first image data;

a second output device for outputting second image data and second distance data based on virtual space information, said second distance data representing a distance from an observer to an apparent location in virtual space corresponding to at least a portion of the second image data;

a comparing and selecting device for comparing at least one datum of said first distance data with a corresponding at least one datum of said second distance data to determine a smaller one thereof and for selecting whichever of said at least a portion of said first image data and the corresponding respective portion of said second image data which corresponds to said smaller one; and

a display device for displaying thus selected image data supplied by said comparing and selecting device.

25. A display apparatus according to claim 24, wherein said first image data comprises a plurality of first pixels, and said second image data comprises a plurality of second pixels, each individual one of said second pixels corresponding to an individual one of said first pixels;

and

wherein, for each individual one of said second pixels, said comparing and selecting device compares first distance data to corresponding second distance data.

26. A display apparatus according to claim 24, wherein said first image data outputted by said first output device corresponds to an image of real space picked up at a position conjugate to eyes of an observer.