Apparatus and method for providing images of real and virtual objects in a head mounted display

• US 6,411,266 B1
• Filed: 11/17/1995
• Issued: 06/25/2002
• Est. Priority Date: 08/23/1993
• Status: Expired due to Term

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Francis J. Maguire
• Inventors: Maguire, Francis J. Jr.
• Primary Examiner(s): CHOW, DOON Y
• Application Number:
  US08/560,108
• Time in Patent Office: 2,412 Days
• Field of Search: 348/373, 348/376, 348/53, 348/158, 348/838, 348/383, 348/578, 348/584, 348/586, 345/7, 345/8, 345/9, 340/980
• US Class Current: 345/8
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Actual images of actual objects in an object space, taken from a moving perspective of a viewer in the object space and in registration with the actual objects, are provided with retrieved images imitative of stationary or moving “virtual” objects in the object space for providing both actual and retrieved images to the viewer as if the retrieved images were also actual and integrated into the object space. The viewer may move about in the actual object space and physically navigate among the actual objects therein, by means of their images. The viewer may touch and move the objects and may perceive the integrated virtual objects as if real, albeit without substance.

Claims

1. Apparatus for providing a virtual reality environment with images of virtual objects provided in response to sensed actions of a viewer wherein the images of virtual objects lag after the sensed actions of the viewer, comprising:

a head mounted camera (10) for mounting on a head (12) of the viewer acting as a cameraman for gathering images of real objects from a moving perspective of the viewer, said camera responsive to reflected light from the real objects in an object space, for providing a real object image signal (14) indicative of the real objects;

an image store (22), responsive to a selection signal (20), for providing a prestored virtual object image signal (24) indicative of moving virtual objects;

a computer and image processor (16) having positions and geometric features of said real objects prestored in a computer spatial model of the object space, responsive to a plurality of monitor signals for providing the selection signal (20) and responsive to the real object image signal (14) and the virtual object image signal (24), for providing an integrated image signal (26) indicative of both said real objects and said virtual objects;

one or more monitors (18), responsive to one or more corresponding actions of the viewer acting as a cameraman in the object space, for providing the plurality of monitor signals; and

a head mounted display (28) for mounting on the head of the viewer, responsive to the integrated image signal (26), for providing integrated images of the object space with real and virtual objects for viewing by the viewer with the images of the real objects in registration with the real objects from the moving point of view of the viewer and for viewing by the viewer with the images of the virtual objects integrated therewith, wherein the moving virtual objects move with respect to the real objects positioned in the spatial model for interacting with the viewer and whereby the lag of the images of virtual objects after the sensed actions of the viewer is reduced.

2. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a horizontal position monitor (30), for providing a horizontal position signal (32).

3. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a vertical position monitor (34), for providing a vertical position signal (36).

4. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a head attitude monitor (38), for providing a head attitude signal (40).

5. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a viewer body part monitor (46), for providing a body part signal (48).

6. The apparatus of claim 1, wherein said one or more monitors include an eye attitude monitor for mounting on the head of the viewer for monitoring an eye of the viewer for providing an eye attitude signal (44) indicative of a direction of a visual axis of the eye of the viewer, for providing the integrated images having nonuniform resolution with a highly detailed portion of each image coinciding with the visual axis of the eye of the viewer.

7. The apparatus of claim 1, wherein said integrated images have a seamless integration of real and virtual objects by occluding real object images taken up by virtual object images retrieved from storage.

8. The apparatus of claim 1, wherein said one or more monitors include an eye attitude monitor for mounting on the head of the viewer for monitoring an eye of the viewer for providing an eye attitude signal (44) indicative of a direction of a visual axis of the eye of the viewer for providing the integrated images having nonuniform resolution with a highly detailed portion of each image coinciding with the visual axis of the eye of the viewer.

9. The apparatus of claim 1, wherein said one or more monitors include an eye attitude monitor for mounting on the head of the viewer for monitoring an eye of the viewer for providing an eye attitude signal (44) indicative of a direction of a visual axis of the eye of the viewer for providing the integrated images having nonuniform resolution with a highly detailed portion of each image coinciding with the visual axis of the eye of the viewer.

10. The apparatus of claim 1, wherein said prestored virtual object image signal is selected by said selection signal from among a plurality of prestored virtual object image signals indicative of a corresponding plurality of alternative preprogrammed scenarios.

11. The apparatus of claim 1, wherein said head mounted camera is a pair of head mounted cameras for mounting on said head of said viewer for gathering said images as stereo images.

12. Apparatus, comprising:

a head mounted camera for mounting on a viewer'"'"'s head, for acquiring images of actual objects in an object space;

display means for mounting on the viewer'"'"'s head for providing the images of actual objects in the object space, taken from a moving perspective of the viewer in the object space, wherein the display means is for providing the images of the actual objects for viewing by the viewer, from the viewer'"'"'s point of view, in registration with the actual objects and in substitution therefor; and

an image processor, having positions and geometric features of said actual objects prestored in a spatial model of the object space, for combining the images of actual objects with retrieved images selected from among a plurality of prestored virtual objects in the object space according to monitored head movements of the viewer, for providing the combined images to the viewer for enabling the viewer to perceive the object space with both actual and virtual objects by means of the combined images and wherein the virtual objects move with respect to the actual objects for interacting with the viewer.

13. The apparatus of claim 12, wherein said head mounted camera is a pair of head mounted cameras for mounting on said viewer'"'"'s head for acquiring stereo images of said actual objects.

14. The apparatus of claim 13, wherein said one or more monitors include an eye attitude monitor for mounting on the head of the viewer for monitoring an eye of the viewer for providing an eye attitude signal (44) indicative of a direction of a visual axis of the eye of the viewer, for providing the integrated images having nonuniform resolution with a highly detailed portion of each image coinciding with the visual axis of the eye of the viewer.

15. The apparatus of claim 12, wherein said retrieved images are selected from among said plurality of prestored virtual objects indicative of a corresponding plurality of alternative preprogrammed scenarios.

16. A method, comprising the steps of:

monitoring position and attitude of a head of a viewer as the viewer moves about in an object space;

acquiring images, taken from a moving point of view of the head of the viewer in the object space, of actual objects in the object space, said actual objects having positions and geometric features prestored in a computer model of the object space;

providing the images of actual objects in the object space, in registration, from the moving point of view of the viewer, with the actual objects for viewing by the viewer as substitutes for viewing the real objects;

retrieving from among a plurality of prestored images of virtual objects, selected prestored images of virtual objects imitative of actual objects moving in the object space, said selected prestored images selected according to the monitored position and attitude of the head of the viewer; and

combining the retrieved images of virtual objects imitative of actual objects moving in the object space with respect to the positions and geometric features of the actual objects as substituted by the images of actual objects in the object space, for providing combined images for viewing by the viewer for enabling the viewer to perceive the object space by means of the combined images as if the retrieved images were also actual and integrated within the object space.

17. The method of claim 16, wherein said step of acquiring images comprises the step of acquiring stereo pairs of images from said moving point of view.

18. The method of claim 17, further comprising the step of:

monitoring direction of a visual axis of an eye of the viewer, for providing the combined images having nonuniform resolution with a highly detailed portion of each image coinciding with the visual axis of the eye of the viewer.

19. The method of claim 16, further comprising the step of:

monitoring direction of a visual axis of an eye of the viewer, for providing the combined images having nonuniform resolution with a highly detailed portion of each image coinciding with the visual axis of the eye of the viewer.

20. The method of claim 16, wherein said prestored images of virtual objects are selected from among a plurality of alternative scenarios.