Apparatus and method for providing images of real and virtual objects in a head mounted display

• US 6,690,338 B1
• Filed: 06/25/2002
• Issued: 02/10/2004
• Est. Priority Date: 08/23/1993
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Francis J. Maguire
• Inventors: Maguire, Francis J. Jr.
• Primary Examiner(s): Chow, Dennis-Doon
• Application Number:
  US10/183,114
• Time in Patent Office: 595 Days
• Field of Search: 345/8, 345/9, 345/7, 348/373, 348/376, 348/53, 348/158, 348/838, 348/383, 348/578, 348/584, 348/586, 340/980
• US Class Current: 345/8
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Actual images of actual objects in an object space, taken from a moving perspective of a viewer in the object space and in registration with the actual objects, are provided with retrieved images imitative of stationary or moving “virtual” objects in the object space for providing both actual and retrieved images to the viewer as if the retrieved images were also actual and integrated into the object space. The viewer may move about in the actual object space and physically navigate among the actual objects therein, by means of their images. The viewer may touch and move the objects and may perceive the integrated virtual objects as if real, albeit without substance.

Claims

1. Apparatus, comprising:

a head mounted camera apparatus (10) for mounting on a head of a viewer (12), responsive to reflected light in an object space, for providing a real object image signal (14);

an image store (22), responsive to a selection signal (20), for providing a virtual object image signal (24);

a computer and image processor (16), responsive to the real object image signal (14), to the virtual object image signal (24), and to a plurality of viewer monitor signals, for providing an integrated image signal having both real and virtual objects;

a plurality of viewer monitors (18), responsive to viewer actions in the object space, for providing the plurality of viewer monitor signals; and

a head mounted display (28), responsive to the integrated image signal (26), for providing images of the object space with both real and virtual objects.

2. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a horizontal position monitor (30), for providing a horizontal position signal (32).

3. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a vertical position monitor (34), for providing a vertical position signal (36).

4. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a head attitude monitor (38), for providing a head attitude signal (40).

5. The apparatus of claim 1, wherein the plurality of viewer monitors (18) includes a viewer body part monitor (46), for providing a body part signal (48).

6. Apparatus, comprising:

a head mounted camera for mounting on a viewer'"'"'s head, for acquiring images of actual objects in an object space;

display means for providing the images of actual objects in the object space, taken from a moving perspective of the viewer in the object space, wherein the display is for providing the images of the actual objects, from the viewer'"'"'s point of view, in registration with the actual objects; and

an image processor for combining the images of actual objects with retrieved images imitative of objects in the object space, depending on acts of the viewer, for providing the combined images to the viewer for enabling the viewer to perceive the object space by means of the combined images as if the retrieved images were also actual.

7. A method, comprising the steps of:

acquiring images, taken from a moving point of view of a viewer in an object space, of actual objects in the object space;

providing the images of actual objects in the object space, in registration, from the point of view of the viewer, with the actual objects;

retrieving images imitative of objects in the object space, according to acts of the viewer; and

combining the retrieved images imitative of objects in the object space with the images of actual objects in the object space, for providing combined images to the viewer for enabling the viewer to perceive the object space by means of the combined images as if the retrieved images were also actual and integrated within the object space.