Stereoscopic user interface method and apparatus

• US 6,243,054 B1
• Filed: 03/31/2000
• Issued: 06/05/2001
• Est. Priority Date: 07/01/1998
• Status: Expired due to Term

• Current Assignee: Vuzix Corporation
• Original Assignee: Michael Deluca
• Inventors: DeLuca, Michael
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): Laneau, Ronald
• Application Number:
  US09/539,536
• Time in Patent Office: 431 Days
• Field of Search: 345/7, 345/8, 345/419, 348/42, 348/47, 348/49
• US Class Current: 345/7
• CPC Class Codes: G02B 30/22 of the stereoscopic type
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/334 using spectral multiplexing
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/371 for tracking viewers with different interocular distances; for tracking rotational head movements around the vertical axis
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

A computer system stereoscopically projects a three dimensional object having an interface image in a space observable by a user. The user controls the movement of a physical object within the space while observing both the three dimensionally projected object and the physical object. The computer system monitors the position of the user to determine the position of the interface image within the space and further monitors the movement of the physical object to determine its position. A control signal is generated in response to the position of the physical object intersecting the position of the interface image. For example, a word processing program is indicated by an interface image such as an icon including the letter “W” three dimensionally projected within the space. The word processing program is activated when the user'"'"'s finger moves within the space to touch the projected icon. The interface allows the user to observe the projected icon, physical finger and their intersection within the space. The physical object may also be extended with a stereoscopic extension image generated by the computer system in response to determining the position and orientation of the physical object.

Claims

1. A method of generating a control signal comprising:

projecting a stereoscopic interface image in a space observable by a user;

enabling a physical object within in the space to be observable by the user in addition to the stereoscopic interface image;

determining an intersection of the physical object with the stereoscopic interface image; and

generating the control signal in response to said step of determining.

2. The method according to claim 1 wherein

said step of projecting further projects an observable image including the stereoscopic interface image and the method comprises the step of

modifying the observable image in response to the control signal.

3. The method according to claim 1 wherein the physical object includes a stereoscopic extension image and the method further comprises the steps of:

determining a position and orientation of the physical object; and

displaying the stereoscopic extension image as an extension of the physical object in response thereto, wherein

said step of determining the intersection further comprises the step of determining an intersection of the stereoscopic extension image with the stereoscopic interface image.

4. The method according to claim 3 further comprising the step of

determining a position of the user, wherein

said step of determining the intersection determines the intersection of the physical object and the stereoscopic interface image relative to the position of the user.

5. The method according to claim 4 further comprising the step of

visually monitoring the user and the physical object, and wherein

said step of determining the position of the user is determined in response to said step of visually monitoring, and

said step of determining the intersection of the physical object with the stereoscopic interface image is determined in response to said step of visually monitoring.

6. The method according to claim 5 wherein

said step of visually monitoring includes the step of receiving a plurality of images from a plurality of cameras each having a predetermined position, and

said step of determining the position of the user further includes the steps of:

recognizing the user in each of the plurality of images; and

determining the position of the user by triangulation.

7. The method according to claim 1 further comprising the step of

visually monitoring the physical object, and wherein

said step of determining the intersection of the physical object with the stereoscopic interface image is determined in response to said step of visually monitoring.

8. The method according to claim 7 wherein

said step of visually monitoring includes the step of receiving a plurality of images from a plurality of cameras each having a predetermined position, and

said step of determining the intersection of the physical object further includes the steps of:

recognizing the physical object in each of the plurality of images; and

determining the position of the physical object by triangulation.

9. A stereoscopic user interface means for generating a control signal comprising:

a display means for projecting a stereoscopic interface image in a space observable by a user wherein said display means enables a physical object within in the space to be observable by the user in addition to the stereoscopic interface image; and

a monitoring means for determining an intersection of the physical object with the stereoscopic interface image and for generating the control signal in response thereto.

10. The stereoscopic user interface means according to claim 9 wherein

said display means further projects an observable image including the stereoscopic interface image, and the stereoscopic user interface means comprises

an image generation means for modifying the observable image in response to the control signal.

11. The stereoscopic user interface means according to claim 10 wherein the physical object includes a stereoscopic extension image and the stereoscopic user interface means further comprises

a coordinate determining means for determining a position and orientation of the physical object, and wherein

said display means displays the stereoscopic extension image as an extension of the physical object in response to said coordinate determining means, and

said monitoring means determines an intersection of the stereoscopic extension image with the stereoscopic interface image to generate the control signal.

12. The stereoscopic user interface means according to claim 9 further comprising

a coordinate determining means for determining a position of the user, wherein

said monitoring means determines the intersection of the physical object and the stereoscopic interface image relative to the position of the user.

13. The stereoscopic user interface means according to claim 9 wherein said monitoring means includes a video camera means for visually monitoring the physical object and wherein the intersection of the physical object with the stereoscopic interface image is determined in response to visually monitoring the physical object.

14. A method of displaying a stereoscopic extension image as an extension of a physical object observable by a user comprising the steps of:

determining a position and orientation of the physical object; and

displaying the stereoscopic extension image also observable by the user as the extension of the physical object in response thereto.

15. The method according to claim 14 wherein said step of displaying further comprises the step of

determining a position of the user, and

said step of displaying includes projecting the stereoscopic extension image relative to the determined position of the user.

16. The method according to claim 15 wherein said step of determining the orientation of the physical object further includes the steps of:

visually recognizing a first and a second point on the physical object;

determining a position of the first point and the position of the second point; and

determining coordinates of a line defined by the positions of first and second points;

wherein

said step of displaying projects the stereoscopic extension image substantially along the line as observed by the user.

17. The method according to claim 16 wherein the physical object is a handle having at least the first and second points and the stereoscopic extension image is a projection of an end effect on the handle.