Stereoscopic user interface method and apparatus

• US 6,064,354 A
• Filed: 07/01/1998
• Issued: 05/16/2000
• Est. Priority Date: 07/01/1998
• Status: Expired due to Fees

• Current Assignee: Michael J. Deluca
• Original Assignee: Michael J. Deluca
• Inventors: DeLuca, Michael Joseph
• Primary Examiner(s): Hjerpe, Richard A.
• Assistant Examiner(s): Laneau, Ronald
• Application Number:
  US09/108,814
• Time in Patent Office: 685 Days
• Field of Search: 345/7, 345/8, 345/9, 345/419, 348/42, 348/47, 348/51
• US Class Current: 345/7
• CPC Class Codes: G02B 30/22 of the stereoscopic type
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/334 using spectral multiplexing
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/371 for tracking viewers with different interocular distances; for tracking rotational head movements around the vertical axis
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

A computer system stereoscopically projects a three dimensional object having an interface image in a space observable by a user. The user controls the movement of a physical object within the space while observing both the three dimensionally projected object and the physical object. The computer system monitors the position of the user to determine the position of the interface image within the space and further monitors the movement of the physical object to determine its position. A control signal is generated in response to the position of the physical object intersecting the position of the interface image. For example, a word processing program is indicated by an interface image such as an icon including the letter "W" three dimensionally projected within the space. The word processing program is activated when the user'"'"'s finger moves within the space to touch the projected icon. The interface allows the user to observe the projected icon, physical finger and their intersection within the space. The physical object may also be extended with a stereoscopic extension image generated by the computer system in response to determining the position and orientation of the physical object.

Claims

1. A method of displaying a stereoscopic extension image as an extension of a physical object observable by a user comprising the steps of:

determining a position and orientation of the physical object; and

displaying the stereoscopic extension image also observable by the user as the extension of the physical object in response thereto, wherein

said step of displaying further comprises the step of determining a position of the user, and includes projecting the stereoscopic extension image relative to the determined position of the user and

said step of determining the orientation of the physical object further comprises the steps of;

visually recognizing a first and a second point on the physical object;

determining a position of the first point and the position of the second point; and

determining coordinates of a line defined by the positions of first and second points; and

further wherein

said step of displaying projects the stereoscopic extension image substantially along the line as observed by the user.

2. The method according to claim 1 wherein the physical object is a handle having at least the first and second points and the stereoscopic extension image is a projection of an end effect on the handle.

3. The method according to claim 1 wherein the stereoscopic extension image is a projection of one of a plurality of selectable end effects.

4. The method according to claim 1 further comprising the steps of:

displaying a stereoscopic interface image observable by the user;

determining an intersection of the stereoscopic extension image with the stereoscopic interface image; and

generating the control signal in response thereto.

5. A method of generating a control signal comprising:

projecting a stereoscopic interface image in a space observable by a user;

enabling a physical object within in the space to be observable by the user in addition to the stereoscopic interface image;

determining an intersection of the physical object with the stereoscopic interface image; and

generating the control signal in response to said step of determining wherein the physical object includes a stereoscopic extension image and the method further comprises the steps of;

determining a position and orientation of the physical object; and

displaying the stereoscopic extension image as an extension of the physical object in response thereto, wherein

said step of determining the intersection further comprises the step of determining an intersection of the stereoscopic extension image with the stereoscopic interface image and further wherein

said step of projecting the stereoscopic interface image is projected by a display having a display surface and the space includes a front space between the display surface and the user and a behind space behind the display surface and wherein the stereoscopic interface image is projected in either the front space or the behind space,

said step of displaying the stereoscopic extension image displays the stereoscopic extension image in either the front space or the behind space, and

said step of determining the intersection further comprises the step of determining the intersection of the stereoscopic extension image with the stereoscopic interface image in either the front space or the behind space.

6. The method according to claim 5 wherein

said step of projecting further projects an observable image including the stereoscopic interface image and the method comprises the step of

modifying the observable image in response to the control signal.

7. The method according to claim 6 further comprising the step of

determining a position of the user, wherein

said step of determining the intersection determines the intersection of the physical object and the stereoscopic interface image relative to the position of the user.

8. The method according to claim 7 further comprising the step of

visually monitoring the user and the physical object, and wherein

said step of determining the position of the user is determined in response to said step of visually monitoring, and

said step of determining the intersection of the physical object with the stereoscopic interface image is determined in response to said step of visually monitoring.

9. The method according to claim 5 further comprising the step of

visually monitoring the physical object, and wherein

said step of determining the intersection of the physical object with the stereoscopic interface image is determined in response to said step of visually monitoring.

10. A method of generating a control signal comprising:

projecting a stereoscopic interface image in a space observable by a user;

enabling a physical object within in the space to be observable by the user in addition to the stereoscopic interface image;

determining an intersection of the physical object with the stereoscopic interface image; and

generating the control signal in response to said step of determining wherein the physical object includes a stereoscopic extension image and the method further comprises the steps of;

determining a position and orientation of the physical object; and

displaying the stereoscopic extension image as an extension of the physical object in response thereto, wherein

said step of determining the intersection further comprises the step of determining an intersection of the stereoscopic extension image with the stereoscopic interface image and further wherein

said step of projecting the stereoscopic interface image is performed by a display having a display surface viewable by the user and the space includes a behind space behind the display surface and wherein the stereoscopic interface image is projected in the behind space,

said step of determining the position and orientation of the physical object includes determining a position and orientation of the physical object to be between the user and the display surface,

said step of displaying the stereoscopic extension image displays the stereoscopic interface image in the behind space, and

said step of determining the intersection further comprises the step of determining an intersection of the stereoscopic extension image with the stereoscopic interface image in the behind space.

11. The method according claim 10 wherein the display surface substantially inhibits the user'"'"'s observation of physical objects in the behind space.

12. The method according to claim 10 wherein

said step of projecting further projects an observable image including the stereoscopic interface image and the method comprises the step of

modifying the observable image in response to the control signal.

13. The method according to claim 10 further comprising the step of

determining a position of the user, wherein

said step of determining the intersection determines the intersection of the physical object and the stereoscopic interface image relative to the position of the user.

14. The method according to claim 13 further comprising the step of

visually monitoring the user and the physical object, and wherein

said step of determining the position of the user is determined in response to said step of visually monitoring, and

said step of determining the intersection of the physical object with the stereoscopic interface image is determined in response to said step of visually monitoring.

15. The method according to claim 10 further comprising the step of

visually monitoring the physical object, and wherein

said step of determining the intersection of the physical object with the stereoscopic interface image is determined in response to said step of visually monitoring.

16. A stereoscopic user interface means for generating a control signal comprising:

a display means for projecting a stereoscopic interface image in a space observable by a user wherein said display means enables a physical object within in the space to be observable by the user in addition to the stereoscopic interface image; and

a monitoring means for determining an intersection of the physical object with the stereoscopic interface image and for generating the control signal in response thereto, wherein the physical object includes a stereoscopic extension image and the stereoscopic user interface means further comprises

a coordinate determining means for determining a position and orientation of the physical object, and wherein

said display means displays the stereoscopic extension image as an extension of the physical object in response to said coordinate determining means, and

said monitoring means determines an intersection of the stereoscopic extension image with the stereoscopic interface image to generate the control signal and further wherein

said display means has a display surface and the space includes a front space between the display surface and the user and a behind space behind the display surface and wherein the stereoscopic interface image is projected in either the front space or the behind space, the stereoscopic interface image is displayed in either the front space or the behind space, and

said monitoring means determines intersection of the stereoscopic extension image with the stereoscopic interface image in either the front space or the behind space.

17. The stereoscopic user interface means according to claim 16 further comprising

a coordinate determining means for determining a position of the user, wherein

said monitoring means determines the intersection of the physical object and the stereoscopic interface image relative to the position of the user.

18. A stereoscopic user interface means for generating a control signal comprising:

a display means for projecting a stereoscopic interface image in a space observable by a user wherein said display means enables a physical object within in the space to be observable by the user in addition to the stereoscopic interface image; and

a monitoring means for determining an intersection of the physical object with the stereoscopic interface image and for generating the control signal in response thereto, wherein the physical object includes a stereoscopic extension image and the stereoscopic user interface means further comprises

a coordinate determining means for determining a position and orientation of the physical object, and wherein

said display means displays the stereoscopic extension image as an extension of the physical object in response to said coordinate determining means, and

said monitoring means determines an intersection of the stereoscopic extension image with the stereoscopic interface image to generate the control signal and further wherein

said display means has a display surface viewable by the user and the space includes a behind space behind the display surface and wherein the stereoscopic interface image and the stereoscopic extension image are projected in the behind space,

said coordinate determining means determining the position and orientation of the physical object to be between the user and the display surface, and

said monitoring means determines the intersection of the stereoscopic extension image with the stereoscopic interface image in the behind space.

19. The stereoscopic user interface means according to claim 18 wherein

said display means further projects an observable image including the stereoscopic interface image, and the stereoscopic user interface means comprises

an image generation means for modifying the observable image in response to the control signal.

20. The stereoscopic user interface means according to claim 18 further comprising

a coordinate determining means for determining a position of the user, wherein

said monitoring means determines the intersection of the physical object and the stereoscopic interface image relative to the position of the user.