Three-dimensional imaging and display system

• US 8,743,345 B2
• Filed: 08/17/2011
• Issued: 06/03/2014
• Est. Priority Date: 10/21/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Krah, Christoph H.
• Primary Examiner(s): Alsomiri, Isam
• Assistant Examiner(s): Diacou, Ari M
• Application Number:
  US13/212,125
• Publication Number:
  US 20110298704A1
• Time in Patent Office: 1,021 Days
• Field of Search: 356/3.02, 356/5.1
• US Class Current: 356/3.02
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01C 3/085 with electronic parallax measurement
• CPC Class Codes: G01C 3/12 with monocular observation at a single point, e.g. coincidence type G01C3/20 takes precedence
• CPC Class Codes: G01S 17/14 wherein a voltage or current pulse is initiated and terminated in accordance with the pulse transmission and echo reception respectively, e.g. using counters
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/48 Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/4861 Circuits for detection, sampling, integration or read-out
• CPC Class Codes: G01S 7/4865 Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak peak detection in noise, signal conditioning G01S7/487
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A three-dimensional imaging and display system is provided in which user input is optically detected in an imaging volume by measuring the path length of an amplitude modulated scanning beam as a function of the phase shift thereof. Visual image user feedback concerning the detected user input is presented.

Claims

1. A method for detecting a user input in a three-dimensional volume, the method comprising:

optically detecting a user input in an imaging volume;

compensating for parallax in the optically detected user input as a function of a scan coordinate defined by a mirror system position and the measured distance from the mirror system to a target object; and

presenting visual image user feedback concerning the optically detected user input.

2. The method of claim 1 comprising:

calibrating for parallax; and

providing a lookup table that has information relating to one or more prior parallax compensation.

3. The method of claim 1 wherein compensating for parallax comprises:

installing a bottom reference plane;

adjusting the deflection angle (∂

,β

) of a scanning beam to target a selected target object in the bottom reference plane;

determining x0, y0, z0, and L0, where;

the coordinate (x0,y0,z0) is the location at which the scanning beam is reflected from the target object in the bottom reference plane; and

LO is the distance the scanning beam travels to the target object in the bottom reference plane;

saving values for x0, y0, z0, and L0 in a lookup table for later retrieval;

repeating the adjusting, determining, and saving for other deflection angles with respect to other target objects in the bottom reference plane;

uninstalling the bottom reference plane;

installing a top reference plane;

adjusting the deflection angle (∂

,β

) of a scanning beam to target a selected target object in the top reference plane;

determining x1, y1, z1, and L1, where;

the coordinate (x1,y1,z1) is the location at which the scanning beam is reflected from the target object in the top reference plane; and

L1 is the distance the scanning beam travels to the target object in the top reference plane;

saving values for x1, y1, z1, and L1 in a lookup table for later retrieval;

with respect to other target objects in the top reference plane, repeating the adjusting, determining, and saving for other deflection angles; and

uninstalling the top reference plane.

4. The method of claim 1 comprising displaying and manipulating virtual objects located in a three-dimensional imaging volume in response to virtual interactions with the virtual objects, the virtual objects being derived from the detected user input, wherein the objects are manipulated to move, scale, or rotate, wherein the objects are displayed as three-dimensional images.

5. The method of claim 1 comprising displaying externally provided images.

6. The method of claim 1 wherein optically detecting comprises optically detecting with invisible light.

7. The method of claim 1 comprising tracking one or more user'"'"'s heads.

8. The method of claim 1 comprising:

imaging a user'"'"'s head in the imaging volume; and

mapping the image of the user'"'"'s head to at least one interface related function.

9. The method of claim 1 comprising:

imaging one or more user hand gestures in the imaging volume; and

mapping the one or more hand gestures to one or more interface related functions.

10. The method of claim 1 comprising:

detecting whether a user is sitting in front of a display;

identifying which user is present; and

saving power when the user is not present.

11. The method of claim 1 comprising measuring a size of one or more objects.

12. The method of claim 1 comprising substituting or replacing images.

13. The method of claim 1 comprising painting and following a moveable object as it moves.

14. The method of claim 1 comprising generating an interactive virtual keyboard.

15. The method of claim 1 comprising:

detecting one or more object types from the optically detected user input; and

associating one or more predetermined functions to the one or more object types.

16. The method of claim 1 comprising customizing a three-dimensional display environment.

17. The method of claim 1 comprising:

detecting elements of the user input with watershed segmentation;

tracking individual and relative movements and positions of the elements of the user input; and

mapping the tracked movements and positions to predetermined functions.

18. The method of claim 1 wherein presenting comprises displaying the visual image user feedback as a three-dimensional image.

19. The method of claim 1 wherein one or more mirrors of the mirror system operate on a mechanical resonant frequency.

20. The method of claim 1 wherein one or more visual images are presented on an auxiliary display.