Three-dimensional imaging and display system

• US 8,018,579 B1
• Filed: 10/21/2005
• Issued: 09/13/2011
• Est. Priority Date: 10/21/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Krah, Christoph H.
• Primary Examiner(s): ALSOMIRI, ISAM A
• Application Number:
  US11/255,348
• Time in Patent Office: 2,153 Days
• Field of Search: 356/4.01, 356/5.01, 356/28, 382/103, 382106-107
• US Class Current: 356/4.01
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01C 3/085 with electronic parallax measurement
• CPC Class Codes: G01C 3/12 with monocular observation at a single point, e.g. coincidence type G01C3/20 takes precedence
• CPC Class Codes: G01S 17/14 wherein a voltage or current pulse is initiated and terminated in accordance with the pulse transmission and echo reception respectively, e.g. using counters
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/48 Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/4861 Circuits for detection, sampling, integration or read-out
• CPC Class Codes: G01S 7/4865 Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak peak detection in noise, signal conditioning G01S7/487
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A three-dimensional imaging and display system is provided in which user input is optically detected in an imaging volume by measuring the path length of an amplitude modulated scanning beam as a function of the phase shift thereof. Visual image user feedback concerning the detected user input is presented.

Claims

1. A three-dimensional imaging and display apparatus, comprising:

a processor;

a first circuit coupled to the processor for optically detecting user input in an imaging volume;

a second circuit coupled to the processor for compensating for parallax in the optically detected user input, wherein the second circuit compensates for parallax as a function of a scan coordinate defined by a mirror system position and at least the measured distance from the minor system to a target object; and

a third circuit coupled to the processor for presenting visual image user feedback concerning the optically detected user input.

2. The apparatus as claimed in claim 1:

wherein the second circuit is adapted to compensate for parallax by calibrating to compensate for parallax, and

wherein the apparatus further comprises a lookup table having prior parallax compensation calibration information therein.

3. The apparatus as claimed in claim 1 wherein c the second circuit compensates for parallax by:

installing a bottom reference plane;

adjusting the deflection angle (∂

,β

) of a scanning beam to target a selected target object in the bottom reference plane;

determining x0, y0, z0, and L0, where;

the coordinate (x0,y0,z0) is the location at which the scanning beam is reflected from the target object in the bottom reference plane; and

L0 is at least the distance the scanning beam travels to the target object in the bottom reference plane;

saving values for x0, y0, z0, and L0 in a lookup table for later retrieval;

with respect to other target objects in the bottom reference plane, repeating the adjusting, determining, and saving for other deflection angles;

uninstalling the bottom reference plane;

installing a top reference plane;

adjusting the deflection angle (∂

,β

) of a scanning beam to target a selected target object in the top reference plane;

determining x1, y1, z1, and L1, where;

the coordinate (x1,y1,z1) is the location at which the scanning beam is reflected from the target object in the top reference plane; and

L1 is at least the distance the scanning beam travels to the target object in the top reference plane;

saving values for x1, y1, z1, and L1 in a lookup table for later retrieval;

with respect to other target objects in the top reference plane, repeating the adjusting, determining, and saving for other deflection angles; and

uninstalling the top reference plane.

4. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for displaying and manipulating virtual objects located in a three-dimensional imaging volume in response to virtual interactions with the virtual objects, the virtual interactions being derived from the detected user input, and the manipulating being selected at least from moving objects, scaling objects, rotations in a volume, and a combination thereof.

5. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for displaying externally provided images.

6. The apparatus as claimed in claim 1 wherein the first circuit is adapted to perform optical detection with invisible light.

7. The apparatus as claimed in claim 1 wherein the first circuit is further adapted to perform surveillance.

8. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for performing head tracking.

9. The apparatus as claimed in claim 1 further comprising:

a fourth circuit coupled the processor for imaging the user'"'"'s head in the imaging volume; and

a fifth circuit coupled to the processor for mapping the image of the user'"'"'s head to at least one interface related function.

10. The apparatus as claimed in claim 1 further comprising:

a fourth circuit coupled to the processor for imaging hand gestures in the imaging volume; and

a fifth circuit coupled to the processor for mapping the hand gestures to at least one interface related function.

11. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for detecting user presence, the detecting being selected from detecting whether a user is sitting in front of a display, identifying which user is present, saving power when the user is away, and a combination thereof.

12. The apparatus as claimed in claim 1 wherein the first circuit is further adapted to perform object measuring.

13. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for performing image substitution or replacement.

14. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for painting a moveable object and following it as it moves.

15. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for generating an interactive virtual keyboard.

16. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for assigning functions by detecting the kinds of objects in the optically detected user input and associating a predetermined function or set of functions with at least some of those objects.

17. The apparatus as claimed in claim 1 further comprising a fourth circuit coupled to the processor for customizing the environment in which the three-dimensional imaging and display apparatus is deployed.

18. The apparatus as claimed in claim 1 further comprising:

a fourth circuit coupled to the processor for utilizing watershed segmentation to detect elements of the user input;

a fifth circuit coupled to the processor for tracking individual and relative movements and positions of the elements of the user input; and

a sixth circuit coupled to the processor for mapping the tracked movements and positions to predetermined functions.

19. The apparatus as claimed in claim 1 further comprising:

a photodiode amplifier;

a clip amp coupled to the photodiode amplifier;

a high pass filter coupled to the clip amp;

a mixer coupled to the high pass filter;

a low pass filter coupled to the mixer;

an analog-to-digital converter coupled to the low pass filter;

the processor coupled to the analog-to-digital converter;

a minor driver coupled to the processor; and

a video subsystem coupled to the processor, wherein the processor comprises a digital signal processor.

20. A three-dimensional imaging and display system, comprising:

structure and circuitry for optically detecting user input in an imaging volume;

structure and circuitry for compensating for parallax in the optically detected user input, wherein the structure and circuitry for compensating further comprises structure and circuitry for compensating for parallax as a function of a scan coordinate defined by a minor system position and at least the measured distance from the minor system to a target object; and

circuitry for presenting visual image user feedback concerning the optically detected user input.

21. The system as claimed in claim 20 further comprising a lookup table having prior parallax compensation calibration information therein.

22. The system as claimed in claim 20 wherein the structure and circuitry for parallax by:

installing a bottom reference plane;

adjusting the deflection angle (∂

,β

) of a scanning beam to target a selected target object in the bottom reference plane;

determining x0, y0, z0, and L0, where;

the coordinate (x0,y0,z0) is the location at which the scanning beam is reflected from the target object in the bottom reference plane; and

L0 is at least the distance the scanning beam travels to the target object in the bottom reference plane;

saving values for x0, y0, z0, and L0 in a lookup table for later retrieval;

with respect to other target objects in the bottom reference plane, repeating the adjusting, determining, and saving for other deflection angles;

uninstalling the bottom reference plane;

installing a top reference plane;

adjusting the deflection angle (∂

,β

) of a scanning beam to target a selected target object in the top reference plane;

determining x1, y1, z1, and L1, where;

the coordinate (x1,y1,z1) is the location at which the scanning beam is reflected from the target object in the top reference plane; and

L1 is at least the distance the scanning beam travels to the target object in the top reference plane;

saving values for x1, y1, z1, and L1 in a lookup table for later retrieval;

with respect to other target objects in the top reference plane, repeating the adjusting, determining, and saving for other deflection angles; and

uninstalling the top reference plane.

23. The system as claimed in claim 20 further comprising structure and circuitry for:

deriving virtual interactions from the detected user input;

displaying virtual objects located in a three-dimensional imaging volume;

manipulating the virtual objects in response to the virtual interactions; and

selecting the manipulations of the virtual objects at least from moving objects, scaling objects, rotations in a volume, and a combination thereof.

24. The system as claimed in claim 20 further comprising circuitry for displaying externally provided images.

25. The system as claimed in claim 20 wherein the structure and circuitry for optically detecting user input further comprises structure and circuitry for optically detecting user input with invisible light.

26. The system as claimed in claim 20 wherein the structure and circuitry for optically detecting user input further comprises structure and circuitry for performing surveillance.

27. The system as claimed in claim 20 wherein the structure and circuitry for optically detecting user input further comprises structure and circuitry for performing head tracking.

28. The system as claimed in claim 20 further comprising structure and circuitry for:

imaging the user'"'"'s head in the imaging volume; and

mapping the image of the user'"'"'s head to at least one interface related function.

29. The system as claimed in claim 20 further comprising structure and circuitry for:

imaging hand gestures in the imaging volume; and

mapping the hand gestures to at least one interface related function.

30. The system as claimed in claim 20 further comprising structure and circuitry for detecting user presence, the detecting being selected from detecting whether a user is sitting in front of a display, identifying which user is present, saving power when the user is away, and a combination thereof.

31. The system as claimed in claim 20 further comprising structure and circuitry for object measuring.

32. The system as claimed in claim 20 further comprising structure and circuitry for performing image substitution or replacement.

33. The system as claimed in claim 20 further comprising structure and circuitry for painting a moveable object and following it as it moves.

34. The system as claimed in claim 20 further comprising structure and circuitry for generating an interactive virtual keyboard.

35. The system as claimed in claim 20 further comprising structure and circuitry for assigning functions by detecting the kinds of objects in the optically detected user input and associating a predetermined function or set of functions with at least some of those objects.

36. The system as claimed in claim 20 further comprising structure and circuitry for customizing the environment in which the three-dimensional imaging and display system is deployed.

37. The system as claimed in claim 20 further comprising structure and circuitry for:

utilizing watershed segmentation to detect elements of the user input;

tracking individual and relative movements and positions of the elements of the user input; and

mapping the tracked movements and positions to predetermined functions.

38. The system as claimed in claim 20 further comprising:

a photodiode amplifier;

a clip amp connected to the photodiode amplifier;

a high pass filter connected to the clip amp;

a mixer connected to the high pass filter;

a low pass filter connected to the mixer;

an analog-to-digital converter connected to the low pass filter;

a digital signal processor connected to the analog-to-digital converter;

a minor driver connected to the digital signal processor; and

a video subsystem connected to the digital signal processor.

39. A three-dimensional imaging and display apparatus, comprising:

a processor;

an imaging mode circuit coupled to the processor and adapted to optically detect three-dimensional user input in a three-dimensional imaging volume by measuring the path length of a modulated scanning beam as a function of the phase shift thereof, and further adapted to

compensate for parallax in the optically detected three-dimensional user input by;

calibrating to compensate for parallax as a function of

a scan coordinate defined by a mirror system position, and

at least the measured distance from the minor system to a target object; and

utilizing a lookup table having prior parallax compensation calibration information therein; and

a display mode circuit coupled to the processor and adapted to simultaneously visually display an image of the detected user input,

present immediate visual image user feedback concerning the optically detected user input occurring in the imaging volume, and

alternately energize an imaging mode and a display mode associated with the apparatus.

40. The apparatus as claimed in claim 39 wherein the imaging mode circuit further comprises:

a receiver coupled to the processor for the modulated scanning beam following reflection by the user input;

a mixer coupled to the receiver for homodyne mixing the modulation frequency of the scanning beam with the signal coming from the receiver;

an isolation circuit coupled to the mixer for isolating the phase shift from the homodyne mixing; and

a derivation circuit coupled to the isolation circuit for deriving from the phase shift the distance that the scanning beam has traveled.