Three-dimensional imaging and display system

• US 8,780,332 B2
• Filed: 08/17/2011
• Issued: 07/15/2014
• Est. Priority Date: 10/21/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Krah, Christoph H.
• Primary Examiner(s): Ratcliffe, Luke
• Assistant Examiner(s): Diacou, Ari M
• Application Number:
  US13/212,122
• Publication Number:
  US 20110298798A1
• Time in Patent Office: 1,063 Days
• Field of Search: 356/5.04
• US Class Current: 356/5.04
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01C 3/085 with electronic parallax measurement
• CPC Class Codes: G01C 3/12 with monocular observation at a single point, e.g. coincidence type G01C3/20 takes precedence
• CPC Class Codes: G01S 17/14 wherein a voltage or current pulse is initiated and terminated in accordance with the pulse transmission and echo reception respectively, e.g. using counters
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/48 Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/4861 Circuits for detection, sampling, integration or read-out
• CPC Class Codes: G01S 7/4865 Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak peak detection in noise, signal conditioning G01S7/487
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A three-dimensional imaging and display system is provided in which user input is optically detected in an imaging volume by measuring the path length of an amplitude modulated scanning beam as a function of the phase shift thereof. Visual image user feedback concerning the detected user input is presented.

Claims

1. A three-dimensional imaging and display apparatus comprising:

a processor;

circuitry coupled to the processor, wherein the first circuit is configured to;

optically detect one or more user inputs in an imaging volume by measuring a path length of a modulated scanning beam as a function of a phase shift, wherein the path length of the modulated scanning beam is separately measured at a plurality of angles, wherein measuring the path length L from the phase shift φ and

the wavelength λ

wherein;

2. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to amplitude modulate the output intensity of the modulated scanning beam.

3. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to optically detect the modulated scanning beam with a PIN diode detector.

4. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to calibrate the phase shift with a distance the modulated scanning beam travels.

5. The three-dimensional imaging and display apparatus of claim 1 comprising:

a receiver coupled to the circuitry, wherein the receiver is configured to receive the modulated scanning beam after the modulated scanning beam has been reflected by the user input;

a mixer coupled to the circuitry, wherein the mixer is configured to homodyne mix a modulation frequency of the modulated scanning beam with a signal from the receiver, wherein the circuitry is configured to isolate the phase shift from the homodyne mixed signal and configured to derive a distance from the phase shift, wherein the distance is a distance that the modulated scanning beam has traveled.

6. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to display and manipulate virtual objects located in a three-dimensional imaging volume in response to virtual interactions with the virtual objects, the virtual interactions being derived from the detected user input, wherein the virtual objects are moved, scaled, or rotated in the imaging volume.

7. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to display externally provided images.

8. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to optically detect with invisible light.

9. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to track one or more heads.

10. The three-dimensional imaging and display apparatus of claim 1, wherein the circuitry is configured to image a user'"'"'s head in the imaging volume and, wherein the circuitry is configured to map an image of the user'"'"'s head to at least one interface related function.

11. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to image hand gestures in the imaging volume to map hand gestures to at least one interface related function.

12. The three-dimensional imaging and display apparatus of claim 1 wherein the circuitry is configured to measure the size of objects.

13. The three-dimensional imaging and display apparatus of claim 1, wherein the circuitry is configured to substitute or replace images.

14. The three-dimensional imaging and display apparatus of claim 1, wherein the circuitry is configured to paint and follow a moveable object as it moves in space.

15. The three-dimensional imaging and display apparatus of claim 1, wherein the circuitry is configured to generate an interactive virtual keyboard.

16. The three-dimensional imaging and display apparatus of claim 1, wherein the circuitry is configured to assign functions to objects, wherein a type of one or more objects is optically detected in the imaging volume and a predetermined function or set of functions is assigned with the one or more objects detected.

17. The three-dimensional imaging and display apparatus of claim 1, wherein the circuitry is configured to customize an environment in which the three-dimensional imaging and display apparatus is deployed.

18. The three-dimensional imaging and display apparatus of claim 1, wherein the circuitry is configured to:

detect elements of the user input with watershed segmentation;

track individual and relative movements and positions of the elements of the user input; and

map tracked movements and positions to predetermined functions.

19. The three-dimensional imaging and display apparatus of claim 1 comprising a mirror operating at a mechanical resonant frequency.

20. The three-dimensional imaging and display apparatus of claim 1 comprising an auxiliary display configured to display one or more three-dimensional or two-dimensional images.

21. A method for detecting and providing image feedback comprising:

optically detecting a user input via watershed segmentation by measuring a path length of a modulated scanning beam as a function of a phase shift, wherein the path length of the modulated scanning beam is separately measured at a plurality of angles; and

presenting visual images on a display area, wherein the visual images are presented in response to the user input.

22. The method of claim 21 wherein optically detecting comprises amplitude modulating the output intensity of a single scanning laser beam.

23. The method of claim 21 wherein optically detecting comprises detecting the modulated scanning beam with a PIN diode detector.

24. The method of claim 21 wherein optically detecting comprises calibrating the phase shift with a distance that the modulated scanning beam travels.

25. The method of claim 21 wherein optically detecting comprises determining the path length L from the phase shift φ

and the wavelength λ

wherein;

26. The method of claim 21 comprising:

receiving the modulated scanning beam on a receiver after the modulated scanning beam has been reflected by the user input;

homodyne mixing a modulation frequency of the scanning beam with a signal from the receiver;

isolating a phase shift from the homodyne mixed signal; and

deriving a distance from the phase shift, wherein the distance is a distance that the scanning beam has traveled.

27. The method of claim 21 comprising:

displaying virtual objects located in a three-dimensional imaging volume, wherein the virtual interactions are derived from the detected user input; and

manipulating the virtual objects in response to virtual interactions with the virtual objects, wherein the manipulated objects are moved, scaled, or rotated.

28. The method of claim 21 wherein presenting visual images further comprises displaying images that have been externally provided.

29. The method of claim 21 wherein optically detecting comprises detecting with invisible light.

30. The method of claim 21 comprising tracking one or more heads.

31. The method of claim 21 comprising:

imaging a user'"'"'s head in the imaging volume; and

mapping an image of the user'"'"'s head to one or more interface related function.

32. The method of claim 21 comprising:

imaging hand gestures in the imaging volume; and

mapping hand gestures to one or more interface related functions.

33. The method of claim 21 comprising measuring the size of the object.

34. The method of claim 21 comprising substituting or replacing the image presented.

35. The method of claim 21 comprising painting and following a movable object.

36. The method of claim 21 comprising generating an interactive virtual keyboard.

37. The method of claim 21 comprising:

optically detecting a type of one or more objects in the imaging volume; and

assigning one or more pre-determined functions to the one or more objects detected.

38. The method of claim 21 comprising customizing an environment for three dimensional imaging.

39. The method of claim 21 comprising:

tracking individual and relative movements and positions of the elements of the user input; and

mapping tracked movements and positions to predetermined functions.

40. The method of claim 21 wherein optically detecting comprises optically detecting on one or more mirrors having a mechanical resonant frequency.

41. The method of claim 21 comprising presenting one or more images on an auxiliary display.