Three-dimensional tracking of a user control device in a volume

• US 8,970,625 B2
• Filed: 12/21/2011
• Issued: 03/03/2015
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: zSpace Inc.
• Original Assignee: zSpace Inc.
• Inventors: Chavez, David A., Paranjpe, Milind, Tu, Jerome C.
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Cain, II, Leon T
• Application Number:
  US13/333,299
• Publication Number:
  US 20120162214A1
• Time in Patent Office: 1,168 Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/046 by electromagnetic means
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Tracking objects presented within a stereo three-dimensional (3D) scene. The user control device may include one or more visually indicated points for at least one tracking sensor to track. The user control device may also include other position determining devices, for example, an accelerometer and/or gyroscope. Precise 3D coordinates of the stylus may be determined based on location information from the tracking sensor(s) and additional information from the other position determining devices. A stereo 3D scene may be updated to reflect the determined coordinates.

Claims

1. A system, comprising:

a display configured to display a stereoscopic three-dimensional (3D) scene within a virtual space;

a user control device configured to manipulate objects within the stereoscopic 3D scene, wherein the user control device includes at least one visually detectable point;

at least one tracking sensor configured to track the at least one visually detectable point in a physical space; and

a processing subsystem configured to;

provide the stereoscopic 3D scene to the display;

receive tracking information for the at least one visually detectable point from the at least one tracking sensor;

receive additional information from the user control device, wherein the additional information comprises measurement information from at least one sensor regarding dynamic state of the user control device;

determine a position and an orientation of the user control device in the physical space based on the received tracking information and the additional information; and

provide an updated stereoscopic 3D scene to the display based on the determined position and orientation of the user control device in the physical space, wherein the updated 3D stereoscopic scene in the virtual space correlates to the physical space within 1 mm in each of an x, y, and z axis.

2. The system of claim 1, wherein the at least one visually detectable point includes first and second visually detectable points, wherein the first and second visually detectable points are non-adjacent, and wherein the first visually detectable point is configured to emit light having a first timing condition, and wherein the second visually detectable point is configured to emit light having a second timing condition.

3. The system of claim 1, wherein the at least one visually detectable point includes first and second visually detectable points, wherein the first visually detectable point is configured to produce a first light field having a first opaque portion, wherein the second visually detectable point is configured to produce a second light field having a second opaque portion different than the first opaque portion.

4. The system of claim 1, wherein the user control device further includes a field independent change-in-orientation mechanism and a constant direction determination mechanism, and wherein the measurement information comprises information from the field independent change in orientation mechanism and the constant direction determination mechanism.

5. The system of claim 4, wherein the constant direction determination mechanism is a constant gravity determination mechanism.

6. The system of claim 4, wherein the field independent change-in-orientation mechanism and the constant direction determination mechanism are employed with a gyroscope and an accelerometer, respectively.

7. The system of claim 1, wherein to perform said determining the position and the orientation of the user control device, the processing subsystem is further configured to determine a center of the at least one visually detectable point.

8. The system of claim 1, wherein the processing subsystem is further configured to compensate for a drift or bias of a measurement device of the user control device based on an inconsistency between the tracking information of the at least one visually detectable point and the additional information.

9. The system of claim 1, wherein the processing subsystem is further configured to resolve an inconsistency between the tracking information of the at least one visually detectable point and the additional information.

10. The system of claim 1, wherein to provide an updated stereoscopic 3D scene, the processor subsystem is further configured to render a virtual representation of the user control device, wherein the virtual representation of the user control device in the virtual space correlates to the user control device'"'"'s position and orientation in the physical space relative.

11. The system of claim 1, wherein the determined position and orientation of the user control device is further based on one or more properties of the at least one tracking sensor.

12. The system of claim 1, wherein the at least one tracking sensor includes first and second cameras, wherein each of the first and second cameras is configured to track a distinct perspective of the at least one visually detectable point.

13. The system of claim 12, wherein the first and second cameras are configured for three-axis triangulation determination, and wherein the first and second cameras are positioned relative to a predefined position and orientation of the display.

14. The system of claim 1, wherein the user control device further includes a magnetometer, and wherein the measurement information comprises information from the magnetometer.

15. The system of claim 1, wherein the user control device comprises a stylus with a pen grip form factor.

16. A method, comprising:

presenting a stereoscopic 3D scene within a virtual space by at least one display, wherein said presenting the stereoscopic 3D scene comprises displaying at least one stereoscopic image of the stereoscopic 3D scene by the at least one display, wherein a user control device is used to manipulate objects within the stereoscopic 3D scene;

determining first location information of the user control device in a physical space based on two or more captured images of at least one visually detectable point of the user control device, wherein each of the two or more captured images is captured from a distinct perspective;

determining second location information of the user control device in the physical space based on additional information from the user control device, wherein the additional information comprises measurement information from at least one sensor regarding dynamic state of the user control device;

determining a position and an orientation of the user control device in the physical space based on the first and second location information; and

updating the stereoscopic 3D scene based on said determining the position and orientation of the user control device, wherein the updated 3D stereoscopic scene in the virtual space correlates to the physical space within 1 mm in each of an x, y, and z axis.

17. The method of claim 16, wherein said determining the position and orientation of the user control device includes resolving an inconsistency between the first and second location information.

18. The method of claim 16, wherein the at least one visually detectable point includes non-adjacent first and second visually detectable points of the user control device.

19. The method of claim 18, wherein the determined first location information of the user control device includes information regarding each of the first and second visually detectable points, wherein said determining the first location information includes determining respective location information corresponding to the first and second visually detectable points based on captured images of the first and second visually detectable points from each of the at least two distinct perspectives.

20. The method of claim 19, wherein the first and second visually detectable points are configured to emit light having a first and a second timing condition, respectively, wherein the first and second timing conditions are different, wherein said determining the first location information includes identifying which captured images correspond to which visually detectable point.

21. The method of claim 19, wherein a light field of the first visually detectable point includes a first opaque portion, wherein a light field of the second visually detectable point includes a second opaque portion that is a different shape than the first opaque portion, wherein said determining the first location information includes identifying quadrants of the user control device based on a presence of the first and second opaque portions in the captured images.

22. The method of claim 16, wherein a first captured image and a second captured image of the two or more captured images are taken from first and second spatially separated cameras, respectively.

23. A non-transitory computer accessible memory medium storing program instructions, wherein the program instructions are executable by a processor to:

provide a stereoscopic three-dimensional (3D) scene to a display;

wherein the stereoscopic 3D scene is display within a virtual space;

receive tracking information for at least one visually detectable point of a user control device from at least one tracking sensor;

receive additional information from the user control device, wherein the additional information comprises measurement information from at least one sensor regarding dynamic state of the user control device;

determine a position and an orientation of the user control device in physical space based on the tracking information for the at least one visually detectable point and the additional information; and

provide an updated stereoscopic 3D scene to the display based on the determined position and orientation of the user control device in the physical space, wherein the updated 3D stereoscopic scene in the virtual space correlates to the physical space within 1 mm in each of an x, y, and z axis.