Three-dimensional tracking of a user control device in a volume

• US 9,201,568 B2
• Filed: 03/02/2015
• Issued: 12/01/2015
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: zSpace Inc.
• Original Assignee: zSpace Inc.
• Inventors: Chavez, David A., Paranjpe, Milind, Tu, Jerome C.
• Primary Examiner(s): Cain, II, Leon T
• Application Number:
  US14/635,654
• Publication Number:
  US 20150169173A1
• Time in Patent Office: 274 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/046 by electromagnetic means
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Tracking objects presented within a stereo three-dimensional (3D) scene. The user control device may include one or more visually indicated points for at least one tracking sensor to track. The user control device may also include other position determining devices, for example, an accelerometer and/or gyroscope. Precise 3D coordinates of the stylus may be determined based on location information from the tracking sensor(s) and additional information from the other position determining devices. A stereo 3D scene may be updated to reflect the determined coordinates.

Claims

1. A system, comprising:

a display configured to display a stereoscopic three-dimensional (3D) scene within a virtual space;

a user control device configured to manipulate objects within the stereoscopic 3D scene, wherein the user control device includes one or more position or orientation sensors, and at least one visually detectable point;

at least one tracking sensor configured to track the at least one visually detectable point in a physical space; and

a processing subsystem configured to;

provide the stereoscopic 3D scene to the display;

receive first location information for the at least one visually detectable point from the at least one tracking sensor;

receive second location information from the one or more position or orientation sensors of the user control device;

determine a position and an orientation of the user control device based on the first and second location information; and

provide an updated stereoscopic 3D scene to the display based on the determined position and orientation of the user control device in the physical space, wherein the updated 3D stereoscopic scene in the virtual space correlates to the physical space.

2. The system of claim 1, wherein the at least one visually detectable point includes first and second visually detectable points, wherein the first and second visually detectable points are non-adjacent, and wherein the first visually detectable point is configured to emit light having a first timing condition, and wherein the second visually detectable point is configured to emit light having a second timing condition.

3. The system of claim 1, wherein the at least one visually detectable point includes first and second visually detectable points, wherein the first visually detectable point is configured to produce a first light field having a first opaque portion, wherein the second visually detectable point is configured to produce a second light field having a second opaque portion different than the first opaque portion.

4. The system of claim 1, wherein the one or more position or orientation sensors of the user control device comprise a field independent change-in-orientation mechanism and a constant direction determination mechanism, and wherein the second location information comprises information from the field independent change in orientation mechanism and the constant direction determination mechanism.

5. The system of claim 4, wherein the constant direction determination mechanism is a constant gravity determination mechanism.

6. The system of claim 4, wherein the field independent change-in-orientation mechanism and the constant direction determination mechanism are implemented with a gyroscope and an accelerometer, respectively.

7. The system of claim 1, wherein to perform said determining the position and the orientation of the user control device, the processing subsystem is further configured to determine a center of the at least one visually detectable point.

8. The system of claim 1, wherein the processing subsystem is further configured to compensate for a drift or bias of a measurement device of the user control device based on an inconsistency between the tracked at least one visually detectable point and the second location information.

9. The system of claim 1, wherein the processing subsystem is further configured to resolve an inconsistency between the tracked at least one visually detectable point and the second location information.

10. The system of claim 1, wherein to provide an updated 3D scene, the processor subsystem is further configured to render a virtual representation of the user control device, wherein the virtual representation of the user control device reflects a correspondence to the user control device'"'"'s position and orientation in the physical space relative to that of the graphic space.

11. The system of claim 1, wherein the at least one tracking sensor includes first and second cameras, wherein each of the first and second cameras is configured to capture a distinct perspective of the at least one visually detectable point.

12. The system of claim 11, wherein the first and second cameras are separated far enough from each other to enable three-axis triangulation determination, and wherein the first and second cameras are positioned relative to a predefined position and orientation of the display.

13. The system of claim 1, wherein the one or more position or orientation sensors of the user control device comprise a magnetometer, and wherein the second location information comprises information from the magnetometer.

14. A method, comprising:

presenting a stereoscopic three-dimensional (3D) scene within a virtual space by at least one display, wherein said presenting the stereoscopic 3D scene comprises displaying at least one stereoscopic image of the stereoscopic 3D scene by the at least one display, wherein a user control device is used to manipulate objects within the stereoscopic 3D scene, and wherein the user control device includes one or more position or orientation sensors, and at least one visually detectable point;

receiving first location information for the at least one visually detectable point from at least one tracking sensor configured to track the at least one visually detectable point in a physical space;

determining second location information of the user control device based on the one or more position or orientation sensors of the user control device;

determining a position and an orientation of the user control device based on the first and second location information; and

updating the stereoscopic 3D scene based on said determining the position and orientation of the user control device in the physical space, wherein the updated stereoscopic 3D scene in the virtual space correlates to the physical space.

15. The method of claim 14, wherein said determining the position and orientation of the user control device includes resolving an inconsistency between the first and second location information.

16. The method of claim 14, wherein the at least one visually detectable point includes non-adjacent first and second visually detectable points of the user control device.

17. The method of claim 16, wherein the received first location information of the user control device includes information regarding each of the first and second visually detectable points, wherein the first location information includes respective location information corresponding to the first and second visually detectable points based on captured images of the first and second visually detectable points from each of at least two distinct perspectives.

18. The method of claim 17, wherein the first and second visually detectable points are configured to emit light having a first and a second timing condition, respectively, wherein the first and second timing conditions are different, wherein said determining the first location information includes identifying which captured images correspond to which visually detectable point.

19. The method of claim 17, wherein a light field of the first visually detectable point includes a first opaque portion, wherein a light field of the second visually detectable point includes a second opaque portion that is a different shape than the first opaque portion, wherein said determining the first location information includes identifying quadrants of the user control device based on a presence of the first and second opaque portions in the captured images.

20. A non-transitory computer accessible memory medium storing program instructions, wherein the program instructions are executable by a processor to perform:

providing a stereoscopic 3D scene for display within a virtual space by at least one display, wherein said providing the stereoscopic 3D scene comprises providing at least one stereoscopic image of the stereoscopic 3D scene to the at least one display, wherein a user control device is used to manipulate objects within the stereoscopic 3D scene, and wherein the user control device includes one or more position or orientation sensors, and at least one visually detectable point;

receiving first location information for the at least one visually detectable point from at least one tracking sensor configured to track the at least one visually detectable point in a physical space;

determining second location information of the user control device based on one or more position or orientation sensors of the user control device;

determining a position and an orientation of the user control device based on the first and second location information; and

updating the stereoscopic 3D scene based on said determining the position and orientation of the user control device in the physical space, wherein the updated stereoscopic 3D scene in the virtual space correlates to the physical space.