Passive optical and inertial tracking in slim form-factor

• US 10,521,026 B2
• Filed: 11/15/2018
• Issued: 12/31/2019
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru Octavian, Dulu, Constantin, Edmonds, Christopher Douglas, Finocchio, Mark James
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US16/192,510
• Publication Number:
  US 20190087021A1
• Time in Patent Office: 411 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/001 : Constructional or mechanical details

Abstract

Systems are provided that include a wireless hand-held inertial controller with passive optical and inertial tracking in a slim form-factor. These systems are configured for use with a head mounted virtual or augmented reality display device (HMD) that operates with six degrees of freedom by fusing (i) data related to the position of the controller derived from a forward-facing optical sensor located in the HMD with (ii) data relating to the orientation of the controller derived from an inertial measurement unit located in the controller.

Claims

1. A hand-held controller for use in a virtual or augmented reality computing environment that includes a head mounted display (HMD) device that determines at least a position of the hand-held controller based on detected position data associated with at least two different optical reflectors positioned on the hand-held controller, the hand-held controller comprising:

a body having a first end, a second end and a middle portion between the first end and the second end, the body also including a first optically reflective marker at or proximate the first end and a second optically reflective marker positioned separately from the first optically reflective marker at or proximate the second end, the first and second optically reflective markers being adapted to reflect IR (infrared) light emitted from the HMD device, toward an optical sensor for the HMD device, wherein the IR light that is reflected to the HMD device is used to generate position data of the hand-held controller by the HMD device; and

the middle portion enclosing at least one of (1) an inertial measurement unit (IMU) that tracks orientation data of the hand-held inertial controller or (2) one or more buttons configured and positioned for selective actuation by a finger or thumb of a user to receive user inputs, as well as a communications radio configured to provide providing wireless communications between the hand-held inertial controller and the HMD device, so as to provide the orientation data and/or user inputs to the HMD device.

2. The hand-held controller of claim 1, wherein the first optically reflective marker and the second optically reflective marker each comprise an infrared marker.

3. The hand-held controller of claim 2 further comprising a material covering the first and second optically reflective markers that is transmissive to light in the IR spectrum but opaque to light in the visible spectrum.

4. The hand-held controller of claim 3, wherein the hand-held controller is shaped as an elongate stylus.

5. The hand-held controller of claim 4, wherein the IMU comprises one or more of a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer.

6. The hand-held controller of claim 5, wherein the orientation data comprises θ

_pitch, θ

_yaw and θ

_roll coordinates.

7. A system comprising:

a head mounted display (HMD) device having a processor; and

a wireless hand-held inertial controller configured to communicate with the processor to selectively provide at least one of user inputs or orientation data to the processor, the hand-held inertial controller comprising;

a body having a first end, a second end and a middle portion between the first end and the second end, the body also including a first optically reflective marker at or proximate the first end and a second optically reflective marker positioned separately from the first optically reflective marker at or proximate the second end, the first and second optically reflective markers being adapted to reflect IR (infrared) light emitted from the HMD device, toward an optical sensor for the HMD device, wherein the IR light that is reflected to the HMD device is used to generate position data of the hand-held controller by the HMD device; and

the middle portion enclosing at least one of (1) an inertial measurement unit (IMU) that tracks orientation data of the hand-held inertial controller and a communications radio configured to provide wireless communications between the hand-held inertial controller and the HMD device, so as to provide the orientation data the HMD device;

wherein the HMD device further includes an optical sensor configured for determining the position of each of the first and second optically reflective markers relative to the HMD device based on reflected light that is detected by the optical sensor after it is emitted by the light source and reflected off of at least one of the first or second optically reflective markers, and

wherein the processor uses the orientation data and the position data obtained from the optical sensor to track the hand-held inertial controller within three dimensional space with six degrees of freedom.

8. The system of claim 7, wherein the HMD device comprises a virtual reality display.

9. The system of claim 7, wherein the HMD device comprises a three dimensional, augmented reality display.

10. The system of claim 9, wherein at least one of the first optically reflective marker or the second optically reflective marker comprise an infrared marker.

11. The system of claim 10, further comprising a material covering the first optically reflective marker and the second optically reflective marker that is transmissive to light in the IR spectrum but opaque to light in the visible spectrum.

12. The system of claim 10, wherein the hand-held inertial controller is shaped in form of an object used in one or more video games and wherein the hand-held inertial controller further includes one or more buttons configured and positioned for selective actuation by a finger or thumb of a user to receive user inputs that are transmitted to the processor by the communications radio.

13. The system of claim 9, wherein the IMU comprises one or more of a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer.

14. The system of claim 13, wherein the orientation data comprises θ

_pitch, θ

_yaw and θ

_roll coordinates.

15. The system of claim 9, wherein the optical sensor comprises a depth camera.

16. The system of claim 15, wherein the depth camera is an infrared depth camera.

17. The system of claim 16, wherein the depth camera is mounted on the HMD device in a forward facing orientation relative to the HMD device.

18. The system of claim 9, wherein the position data comprises x, y and z coordinates.

19. The system of claim 9, wherein the orientation data and the position data of the hand-held inertial controller is transformed by the processor into x, y, z, θ

_pitch, θ

_yaw and θ

_roll coordinates within a real world frame of reference.

20. The system of claim 9, wherein the communications radio is a Bluetooth radio and the wireless communications are Bluetooth transmissions.