SIX DOF MIXED REALITY INPUT BY FUSING INERTIAL HANDHELD CONTROLLER WITH HAND TRACKING

• US 20170357332A1
• Filed: 09/23/2016
• Published: 12/14/2017
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru Octavian, Dulu, Constantin
• Granted Patent: US 10,078,377 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The invention is directed to systems and methods for providing a wireless hand-held inertial controller for use with a head-mounted, augmented or virtual reality display or other conventional display that operates with six degrees of freedom by fusing (i) data related to the position of the hand-held inertial controller derived from a depth camera located on the display with (ii) data relating to the orientation of the hand-held inertial controller derived from an inertial measurement unit located in the hand-held inertial controller.

Claims

1. A system comprising:

a display connected to a processor;

a hand-held input device configured to communicate with the processor to selectively provide one or more user inputs, the hand-held input device also including a first sensor for determining the orientation of the hand-held input device relative to a predetermined frame of reference and providing orientation data to the processor;

a second sensor located in a known location relative to the display for determining the position of one or more hands of a user relative to the display and for providing position data to the processor, and

wherein the processor uses the orientation data and the position data to track the one or more hands of the user within three dimensional space with six degrees of freedom.

2. The system of claim 1, wherein the display comprises a conventional computer monitor or flat panel display.

3. The system of claim 1, wherein the display comprises a head mounted display (HMD).

4. The system of claim 3, wherein the HMD comprises a virtual reality display.

5. The system of claim 3, wherein the HMD comprises a three dimensional, augmented reality display.

6. The system of claim 5, wherein the hand-held input device comprises a microcontroller operatively connected to the first sensor, one or more momentary switches that are operatively connected to, and selectively provide inputs to, the microcontroller, one more status LEDs operatively connected to the microcontroller, and a wireless radio operatively connected to the microcontroller for transmitting user inputs and orientation data to the processor of the HMD.

7. The system of claim 6, wherein the first sensor comprises an inertial measurement unit (IMU).

8. The system of claim 7, wherein the IMU comprises one or more of a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer.

9. The system of claim 8, wherein the orientation data comprises θ

_pitch, θ

_yaw and θ

_roll coordinates.

10. The system of claim 9, wherein the second sensor comprises a depth camera.

11. The system of claim 10, wherein the depth camera is an infrared depth camera.

12. The system of claim 11, wherein the depth camera is mounted on the HMD in a forward facing orientation relative to the HMD.

13. The system of claim 12, wherein the position data comprises x, y and z coordinates.

14. The system of claim 13, wherein the positional tracking of the one or more hands of a user is transformed by the processor into x, y, z, θ

_pitch, θ

_yaw and θ

_roll coordinates within a real world frame of reference.

15. In a system comprising a head mounted virtual or augmented reality display device (HMD) having a forward facing optical sensor having a field of view, and wherein the display device interfaces with a wireless hand-held inertial controller for providing user input to the display device, a method for recovering six degrees of freedom relative to the wireless hand-held inertial controller, the method comprising acts of:

detecting, by the optical sensor of the head mounted display device, the presence of a user'"'"'s hand within the field of view of the optical sensor;

determining, by the head mounted display device, if a wireless hand-held inertial controller is active and paired with the head mounted display device;

tracking, by the optical sensor of the head mounted display device, movement of the user'"'"'s hand relative to the head mounted display over a period of time to derive trajectory data representative of the trajectory of the user'"'"'s hand during the period of time;

receiving, by the head mounted display device, acceleration data for the period of time from the wireless hand-held inertial controller as derived by the inertial measurement unit of the wireless hand-held inertial controller;

comparing the trajectory data with the acceleration data to compute a confidence level that the wireless hand-held device is located in the user'"'"'s hand; and

if the confidence level meets or exceeds a predetermined minimum threshold, fusing location data representative of the location of the user'"'"'s hand as derived from the optical sensor of the head mounted display device with orientation data representative of the orientation of the user'"'"'s hand as derived from the inertial measurement unit of the wireless hand-held inertial controller to track the user'"'"'s hand within three dimensional space with six degrees of freedom.

16. The method of claim 15, wherein the wireless hand-held inertial controller includes an inertial measurement unit (IMU).

17. The method of claim 16, wherein the act for detecting the presence of a user'"'"'s hand within the field of view of the optical sensor comprises:

capturing, by the optical sensor of the HMD, a plurality of successive frames of video; and

for each frame of video captured by the optical sensor of the HMD, analyzing video data captured by the HMD to determine if a user'"'"'s hand is located within the field of view of the HMD'"'"'s optical sensor.

18. The method of claim 17 further comprising acts for:

receiving, by the HMD, from the wireless hand-held inertial controller first orientation data representative of the orientation of the wireless hand-held inertial controller derived from the IMU;

computing second orientation data representative of the orientation of the wireless hand-held inertial controller based on pose data detected by the optical sensors of the HMD; and

comparing the first orientation data with the second orientation to compute a confidence level.

19. The method of claim 15 further comprising an act for creating an association between the user'"'"'s hand and the controller if the confidence level meets or exceeds a predetermined minimum threshold.

20. The method of claim 19 further comprising acts for:

periodically monitoring the confidence level of the association between the user'"'"'s hand and the associated controller;

if the confidence level remains above the predetermined minimum threshold, persisting the association; and

if the confidence level drops below the predetermined minimum threshold, removing the association.