Six DOF mixed reality input by fusing inertial handheld controller with hand tracking

• US 10,078,377 B2
• Filed: 09/23/2016
• Issued: 09/18/2018
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru Octavian, Dulu, Constantin
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US15/274,294
• Publication Number:
  US 20170357332A1
• Time in Patent Office: 725 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Systems and methods are described that provide a wireless hand-held inertial controller for use with a head-mounted, augmented reality (AR) or virtual reality (VR) display or other conventional display that operates with six degrees of freedom by fusing (i) data related to the position of the hand-held inertial controller derived from a depth camera located on the display with (ii) data relating to the orientation of the hand-held inertial controller derived from an inertial measurement unit located in the hand-held inertial controller.

Claims

1. A system comprising:

a head-mounted display device (HMD′

) comprising;

a processor;

a wearable stereoscopic display adapted for displaying one of an augmented reality (AR) environment or a full virtual reality (VR) environment; and

an optical sensor system that produces location data at a particular instant of time for a wireless hand-held input controller, wherein the location data is obtained from a depth camera mounted on the HMD, and wherein the location data is defined by x, v and z coordinates;

the wireless hand-held input controller comprising;

an inertial measurement unit (IMU) comprising one or more sensors for determining orientation data of the wireless hand-held input controller and wherein the orientation data is defined by pitch (elevation angle), yaw (azimuth angle) and roll (rotation) relative to a predetermined frame of reference; and

a microcontroller that communicates to the processor of the HMD one or more user inputs relative to the orientation data; and

wherein the processor of the HMD uses the orientation data at the particular instant of time as provided by the IMU of the wireless hand-held input controller and the location data at the particular instant of time as provided by the optical sensor system of the HMD to determine location and orientation of the wireless hand-held input controller in reference to the HMD with six degrees of freedom as derived from the x, v and z coordinates determined at the HMD and the yaw, pitch and roll coordinates determined at the IMU of the wireless hand-held input controller.

2. The system of claim 1, wherein the HMD further comprises an on-board image production system mounted on the HMD.

3. The system of claim 1, wherein the HMD comprises a virtual reality display.

4. The system of claim 1, wherein the HMD comprises a three dimensional, augmented reality display.

5. The system of claim 1, wherein the wireless hand-held input controller comprises one or more momentary switches that are operatively connected to, and selectively provide inputs to, the microcontroller, one more status LEDs operatively connected to the microcontroller, and a wireless radio operatively connected to the microcontroller for transmitting user inputs and orientation data to the processor of the HMD.

6. The system of claim 5, wherein the IMU comprises one or more of a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer.

7. The system of claim 6, wherein the orientation data comprises θ

_pitch, θ

_yaw and θ

_roll coordinates.

8. The system of claim 7, wherein the depth camera is an infrared camera.

9. The system of claim 1, wherein the HMD further comprises an inward facing optical sensor that detects a gaze direction of a user'"'"'s eyes.

10. The system of claim 1, wherein the HMD further comprises an on-board position sensor system mounted to the HMD.

11. The system of claim 10, wherein the position sensor system mounted to the HMD comprises an inertial measurement unit (IMU) comprising one or more of:

a 3-axis accelerometer;

a 3-axis gyroscope;

a 3-axis magnetometer;

a global positioning system; and

a multilateration tracker.

12. The system of claim 1, wherein the determined location and orientation of the wireless hand-held input controller is determined in reference to x, y, z, θ

_pitch, θ

_yaw and θ

_roll coordinates within a real world frame of reference.

13. In a system comprising a head mounted display device (HMD) configured to display a three-dimensional space for an augmented reality (AR) environment or a full virtual reality (VR) environment, wherein the HMD comprises a forward facing optical sensor having a field of view, and wherein the HMD interfaces with a wireless hand-held input controller that provides user input to the HMD, a computer-implemented method for determining and tracking location and orientation of the wireless hand-held input controller in reference to the HMD, and wherein the determined location and orientation are determined with six degrees of freedom, the computer-implemented method comprising:

detecting with the forward facing optical sensor the presence of a user'"'"'s hand within the field of view;

determining if the wireless hand-held input controller is active and paired with the HMD;

tracking with the forward facing optical sensor movement of the user'"'"'s hand relative to the HMD over a period of time;

generating trajectory data representative of the trajectory of the user'"'"'s hand during the period of time;

generating acceleration data for the period of time from the wireless hand-held input controller, wherein the acceleration data is derived by an inertial measurement unit (IMU) of the wireless hand-held input controller;

receiving at a processor of the HMD the trajectory data and the acceleration data and comparing the trajectory data with the acceleration data to compute a confidence level that the wireless hand-held input controller is located in the user'"'"'s hand; and

if the confidence level meets or exceeds a predetermined minimum threshold, combining location data representative of a location of the user'"'"'s hand as derived from the forward facing optical sensor of the HMD with orientation data representative of an orientation of the user'"'"'s hand as derived from the IMU of the wireless hand-held input controller in order to track the user'"'"'s hand within the three dimensional space of the AR or VR environment with six degrees of freedom.

14. The method of claim 13, wherein detecting the presence of the user'"'"'s hand within the field of view comprises:

capturing, by the forward facing optical sensor of the HMD, a plurality of successive frames of video; and

for each frame of video captured by the forward facing optical sensor of the HMD, analyzing video data captured by the HMD to determine if a user'"'"'s hand is located within the field of view of the HMD'"'"'s forward facing optical sensor.

15. The method of claim 13 further comprising:

receiving from the wireless hand-held input controller first orientation data from the IMU of the wireless hand-held input controller, wherein the first orientation data is received at the processor of the HMD and is representative of the orientation of the wireless hand-held input controller derived from the IMU of the wireless hand-held input controller;

computing at the processor of the HMD second orientation data representative of the orientation of the wireless hand-held input controller, wherein the second orientation data is computed based on pose data detected by the forward facing optical sensor of the HMD; and

computing at the processor of the HMD a confidence level by comparing the first orientation data with the second orientation data.

16. The method of claim 15 further comprising creating an association between the user'"'"'s hand and the wireless hand-held input controller if the confidence level meets or exceeds a predetermined minimum threshold.

17. The method of claim 16 further comprising:

periodically monitoring the confidence level of the association between the user'"'"'s hand and the associated wireless hand held input controller;

if the confidence level remains above the predetermined minimum threshold, persisting the association; and

if the confidence level drops below the predetermined minimum threshold, removing the association.

18. A system used for augmented reality (AR) or full virtual reality (VR) in which a head-mounted display device (HMD) is aligned with a wireless hand-held controller in a manner so that the HMD and wireless hand-held controller are rotationally invariant relative to one another, the system comprising:

a head-mounted display device (HMD) comprising;

an on-board processor mounted to the HMD;

a wearable stereoscopic display adapted for displaying one of an augmented reality (AR) environment or a full virtual reality (VR) environment;

an optical sensor system that produces location data at a particular instant of time for the wireless hand-held controller, wherein the optical sensor system comprises;

an outward facing optical sensor that senses a field of view of the HMD within the environment such that location data is obtained from the outward facing optical sensor, and wherein the location data is defined by x, y and z coordinates; and

a position sensor system comprising one or more sensors for determining orientation data of the HMD and wherein the orientation data for the HMD is defined by pitch (elevation angle), yaw (azimuth angle) and roll (rotation) relative to a predetermined frame of reference;

a wireless hand-held controller comprising;

an inertial measurement unit (IMU) comprising one or more sensors for determining orientation data of the wireless hand-held controller and wherein the orientation data of the wireless hand-held controller is defined by pitch (elevation angle), yaw (azimuth angle) and roll (rotation) relative to the particular frame of reference; and

a microcontroller that communicates to the on-board processor of the HMD one or more user inputs relative to the orientation data of the wireless handheld controller; and

wherein the on-board processor of the HMD performs at least the following;

processes the orientation data at the particular instant of time as provided by the IMU of the wireless hand-held controller and the location data at the particular instant of time as provided by the optical sensor system of the HMD to determine location and orientation of the wireless hand-held controller in reference to the HMD with six degrees of freedom as derived from the x, y and z coordinates determined at the HMD and the yaw, pitch and roll determined at the IMU of the wireless hand-held controller; and

processes the orientation data at the particular instant of time as provided by the position sensor system of the HMD and the orientation data at the particular instant of time as provided by the IMU of the wireless hand-held controller such that the HMD and the wireless hand-held controller are gravity aligned and north aligned so as to be rotationally invariant.