Modular extension of inertial controller for six DOF mixed reality input

• US 10,146,335 B2
• Filed: 09/23/2016
• Issued: 12/04/2018
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru Octavian, Dulu, Constantin, Edmonds, Christopher Douglas, Finocchio, Mark James
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US15/274,387
• Publication Number:
  US 20170357334A1
• Time in Patent Office: 802 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

A modular holding fixture for selectively coupling to a wireless hand-held inertial controller to provide passive optical and inertial tracking in a slim form-factor for use with a head mounted display that operates with six degrees of freedom by fusing (i) data related to the position of the controller derived from a forward-facing depth camera located in the head mounted display with (ii) data relating to the orientation of the controller derived from an inertial measurement unit located in the controller.

Claims

1. An apparatus for holding a wireless hand-held input controller being used in a three dimensional (3D) environment for augmented reality or for virtual reality, the apparatus comprising:

a housing comprising;

a first end and a second end; and

a central coupling portion located between said first and second ends, and wherein the wireless hand-held input controller and the central coupling portion are configured relative to one another so that the wireless hand-held input controller is able to be removably attached to the central coupling portion;

a first optically reflective marker at or proximate the first end of the housing;

a second optically reflective marker at or proximate the second end of the housing; and

wherein the central coupling portion is positioned between the first and second optically reflective markers and wherein the position is defined by a fixed geometrical relationship between the central coupling portion and each of the first and second optically reflective markers, such that optical tracking of the wireless hand-held input controller when located in the 3D environment is enhanced when coupled to the central coupling portion, an enhancement occurring based on a determination of the position of the central coupling portion relative to the location of the first and second optically reflective markers as tracked in the 3D environment.

2. The apparatus of claim 1, wherein the first and second optically reflective markers are each spherical.

3. The apparatus of claim 2, wherein the first and second spherical optically reflective markers are each trackable in the 3D environment using an optical sensor system comprising an infrared depth camera mounted on a head-mounted display device (HMD).

4. The apparatus of claim 3 wherein, the first and second ends of the housing form an elongated stylus that establishes a sufficient baseline separating the first and second spherical optically reflective markers so that at least one of the first and second spherical optically reflective markers is visible by the infrared depth camera across a range of hand orientations used to hold the elongated stylus.

5. The apparatus of claim 4 wherein, the elongated stylus comprises a pistol grip attached to the elongated stylus adjacent the central coupling portion.

6. The apparatus of claim 3, wherein the HMD comprises a processor mounted to the HMD, and wherein the wireless hand-held input controller comprises an inertial measurement unit (IMU) comprising one or more sensors for determining orientation data of the wireless hand-held input controller, and a microprocessor that communicates to the processor of the HMD inputs derived from the IMU relative to the orientation data.

7. The apparatus of claim 6, wherein the processor of the HMD uses the orientation data as provided by the IMU of the wireless hand-held input controller and the location data of the wireless hand-held input controller as determined from the locations of the first and second optically reflective markers by the optical sensor system of the HMD to determine location and orientation of the wireless hand-held input controller in reference to the HMD with six degrees of freedom.

8. The apparatus of claim 7, wherein the HMD further comprises a position sensor system mounted to the HMD, the position sensor system comprising one or more sensors for determining orientation data of the HMD.

9. The apparatus of claim 8, wherein the processor of the HMD processes the orientation data as provided by the position sensor system of the HMD and the orientation data as provided by the IMU of the wireless hand-held input controller such that the HMD and wireless hand-held input controller are gravity aligned and north aligned.

10. The apparatus of claim 6, wherein the wireless hand-held input controller further comprises a wireless radio, one or more momentary switches and one or more status LED(s).

11. A system comprising:

a head-mounted display device (HMD) comprising;

a processor;

a wearable stereoscopic display adapted for displaying one of an augmented reality (AR) environment or a virtual reality (VR) environment; and

an optical sensor system that produces location data at a particular instant of time for a wireless hand-held input controller, wherein the location data is defined by x, v and z coordinates;

the wireless hand-held input controller comprising;

an inertial measurement unit (IMU) comprising one or more sensors for determining orientation data of the wireless hand-held input controller and wherein the orientation data is defined by pitch (elevation angle), yaw (azimuth angle) and roll (rotation) relative to a predetermined frame of reference; and

a microcontroller that communicates to the processor of the HMD one or more user inputs relative to the orientation data; and

a holding fixture comprising;

a housing comprising;

a first end and a second end; and

a central coupling portion located between said first and second ends, and wherein the wireless hand-held input controller and the central coupling portion are configured relative to one another so that the wireless hand-held input controller is able to be removably attached to the central coupling portion;

a first optically reflective marker at or proximate the first end of the housing;

a second optically reflective marker at or proximate the second end of the housing;

wherein the location of the central coupling portion is between the first and second optically reflective markers and is defined by a fixed geometrical relationship between the central coupling portion and each of the optically reflective markers; and

wherein the processor of the HMD uses the orientation data as provided by the IMU of the wireless hand-held input controller and the location data of the wireless hand-held input controller as determined by the processor of the HMD from the locations of the first and second optically reflective markers by the optical sensor system of the HMD to determine location and orientation of the wireless hand-held input controller in reference to the HMD with six degrees of freedom.

12. The system of claim 11, wherein the first and second optically reflective markers are each spherical and wherein the first and second spherical optically reflective markers are each trackable in the 3D environment using an infrared depth camera of the optical sensor system mounted on the head-mounted display device (HMD).

13. The system of claim 12, wherein the first and second ends of the housing form an elongated stylus that establishes a sufficient baseline separating the first and second spherical optically reflective markers so that at least one of the first and second spherical optically reflective markers is visible by the infrared depth camera across a range of hand orientations used to hold the elongated stylus.

14. The system of claim 13, wherein the elongated stylus comprises a pistol grip attached to the elongated stylus adjacent the central coupling portion.

15. The system of claim 12, wherein the HMD further comprises a position sensor system mounted to the HMD, the position sensor system comprising one or more sensors for determining orientation data of the HMD.

16. The system of claim 15 wherein the processor of the HMD processes the orientation data as provided by the position sensor system of the HMD and the orientation data as provided by the IMU of the wireless hand-held input controller such that the HMD and wireless hand-held input controller are gravity aligned and north aligned.

17. The system of claim 16 wherein the wireless hand-held input controller further comprises a wireless radio, one or more momentary switches and one or more status LED(s).

18. A holding fixture for use with a wireless hand-held controller that includes a microcontroller, an inertial measurement unit, a communications radio and one or more buttons configured and positioned for selective actuation with a finger or thumb of a user, the holding fixture comprising:

a housing having a first end and a second end;

a first optically reflective marker at or proximate the first end of the housing;

a second optically reflective marker at or proximate the second end of the housing; and

a docking portion located on the housing between the first optically reflective marker and the second optically reflective marker, the docking portion having a size and shape that is complementary of the size and shape of the wireless hand-held controller and is configured to selectively couple the wireless hand-held controller to the holding fixture in a fixed spatial relationship relative to the first optically reflective marker and the second optically reflective marker.

19. The holding fixture of claim 18, wherein the first and second optically reflective markers are each spherical and comprise an IR transmissive material that blocks light in the visible spectrum but allows light in the IR range to pass through the material in order to allow the first and second spherical optically reflective markers to be hidden from view without limiting IR detection of the first and second spherical optically reflective markers, and wherein the first and second spherical optically reflective markers are each trackable in the 3D environment using an infrared depth camera mounted on a head-mounted display device (HMD).

20. The holding fixture of claim 19, wherein the first and second ends of the housing form an elongated stylus that establishes a sufficient baseline separating the first and second spherical optically reflective markers so that at least one of the first and second spherical optically reflective markers is visible by the infrared depth camera across a range of hand orientations used to hold the elongated stylus.