Robust optical disambiguation and tracking of two or more hand-held controllers with passive optical and inertial tracking

• US 10,249,090 B2
• Filed: 09/23/2016
• Issued: 04/02/2019
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru Octavian, Dulu, Constantin
• Primary Examiner(s): Nguyen, Anh-Tuan V
• Application Number:
  US15/274,361
• Publication Number:
  US 20170358139A1
• Time in Patent Office: 921 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Methods for disambiguation and tracking of two or more wireless hand-held controllers with passive optical and inertial tracking within a system having a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with wireless hand-held inertial controllers for providing user input to the display device, with each controller two passive optically reflective markers, one marker being position at or adjacent each end of the controller and being separated by a known distance, and each controller also including an onboard inertial measurement unit for providing inertial data corresponding to its orientation.

Claims

1. In a system comprising a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with at least two wireless hand-held inertial controllers, for providing user input to the display device, each of the at least two controllers having at least two passive optically reflective markers, with one marker being positioned at or adjacent to each end of the respective controller and being separated by a known distance, and each controller also including an onboard inertial measurement unit (IMU) for providing inertial data corresponding to its orientation, a method for disambiguation and tracking of the passive optically reflective markers by passive optical and inertial tracking, the method comprising at least:

activating the at least two controllers;

establishing a wireless connection between the at least two controllers and the display device and assigning a unique controller ID to each controller;

locating, by the optical sensor, each marker within the field of view of the display device;

computing a distance separating each marker from each other marker, comparing each such distance to the known distance, and assigning an energy value to each possible marker pair that is proportional to a correspondence or lack thereof determined from the comparison; and

localizing and uniquely identifying each marker, including at least;

for each possible marker pair, comparing a vertical tilt between each such marker as detected by the optical system with a vertical tilt of each controller, based on its inertial data, and assigning a confidence value proportional to the correspondence or lack thereof;

for each possible marker pair, comparing a horizontal azimuth angle between each such marker as detected by the optical system and with a horizontal azimuth angle of each controller, based on its inertial data, and assigning a confidence value proportional to the correspondence or lack thereof;

comparing any movement of each marker as detected by the optical sensor during a specified period of time, against inertial data for each controller indicating acceleration of such controller during the same period of time and assigning a confidence value proportional to the correspondence or lack thereof; and

using the confidence values to uniquely identify each marker, create associations between specific markers, and/or create associations between specific markers and specific controllers.

2. The method of claim 1, wherein localizing and identifying each marker comprises:

temporal tracking, by the optical sensor, to generate location data representative of the location of each marker; and

inertial tracking, by the IMU, to generate orientation data representative of the orientation of the controller;

uniquely identifying each marker based on comparing the location data and the orientation data; and

associating a unique marker ID with each marker.

3. The method of claim 2 further comprising creating an association between each unique marker ID and the unique controller ID.

4. The method of claim 3, wherein uniquely identifying each marker further comprises one or more of the following:

tracking, by the optical sensor, movement of each marker during a specified period of time to generate location data representative of the location and movement of each marker;

tracking, by the IMU sensor, movement of the controller during the specified period of time to generate orientation data representative of the movement of each marker;

or

comparing the location data with the orientation data to uniquely identify each marker.

5. The method of claim 3, wherein uniquely identifying each marker further comprises one or more of the following:

tracking, by the optical sensor, movement of each marker during a specified period of time to generate first acceleration data representative of the movement of each marker;

tracking, by the IMU sensor, movement of the controller during the specified period of time to generate second acceleration data representative of the movement of each marker;

or

comparing the first acceleration data with the second acceleration data to uniquely identify each marker.

6. The method of claim 3, wherein uniquely identifying each marker comprises one or more of the following:

utilizing acceleration data sampled over a period of time to predict a future location of one of the markers;

or

comparing a detected location of the marker with the predicted location of the marker.

7. In a system comprising a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with at least two wireless hand-held inertial controllers for providing user input to the display device, each of the at least two controllers having at least two passive optically reflective markers, with one marker being positioned at or adjacent each end of the respective controller and being separated by a known distance, and each controller also including an onboard inertial measurement unit (IMU) for providing inertial data corresponding to its orientation, a method for disambiguation and tracking of the passive optically reflective markers by passive optical and inertial tracking, the method comprising at least:

establishing a wireless connection between the at least two controllers and the display device and assigning a unique controller ID to each controller;

locating, by the optical sensor, each marker within the field of view of the display device;

computing a distance separating each marker from each other marker, comparing each such distance to the known distance, and assigning an energy value to each possible marker pair that is proportional to a correspondence of the comparison;

uniquely identifying each marker by performing at least;

for each possible marker pair, comparing a vertical tilt between each such marker as detected by the optical system with a vertical tilt of each controller, based on its inertial data, and assigning a confidence value proportional to the correspondence of the vertical tilt comparison;

for each possible marker pair, comparing a horizontal azimuth angle between each such marker as detected by the optical system with a horizontal azimuth angle of each controller, based on its inertial data, and assigning a confidence value proportional to the correspondence of the horizontal azimuth angle comparison;

comparing any movement of each marker as detected by the optical sensor during a specified period of time, against inertial data for each controller indicating acceleration of such controller during the same period of time and assigning a confidence value proportional to the correspondence of the acceleration comparison; and

using the confidence values to uniquely identify each marker, create associations between specific markers, and/or create associations between specific markers and specific controllers.

8. The method of claim 7, wherein uniquely identifying each marker further comprises:

utilizing acceleration data sampled over a period of time to predict a future location of one of the markers; and

comparing a detected location of the marker with the predicted location of the marker.

9. The method of claim 8 further comprising assigning an energy value proportional to the correspondence or lack of correspondence between the predicted location and the detected location.

10. The method of claim 7 further comprising assigning a unique marker ID to each of the markers.

11. The method of claim 10 further comprising creating an association between unique controller ID and each of the unique marker IDs.

12. In a system comprising a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with two or more wireless hand-held inertial controllers for providing user input to the display device, each controller having at least two passive optically reflective markers, one marker being positioned at or adjacent each end of the controller and being separated by a known distance, and each controller also including an onboard inertial measurement unit (IMU) for providing inertial data corresponding to its orientation, a method for disambiguation and tracking of the two or more wireless hand-held controllers with passive optical and inertial tracking, the method comprising at least:

for each controller, establishing a wireless connection with the display device and assigning a unique controller ID to each controller;

locating, by the optical sensor, each marker within the field of view of the display device;

computing a distance separating each marker from each other marker, comparing each such distance to the known distance, and assigning an energy value to each possible marker pair that is proportional to a correspondence or lack thereof determined from the comparison;

uniquely identifying each marker by performing at least the following;

for each possible marker pair, comparing a vertical tilt between each such marker as detected by the optical system with a vertical tilt of each controller based on its inertial data and assigning a confidence value proportional to the correspondence or lack thereof;

for each possible marker pair, comparing a horizontal azimuth angle between each such marker as detected by the optical system with a horizontal azimuth angle of each controller based on its inertial data and assigning a confidence value proportional to the correspondence or lack thereof;

comparing any movement of each marker as detected by the optical sensor during a specified period of time, against inertial data for each controller indicating acceleration of such controller during the same period of time and assigning a confidence value proportional to the correspondence or lack thereof; and

using the confidence values to uniquely identify each marker, create associations between specific markers, and/or create associations between specific markers and specific controllers.