ROBUST OPTICAL DISAMBIGUATION AND TRACKING OF TWO OR MORE HAND-HELD CONTROLLERS WITH PASSIVE OPTICAL AND INERTIAL TRACKING

• US 20170358139A1
• Filed: 09/23/2016
• Published: 12/14/2017
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Balan, Alexandru Octavian, Dulu, Constantin
• Granted Patent: US 10,249,090 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Methods for disambiguation and tracking of two or more wireless hand-held controllers with passive optical and inertial tracking within a system having a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with wireless hand-held inertial controllers for providing user input to the display device, with each controller two passive optically reflective markers, one marker being position at or adjacent each end of the controller and being separated by a known distance, and each controller also including an onboard inertial measurement unit for providing inertial data corresponding to its orientation.

Claims

1. In a system comprising a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with a wireless hand-held inertial controller for providing user input to the display device, the controller having at least two passive optically reflective markers, one marker being positioned at or adjacent each end of the controller and being separated by a known distance, and the controller also including an onboard inertial measurement unit (IMU) for providing inertial data corresponding to its orientation, a method for disambiguation and tracking of the two passive optically reflective markers by passive optical and inertial tracking, the method comprising acts of:

activating the controller;

establishing a wireless connection between the controller and the display device and assigning a unique controller ID to the controller; and

localizing and identifying each marker.

2. The method of claim 1, wherein the act of localizing and identifying each marker comprises:

temporal tracking, by the optical sensor, to generate location data representative of the location of each marker; and

inertial tracking, by the IMU, to generate orientation data representative of the orientation of the controller;

uniquely identifying each marker based on the location data and the orientation data; and

associating a unique marker ID with each marker.

3. The method of claim 2 further comprising an act for creating an association between each unique marker ID and the unique controller ID.

4. The method of claim 3, wherein the act of uniquely identifying each marker comprises one or more of the following acts:

upon detecting two markers located within the field of view of the display device, computing the distance separating the two markers; and

comparing the computed distance separating the detected markers with the known distance.

5. The method of claim 3, wherein the act of uniquely identifying each marker comprises one or more of the following acts:

detecting, based on the inertial data received from the IMU, vertical tilt data representative of the vertical tilt of the controller;

detecting, by the optical sensor, position data representative of the position of each marker relative to the other marker to identify which one of the markers is located at a point that is higher than the other one of the markers; and

comparing the vertical tilt data with the position data to uniquely identify each marker.

6. The method of claim 3, wherein the act of uniquely identifying each marker comprises one or more of the following acts:

tracking, by the optical sensor, movement of each marker during a specified period of time to generate location data representative of the location and movement of each marker;

tracking, by the IMU sensor, movement of the controller during the specified period of time to generate orientation data representative of the movement of each marker; and

comparing the location data with the orientation data to uniquely identify each marker.

7. The method of claim 3, wherein the act of uniquely identifying each marker comprises one or more of the following acts:

tracking, by the optical sensor, movement of each marker during a specified period of time to generate first acceleration data representative of the movement of each marker;

tracking, by the IMU sensor, movement of the controller during the specified period of time to generate second acceleration data representative of the movement of each marker; and

comparing the first acceleration data with the second acceleration data to uniquely identify each marker.

8. The method of claim 3, wherein the act of uniquely identifying each marker comprises one or more of the following acts:

utilizing acceleration data sampled over a period of time to predict a future location of one of the markers; and

comparing a detected location of the marker with the predicted location of the marker.

9. In a system comprising a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with a wireless hand-held inertial controller for providing user input to the display device, the controller having at least two passive optically reflective markers, one marker being positioned at or adjacent each end of the controller and being separated by a known distance, and the controller also including an onboard inertial measurement unit (IMU) for providing inertial data corresponding to its orientation, a method for disambiguation and tracking of the two passive optically reflective markers by passive optical and inertial tracking, the method comprising acts of:

establishing a wireless connection between the controller and the display device and assigning a unique controller ID to the controller;

locating, by the optical sensor, each marker within the field of view of the display device;

uniquely identifying each marker by performing one or more of the following acts;

upon detecting two markers located within the field of view of the display device, computing the distance separating the two markers;

comparing the computed distance separating the detected markers to the known distance; and

if the computed distance is approximately equal to the known distance, creating an association between the two markers and the controller.

10. The method of claim 9, wherein the act of uniquely identifying each marker further comprises:

detecting, based on the inertial data received from the IMU, vertical tilt data representative of the vertical tilt of the controller;

detecting, by the optical sensor, position data representative of the position of each marker relative to the other marker to identify which one of the markers is located at a point that is higher than the other one of the markers;

comparing the vertical tilt data with the position data; and

assigning a first energy value proportional to the correspondence or lack of correspondence between the vertical tilt data with the position data.

11. The method of claim 10, wherein the act of uniquely identifying each marker further comprises one or more of the following acts:

tracking, by the optical sensor, movement of each marker during a specified period of time to generate location data representative of the location and movement of each marker;

tracking, by the IMU sensor, movement of the controller during the specified period of time to generate orientation data representative of the movement of each marker;

comparing the location data with the orientation data; and

assigning a second energy value proportional to the correspondence or lack of correspondence between the location data and the orientation data.

12. The method of claim 11, wherein the act of uniquely identifying each marker further comprises the following acts:

tracking, by the optical sensor, movement of each marker during a specified period of time to generate first acceleration data representative of the movement of each marker;

tracking, by the IMU sensor, movement of the controller during the specified period of time to generate second acceleration data representative of the movement of each marker;

comparing the first acceleration data with the second acceleration data; and

assigning a third energy value proportional to the correspondence or lack of correspondence between the first acceleration data with the second acceleration data.

13. The method of claim 12, wherein the act of uniquely identifying each marker further comprises the following acts:

utilizing acceleration data sampled over a period of time to predict a future location of one of the markers; and

comparing a detected location of the marker with the predicted location of the marker.

14. The method of claim 13 further comprising an act for assigning a fourth energy value proportional to the correspondence or lack of correspondence between the predicted location and the detected location.

15. The method of claim 14 further comprising acts:

summing the first, second, third and fourth energy values; and

comparing the sum to a predetermined threshold value.

16. The method of claim 15 further comprising an act for establishing a wireless connection between the controller and the display device.

17. The method of claim 16 further comprising an act for assigning a unique controller ID to the controller.

18. The method of claim 17 further comprising an act for assigning a unique marker ID to each of the markers.

19. The method of claim 18 further comprising an act for creating an association between unique controller ID and each of the unique marker IDs.

20. In a system comprising a head mounted virtual or augmented reality display device having a forward facing optical sensor having a field of view, and wherein the display device interfaces with wireless hand-held inertial controllers for providing user input to the display device, with each controller having at least two passive optically reflective markers, one marker being positioned at or adjacent each end of the controller and being separated by a known distance, and each controller also including an onboard inertial measurement unit (IMU) for providing inertial data corresponding to its orientation, a method for disambiguation and tracking of two or more wireless hand-held controllers with passive optical and inertial tracking, the method comprising acts of:

for each controller, establishing a wireless connection with the display device and assigning a unique controller ID to each controller;

locating, by the optical sensor, each marker within the field of view of the display device;

uniquely identifying each marker by performing one or more of the following acts;

computing the distance separating each marker from each other marker, comparing each such distance to the known distance, and assigning an energy value to each possible marker pair that is proportional to the correspondence or lack thereof;

for each possible marker pair, comparing the vertical pitch between each such marker as detected by the optical system and with the vertical pitch of each controller based on its inertial data and assigning an energy value proportional to the correspondence or lack thereof;

for each possible marker pair, comparing the horizontal azimuth angle between each such marker as detected by the optical system and with the horizontal azimuth angle of each controller based on its inertial data and assigning an energy value proportional to the correspondence or lack thereof;

comparing any movement of each marker as detected by the optical sensor during a specified period of time, against inertial data for each controller indicating acceleration of such controller during the same period of time and assigning an energy value proportional to the correspondence or lack thereof; and

using the energy values to uniquely identify each marker, create associations between specific markers and/or create associations between specific markers and specific controllers.