TRACKING POSE OF HANDHELD OBJECT

• US 20190318501A1
• Filed: 04/16/2018
• Published: 10/17/2019
• Est. Priority Date: 04/16/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: BALAN, Alexandru Octavian, NAGARAJ, Yogeshwar Narayanan, DULU, Constantin, GUYMAN, William, RAZUMENIC, Ivan
• Granted Patent: US 10,740,924 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/405 being a wireless ad hoc network, e.g. Bluetooth, Wi-Fi, Pico net
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Examples are disclosed that relate to tracking a pose of a handheld object used with a head-mounted display device. In one example, a method comprises: receiving image data from an image sensing system; detecting a plurality of feature points of the handheld object in a frame of the image data; receiving inertial measurement unit (IMU) data from an IMU of the handheld object; based on detecting the plurality of feature points and receiving the IMU data, determining a first pose of the handheld object; determining that at least a portion of the plurality of feature points is not detected in another frame of the image data; using the IMU data, updating the first pose of the handheld object to a second pose; and body-locking the second pose of the handheld object to a body location on a user wearing the head-mounted display device.

Claims

1. At a computing device, a method for tracking a pose of a handheld object used with a head-mounted display device, the method comprising:

receiving image data from an image sensing system;

detecting a plurality of feature points of the handheld object in a frame of the image data;

receiving inertial measurement unit (IMU) data from an IMU of the handheld object;

based on detecting the plurality of feature points and receiving the IMU data, determining a first pose of the handheld object;

determining that at least a portion of the plurality of feature points is not detected in another frame of the image data;

using the IMU data, updating the first pose of the handheld object to a second pose; and

body-locking the second pose of the handheld object to a body location on a user wearing the head-mounted display device.

2. The method of claim 1, wherein body-locking the second pose of the handheld object further comprises:

determining that a first threshold time has been reached;

determining that the handheld object is not stationary; and

based on determining that the first threshold time has been reached and that the handheld object is not stationary, body-locking the second pose of the handheld object to the body location on the user wearing the head-mounted display device.

3. The method of claim 2, further comprising:

determining that a second threshold time has been reached after expiration of the first threshold time; and

based on determining that the first threshold time has been reached, the handheld object is not stationary, and the second threshold time has been reached, body-locking the second pose of the handheld object to the body location on the user wearing the head-mounted display device.

4. The method of claim 2, further comprising:

determining that the handheld object is stationary after determining that the handheld object is not stationary; and

based on determining that the handheld object is stationary after determining that the handheld object is not stationary, using the IMU data to update an orientation of the second pose of the handheld object, and marking a position of the second pose as invalid.

5. The method of claim 1, further comprising:

prior to body-locking the second pose of the handheld object, determining that a velocity of the handheld object is less than a threshold velocity; and

based on determining that the velocity of the handheld object is less than the threshold velocity, body-locking the second pose of the handheld object to the body location on the user wearing the head-mounted display device.

6. The method of claim 1, further comprising:

determining that the handheld object is stationary; and

based on determining that the handheld object is stationary, world-locking the second pose of the handheld object.

7. The method of claim 1, wherein the body location on the user wearing the head-mounted display device comprises a neck or an elbow of the user.

8. The method of claim 1, further comprising detecting additional feature points in one or more additional frames of the image data;

and

based on detecting the additional feature points, ceasing to body-lock the second pose of the handheld object and updating the second pose to a third pose.

9. The method of claim 1, wherein the pose of the handheld object comprises a position and an orientation, and body-locking the second pose of the handheld object comprises locking the position and not locking the orientation.

10. The method of claim 1, wherein the image sensing system is mounted on the head-mounted display device.

11. A computing device for tracking a pose of a handheld object used with a head-mounted display device, the computing device comprising:

a logic device configured to execute instructions; and

a storage device comprising instructions executable by the logic device to;

receive image data from an image sensing system;

detect a plurality of feature points of the handheld object in a frame of the image data;

receive inertial measurement unit (IMU) data from an IMU of the handheld object;

based on detecting the plurality of feature points and receiving the IMU data, determine a first pose of the handheld object;

determine that at least a portion of the plurality of feature points is not detected in another frame of the image data;

using the IMU data, update the first pose of the handheld object to a second pose; and

body-lock the second pose of the handheld object to a body location on a user wearing the head-mounted display device.

12. The computing device of claim 11, wherein the instructions are executable to:

determine that a first threshold time has been reached;

determine that the handheld object is not stationary; and

based on determining that the first threshold time has been reached and that the handheld object is not stationary, body-lock the second pose of the handheld object to the body location on the user wearing the head-mounted display device.

13. The computing device of claim 12, wherein the instructions are executable to:

determine that a second threshold time has been reached after expiration of the first threshold time; and

based on determining that the first threshold time has been reached, the handheld object is not stationary, and the second threshold time has been reached, body-lock the second pose of the handheld object to the body location on the user wearing the head-mounted display device.

14. The computing device of claim 12, wherein the instructions are executable to:

determine that the handheld object is stationary after determining that the handheld object is not stationary; and

based on determining that the handheld object is stationary after determining that the handheld object is not stationary, use the IMU data to update an orientation of the second pose of the handheld object, and marking a position of the second pose as invalid.

15. The computing device of claim 11, wherein the instructions are executable to:

prior to body-locking the second pose of the handheld object, determine that a velocity of the handheld object is less than a threshold velocity; and

based on determining that the velocity of the handheld object is less than the threshold velocity, body-lock the second pose of the handheld object to the body location on the user wearing the head-mounted display device.

16. The computing device of claim 11, wherein the instructions are executable to:

determine that the handheld object is stationary; and

based on determining that the handheld object is stationary, world-lock the second pose of the handheld object.

17. The computing device of claim 11, wherein the body location on the user wearing the head-mounted display device comprises a neck or an elbow of the user.

18. The computing device of claim 11, wherein the instructions are executable to:

detect additional feature points in one or more additional frames of the image data; and

based on detecting the additional feature points in the image data, cease body-locking the second pose of the handheld object and update the second pose to a third pose.

19. The computing device of claim 11, wherein the pose of the handheld object comprises a position and an orientation, and body-locking the second pose of the handheld object comprises locking the position and not locking the orientation.

20. A head-mounted display device, comprising:

an image sensing system; and

a computing device comprising;

a logic device configured to execute instructions; and

a storage device comprising instructions executable by the logic device to;

receive image data from the image sensing system;

detect a plurality of feature points of the handheld object in a frame of the image data;

receive inertial measurement unit (IMU) data from an IMU of the handheld object;

based on detecting the plurality of feature points and receiving the IMU data, determine a first pose of the handheld object;

determine that at least a portion of the plurality of feature points is not detected in another frame of the image data;

using the IMU data, update the first pose of the handheld object to a second pose;

determine that the handheld object is not stationary;

determine that a first threshold time has been reached; and

based on determining that the first threshold time has been reached and that the handheld object is not stationary, body-lock the second pose of the handheld object to a body location on a user wearing the head-mounted display device.