Pose tracking system with physical tracking enhancement tags

• US 10,636,170 B1
• Filed: 01/23/2019
• Issued: 04/28/2020
• Est. Priority Date: 03/13/2017
• Status: Active Grant

• Current Assignee: Xrpro
• Original Assignee: Occipital, Inc.
• Inventors: Reddy, Vikas M., Powers, Jeffrey Roger, Schroeder, Paul Jakob, Slaughter, Chris, Mallya, Ganesh
• Primary Examiner(s): Couso, Jose L
• Application Number:
  US16/255,089
• Time in Patent Office: 461 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models

Abstract

A virtual environment or mixed reality environment system including a physical tracking enhancement tag that may be identified by the system to assist with localization of the system with respect to the physical environment. The tracking enhancement tags may be encoded or coded in a high frequency and low frequency that is readable by the system to provide different types of information related the environment. For example, the encoded information may include a unique identifier of the tag and a relative direction with regards to the physical environment.

Claims

1. A method comprising:

generating a virtual scene map representative of a physical environment based at least in part on image data captured by an image device of a headset device and orientation data captured by an inertial measurement unit (IMU) of the headset device;

determining a six-degree of freedom (6DOF) pose of the headset device relative to the virtual scene map based at least in part on the orientation data and the image data, wherein the virtual scene map is generated and the 6DOF pose of the headset device relative to the virtual scene map is determined using a simultaneous location and mapping (SLAM) technique;

determining a 6DOF pose of a tracking enhancement tag relative to the headset device based at least in part on the image data, the tracking enhancement tag within the physical environment;

determining a 6DOF pose of the tracking enhancement tag relative to the virtual scene map based at least in part on the 6DOF pose of the headset device relative to the virtual scene map and the 6DOF pose of the tracking enhancement tag relative to the headset device; and

locating the tracking enhancement tag within the virtual scene map based at least in part on the 6DOF pose of the tracking enhancement tag relative to the virtual scene map.

2. The method as recited in claim 1, wherein the 6DOF pose of the tracking enhancement tag relative to the headset device is determined using object tracking.

3. The method as recited in claim 1, wherein the 6DOF pose of the tracking enhancement tag relative to the virtual scene map is determined by composing the 6DOF pose of the headset device relative to the virtual scene map with the 6DOF pose of the tracking enhancement tag relative to the headset device.

4. The method as recited in claim 1, further comprising:

re-initializing the virtual scene map on the headset device;

capturing, by the image device, second image data of the physical environment, the second image data including data representative of the tracking enhancement tag; and

determining a second 6DOF pose of the headset device relative to the scene map based at least in part on second image data and the 6DOF pose of the tracking enchantment tag relative to the virtual scene.

5. The method as recited in claim 4, wherein the second 6DOF pose of the headset device relative to the scene map is determined by:

determining a 6DOF pose of the tracking enhancement tag relative to the headset device using object tracking; and

composing the 6DOF pose of the tracking enhancement tag relative to the headset device with the 6DOF pose of the tracking enchantment tag relative to the virtual scene map.

6. The method as recited in claim 1, wherein the tracking enhancement tag includes at least one of:

a retroreflective element;

an infrared coded element;

a frequency encoded element;

a patterned element;

or

a light emitting element.

7. A system comprising:

one or more image components for capturing image data associated with a physical environment surrounding the user, the physical environment including a tracking enhancement tag;

one or more inertial measurement units (IMU) for capturing orientation data associated with the system;

one or more processors;

non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to;

accessing a virtual scene map representative of a physical environment including a tracking enhancement tag;

determining the identity of the tracking enhancement tag from image data including a representation of the tracking enhancement tag;

determining a position of the tracking enhancement tag relative to the virtual scene map based at least in part on the identity;

determining a position of the tracking enhancement tag relative to the system based at least in part on a six-degree of freedom (6DOF) pose of the system relative to the virtual scene map and the position of the tracking enhancement tag relative to the virtual scene map;

determining a 6DOF pose of the system relative to the physical environment based at least in part on the position of the tracking enhancement tag relative to the system;

re-initializing the virtual scene map;

capturing, by the one or more image components, second image data of the physical environment, the second image data including data representative of the tracking enhancement tag; and

determining a second 6DOF pose of the system relative to the scene map based at least in part on second image data and the position of the tracking enchantment tag relative to the virtual scene.

8. The system as recited in claim 7, wherein determining the position of the tracking enhancement tag relative to the system includes composing the 6DOF pose of the system relative to the virtual scene map and the position of the tracking enhancement tag relative to the virtual scene map.

18. The system as recited in claim 7, wherein the position of the tracking enhancement tag is a 6DOF pose of the tracking enhancement tag relative to the virtual scene map.

19. The system as recited in claim 7, wherein the position of the tracking enhancement tag relative to the headset device is determined using object tracking.

20. The system as recited in claim 7, wherein the second 6DOF pose of the system relative to the scene map is determined based at least in part on a visual indication on the tracking enhancement tag, the visual indication indicating an upright position with respect to the physical environment.

9. A non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform actions comprising:

generating a virtual scene map representative of a physical environment based at least in part on image data captured by an image device of a headset device and orientation data captured by an inertial measurement unit (IMU) of the headset device;

determining a six-degree of freedom (6DOF) pose of the headset device relative to the virtual scene map based at least in part on the orientation data and the image data;

determining a 6DOF pose of a tracking enhancement tag relative to the headset device based at least in part on the image data, the tracking enhancement tag within the physical environment;

determining a 6DOF pose of the tracking enhancement tag relative to the virtual scene map based at least in part on the 6DOF pose of the headset device relative to the virtual scene map and the 6DOF pose of the tracking enhancement tag relative to the headset device;

locating the tracking enhancement tag within the virtual scene map based at least in part on the 6DOF pose of the tracking enhancement tag relative to the virtual scene map;

re-initializing the virtual scene map on the headset device;

capturing, by the image device, second image data of the physical environment, the second image data including data representative of the tracking enhancement tag; and

determining a second 6DOF pose of the headset device relative to the scene map based at least in part on second image data and the 6DOF pose of the tracking enchantment tag relative to the virtual scene.

10. The non-transitory computer-readable media as recited in claim 9, wherein the virtual scene map is generated and the 6DOF pose of the headset device relative to the virtual scene map is determined using a simultaneous location and mapping (SLAM) technique.

11. The non-transitory computer-readable media as recited in claim 9, wherein the 6DOF pose of the tracking enhancement tag relative to the headset device is determined using object tracking.

12. The non-transitory computer-readable media as recited in claim 9, wherein the 6DOF pose of the tracking enhancement tag relative to the virtual scene map is determined by composing the 6DOF pose of the headset device relative to the virtual scene map with the 6DOF pose of the tracking enhancement tag relative to the headset device.

13. The non-transitory computer-readable media as recited in claim 9, wherein the second 6DOF pose of the headset device relative to the scene map is determined by:

determining a 6DOF pose of the tracking enhancement tag relative to the headset device using object tracking; and

composing the 6DOF pose of the tracking enhancement tag relative to the headset device with the 6DOF pose of the tracking enchantment tag relative to the virtual scene map.

14. The non-transitory computer-readable media as recited in claim 9, wherein the second 6DOF pose of the headset device relative to the scene map is determined based at least in part on a visual indication on the tracking enhancement tag, the visual indication indicating an upright position with respect to the physical environment.

15. The non-transitory computer-readable media as recited in claim 14, wherein the visual indication is encoded in the infrared spectrum.

16. The non-transitory computer-readable media as recited in claim 14, wherein the visual indication is the shape of the tracking enhancement tag.

17. The non-transitory computer-readable media as recited in claim 9, wherein the second 6DOF pose of the headset device relative to the scene map is determined based at least in part on an unique identifier associated with the tracking enhancement tag.