Mixed reality controller and headset tracking system

• US 10,679,378 B1
• Filed: 01/23/2019
• Issued: 06/09/2020
• Est. Priority Date: 03/13/2017
• Status: Active Grant

• Current Assignee: Xrpro
• Original Assignee: Occipital, Inc.
• Inventors: Reddy, Vikas M., Powers, Jeffrey Roger, Schroeder, Paul Jakob, Slaughter, Chris, Mallya, Ganesh
• Primary Examiner(s): Couso, Jose L
• Application Number:
  US16/254,824
• Time in Patent Office: 503 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models

Abstract

A system configured to determine a six-degree of freedom pose of a physical object in a physical environment and to utilize the six-degree of freedom pose as an within a virtual environment or mixed reality environment. In some cases, the system may utilize one or more cameras on a headset device to track the pose of a controller or other objects and one or more cameras on the controller itself to track the pose of the headset device or the user. In one example, the system may capture image data of a physical object having a constellation or pattern on the external source. The system may analyze the image data to identify image points associated with the constellation or pattern and to determine the pose of the object based on a location of the points in the image.

Claims

1. A system comprising:

a display for presenting a virtual environment to a user;

one or more monochromic image components for capturing monochromic image data associated with a physical environment surrounding the user;

one or more processors;

non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to;

receive a monochromic image from the one or more monochromic image components, the monochromic image including data representatives of an object in the physical environment;

identify image points within the monochromic image;

generate a plurality of image point sets, each image point set including four or more image points;

generate a candidate 6DOF pose for individual ones of the image point sets;

project each of the candidate 6DOF poses into the monochromic image;

determine a number of inliers for each of the candidate 6DOF poses; and

output the candidate 6DOF pose having the largest number of inliers as the 6DOF pose of the object.

2. The system as recited in claim 1, wherein the non-transitory computer-readable media stores additional computer-executable instructions, which when executed by the one or more processors cause the one or more processors to:

segment the monochromic image into a plurality of patches;

identify using a low-resolution search a set of patches containing an image point; and

determine a location of the image point within each patch by applying a weighted average via a sliding window at a high resolution.

3. The system as recited in claim 1, wherein the non-transitory computer-readable media stores additional computer-executable instructions, which when executed by the one or more processors cause the one or more processors to:

receiving orientation data of the object from the object;

determine for each of the candidate 6DOF poses a feasibility metric based at least in part on the orientation data of the object; and

discard, prior to projecting the candidate 6DOF poses, at least one of the candidate 6DOF poses, the discarded candidate 6DOF pose having a feasibility metric below a threshold.

4. The system as recited in claim 1, wherein the non-transitory computer-readable media stores additional computer-executable instructions, which when executed by the one or more processors cause the one or more processors to:

render the controller based at least in part on the candidate 6DOF pose within the virtual environment presented on the display.

5. The system as recited in claim 1, wherein determining the number of inliers for each candidate 6DOF pose includes determining a number of projected points of the candidate 6DOF pose within a threshold pixel distance of an image point of the image points.

6. The system as recited in claim 1, wherein the non-transitory computer-readable media stores additional computer-executable instructions, which when executed by the one or more processors cause the one or more processors to:

determine a reprojection error associated with each candidate 6DOF pose; and

wherein the candidate 6DOF pose having the largest number of inliers and a lowest reprojection error is output as the 6DOF pose of the object.

7. The system as recited in claim 1, wherein the object is a hand-held controller associated with the system.

14. The system as recited in claim 1, wherein the non-transitory computer-readable media stores additional computer-executable instructions, which when executed by the one or more processors cause the one or more processors to:

generating a refined 6DOF pose for each of the plurality of candidate 6DOF poses by applying a perspective-n-point (PNP) operation to the candidate 6DOF pose.

8. A system comprising:

a display for presenting a virtual environment to a user;

one or more monochromic image components for capturing image data associated with a physical environment surrounding the user;

one or more wireless communication interfaces for receiving data;

one or more processors;

non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to;

receive a monochrome image from the one or more monochromic image components, the image including data representatives of an object in the physical environment;

determine an initial 6DOF pose prediction of the object;

generate an initial plurality of candidate 6DOF poses based at least in part on the initial 6DOF pose prediction;

receive orientation data of the object;

determine for each pose of the initial plurality of candidate 6DOF poses a feasibility metric based at least in part on the orientation data of the object; and

discard at least one of the initial plurality of candidate 6DOF poses to generate a remaining plurality of candidate 6DOF poses, the discarded candidate 6DOF pose having a feasibility metric below a threshold;

project each candidate 6DOF pose of the remaining plurality of candidate 6DOF poses into the monochrome image;

determine a number of inliers for each candidate 6DOF pose of the remaining plurality of candidate 6DOF poses; and

output the candidate 6DOF pose of the remaining plurality of candidate 6DOF poses having the largest number of inliers as the 6DOF pose of the object.

9. The system as recited in claim 8, wherein the orientation data of the object is received from the object via the one or more wireless communication interfaces.

10. The system as recited in claim 8, wherein the initial 6DOF pose prediction is based at least in part on a known prior 6DOF pose of the object.

11. The system as recited in claim 8, wherein generating the plurality of candidate 6DOF poses includes perturbing the initial 6DOF pose along three angles and three translations.

12. The system as recited in claim 8, wherein the non-transitory computer-readable media stores additional computer-executable instructions, which when executed by the one or more processors cause the one or more processors to:

manipulate a second object in the virtual environment based at least in part on the 6DOF pose of the first object.

13. The system as recited in claim 8, wherein determine the number of inliers for each candidate 6DOF pose of the remaining plurality of candidate 6DOF poses includes determining a number of projected points of the candidate 6DOF poses of the remaining plurality of candidate 6DOF poses that are within a threshold distance of an image point of the image points.

15. A system comprising:

a display for presenting a virtual environment to a user;

one or more image components for capturing image data associated with a physical environment surrounding the user;

one or more wireless communication interfaces for receiving data;

one or more processors;

non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to;

receive an image from the one or more image components, the image including data representatives of an object in the physical environment;

receive orientation data from the object;

approximate a rotation of the object in relationship to a previous known position of the object based at least in part on the orientation data;

approximate a depth of the object in relationship to a known position of the system based at least in part on the image and a stored model of the object;

determine a translation of the object in relationship to the previous known position the object based at least in part on a prior 6DOF pose, the rotation, and the depth;

determine 6DOF pose of the object based at least in part on the translation, the rotation, and the depth; and

output the 6DOF pose of the object.

16. The system as recited in claim 15, wherein approximating the depth is based at least in part on the orientation data.

17. The system as recited in claim 15, wherein approximating the depth is based at least in part on determining a weak perspective using the stored model and a length of the object within the image.

18. The system as recited in claim 15, wherein approximating the depth is based at least in part on an edge length of the stored model and an edge length of the object within the image.

19. The system as recited in claim 15, wherein the image components include monochrome image components and the image is a monochrome image.

20. The system as recited in claim 15, wherein the object is a controller associated with the system.