Environmental mapping for augmented reality

• US 10,665,029 B2
• Filed: 10/10/2018
• Issued: 05/26/2020
• Est. Priority Date: 10/10/2018
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Yeung, Jason A., Goslin, Michael P., Drake, Corey D., Baumbach, Elliott H., Barone, Nicholas F.
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): Imperial, Jed-Justin
• Application Number:
  US16/156,191
• Publication Number:
  US 20200118337A1
• Time in Patent Office: 594 Days
• Field of Search: None
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Embodiments provide for reconciling a first map of an environment created by a first device with a second map of the environment created by a second device by determining a first confidence score for a first location of a given object in the first map; determining a second confidence score for a second location of the given object in the second map; inserting the given object into a reconciled map at a position based on the first confidence score and the second confidence score; anchoring an Augmented Reality object in the reconciled map at coordinates based on the position of the given object; and outputting the reconciled map to an Augmented Reality device.

Claims

1. A method of reconciling a first environmental map of an environment created by a first device with a second environmental map of the environment created by a second device, the method comprising:

identifying one real-world object represented with first positional data in the first environmental map and with second positional data in the second environmental map;

determining a first confidence score for a first location of the one real-world object in the first environmental map, and a second confidence score for a second location of the one real-world object in the second environmental map;

reconciling the first environmental map with the second environmental map into a reconciled environmental map based on the first positional data and the second positional data;

anchoring a reconciled AR object associated with the one real-world object in the reconciled environmental map based on the first positional data and the first confidence score relative to the second positional data and the second confidence score; and

outputting the reconciled environmental map to an AR device, wherein the reconciled AR object is located closer to the first location than the second location in the reconciled environmental map when the first confidence score is higher than the second confidence score.

2. The method of claim 1, further comprising:

wherein the first positional data is provided from a first point of view and includes a first partial mesh of the one real-world object and the second positional data is provided from a second point of view and includes a second partial mesh of the one real-world object; and

wherein reconciling the first environmental map with the second environmental map includes applying the first positional data and the second positional data to a shared skeleton for the reconciled AR object.

3. The method of claim 2, wherein the first positional data and the second positional data each include:

a location of the one real-world object in the environment;

an orientation of the one real-world object in the environment; and

positions of various components of the one real-world object in the environment.

4. The method of claim 1, wherein the first environmental map is created at a first time and the second environmental map is created at a second time, and wherein a difference in the first confidence score and the second confidence score is based at least in part on a difference between the first time and the second time.

5. The method of claim 1, wherein a difference in the first confidence score and the second confidence score is based at least in part on a difference between a first distance between the first device and the one real-world object when the first environmental map is created and a second distance between the second device and the one real-world object when the second environmental map is created.

6. The method of claim 1, wherein the first environmental map includes an area not included in the second environmental map, wherein the reconciled environmental map includes the area.

7. A method, comprising:

constructing, by a first video device, a first map of an environment based on a first field of view of the first video device;

retrieving a second map of the environment produced by a second video device based on a second field of view of the second video device, different from the first field of view;

identifying, by the first video device, a third field of view in the environment for a third device of an AR (Augmented Reality) device, wherein the third field of view is different from both the first field of view and the second field of view;

reconciling the first map with the second map to produce a reconciled map, wherein objects identified at a shared position in the environment in both the first map and the second map are included in the reconciled map at the shared position, and objects identified at different positions in the first map and the second map are included in the reconciled map at a given position specified in one of the first map and the second map based on which of the first map and the second map is associated with a higher confidence score for the given position;

anchoring an AR object at a given position in the reconciled map; and

in response to the third field of view including the given position, outputting the AR object in a video feed produced by the AR device.

8. The method of claim 7, wherein reconciling the first map with the second map to produce a reconciled map further comprises:

identifying a first object in the first map and a second object in the second map that respectively represent a first partial mesh and a second partial mesh of one real-world object;

applying first positional data from the first partial mesh and second positional data from the second partial mesh to a shared skeleton for the one real-world object; and

applying the shared skeleton to the AR object anchored in the reconciled map to represent the one real-world object.

9. The method of claim 7, wherein the AR object is inserted into the video feed to highlight a given object absent from the second map and present in the first map.

10. The method of claim 7, wherein the AR object is overlaid into the video feed in a line-of-sight to the AR device over an obstructing object in the environment.

11. The method of claim 7, wherein the second video device is a virtual video device and the second map is a virtual map of the environment.

12. The method of claim 7, wherein a confidence score for a first position of a given object is based on factors selected from a group comprising:

a speed of travel of the first video device when determining a relative distance to a particular object in the environment.

13. The method of claim 12, wherein the confidence score for the first position of the given object is based on:

whether a weather condition was present when determining the relative distance to the particular object.

14. The method of claim 7, wherein the confidence score for the first position of the given object is based on:

whether a given location was determined by measurement or estimation by the first video device.

15. The method of claim 7, wherein the AR object is output at a given scale based on a distance between the third device and the given position.

16. A method of reconciling a first environmental map of an environment created by a first video device with a second environmental map of the environment created by a second video device, the method comprising:

receiving a video feed of the environment from the first video device having a real-world object in a field of view of the first video device, the video feed providing first positional data on the real-world object, wherein the first positional data are associated with a first confidence score and a first location;

determining second positional data for the real-world object from the second environmental map, wherein the second positional data are associated with a second confidence score and a second location, different than the first location;

anchoring an Augmented Reality (AR) object that represents the real-world object into a reconciled environmental map at the second position when the first confidence score is lower than the second confidence score; and

outputting the video feed for display with the AR object overlaid onto the video feed when the position is in the field of view of the first video device.

17. The method of claim 16, wherein the AR object is overlaid onto the video feed to indicate that the real-world object is absent from the field of view and present in the second environmental map.

18. The method of claim 16, wherein the first positional data and the second positional data each include:

a location for the real-world object in the environment;

an orientation for the real-world object in the environment; and

a pose of the real-world object.

19. The method of claim 16, wherein the first positional data provide a first partial mesh of the real-world object and the second positional data provide a second partial mesh of the real-world object that are reconciled into the AR object via a shared skeleton.

20. The method of claim 16, wherein the first positional data are associated with a first confidence score and the second positional data are associated with a second confidence score, wherein the position is set based on an average of the first positional data weighted by the first confidence score and the second positional data weighted by the second confidence score.