Image based tracking in augmented reality systems

• US 10,733,802 B2
• Filed: 06/11/2019
• Issued: 08/04/2020
• Est. Priority Date: 10/30/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Jurgenson, Nathan, Luo, Linjie, Rodriguez, II, Jonathan M, Sheth, Rahul, Li, Jia, Lv, Xutao
• Primary Examiner(s): Seth, Manav
• Application Number:
  US16/438,226
• Publication Number:
  US 20190295326A1
• Time in Patent Office: 420 Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/07 Target detection

Abstract

Systems and methods for image based location estimation are described. In one example embodiment, a first positioning system is used to generate a first position estimate. A set of structure façade data describing one or more structure façades associated with the first position estimate is then accessed. A first image of an environment is captured, and a portion of the image is matched to part of the structure façade data. A second position is then estimated based on a comparison of the structure façade data with the portion of the image matched to the structure façade data.

Claims

1. A method for determining a device location comprising:

accessing, by one or more processors of the device, building location information describing one or more location points for at least a first building, wherein the building location information is associated with a first position estimate of the device;

processing, using the one or more processors, a first image of an environment comprising the first building using the building location information to identify a match between a first portion of the first image of the environment and the one or more location points of the building location information; and

calculating, using the one or more processors, a second position estimate of the device, wherein the second position estimate is calculated based at least in part on a perspective associated with the first portion of the first image of the environment and the match between the first portion and the one or more location points.

2. The method of claim 1, further comprising, prior to the accessing of the building location information:

capturing a plurality of model images of the environment;

capturing a model location for each of the plurality of model images of the environment; and

processing the plurality of model images and the model location for each of the plurality of model images to generate a database of building location points comprising the one or more location points for the building;

wherein the one or more location points as processed describe one or more faç

ade surface locations and a two-dimensional faç

ade pattern.

3. The method of claim 1, further comprising:

receiving, at the device, in response to the first position estimate, the building location information, wherein the one or more location points of the building location information comprises one or more faç

ade surface locations and a two-dimensional faç

ade pattern.

4. The method of claim 3, wherein processing the first image of the environment comprises:

calculating, from the first position estimate and the one or more faç

ade surface locations, an expected perspective for the two-dimensional faç

ade pattern;

processing the two-dimensional faç

ade pattern to generate a first perspective pattern;

processing the first image of the environment using the first perspective pattern to identify a first surface.

5. The method of claim 4, wherein calculating the second position estimate of the device, comprises:

calculating an error value between the first perspective pattern and the first surface of the first image;

determining a relative location value associated with the error value; and

determining the second position estimate from the first position estimate using the relative location value.

6. The method of claim 5, further comprising:

accessing a set of augmented reality image data associated with an augmented reality image item;

capturing, by the image sensor, a second image of the environment;

calculating a position for the augmented reality image item using the second position estimate;

generating an augmented reality image using the second image, the set of augmented reality image data, and the second position estimate; and

outputting the augmented reality image on a display of the device.

7. The method of claim 4, wherein identifying the first surface of the first image comprises:

calculating, from the first position estimate and the one or more faç

ade surface locations, an expected perspective for the two-dimensional faç

ade pattern;

processing the two-dimensional faç

ade pattern to generate a first perspective pattern;

determining that the first perspective pattern does not meet a threshold match level for the first image;

requesting a supplemental set of faç

ade data from a first server computer; and

receiving the supplemental set of faç

ade data from the first server computer;

wherein the identifying of the first surface is based on the supplemental set of faç

ade data.

8. The method of claim 7, wherein requesting the supplemental set of faç

ade data from the first server computer comprises communicating the first image of the environment to the first server computer; and

wherein receiving the supplemental set of faç

ade data comprises receiving a supplemental position estimate from the first server computer based on the first image.

9. The method of claim 8, further comprising:

capturing a second image of the environment;

identifying a first surface of the second image; and

calculating a perspective difference between the first surface of the second image and the first surface of the first image; and

generating a third position estimate using the second position estimate and the perspective difference.

10. The method of claim 7 further comprising:

receiving, at the device, two-dimensional augmented reality data for a sticker object;

determining an augmented realty placement of the sticker object as positioned on a plane with the one or more faç

ade surface locations;

using the expected perspective for the two-dimensional faç

ade pattern as a sticker object perspective; and

generating an augmented reality image comprising the sticker object on the plane with the one or more faç

ade surface locations based on the expected perspective.

11. The embodiment of claim 10 wherein the two-dimensional augmented reality data represents an emoji object selected by a user input from a set of augmented reality system pictograms.

12. The method of claim 1, further comprising:

identifying, using the building location information, a second surface of the first image of the environment;

wherein the second surface matches second location points of the building location information;

wherein the second position estimate is further calculating based at least in part on a second perspective of the second surface of the first image of the environment.

13. The method of claim 12, further comprising:

updating the building location information using the first image and the second image to adjust the one or more faç

ade surface locations and the two-dimensional faç

ade pattern.

14. The method of claim 13, further comprising:

communicating at least a portion of the first image and at least a portion of the second image to the first server computer with the second location estimate and the third location estimate.

15. A device comprising:

a memory; and

one or more processors coupled to the memory, the one or more processors configured to;

access building location information describing one or more location points for at least a first building, wherein the building location information is associated with a first position estimate of the device;

process a first image of an environment comprising the first building using the building location information to identify a match between a first portion of the first image of the environment and the one or more location points of the building location information; and

calculate a second position estimate of the device, wherein the second position estimate is calculated based at least in part on a perspective associated with the first portion of the first image of the environment and the match between the first portion and the one or more location points.

16. The device of claim 15, wherein the one or more processors are further configured to:

manage receipt, in response to the first position estimate, of the building location information, wherein the one or more location points of the building location information comprises one or more faç

ade surface locations and a two-dimensional faç

ade pattern.

17. The device of claim 16, wherein the one or more processors are configured to process the first image of the environment with operations to:

calculate, from the first position estimate and the one or more faç

ade surface locations, an expected perspective for the two-dimensional faç

ade pattern;

process the two-dimensional faç

ade pattern to generate a first perspective pattern; and

process the first image of the environment using the first perspective pattern to identify the first surface.

18. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a device, cause the device to perform operations comprising:

accessing building location information describing one or more location points for at least a first building, wherein the building location information is associated with a first position estimate of the device;

processing a first image of an environment comprising the first building using the building location information to identify a match between a first portion of the first image of the environment and the one or more location points of the building location information; and

calculating, using the one or more processors, a second position estimate of the device, wherein the second position estimate is calculated based at least in part on a perspective associated with the first portion of the first image of the environment and the match between the first portion and the one or more location points.

19. The non-transitory computer readable medium of claim 18 wherein the instructions further cause the device to perform operations comprising:

managing receipt at the device, in response to the first position estimate, of the building location information, wherein the one or more location points of the building location information comprises one or more faç

ade surface locations and a two-dimensional faç

ade pattern;

wherein processing the first image of the environment comprises;

calculating, from the first position estimate and the one or more faç

ade surface locations, an expected perspective for the two-dimensional faç

ade pattern;

processing the two-dimensional faç

ade pattern to generate a first perspective pattern;

processing the first image of the environment using the first perspective pattern to identify a first surface; and

wherein calculating the second position estimate of the device, comprises;

calculating an error value between the first perspective pattern and the first surface of the first image;

determining a relative location value associated with the error value; and

determining the second position estimate from the first position estimate using the relative location value.

20. The non-transitory computer readable medium of claim 18 wherein identifying the first surface of the first image comprises:

calculating, from the first position estimate and the one or more faç

ade surface locations, an expected perspective for the two-dimensional faç

ade pattern;

processing the two-dimensional faç

ade pattern to generate a first perspective pattern;

determining that the first perspective pattern does not meet a threshold match level for the first image;

requesting a supplemental set of faç

ade data from a first server computer; and

receiving the supplemental set of faç

ade data from the first server computer;

wherein the identifying of the first surface is based on the supplemental set of faç

ade data;

wherein requesting the supplemental set of faç

ade data from the first server computer comprises communicating the first image of the environment to the first server computer; and

wherein receiving the supplemental set of faç

ade data comprises receiving a supplemental position estimate from the first server computer based on the first image.