Image and point cloud based tracking and in augmented reality systems

• US 10,657,708 B1
• Filed: 05/04/2018
• Issued: 05/19/2020
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Jurgenson, Nathan, Luo, Linjie, Rodriguez, II, Jonathan M, Sheth, Rahul, Li, Jia, Lv, Xutao
• Primary Examiner(s): Lhymn, Sarah
• Application Number:
  US15/971,566
• Time in Patent Office: 746 Days
• Field of Search: 345418, 345419, 345420, 345619
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering

Abstract

Systems and methods for image based location estimation are described. In one example embodiment, a first positioning system is used to generate a first position estimate. A 3D point cloud data describing an environment is then accessed. A first image of an environment is captured, and a portion of the image is matched to a portion of key points in the 3D point cloud data. An augmented reality object is then aligned within one or more images of the environment based on the match of the 3D point cloud with the image. In some embodiments, building façade data may additionally be used to determine a device location and place the augmented reality object within an image.

Claims

1. A method for reducing augmented reality perspective position error comprising:

accessing three-dimensional (3D) point cloud data describing an environment associated with a client device and a first position estimate for an image sensor of a companion device associated with the client device;

accessing a first image of the environment captured by the image sensor of the companion device, wherein the companion device is separate from the client device and associated with a different location than the first position estimate;

processing the first image to match at least a portion of a set of key points of the 3D point cloud to the first image;

determining, based on the match of the portion of the set of key points of the 3D point cloud to the first image, a position error associated with the first position estimate along with a second position estimate for the image sensor of the companion device;

generating a model of a virtual object within the 3D point cloud; and

generating a first augmented reality image comprising the virtual object in the environment using the second position estimate for the client device, the model of the virtual object within the 3D point cloud, and the match of the portion of the set of key points of the 3D point cloud to the first image.

2. The method of claim 1 further comprising:

communicating the first position estimate and the first image together as part of a first communication from the client device to a cloud server computer;

wherein the processing of the image to match at least a portion of a set of key points of the 3D point cloud to the first image and the determining of the second position estimate are performed by the cloud server computer.

3. The method of claim 2 further comprising:

tracking, at the client device, motion of the companion device;

receiving, at the client device from the cloud server computer, the second position estimate; and

generating, at the client device, using the second position estimate and the motion of the companion device from an image capture time to a second position receipt time, a third position estimate;

wherein the first augmented reality image is further generated using the third position estimate to align the virtual object within a second image of the environment.

4. The method of claim 2 further comprising:

accessing, by the client device, a set of structure facade data describing one or more structure facades associated with the environment;

identifying, using the set of structure facade data, a first structure facade portion of the first image of the environment, wherein the first structure facade portion matches first structure facade data of the set of structure facade data; and

calculating, based at least in part on the first structure facade portion of the first image of the environment, a third position estimate of the device;

wherein the first augmented reality image is further generated using the third position estimate along with the second position estimate to align the virtual object within the first image of the environment.

5. The method of claim 4 wherein the set of structure facade data is received at the client device from the cloud server computer with the second position estimate.

6. The method of claim 5 further comprising:

capturing, at the companion device, a plurality of video frames of the environment;

for each video frame of the plurality of video frames;

generating an updated position estimate based on at least one structure facade match with the set of structure facade data; and

positioning the virtual reality object within the video frame based on the updated position estimate;

generate a corresponding augmented reality image comprising the virtual reality object and a corresponding portion of the environment; and

outputting the corresponding augmented reality image on an output display of the companion device.

7. The method of claim 6 further comprising:

receiving a compressed 3D cloud point model from the cloud server computer with the second position estimate;

wherein for each video frame, the positioning of the virtual reality object is further performed based on a video frame match with the compressed 3D cloud point model.

8. The method of claim 1 wherein the processing of the image to match at least a portion of a set of key points of the 3D point cloud to the first image and the determining of the second position estimate for the client device are performed by the client device.

9. The method of claim 8 further comprising:

accessing, from the memory of the client device, a set of structure facade data describing one or more structure facades associated with the environment;

identifying by the client device, using the processor and the set of structure facade data, a first structure facade portion of the first image of the environment, wherein the first structure facade portion matches first structure facade data of the set of structure facade data; and

calculating, by the client device based at least in part on the first structure facade portion of the first image of the environment, a third position estimate of the client device;

wherein the first augmented reality image is further generated using the third position estimate along with the second position estimate to align the virtual object within the first image of the environment.

10. The method of claim 9, further comprising, prior to the accessing of the structure facade data:

capturing a plurality of model images of the environment;

capturing a model location for each of the plurality of model images of the environment; and

processing the plurality of model images and the model location for each of the plurality of model images to generate a database of facade data comprising the first structure facade data and to generate the 3D point cloud of the environment;

wherein the first structure facade data as processed comprises one or more facade surface locations and a two-dimensional facade pattern.

11. The method of claim 9, further comprising:

receiving, at the client device, in response to the first position estimate, the set of structure facade data, wherein the first structure facade data of the set of structure facade data comprises one or more facade surface locations and a two-dimensional facade pattern;

calculating, from the first position estimate and the one or more facade surface locations, an expected perspective for the two-dimensional facade pattern;

processing the two-dimensional facade pattern to generate a first perspective pattern;

processing the first image of the environment using the first perspective pattern to identify the first structure facade portion.

12. The method of claim 11, wherein calculating, based at least in part on a perspective of the first structure facade portion of the first image of the environment, the second position estimate of the client device, comprises:

calculating an error value between the first perspective pattern and the first structure facade portion;

determining a relative location value associated with the error value; and

determining the second position estimate from the first position estimate using the relative location value.

13. A device comprising:

a memory; and

one or more processors coupled to the memory and configured to;

access three-dimensional (3D) point cloud data describing an environment associated with the device, a companion device associated with the device, and a first position estimate for an image sensor of the companion device;

access a first image of the environment captured by the image sensor of the companion device, wherein the companion device is separate from the device and associated with a different location than the first position estimate;

process the first image to match at least a portion of a set of key points of the 3D point cloud to the first image;

determine, based on the match of the portion of the set of key points of the 3D point cloud to the first image, a position error associated with the first position estimate and a second position estimate for the image sensor of the companion device;

generate a model of a virtual object within the 3D point cloud; and

generate a first augmented reality image comprising the virtual object in the environment using the second position estimate for the device, the model of the virtual object within the 3D point cloud, and the match of the portion of the set of key points of the 3D point cloud to the first image.

14. The device of claim 13 wherein the one or more processors are further configured to match the portion of the set of key points of the 3D point cloud to the first image by processing the first image to identify one or more key features of the first image, and performing a 3D matching process to search for a match between the key features of the first image and the portion of the set of key points of the 3D point cloud data.

15. The device of claim 13 wherein the one or more processors are further configured to receive the first image from an image sensor of the companion device, and wherein the companion device comprises a first wearable device.

16. The device of claim 15 wherein the first wearable device comprises a device selected from the set of:

an augmented reality helmet, an augmented reality visor, augmented reality glasses, and an augmented reality glasses attachment.

17. The device of claim 16 wherein the device comprises a smartphone.

18. A non-transitory computer readable medium comprising instructions that, when performed by a device, cause the device to perform a method comprising:

accessing three-dimensional (3D) point cloud data describing an environment associated with the device, a companion device associated with the device, and a first position estimate for an image sensor of the companion device;

accessing a first image of the environment captured by the image sensor of the companion device, wherein the companion device is separate from the device and associated with a different location than the first position estimate;

processing the first image to match at least a portion of a set of key points of the 3D point cloud to the first image;

determining, based on the match of the portion of the set of key points of the 3D point cloud to the first image, a position error associated with the first position estimate and a second position estimate for the image sensor of the companion device;

generating a model of a virtual object within the 3D point cloud; and

generating a first augmented reality image comprising the virtual object in the environment using the second position estimate for the device, the model of the virtual object within the 3D point cloud, and the match of the portion of the set of key points of the 3D point cloud to the first image.

19. The non-transitory computer readable medium of claim 18 wherein the instructions further cause the device to perform the method comprising:

accessing a set of structure facade data describing one or more structure facades associated with the environment;

identifying, using the set of structure facade data, a first structure facade portion of the first image of the environment, wherein the first structure facade portion matches first structure facade data of the set of structure facade data; and

calculating, based at least in part on the first structure faç

ade portion of the first image of the environment, third position estimate of the device;

wherein the first augmented reality image is further generated using the third position estimate along with the second position estimate to align the virtual object within the first image of the environment.

20. The non-transitory computer readable medium of claim 18 wherein the instructions further cause the device to perform the method comprising:

capturing, at the companion device, a plurality of video frames of the environment;

for each video frame of the plurality of video frames;

generating an updated position estimate based on at least one structure facade match with the set of structure facade data;

positioning the virtual reality object within the video frame based on the updated position estimate;

generate a corresponding augmented reality image comprising the virtual reality object and a corresponding portion of the environment; and

outputting the corresponding augmented reality image on an output display of the companion device.