Online reference generation and tracking for multi-user augmented reality

• US 9,558,557 B2
• Filed: 03/19/2015
• Issued: 01/31/2017
• Est. Priority Date: 09/09/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Jiang, Bolan, Swaminathan, Ashwin, Spindola, Serafin Diaz, Ahuja, Dheeraj, Sweet, Charles Wheeler III, Choudhury, Prasun
• Primary Examiner(s): Li, Tracy Y
• Assistant Examiner(s): Rahman, Mohammad J
• Application Number:
  US14/662,555
• Publication Number:
  US 20150193935A1
• Time in Patent Office: 684 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A multi-user augmented reality (AR) system operates without a previously acquired common reference by generating a reference image on the fly. The reference image is produced by capturing at least two images of a planar object and using the images to determine a pose (position and orientation) of a first mobile platform with respect to the planar object. Based on the orientation of the mobile platform, an image of the planar object, which may be one of the initial images or a subsequently captured image, is warped to produce the reference image of a front view of the planar object. The reference image may be produced by the mobile platform or by, e.g., a server. Other mobile platforms may determine their pose with respect to the planar object using the reference image to perform a multi-user augmented reality application.

Claims

1. A method comprising:

receiving a first image of a planar object captured by a first mobile platform from the first mobile platform;

receiving a second image of the planar object captured by a second mobile platform from the second mobile platform;

determining a first pose, comprising a position and an orientation, of the first mobile platform with respect to the planar object using the first and second images;

warping the first image of the planar object using the orientation of the first mobile platform with respect to the planar object to produce a reference image of a front view of the planar object; and

transmitting the reference image to the first and second platforms, wherein the second platform determines a second pose of the second platform with respect to the planar object based at least in part on the reference image and the second image captured by the second platform, wherein determining the first pose, comprising the position and the orientation, of the first mobile platform with respect to the planar object comprises;

extracting features from the first and second images;

computing a 2D-2D homography between the first and second images using the features;

computing 3D coordinates for the features to define a 3D plane that corresponds to a plane of the planar object;

computing a 3D-2D homography between the 3D plane and one of the first and second images using the features and the 3D plane; and

using the 3D-2D homography to determine the first pose.

2. The method of claim 1, further comprising performing an augmented reality application with the second mobile platform based on the second pose of the second mobile platform with respect to the planar object.

3. The method of claim 1, wherein the second platform determines the second pose of the second platform based at least in part on the reference image and the second image of the planar object captured by the second platform by performing a method comprising:

extracting features from the reference image;

defining 3D points on a 3D plane that correspond to the features extracted from the reference image;

finding points in the second image of the planar object captured by the second mobile platform that match with 2D projections of the 3D points; and

using matched 3D-2D point pairs to estimate the second pose of the second mobile platform.

4. The method of claim 3, wherein the features extracted from the reference image comprise point features or line features.

5. The method of claim 1, further comprising:

receiving a new image captured by the second mobile platform from the second mobile platform that includes a portion of the planar object that is not included in the reference image of the front view of the planar object; and

merging the new image and the reference image to generate a new reference image of the front view of the planar object; and

transmitting the new reference image to the first and second mobile platforms, wherein the second mobile platform determines an updated second pose of the second mobile platform with respect to the planar object using a subsequent image of the planar object captured by the second mobile platform and the new reference image.

6. The method of claim 5, wherein merging the new image and the reference image to generate the new reference image of the front view of the planar object comprises transforming pixels of the reference image to the new reference image through an identity transformation and transforming pixels of the new image to the new reference image.

7. The method of claim 6, wherein transforming pixels of the new image to the new reference image is performed using at least one of a computed homography or affine transformation.

8. An apparatus comprising:

means for receiving a first image of a planar object captured by a first mobile platform from the first mobile platform;

means for receiving a second image of the planar object captured by a second mobile platform from the second mobile platform;

means for determining a first pose, comprising a position and an orientation, of the first mobile platform with respect to the planar object using the first and second images;

means for warping the first image of the planar object using the orientation of the first mobile platform with respect to the planar object to produce a reference image of a front view of the planar object; and

means for transmitting the reference image to the first and second platforms, wherein the second platform determines a second pose of the second platform with respect to the planar object based at least in part on the reference image and the second image captured by the second platform, wherein the means for determining the first pose, comprising the position and the orientation, of the first mobile platform with respect to the planar object comprises;

means for extracting features from the first and second images;

means for computing a 2D-2D homography between the first and second images using the features;

means for computing 3D coordinates for the features to define a 3D plane that corresponds to a plane of the planar object;

means for computing a 3D-2D homography between the 3D plane and one of the first and second images using the features and the 3D plane; and

means for using the 3D-2D homography to determine the first pose.

9. The apparatus of claim 8, further comprising means for performing an augmented reality application with the second mobile platform based on the second pose of the second mobile platform with respect to the planar object.

10. The apparatus of claim 8, wherein the second platform determines the second pose of the second platform based at least in part on the reference image and the second image of the planar object captured by the second platform by performing a method comprising:

extracting features from the reference image;

defining 3D points on a 3D plane that correspond to the features extracted from the reference image;

finding points in the second image of the planar object captured by the second mobile platform that match with 2D projections of the 3D points; and

using matched 3D-2D point pairs to estimate the second pose of the second mobile platform.

11. The apparatus of claim 10, wherein the features extracted from the reference image comprise point features or line features.

12. The apparatus of claim 8, further comprising:

means for receiving a new image captured by the second mobile platform from the second mobile platform that includes a portion of the planar object that is not included in the reference image of the front view of the planar object; and

means for merging the new image and the reference image to generate a new reference image of the front view of the planar object; and

means for transmitting the new reference image to the first and second mobile platforms, wherein the second mobile platform determines an updated second pose of the second mobile platform with respect to the planar object using a subsequent image of the planar object captured by the second mobile platform and the new reference image.

13. The apparatus of claim 12, wherein the means for merging the new image and the reference image to generate the new reference image of the front view of the planar object comprises means for transforming pixels of the reference image to the new reference image through an identity transformation and means for transforming pixels of the new image to the new reference image.

14. The apparatus of claim 13, wherein transforming pixels of the new image to the new reference image is performed using at least one of a computed homography or affine transformation.

15. A server apparatus comprising:

a memory; and

a processor coupled to the memory, the processor to;

receive a first image of a planar object captured by a first mobile platform from the first mobile platform;

receive a second image of the planar object captured by a second mobile platform from the second mobile platform;

determine a first pose, comprising a position and an orientation, of the first mobile platform with respect to the planar object using the first and second images;

warp the first image of the planar object using the orientation of the first mobile platform with respect to the planar object to produce a reference image of a front view of the planar object; and

transmit the reference image to the first and second platforms, wherein the second platform determines a second pose of the second platform with respect to the planar object based at least in part on the reference image and the second image captured by the second platform, wherein to determine the first pose, comprising the position and the orientation, of the first mobile platform with respect to the planar object, the processor is to;

extract features from the first and second images;

compute a 2D-2D homography between the first and second images using the features;

compute 3D coordinates for the features to define a 3D plane that corresponds to a plane of the planar object;

compute a 3D-2D homography between the 3D plane and one of the first and second images using the features and the 3D plane; and

use the 3D-2D homography to determine the first pose.

16. The server apparatus of claim 15, wherein the second mobile platform performs an augmented reality application based on the second pose of the second mobile platform with respect to the planar object.

17. The server apparatus of claim 15, wherein the second platform determines the second pose of the second platform based at least in part on the reference image and the second image of the planar object captured by the second platform by performing a method comprising:

extracting features from the reference image;

defining 3D points on a 3D plane that correspond to the features extracted from the reference image;

finding points in the second image of the planar object captured by the second mobile platform that match with 2D projections of the 3D points; and

using matched 3D-2D point pairs to estimate the second pose of the second mobile platform.

18. The server apparatus of claim 17, wherein the features extracted from the reference image comprise point features or line features.

19. The server apparatus of claim 15, wherein the processor is further to:

receive a new image captured by the second mobile platform from the second mobile platform that includes a portion of the planar object that is not included in the reference image of the front view of the planar object; and

merge the new image and the reference image to generate a new reference image of the front view of the planar object; and

transmit the new reference image to the first and second mobile platforms, wherein the second mobile platform determines an updated second pose of the second mobile platform with respect to the planar object using a subsequent image of the planar object captured by the second mobile platform and the new reference image.

20. The server apparatus of claim 19, wherein to merge the new image and the reference image to generate the new reference image of the front view of the planar object, the processor is to transform pixels of the reference image to the new reference image through an identity transformation and transform pixels of the new image to the new reference image.

21. A non-transitory computer-readable medium comprising code which, when executed by a processor, causes the processor to perform a method comprising:

receiving a first image of a planar object captured by a first mobile platform from the first mobile platform;

receiving a second image of the planar object captured by a second mobile platform from the second mobile platform;

determining a first pose, comprising a position and an orientation, of the first mobile platform with respect to the planar object using the first and second images;

warping the first image of the planar object using the orientation of the first mobile platform with respect to the planar object to produce a reference image of a front view of the planar object; and

transmitting the reference image to the first and second platforms, wherein the second platform determines a second pose of the second platform with respect to the planar object based at least in part on the reference image and the second image captured by the second platform, wherein the code for determining the first pose, comprising the position and the orientation, of the first mobile platform with respect to the planar object comprises code for;

extracting features from the first and second images;

computing a 2D-2D homography between the first and second images using the features;

computing 3D coordinates for the features to define a 3D plane that corresponds to a plane of the planar object;

computing a 3D-2D homography between the 3D plane and one of the first and second images using the features and the 3D plane; and

using the 3D-2D homography to determine the first pose.

22. The non-transitory computer-readable medium of claim 21, further comprising code for performing an augmented reality application with the second mobile platform based on the second pose of the second mobile platform with respect to the planar object.

23. The non-transitory computer-readable medium of claim 21, wherein the second platform determines the second pose of the second platform based at least in part on the reference image and the second image of the planar object captured by the second platform by performing a method comprising:

extracting features from the reference image;

defining 3D points on a 3D plane that correspond to the features extracted from the reference image;

finding points in the second image of the planar object captured by the second mobile platform that match with 2D projections of the 3D points; and

using matched 3D-2D point pairs to estimate the second pose of the second mobile platform.

24. The non-transitory computer-readable medium of claim 23, wherein the features extracted from the reference image comprise point features or line features.

25. The non-transitory computer-readable medium of claim 21, further comprising code for:

receiving a new image captured by the second mobile platform from the second mobile platform that includes a portion of the planar object that is not included in the reference image of the front view of the planar object; and

merging the new image and the reference image to generate a new reference image of the front view of the planar object; and

transmitting the new reference image to the first and second mobile platforms, wherein the second mobile platform determines an updated second pose of the second mobile platform with respect to the planar object using a subsequent image of the planar object captured by the second mobile platform and the new reference image.

26. The non-transitory computer-readable medium of claim 25, wherein the code for merging the new image and the reference image to generate the new reference image of the front view of the planar object comprises code for transforming pixels of the reference image to the new reference image through an identity transformation and code for transforming pixels of the new image to the new reference image.