Online creation of panoramic augmented reality annotations on mobile platforms

• US 9,204,040 B2
• Filed: 05/20/2011
• Issued: 12/01/2015
• Est. Priority Date: 05/21/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Langlotz, Tobias Martin, Wagner, Daniel, Mulloni, Alessandro, Schmalstieg, Dieter
• Primary Examiner(s): PRINCE, JESSICA MARIE
• Application Number:
  US13/112,329
• Publication Number:
  US 20110285811A1
• Time in Patent Office: 1,656 Days
• Field of Search: 348/36, 348/37
• US Class Current: 1/1
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A mobile platform generates a panoramic map by rotating a camera and stores the panoramic map. The mobile platform performs a computer vision operation on the panoramic map while continuing to generate the panoramic map. The mobile platform may determine the location of the camera when the panoramic map is generated, produce an annotation on the panoramic map and transmit the determined location, annotation, and portion to a storage device. In another example, the mobile platform may receive an annotation, a portion of a source panoramic map associated with the annotation, and a location where the source panoramic map was generated. The mobile platform may compare the portion of the source panoramic map to the panoramic map to identify a position on the panoramic map that corresponds to the portion of the source panoramic map, and display the annotation based on the determined position on the panoramic map.

Claims

1. A method, comprising:

generating a panoramic map, at a location, based on a live video stream;

storing the panoramic map;

performing a computer vision operation on the panoramic map while continuing to generate the panoramic map, wherein performing the computer vision comprises;

receiving from a storage device an annotation, a portion of a source panoramic map associated with the annotation, and a source location where the source panoramic map was generated;

comparing the portion of the source panoramic map to the panoramic map to identify a position on the panoramic map that corresponds to the portion of the source panoramic map, wherein the comparing the portion of the source panoramic map to the panoramic map is performed based on a proximity of the location to the source location; and

displaying the annotation based on the position on the panoramic map that is identified as corresponding to the portion of the source panoramic map.

2. The method of claim 1, wherein the portion of the source panoramic map is a plurality of adjacent patches from the source panoramic map.

3. The method of claim 2, wherein each of the plurality of adjacent patches comprises NxN pixels, where N is no more than 64.

4. The method of claim 1, wherein comparing the portion of the source panoramic map to the panoramic map comprises comparing the portion of the source panoramic map to completed portions of the panoramic map.

5. The method of claim 4, further comprising placing completed portions of the panoramic map in a queue to be compared to the portion of the source panoramic map based on a limited time budget.

6. The method of claim 4, wherein comparing the portion of the source panoramic map to the panoramic map further comprises:

generating integral images of each completed portion of the panoramic map;

creating a Walsh transform for the portion of the source panoramic map and each of the integral images;

comparing the Walsh transforms for the portion of the source panoramic map and each of the integral images to find good matches; and

applying a Normalized Cross Correlation computation for the good matches.

7. The method of claim 6, wherein the portion of the source panoramic map is an arrangement of a plurality of adjacent patches from the source panoramic map, wherein creating the Walsh transform, and comparing the Walsh transform is performed for each of the plurality of adjacent patches, the method further comprising determining that a plurality of the good matches are in a same arrangement as the plurality of adjacent patches.

8. An apparatus, comprising:

a camera;

a processor connected to the camera to receive images from the camera in a live video stream;

memory connected to the processor;

a display controlled by the processor;

a satellite positioning system receiver that provides location data to the processor; and

software held in the memory and run in the processor to cause the processor to generate a panoramic map, at a location, using the images from the camera, store the panoramic map in the memory, and to perform a computer vision operation on the panoramic map in the memory while continuing to generate the panoramic map, wherein the software held in memory and run in the processor to cause the processor to perform the computer vision operation causes the processor to retrieve an annotation, a portion of a source panoramic map associated with the annotation, and a source location where the source panoramic map was generated, compare the portion of the source panoramic map to the panoramic map to identify a position on the panoramic map that corresponds to the portion of the source panoramic map, wherein the compare the portion of the source panoramic map to the panoramic map is performed based on a proximity of the location to the source location, and display on the display the annotation based on the position on the panoramic map that is identified as corresponding to the portion of the source panoramic map.

9. The apparatus of claim 8, wherein the portion of the source panoramic map is a plurality of adjacent patches from the source panoramic map.

10. The apparatus of claim 9, wherein each of the plurality of adjacent patches comprises NxN pixels, where N is no more than 64.

11. The apparatus of claim 8, wherein the processor compares the portion of the source panoramic map to completed portions of the panoramic map.

12. The apparatus of claim 11, wherein the processor places completed portions of the panoramic map in a queue to be compared to the portion of the source panoramic map based on a limited time budget.

13. The apparatus of claim 11, wherein the software further causes the processor to generate integral images of each completed portion of the panoramic map, create a Walsh transform for the portion of the source panoramic map and each of the integral images, compare the Walsh transforms for the portion of the source panoramic map and each of the integral images to find good matches, and apply a Normalized Cross Correlation computation for the good matches.

14. The apparatus of claim 13, wherein the portion of the source panoramic map is an arrangement of a plurality of adjacent patches from the source panoramic map, wherein the processor creates the Walsh transform, and compares the Walsh transform for each of the plurality of adjacent patches, the processor further determines that a plurality of the good matches are in a same arrangement as the plurality of adjacent patches.

15. A system, comprising:

means for generating a panoramic map, at a location, using images from a camera;

means for storing the panoramic map;

means for performing a computer vision operation on the panoramic map while continuing to generate the panoramic map;

means for receiving from an annotation, a portion of a source panoramic map associated with the annotation, and a source location where the source panoramic map was generated;

means for comparing the portion of the source panoramic map to the panoramic map to identify a position on the panoramic map that corresponds to the portion of the source panoramic map, wherein the means for comparing the portion of the source panoramic map to the panoramic map is performed based on a proximity of the location to the source location; and

means for displaying the annotation based on the position on the panoramic map that is identified as corresponding to the portion of the source panoramic map.

16. The system of claim 15, wherein the portion of the source panoramic map is a plurality of adjacent patches from the source panoramic map.

17. The system of claim 15, wherein the means for comparing the portion of the source panoramic map to the panoramic map comprises means for comparing the portion of the source panoramic map to completed portions of the panoramic map.

18. The system of claim 17, further comprising means for placing completed portions of the panoramic map in a queue to be compared to the portion of the source panoramic map based on a limited time budget.

19. The system of claim 15, wherein the means for comparing the portion of the source panoramic map to the panoramic map further comprises:

means for generating integral images of each completed portion of the panoramic map;

means for creating a Walsh transform for the portion of the source panoramic map and each of the integral images;

means for comparing the Walsh transforms for the portion of the source panoramic map and each of the integral images to find good matches; and

means for applying a Normalized Cross Correlation computation for the good matches.

20. The system of claim 19, wherein the portion of the source panoramic map is an arrangement of a plurality of adjacent patches from the source panoramic map, wherein the means for comparing the portion of the source panoramic map to the panoramic map creates the Walsh transform and compares the Walsh transform for each of the plurality of adjacent patches, wherein the means for comparing the portion of the source panoramic map to the panoramic map further comprises means for determining that a plurality of the good matches are in a same arrangement as the plurality of adjacent patches.

21. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to generate a panoramic map, at a location, using images from the camera;

program code to store the panoramic map;

program code to perform a computer vision operation on the panoramic map while continuing to generate the panoramic map;

program code to retrieve an annotation, a portion of a source panoramic map associated with the annotation, and a source location where the source panoramic map was generated;

program code to compare the portion of the source panoramic map to the panoramic map to identify a position on the panoramic map that corresponds to the portion of the source panoramic map, wherein the program code to compare the portion of the source panoramic map to the panoramic map is performed based on a proximity of the location to the source location; and

program code to display the annotation based on the position on the panoramic map that is identified as corresponding to the portion of the source panoramic map.

22. The non-transitory computer-readable medium of claim 21, wherein the portion of the source panoramic map is a plurality of adjacent patches from the source panoramic map.

23. The non-transitory computer-readable medium of claim 21, wherein the program code to compare the portion of the source panoramic map to the panoramic map compares the portion of the source panoramic map to completed portions of the panoramic map.

24. The non-transitory computer-readable medium of claim 23, further comprising program code to place completed portions of the panoramic map in a queue to be compared to the portion of the source panoramic map based on a limited time budget.

25. The non-transitory computer-readable medium of claim 21, wherein the program code to compare the portion of the source panoramic map to the panoramic map further comprises:

program code to generate integral images of each completed portion of the panoramic map,

program code to create a Walsh transform for the portion of the source panoramic map and each of the integral images,

program code to compare the Walsh transforms for the portion of the source panoramic map and each of the integral images to find good matches; and

program code to apply a Normalized Cross Correlation computation for the good matches.

26. The non-transitory computer-readable medium of claim 25, wherein the portion of the source panoramic map is an arrangement of a plurality of adjacent patches from the source panoramic map, further comprising program code to determine that a plurality of the good matches are in a same arrangement as the plurality of adjacent patches.