Visual tracking using panoramas on mobile devices

• US 9,635,251 B2
• Filed: 05/20/2011
• Issued: 04/25/2017
• Est. Priority Date: 05/21/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wagner, Daniel, Mulloni, Alessandro, Schmalstieg, Dieter, Langlotz, Tobias Martin
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): OWENS, TSION B
• Application Number:
  US13/112,876
• Publication Number:
  US 20110285810A1
• Time in Patent Office: 2,167 Days
• Field of Search: 348 36, 348 37
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A mobile platform maps and tracks the orientation of the mobile platform in real-time as the mobile platform rotates. The mobile platform captures a plurality of camera images as the camera rotates and projects the plurality of camera images consecutively to a panoramic cylindrical map. The map is extended by projecting areas of any camera image that correspond to unmapped portions of the panoramic cylindrical map. Keypoints are extracted from mapped portions of the map and using the keypoints the orientation of the camera is determined. Accumulated error is removed when closing the panoramic cylindrical map by extending the map horizontally beyond 360 degrees to produce overlapping regions and extracting and matching keypoints from the overlapping regions. Additionally, an incorrect estimate of the initial orientation of the map may be corrected by determining the correct orientation and re-projecting the map onto a map with the correct orientation.

Claims

1. A method comprising:

capturing a plurality of camera images from a camera as the camera rotates;

projecting the plurality of camera images consecutively to a panoramic cylindrical map, wherein the panoramic cylindrical map is extended by projecting areas of any camera image that correspond to unmapped portions of the panoramic cylindrical map;

extracting keypoints from mapped portions of the panoramic cylindrical map;

using the keypoints to determine an orientation of the camera while capturing the plurality of camera images; and

using the orientation of the camera to assist in the projecting the plurality of camera images consecutively to the panoramic cylindrical map.

2. The method of claim 1, wherein projecting the plurality of camera images consecutively to the panoramic cylindrical map and using the keypoints to determine the orientation of the camera are performed substantially simultaneously.

3. The method of claim 1, wherein capturing the plurality of camera images, projecting the plurality of image frames, extracting keypoints and using the keypoints is performed by a mobile platform.

4. The method of claim 1, wherein when projecting the plurality of camera images consecutively to the panoramic cylindrical map each pixel of the panoramic cylindrical map is filled not more than five times.

5. The method of claim 1, wherein when projecting the plurality of camera images consecutively to the panoramic cylindrical map each pixel of the panoramic cylindrical map is filled only once.

6. The method of claim 1, wherein using keypoints to determine the orientation of the camera comprises extracting keypoints from a captured camera image and comparing the keypoints from the captured camera image to the keypoints from the panoramic cylindrical map to determine the orientation of the camera.

7. The method of claim 1, further comprising removing accumulated error when closing the panoramic cylindrical map, wherein removing accumulated error comprises:

extending the panoramic cylindrical map horizontally beyond 360 degrees to produce overlapping regions;

extracting keypoints from the overlapping regions;

matching keypoints from the overlapping regions to determine correct alignment of the overlapping regions; and

adjusting the panoramic cylindrical map to align the overlapping regions.

8. The method of claim 1, further comprising correcting incorrect initial orientation of the panoramic cylindrical map comprising:

determining a correct orientation in the panoramic cylindrical map; and

re-projecting the panoramic cylindrical map onto a different the panoramic cylindrical map having the correct orientation.

9. The method of claim 8, wherein determining the correct orientation in the panoramic cylindrical map is performed automatically or by dragging a map center on a screen into a desired position.

33. The method of claim 1, wherein the panoramic cylindrical map has a preset height and width.

34. The method of claim 1, wherein projecting the plurality of camera images consecutively to the panoramic cylindrical map comprises:

for each new image in the plurality of camera images, detecting one or more areas of the new image that correspond to unfilled pixels in the panoramic cylindrical map; and

projecting the one or more areas to fill the pixels in the panoramic cylindrical map.

35. The method of claim 1, wherein areas of any image in the plurality of camera images that correspond to areas outside the panoramic cylindrical map are not projected onto the panoramic cylindrical map.

10. An apparatus comprising:

a camera;

a processor connected to receive data from the camera;

memory connected to the processor;

a display connected to the memory; and

software held in the memory and run in the processor to cause the processor to consecutively project a plurality of camera images captured by the camera as the camera rotates onto a panoramic cylindrical map, wherein the panoramic cylindrical map is extended by projecting areas of any camera image that correspond to unmapped portions of the panoramic cylindrical map, extract keypoints from mapped portions of the panoramic cylindrical map, and the keypoints to determine an orientation of the camera while the plurality of camera images are captured by the camera; and

use the orientation of the camera to assist in the projecting the plurality of camera images consecutively to the panoramic cylindrical map.

11. The apparatus of claim 10, wherein the software held in the memory causes the processor to project the plurality of camera images to the panoramic cylindrical map and use the keypoints to determine the orientation of the camera substantially simultaneously.

12. The apparatus of claim 10, wherein the apparatus is a mobile platform.

13. The apparatus of claim 10, wherein the software held in the memory causes the processor to project the plurality of camera images consecutively to the panoramic cylindrical map so that each pixel of the panoramic cylindrical map is filled not more than five times.

14. The apparatus of claim 10, wherein the software held in the memory causes the processor to project the plurality of camera images consecutively to the panoramic cylindrical map so that each pixel of the panoramic cylindrical map is filled only once.

15. The apparatus of claim 10, wherein the software held in the memory causes the processor to extract keypoints from a camera image captured by the camera and compare the keypoints from the camera image to the keypoints from the panoramic cylindrical map to determine the orientation of the camera.

16. The apparatus of claim 10, wherein the software held in the memory further causes the processor to extend the panoramic cylindrical map horizontally beyond 360 degrees to produce overlapping regions, extract keypoints from the overlapping regions, match keypoints from the overlapping regions to determine correct alignment of the overlapping regions, and to adjust the panoramic cylindrical map to align the overlapping regions.

17. The apparatus of claim 10, wherein the software held in the memory further causes the processor to correct an incorrect initial orientation of the panoramic cylindrical map by causing the processor to determine a correct orientation in the panoramic cylindrical map, and to re-project the panoramic cylindrical map onto a different the panoramic cylindrical map having the correct orientation.

18. The apparatus of claim 17, wherein the processor determines the correct orientation in the panoramic cylindrical map automatically or by receiving data from a touch screen coupled to the processor that indicates a map center.

19. A system comprising:

means for capturing a plurality of camera images from a rotating position;

means for projecting the plurality of camera images consecutively to a panoramic cylindrical map, wherein the panoramic cylindrical map is extended by projecting areas of any camera image that correspond to unmapped portions of the panoramic cylindrical map;

means for extracting keypoints from mapped portions of the panoramic cylindrical map;

means for using the keypoints to determine an orientation of the means for capturing the plurality of camera images while capturing the plurality of camera images; and

means for using the orientation of the camera to assist in the projecting the plurality of camera images consecutively to the panoramic cylindrical map.

20. The system of claim 19, wherein the system is a mobile platform.

21. The system of claim 19, wherein the means for projecting the plurality of camera images consecutively to the panoramic cylindrical map fills each pixel of the panoramic cylindrical map not more than five times.

22. The system of claim 19, wherein the means for projecting the plurality of camera images consecutively to the panoramic cylindrical map fills each pixel of the panoramic cylindrical map only once.

23. The system of claim 19, wherein the means for using keypoints to determine the orientation of the means for capturing the plurality of camera images comprises means for extracting keypoints from a captured camera image and means for comparing the keypoints from the captured camera image to the keypoints from the panoramic cylindrical map to determine the orientation of the means for capturing the plurality of camera images.

24. The system of claim 19, further comprising means for removing accumulated error when closing the panoramic cylindrical map, wherein the means for removing accumulated error comprises:

means for extending the panoramic cylindrical map horizontally beyond 360 degrees to produce overlapping regions;

means for extracting keypoints from the overlapping regions;

means for matching keypoints from the overlapping regions to determine correct alignment of the overlapping regions; and

means for adjusting the panoramic cylindrical map to align the overlapping regions.

25. The system of claim 19, further comprising means for correcting incorrect initial orientation of the panoramic cylindrical map comprising:

means for determining a correct orientation in the panoramic cylindrical map; and

means for re-projecting the panoramic cylindrical map onto a different the panoramic cylindrical map having the correct orientation.

26. The system of claim 25, wherein the means for determining the correct orientation in the panoramic cylindrical map comprises a touch screen display that is used to indicate a desired position for a map center.

27. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to consecutively project a plurality of camera images captured by a camera as the camera rotates onto a panoramic cylindrical map, wherein the panoramic cylindrical map is extended by projecting areas of any camera image that correspond to unmapped portions of the panoramic cylindrical map;

program code to extract keypoints from mapped portions of the panoramic cylindrical map, and

program code to use the keypoints to determine an orientation of the camera while capturing the plurality of camera images; and

program code to use the orientation of the camera to assist in the projecting the plurality of camera images consecutively to the panoramic cylindrical map.

28. The computer-readable medium of claim 27, further comprising program code to project the plurality of camera images consecutively to the panoramic cylindrical map so that each pixel of the panoramic cylindrical map is filled not more than five times.

29. The computer-readable medium of claim 27, further comprising program code to project the plurality of camera images consecutively to the panoramic cylindrical map so that each pixel of the panoramic cylindrical map is filled only once.

30. The computer-readable medium of claim 27, further comprising:

program code to extract keypoints from a camera image captured by the camera; and

program code to compare the keypoints from the camera image to the keypoints from the panoramic cylindrical map to determine the orientation of the camera.

31. The computer-readable medium of claim 27, further comprising:

program code to extend the panoramic cylindrical map horizontally beyond 360 degrees to produce overlapping regions;

program code to extract keypoints from the overlapping regions;

program code to match keypoints from the overlapping regions to determine correct alignment of the overlapping regions; and

program code to adjust the panoramic cylindrical map to align the overlapping regions.

32. The computer-readable medium of claim 27, further comprising program code to correct an incorrect initial orientation of the panoramic cylindrical map comprising:

program code to determine a correct orientation in the panoramic cylindrical map; and

program code to re-project the panoramic cylindrical map onto a different the panoramic cylindrical map having the correct orientation.