North centered orientation tracking in uninformed environments

• US 8,933,986 B2
• Filed: 05/20/2011
• Issued: 01/13/2015
• Est. Priority Date: 05/28/2010
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Schall, Gerhard, Mulloni, Alessandro, Reitmayr, Gerhard
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): PHAM, NAM D
• Application Number:
  US13/112,268
• Publication Number:
  US 20110292166A1
• Time in Patent Office: 1,334 Days
• US Class Current: 348/37
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A mobile platform uses orientation sensors and vision-based tracking to track with absolute orientation. The mobile platform generates a panoramic map by rotating a camera, which is compared to an image frame produced by the camera, to determine the orientation of the camera with respect to the panoramic map. The mobile platform also estimates an orientation of the panoramic map with respect to a world reference frame, e.g., magnetic north, using orientation sensors, including at least one accelerometer and a magnetic sensor. The orientation of the camera with respect to the world reference frame is then determined using the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame. A filter, such as a Kalman filter, provides an accurate and stable estimate of the orientation of the panoramic map with respect to the world reference frame.

Claims

1. A method comprising:

generating a panoramic map, at a mobile device, based on images captured from a camera;

using orientation sensors, at a mobile device, to estimate an orientation of the panoramic map with respect to a world reference frame;

determining the orientation of the camera with respect to the panoramic map based on matching at least one feature from an image frame produced by the camera to at least one feature from the panoramic map; and

determining an orientation of the camera with respect to the world reference frame based on the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame.

2. The method of claim 1, further comprises:

filtering data based on the orientation sensors and the orientation of the camera with respect to the panoramic map over time; and

adjusting the orientation of the panoramic map with respect to the world reference frame based on the filtered data.

3. The method of claim 2, wherein the estimate of the orientation of the panoramic map is continuously updated over time.

4. The method of claim 1, wherein the world reference frame is magnetic north.

5. The method of claim 1, wherein determining the orientation of the camera with respect to the world reference frame using the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame is performed in real time.

6. The method of claim 1, wherein the orientation sensors comprise at least one accelerometer and magnetic sensor.

7. The method of claim 1, wherein the panoramic map consists of a plurality of cells, and further comprising:

extracting at least one cell feature from each of the plurality of cells; and

wherein the matching the at least one feature image frame produced by the camera to the at least one feature from the panoramic map comprises;

extracting at least one image feature from the image frame; and

comparing the at least one cell feature to the at least one image feature.

8. The method of claim 1, further comprising:

adjusting the orientation of the panoramic map with respect to the world reference frame based on the orientation of the camera with respect to the panoramic map and the orientation sensors.

9. A mobile device comprising:

orientation sensors configured to provide orientation data;

a camera;

a processor connected to the orientation sensors, wherein the processor configured to receive the orientation data and further configured to access image data acquired by the camera; and

memory in communication with the processor, wherein the memory is configured to store instructions configured to cause the processor to generate a panoramic map using images from the camera, estimate an orientation of the panoramic map with respect to a world reference frame using the orientation data, determine the orientation of the camera with respect to the panoramic map based on matching at least one feature from an image frame produced by the camera to at least one feature from the panoramic map, determine an orientation of the camera with respect to the world reference frame based on the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame.

10. The apparatus of claim 9, wherein the instructions further configured to cause the processor to filter data based on the orientation sensors and the orientation of the camera with respect to the panoramic map over time, and to adjust the orientation of the panoramic map with respect to the world reference frame based on the filtered data.

11. The apparatus of claim 10, wherein the instructions are configured to update the estimate of the orientation of the panoramic map continuously over time.

12. The apparatus of claim 9, wherein the world reference frame is an absolute reference frame.

13. The apparatus of claim 9, wherein the instructions configured to determine in real time the orientation of the camera with respect to the world reference frame using the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame.

14. The apparatus of claim 9, wherein the orientation sensors comprise at least one accelerometer and at least one magnetic sensor.

15. A system comprising:

means for generating a panoramic map, at a mobile device, based on images captures from a camera;

means for using orientation sensors, at a mobile device, to estimate an orientation of the panoramic map with respect to a world reference frame;

means for determining the orientation of the camera with respect to the panoramic map based on matching at least one feature from an image frame produced by the camera to at least one feature from the panoramic map; and

means for determining an orientation of the camera with respect to the world reference frame based on the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame.

16. The system of claim 15, further comprising:

means for filtering data based on the orientation sensors and the orientation of the camera with respect to the panoramic map over time; and

means for adjusting the orientation of the panoramic map with respect to the world reference frame based on the filtered data.

17. The system of claim 16, wherein the estimate of the orientation of the panoramic map is continuously updated over time.

18. The system of claim 15, wherein the world reference frame is magnetic north.

19. The system of claim 15, wherein the means for determining the orientation of the camera with respect to the world reference frame using the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame determines the orientation of the camera with respect to the world reference frame in real time.

20. The system of claim 15, wherein the orientation sensors comprise at least one accelerometer, and at least one magnetic sensor, or any combination thereof.

21. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to generate a panoramic map, at a mobile device, using images from a camera;

program code to estimate an orientation of the panoramic map with respect to a world reference frame using orientation data from orientation sensors at a mobile device;

program code to determine the orientation of the camera with respect to the panoramic map based on matching at least one feature from an image frame produced by the camera to at least one feature from the panoramic map;

program code to determine an orientation of the camera with respect to the world reference frame using the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame.

22. The computer-readable medium of claim 21, further comprising:

program code to filter data based on the orientation sensors and the orientation of the panoramic map with respect to the world reference frame over time; and

program code to adjust the orientation of the panoramic map with respect to the world reference frame based on the filtered data.

23. The computer-readable medium of claim 22, wherein the estimate of the orientation of the panoramic map is continuously updated over time.

24. The computer-readable medium of claim 21, wherein the world reference frame is magnetic north.

25. The computer-readable medium of claim 21, wherein the program code to determine the orientation of the camera with respect to the world reference frame using the orientation of the camera with respect to the panoramic map and the orientation of the panoramic map with respect to the world reference frame determines the orientation of the camera with respect to the world reference frame in real time.

26. The computer-readable medium of claim 21, wherein the orientation sensors comprise at least one accelerometers and magnetic sensor.