Method and system for obtaining pair-wise epipolar constraints and solving for panorama pose on a mobile device

• US 10,510,151 B2
• Filed: 10/28/2016
• Issued: 12/17/2019
• Est. Priority Date: 05/03/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sheridan, Alan, Armstrong, Charles
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US15/308,265
• Publication Number:
  US 20190051000A1
• Time in Patent Office: 1,145 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Aspects of the disclosure generally relate to determining the location and orientation of panoramic images by a computing apparatus. One or more computing devices may receive alignment data between a first panoramic image and second panoramic image and original location data for the first panoramic image and the second panoramic image. The one or more computing devices may determine relative orientations between the pair of panoramic images based on the alignment data and calculate a heading from the first panoramic image to the second panoramic image based on the original location data. The location data and alignment data may be optimized by the one or more computing devices based on the relative orientations between the pair of panoramic images and the original location data. The one or more computing devices may replace the original location data and relative orientations with the optimized relative orientations and optimized location data.

Claims

1. A computer implemented method for determining the location and orientation of panoramic images:

receiving, by one or more computing devices, alignment data between a pair of panoramic images including a first panoramic image and second panoramic image, and original location data for the pair of panoramic images, wherein the original location data includes geographic location data including latitude and longitude coordinates corresponding to where the first panoramic image and the second panoramic image were captured;

calculating, by the one or more computing devices, a heading from the first panoramic image to the second panoramic image based on the original location data;

determining, by the one or more computing devices, relative orientations between the pair of panoramic images based on the alignment data and the heading;

optimizing, by the one or more computing devices, the original location data and the alignment data based on the relative orientations between the pair of panoramic images and the original location data, wherein optimizing includes calculating optimized relative orientations and optimized location data by minimizing costs for a location objective function and an orientation objective function;

replacing, by the one or more computing devices, the original location data and relative orientations with the optimized relative orientations and optimized location data.

2. The method of claim 1, wherein the alignment data is received from a user.

3. The method of claim 1, wherein the relative orientations between the pair of panoramic images includes:

the direction along which the first panoramic image reaches the second panoramic image; and

the direction along which the second panoramic image reaches the first panoramic image.

4. The method of claim 1, comprising the computing apparatus performing calculating the heading, θ

, where θ

=a tan 2(sin Δ

λ

·

cos φ

₂, cos φ

₁·

sin φ

₂−

sin φ

₁·

cos φ

₂·

cos Δ

λ

), and where φ

₁,λ

₁ is the latitude and longitude, respectively, of the first panoramic image and, φ

₂,λ

₂ is the latitude and longitude of the second panoramic image, and Δ

λ

is the difference in longitude between the first panoramic image and the second panoramic image.

5. The method of claim 1, wherein optimizing further includes calculating optimized relative orientations and optimized location data by minimizing costs for a distance objective function, wherein minimizing the cost for the distance objective function includes minimizing the difference of the distance between the optimized locations for the first and second panoramic images and the distance between the original locations of the first and second panoramic images.

6. The method of claim 1, wherein the costs to the one or more objective functions are minimized by solving for the minimum value using a least squares solver.

7. The method of claim 1, wherein minimizing the cost for the location objective function includes minimizing the movement of the optimized locations for the first and second panoramic images from the original locations of the first and second panoramic images, and

wherein minimizing the cost for the orientation objective function includes maintaining the optimized relative orientations at 180 degrees apart.

8. A system for determining the location and orientation of panoramic images, the system comprising:

one or more computing devices having one or more processors coupled to memory, the one or more processors being configured to;

receive alignment data between a pair of panoramic images including a first panoramic image and second panoramic image, and original location data for the pair of panoramic images, wherein the original location data includes geographic location data including latitude and longitude coordinates corresponding to where the first panoramic image and the second panoramic image were captured;

calculate a heading from the first panoramic image to the second panoramic image based on the original location data;

determine relative orientations between the pair of panoramic images based on the alignment data and the heading;

optimize the location data and the alignment data based on the relative orientations between the pair of panoramic images and the original location data, wherein optimizing includes calculating optimized relative orientations and optimized location data by minimizing costs for a location objective function and an orientation objective function; and

replace the original location data and relative orientations with the optimized relative orientations and optimized location data.

9. The system of claim 8, wherein the alignment data is received from a user.

10. The system of claim 8, wherein the relative orientations between the pair of panoramic images includes:

the direction along which the first panoramic image reaches the second panoramic image; and

the direction along which the second panoramic image reaches the first panoramic image.

11. The system of claim 8, wherein the one or more processors are further configured to calculate the heading, θ

, where θ

=a tan 2(sin Δ

λ

·

cos φ

₂, cos φ

₁·

sin φ

₂−

sin φ

₁·

cos φ

₂·

cos Δ

λ

), and where φ

₁,λ

₁ is the latitude and longitude, respectively, of the first panoramic image and, φ

₂,λ

₂ is the latitude and longitude of the second panoramic image, and Δ

λ

is the difference in longitude between the first panoramic image and the second panoramic image.

12. The system of claim 8, wherein the optimizing further includes calculating optimized relative orientations and optimized location data by minimizing costs for a distance objective function, wherein minimizing the cost for the distance objective function includes minimizing the difference of the distance between the optimized locations for the first and second panoramic images and the distance between the original locations of the first and second panoramic images.

13. The system of claim 8, wherein the costs to the one or more objective functions are minimized by solving for the minimum value using a least squares solver.

14. The system of claim 8, wherein minimizing the cost for the location objective function includes minimizing the movement of the optimized locations for the first and second panoramic images from the original locations of the first and second panoramic images, and

wherein minimizing the cost for the orientation objective function includes maintaining the optimized relative orientations at 180 degrees apart.

15. A non-transitory computer-readable medium storing instructions, which when executed by one or more processors, cause the one or more processors to:

receive alignment data between a pair of panoramic images including a first panoramic image and second panoramic image and original location data for the pair of panoramic images, wherein the original location data includes geographic location data including latitude and longitude coordinates corresponding to where the first panoramic image and the second panoramic image were captured;

calculate a heading from the first panoramic image to the second panoramic image based on the original location data;

determine relative orientations between the pair of panoramic images based on the alignment data and the heading;

optimize the original location data and the alignment data based on the relative orientations between the pair of panoramic images and the original location data, wherein optimizing includes calculating optimized relative orientations and optimized location data by minimizing costs for a location objective function and an orientation objective function; and

replace the original location data and relative orientations with the optimized relative orientations and optimized location data.

16. The non-transitory computer-readable medium of claim 15, wherein the alignment data is received from a user.

17. The non-transitory computer-readable medium of claim 15, wherein the relative orientations between the pair of panoramic images includes:

the direction along which the first panoramic image reaches the second panoramic image; and

the direction along which the second panoramic image reaches the first panoramic image.

18. The non-transitory computer-readable medium of claim 15, wherein the instructions further cause the one or more processors to calculate the heading, θ

, where θ

=a tan 2(sin Δ

·

cos φ

₂, cos φ

₁·

sin φ

₂−

sin φ

₁·

cos φ

₂·

cos Δ

λ

), and where φ

₁,λ

₁ is the latitude and longitude, respectively, of the first panoramic image and, φ

₂,λ

₂ is the latitude and longitude of the second panoramic image, and A is the difference in longitude between the first panoramic image and the second panoramic image.

19. The non-transitory computer-readable medium of claim 15, wherein the optimizing further includes calculating optimized relative orientations and optimized location data by minimizing costs for a distance objective function, wherein minimizing the cost for the distance objective function includes minimizing the difference of the distance between the optimized locations for the first and second panoramic images and the distance between the original locations of the first and second panoramic images.

20. The non-transitory computer-readable medium of claim 19, wherein minimizing the cost for the location objective function includes minimizing the movement of the optimized locations for the first and second panoramic images from the original locations of the first and second panoramic images, and

wherein minimizing the cost for the orientation objective function includes maintaining the optimized relative orientations at 180 degrees apart.