Image-based method and system for building spherical panoramas

• US 7,317,473 B2
• Filed: 09/04/2002
• Issued: 01/08/2008
• Est. Priority Date: 05/27/1998
• Status: Expired due to Fees

• Current Assignee: Transpacific IP Group Limited
• Original Assignee: Transpacific IP Group Limited
• Inventors: Huang, Shu-Cheng, Chen, Ju-Wei, Cheng, Tse
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US10/235,609
• Publication Number:
  US 20030063816A1
• Time in Patent Office: 1,952 Days
• Field of Search: 348/39, 348/36, 345/441, 345/419, 345/425, 345/427, 345/649, 382/284, 382/154, 382/153, 382/155, 382/156, 382/157
• US Class Current: 348/39
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

The invention relates to building spherical panoramas for image-based virtual reality systems. The image-based spherical panoramas can be navigated in any desired view direction (360-degrees) by suitable three-dimensional image browsers or viewers. The method and system also includes computing the number of photographs required to be taken and the azimuth angle of the center point of each photograph for building a spherical environment map representative of the spherical panorama. The method and system also includes an algorithm for computing the accurate azimuth angles of these taken photographs and seaming them together to build the spherical environment map.

Claims

1. A method of stitching together a two dimensional array of photographic images to build a spherical panorama, the method comprising:

computing sight directions for each of the photographic images based on vertical registration of contiguous photographic images for each column of the two dimensional array;

adjusting the intensity of the photographic images to an intensity related to an average of the intensities of the photographic images; and

stitching together contiguous ones of the photographic images based on the sight directions to build the spherical panorama.

2. The method of claim 1 wherein computing sight directions comprises computing sight directions based on horizontal registration of contiguous photographic images for each row of the two dimensional array.

3. The method of claim 2 wherein the horizontal and vertical registrations are determined by computing image phase correlation for possible positions of image alignment using a correlation-based algorithm and selecting the position with the minimum alignment difference.

4. The method of claim 2 further comprising inspecting the results of the horizontal and vertical registration with a software tool.

5. The method of claim 2 wherein the vertical registration of contiguous photographic images for each column of the two dimensional array includes:

rotating each photographic image;

image warping the rotated photographic image;

performing horizontal registration of contiguous photographic images for each column after being rotated of the two dimensional array; and

determining sight directions from the image positions after image registration.

6. The method of claim 5 wherein each photographic image is rotated 90°

, the photographic image is image warped with a component of the sight directions φ

_L-set to 90°

, and horizontal registration of contiguous photographic images is performed for each column after being rotated.

7. The method of claim 2 wherein the horizontal registration is performed after the vertical registration.

8. The method of claim 1 wherein stitching together contiguous photographic images further includes image blending overlapping regions of the contiguous photographic images.

9. The method of claim 8 wherein image blending between contiguous photographic images includes determining intensity levels of pixels in the overlapping regions using the following equation:

$I<br><br>\left(r_i\right)=\frac{d_A^{<br><br>t}·<br><br>I<br><br>\left(p_i\right)+d_B^{<br><br>t}·<br><br>I<br><br>\left(q_i\right)}{d_A^{<br><br>t}+d_B^{<br><br>t}}$

where;

p_i is the location of a pixel from a first one of the photographic images;

q_i is the location of a pixel from a corresponding second one of the photographic images contiguous with the first one of the photographic images;

d_A is the distance between location p_i and a boundary of the first one of the photographic images;

d_B is the distance between location q_i and a boundary of the second one of the photographic images;

I(p_i) is the intensity of the pixel at location pi;

I(q_i) is the intensity of the pixel at location qi; and

I(r_i) is the resultant intensity of the pixel.

10. The method of claim 2, further comprising performing a second “

fine-tuning”

registration.

11. A system for stitching together a two dimensional array of photographic images to build a spherical panorama, the system comprising at least one computer-readable storage medium including:

computer instructions for computing sight directions for each of the photographic images based on vertical registration of contiguous photographic images for each column of the two dimensional array;

computer instructions for adjusting the intensity of the photographic images to an intensity related to an average of the intensities of the photographic images; and

computer instructions for stitching together contiguous ones of the photographic images based on the sight directions to build the spherical panorama.

12. The system of claim 11 wherein the computer instructions for computing sight directions comprises computer instructions for computing sight directions based on horizontal registration of contiguous photographic images for each row of the two dimensional array.

13. The system of claim 12 wherein the computer instructions for computing sight directions includes computer instructions for computing image phase correlation for possible positions of image alignment using a correlation-based algorithm and selecting the position with the minimum alignment difference.

14. The system of claim 12 further comprising computer instructions for inspecting the results of the horizontal and vertical registration with a software tool.

15. The system of claim 12 wherein the computer instructions for computing sight directions based on vertical registration of contiguous photographic images for each column of the two dimensional array includes:

computer instructions for rotating each photographic image;

image warping the rotated photographic image;

computer instructions for performing horizontal registration of contiguous photographic images for each column after above rotation of the two dimensional array; and

computer instructions for determining sight directions from the image positions after image registration.

16. The system of claim 15 wherein the computer instructions for computing sight directions based on vertical registration of contiguous photographic images for each column of the two dimensional array includes rotating each photographic image 90°

, and image warping the photographic image with a component of the sight directions φ

_L set to 90°

.

17. The system of claim 12 wherein the computer instructions for computing sight directions based on horizontal registration and vertical registration comprise computer instructions for performing a second “

fine-tuning”

registration.

18. The system of claim 11 wherein the computer instructions for stitching together contiguous on of the photographic images further includes computer instructions for image blending overlapping regions of the contiguous photographic images.

19. The system of claim 18 wherein the computer instructions for image blending between contiguous photographic images includes computer instructions for determining intensity levels of pixels in the overlapping regions using the following equation:

$I<br><br>\left(r_i\right)=\frac{d_A^{<br><br>t}·<br><br>I<br><br>\left(p_i\right)+d_B^{<br><br>t}·<br><br>I<br><br>\left(q_i\right)}{d_A^{<br><br>t}+d_B^{<br><br>t}}$

where;

p_i is the location of a pixel from a first one of the photographic images;

q_i is the location of a pixel from a corresponding second one of the photographic images contiguous with the first one of the photographic images;

d_A is the distance between location p_i and a boundary of the first one of the photographic images;

d_B is the distance between location q_i and a boundary of the second one of the photographic images;

I(p_i) is the intensity of the pixel at location pi;

I(q_i) is the intensity of the pixel at location qi;

and I(r_i) is the resultant intensity of the pixel.

20. One or more computer readable storage media having computer instructions stored thereon that when executed by a processor cause the stitching together of a two dimensional array of photographic images to build a spherical panorama, the computer instructions when executed performing a method comprising:

computing sight directions for each of the photographic images based on vertical registration of contiguous photographic images for each column of the two dimensional array;

adjusting the intensity of the photographic images to an intensity related to an average of the intensities of the photographic images; and

stitching together contiguous ones of the photographic images based on the sight directions to build the spherical panorama.

21. The computer readable storage media of claim 20 wherein computing sight directions comprises computing sight directions based on horizontal registration of contiguous photographic images for each row of the two dimensional array.

22. The computer readable storage media of claim 21 wherein the horizontal and vertical registrations are determined by computing image phase correlation for possible positions of image alignment using a correlation-based algorithm and selecting the position with the minimum alignment difference.

23. The computer readable storage media of claim 21 further comprising inspecting the results of the horizontal and vertical registration with a software tool.

24. The computer readable storage media of claim 21 wherein the vertical registration of contiguous photographic images for each column of the two dimensional array includes:

rotating each photographic image;

image warping the rotated photographic image;

performing horizontal registration of contiguous photographic images for each column after being rotated of the two dimensional array; and

determining sight directions from the image positions after image registration.

25. The computer readable storage media of claim 21 wherein the horizontal registration is performed after the vertical registration.

26. The computer readable storage media of claim 21, further comprising performing a second “

fine-tuning”

registration.

27. The computer readable storage media of claim 20 wherein stitching together contiguous photographic images further includes image blending overlapping regions of the contiguous photographic images.

28. The computer readable storage media of claim 27 wherein image blending between contiguous photographic images includes determining intensity levels of pixels in the overlapping regions using the following equation:

$I<br><br>\left(r_i\right)=\frac{d_A^{<br><br>t}·<br><br>I<br><br>\left(p_i\right)+d_B^{<br><br>t}·<br><br>I<br><br>\left(q_i\right)}{d_A^{<br><br>t}+d_B^{<br><br>t}}$

where;

p_i is the location of a pixel from a first one of the photographic images;

q_i is the location of a pixel from a corresponding second one of the photographic images contiguous with the first one of the photographic images;

d_A is the distance between location p, and a boundary of the first one of the photographic images;

d_B is the distance between location q_i and a boundary of the second one of the photographic images;

I(p_i) is the intensity of the pixel at location pi;

I(q_i) is the intensity of the pixel at location qi; and

I(r_i) is the resultant intensity of the pixel.

29. A system for stitching together a two dimensional array of photographic images to build a spherical panorama, the system comprising:

a datastore having computer instructions stored thereon; and

a processor that executes the computer instructions to compute sight directions for each of the photographic images based on vertical registration of contiguous photographic images for each column of the two dimensional array, to adjust the intensity of the photographic images to an intensity related to an average of the intensities of the photographic images, and to stitch together contiguous ones of the photographic images based on the sight directions to build the spherical panorama.

36. The system of claim 29 wherein processor image blends overlapping regions of the contiguous photographic images.

37. The system of claim 36 wherein image blending between contiguous photographic images includes determining intensity levels of pixels in the overlapping regions using the following equation:

$I<br><br>\left(r_i\right)=\frac{d_A^{<br><br>t}·<br><br>I<br><br>\left(p_i\right)+d_B^{<br><br>t}·<br><br>I<br><br>\left(q_i\right)}{d_A^{<br><br>t}+d_B^{<br><br>t}}$

where;

p_i is the location of a pixel from a first one of the photographic images;

q_i is the location of a pixel from a corresponding second one of the photographic images contiguous with the first one of the photographic images;

d_A is the distance between location p_i and a boundary of the first one of the photographic images;

d_B is the distance between location q_i and a boundary of the second one of the photographic images;

I(p_i) is the intensity of the pixel at location pi;

I(q_i) is the intensity of the pixel at location qi; and

I(r_i) is the resultant intensity of the pixel.

30. The system of 29 wherein the processor computes sight based on horizontal registration of contiguous photographic images for each row of the two dimensional array.

31. The system of claim 30 wherein the horizontal and vertical registrations are determined by computing image phase correlation for possible positions of image alignment using a correlation-based algorithm and selecting the position with the minimum alignment difference.

32. The system of claim 30 wherein datastore has stored thereon a software tool for inspecting the results of the horizontal and vertical registration.

33. The system of claim 30 wherein the vertical registration of contiguous photographic images for each column of the two dimensional array includes:

rotating each photographic image;

image warping the rotated photographic image;

performing horizontal registration of contiguous photographic images for each column after being rotated of the two dimensional array; and

determining sight directions from the image positions after image registration.

34. The system of claim 30 wherein the horizontal registration is performed after the vertical registration.

35. The system of claim 30, wherein the processor performs a second “

fine-tuning”

registration.