Equatorial stitching of hemispherical images in a spherical image capture system

• US 10,650,487 B2
• Filed: 06/24/2019
• Issued: 05/12/2020
• Est. Priority Date: 08/12/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Steel, Joseph, Macmillan, Timothy
• Primary Examiner(s): Chen, Xuemei G
• Application Number:
  US16/450,672
• Publication Number:
  US 20190385274A1
• Time in Patent Office: 323 Days
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/073 Transforming surfaces of revolution to planar images, e.g. cylindrical surfaces to planar images

Abstract

Hyper-hemispherical images may be combined to generate a rectangular projection of a spherical image having an equatorial stitch line along of a line of lowest distortion in the two images. First and second circular images are received representing respective hyper-hemispherical fields of view. A video processing device may project each circular image to a respective rectangular image by mapping an outer edge of the circular image to a first edge of the rectangular image and mapping a center point of the circular image to a second edge of the first rectangular image. The rectangular images may be stitched together along the edges corresponding to the outer edge of the original circular image.

Claims

1. A method for stitching at least hemispherical images to generate a rectangular projection of a spherical image, the method comprising:

projecting a first circular image to a first rectangular image, the first rectangular image comprising (i) a first edge corresponding to a first portion of the first circular image and (ii) a second edge corresponding to a second portion of the first circular image;

projecting a second circular image to a second rectangular image, the second rectangular image comprising (i) a first edge corresponding to a first portion of the second circular image and (ii) a second edge corresponding to a second portion of the second circular image;

generating the rectangular projection of the spherical image by stitching the first edge of the first rectangular image with the first edge of the second rectangular image;

dividing the rectangular projection of the spherical image into at least first and second portions;

rotating each of the at least first and second portions of the rectangular projection clockwise or counterclockwise; and

combining the rotated at least first and second portions, the combining comprising aligning an edge of the rotated first portion and an edge of the rotated second portion to be parallel with respect to each other to generate a re-oriented projection.

2. The method of claim 1, further comprising:

receiving the first circular image, the first circular image corresponding to a first at least hemispherical field of view captured by a first camera facing a first direction; and

receiving the second circular image, the second circular image corresponding to a second at least hemispherical field of view captured by a second camera facing a second direction opposite the first direction.

3. The method of claim 2, wherein:

the at least hemispherical images comprise hyper-hemispherical images having a field of view greater than 180 degrees;

the at least hemispherical field of view captured by the first camera exceeds 180 degrees; and

the at least hemispherical field of view captured by the second camera exceeds 180 degrees.

4. The method of claim 1, wherein:

the projecting of the first circular image to the first rectangular image comprises;

(i) mapping an outer edge of the first circular image to a bottom edge of the first rectangular image, and (ii) mapping a center point of the first circular image to a top edge of the first rectangular image;

the first edge of the first rectangular image comprises the bottom edge of the first rectangular image, and the second edge of the first rectangular image comprises the top edge of the first rectangular image;

the projecting the second circular image to the second rectangular image comprises;

(i) mapping an outer edge of the second circular image to a top edge of the second rectangular image, and (ii) mapping a center point of the second circular image to a bottom edge of the second rectangular image; and

the first edge of the second rectangular image comprises the top edge of the second rectangular image, and the second edge of the second rectangular image comprises the bottom edge of the second rectangular image.

5. The method of claim 4, wherein the center point of the first circular image and the center point of the second circular image each comprise a single pixel.

6. The method of claim 1, wherein:

the rotating of each of the at least first and second portions comprises rotating the first portion clockwise by a prescribed amount and rotating the second portion counterclockwise by the prescribed amount; and

the combining of the rotated at least first and second portions to generate a re-oriented projection comprises recombining the rotated at least first and second sub-images together.

7. The method of claim 1, wherein the projecting of the first circular image to the first rectangular image comprises:

correlating a plurality of concentric rings of pixels from the first circular image to respective ones of a plurality of rows of the first rectangular image; and

stretching at least a portion of the plurality of rows so as to meet a dimension of the first rectangular image.

8. A video processing system comprising:

a first image-capturing device and a second image-capturing device;

at least one processor apparatus; and

a non-transitory computer-readable apparatus in data communication with the at least one processor apparatus and comprising a storage medium having at least one computer program, the at least one computer program comprising a plurality of instructions configured to, when executed by the at least one processor apparatus, cause the video processing system to;

generate a first rectangular image from a first circular image, the first circular image being captured by the first image-capturing device, the first rectangular image comprising;

(i) a first edge corresponding to a first portion of the first circular image and (ii) a second edge corresponding to a second portion of the first circular image;

generate a second rectangular image from a second circular image, the second circular image being captured by the second image-capturing device, the second rectangular image comprising;

(i) a first edge corresponding to a first portion of the second circular image and (ii) a second edge corresponding to a second portion of the second circular image;

stitch the first edge corresponding to the first portion of the first circular image with the first edge corresponding to the first portion of the second circular image to produce a rectangular projection of a spherical image;

derive at least a first image portion and a second image portion based at least on the rectangular projection;

rotate the first image portion to a first orientation and the second image portion to a second orientation; and

generate a new rectangular projection based on a combination of at least the rotated first image portion and the rotated second image portion.

9. The video processing system of claim 8, wherein the first image-capturing device and the second image-capturing device are arranged in a back-to-back configuration such that the first and second image-capturing devices face opposite directions.

10. The video processing system of claim 8, wherein each of the first and second image-capturing devices is configured to capture the respective first and second circular images with respective fields of view each greater than 180 degrees, and enable overlapping regions of the respective fields of view to be aligned along at least one stitch line.

11. The video processing system of claim 10, wherein the rectangular projection of the spherical image comprises the at least one stitch line, the at least one stitch line being associated with a lesser amount of distortion relative to other portions of the rectangular projection.

12. The video processing system of claim 8, wherein:

the generation of the first rectangular image comprises;

(i) mapping an outer edge of the first circular image to a bottom edge of the first rectangular image, and (ii) mapping a center point of the first circular image to a top edge of the first rectangular image; and

the generation of the second rectangular image comprises;

(i) mapping an outer edge of the second circular image to a top edge of the second rectangular image, and (ii) mapping a center point of the second circular image to a bottom edge of the second rectangular image.

13. The video processing system of claim 8, wherein the plurality of instructions are further configured to, when executed by the at least one processor apparatus, cause the video processing system to:

map one or more concentric rings of pixels corresponding to the first circular image to one or more rows of pixels corresponding to the first rectangular image; and

map one or more concentric rings of pixels corresponding to the second circular image to one or more rows of pixels corresponding to the second rectangular image.

14. The video processing system of claim 8, wherein the combination of at least the rotated first and second image portions comprises a stitch of the at least first and second image portions with the first rectangular image, the stitch comprising (i) a stitch along a first edge portion corresponding to the second edge of the first rectangular image, and (ii) a stitch along a second edge portion corresponding to the second edge of the first rectangular image.

15. A computerized apparatus configured to algorithmically stitch at least hemispherical images, the computerized apparatus comprising:

digital processor apparatus;

a non-transitory computer-readable apparatus in data communication with the digital processor apparatus and comprising a storage medium having at least one computer program, the at least one computer program comprising a plurality of instructions configured to, when executed by the digital processor apparatus, cause the computerized apparatus to;

compute a projection of a first non-rectangular image to a first rectangular image, the first rectangular image comprising;

(i) a first edge corresponding to a first portion of the first non-rectangular image, and (ii) a second edge corresponding to a second portion of the first non-rectangular image;

compute a projection of a second non-rectangular image to a second rectangular image, the second rectangular image comprising;

(i) a first edge corresponding to a first portion of the second non-rectangular image, and (ii) a second edge corresponding to a second portion of the second non-rectangular image;

generate a composite rectangular projection by stitching the first edge of the first rectangular image with the first edge of the second rectangular image;

derive a first image portion and a second image portion from the composite rectangular projection;

rotate each of the first and second image portions into respective prescribed orientations; and

combine the rotated first image portion and the rotated second image portion together to generate a re-oriented projection.

16. The computerized apparatus of claim 15, wherein:

the first and second non-rectangular images comprise circular images; and

the computerized apparatus comprises a first camera and a second camera, each of the first and second cameras being configured to capture hyper-hemispherical images, each hyper-hemispherical image representing a field of view n degrees greater than a 180-degree hemisphere in at least one direction.

17. The computerized apparatus of claim 15, wherein:

the projection to the first rectangular image comprises;

(i) a correlation of an outer edge of the first non-rectangular image to a bottom edge of the first rectangular image, and (ii) a correlation of a central region of the first non-rectangular image to a top edge of the first rectangular image; and

the projection to the second rectangular image comprises;

(i) a correlation of an outer edge of the second non-rectangular image to a top edge of the second rectangular image, and (ii) a correlation of a central region of the second non-rectangular image to a bottom edge of the second rectangular image.

18. The computerized apparatus of claim 15, wherein the plurality of instructions are further configured to, when executed by the digital processor apparatus, cause the computerized apparatus to:

correlate one or more regions of pixels corresponding to the first non-rectangular image to one or more rows of pixels corresponding to the first rectangular image, and stretch the one or more rows of pixels corresponding to the first rectangular image so as to fill the first rectangular image;

correlate one or more regions of pixels corresponding to the second non-rectangular image to one or more rows of pixels corresponding to the second rectangular image, and stretch the one or more rows of pixels corresponding to the second rectangular image so as to fill the second rectangular image.

19. The computerized apparatus of claim 15, wherein the derivation of the first and second image portions comprises division of the composite rectangular projection into at least the first and second image portions.

20. The computerized apparatus of claim 15, wherein each of the first and second image portions comprises portions from the first rectangular image and the second rectangular image.