Systems and methods for generating spherical images

• US 8,902,322 B2
• Filed: 11/09/2012
• Issued: 12/02/2014
• Est. Priority Date: 11/09/2012
• Status: Active Grant

• Current Assignee: Bubl Technology Inc.
• Original Assignee: Bubl Technology Inc.
• Inventors: Ramsay, Sean Geoffrey, Mills, Daniel Chantal, Horvath, Dylan Stephen, Bodaly, Scott Andrew Robinson
• Primary Examiner(s): HANNETT, JAMES M
• Application Number:
  US13/673,642
• Publication Number:
  US 20140132788A1
• Time in Patent Office: 753 Days
• Field of Search: 348/218.1
• US Class Current: 348/218.1
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An imaging system comprises four image sensors each pointing outwardly from the vertices of a notional tetrahedron with the optical axes of the image sensors substantially collinear with respective medians of the notional tetrahedron, with the focal plane array of each image sensor positioned between the lens system of its respective image sensor and the centroid of the notional tetrahedron. The imaging system and can be used to obtain image data for generating a spherical image of the space surrounding the imaging system. A method for generating a spherical image from this image data assigns spherical coordinates to the pixels in the images according to a cylindrical projection that is individually aligned with the image plane of each image, and blends overlapping pixels and fills blank pixel spaces. The method can be applied to image data representing outward views from the vertices or centroids of faces of any Platonic solid.

Claims

1. An imaging system, comprising:

a support frame;

four image sensors carried by the support frame;

each image sensor comprising a lens system having an optical axis and a focal plane array aligned with the optical axis of the lens system;

each image sensor having a field of view that is at least substantially coterminous with the field of view of each adjacent image sensor;

the image sensors arranged so that;

the optical axis of each lens system is substantially collinear with a median of a common notional regular tetrahedron;

each focal plane array is positioned between the lens system of its respective image sensor and a centroid of the notional tetrahedron; and

each image sensor faces outwardly relative to the centroid of the notional tetrahedron.

2. The imaging system of claim 1, wherein each image sensor has a field of view of at least 129.5 degrees.

3. The imaging system of claim 1, wherein each image sensor has a field of view that overlaps the field of view of each adjacent image sensor.

4. The imaging system of claim 1, wherein each image sensor has a field of view of at least 135 degrees.

5. The imaging system of claim 1, wherein each image sensor has a field of view of at least 165 degrees.

6. The imaging system of claim 1, wherein each image sensor has a field of view of between about 165 degrees and about 170 degrees.

7. A method for generating a spherical image, comprising:

receiving four images;

each image defining an image plane representing a field of view from a unique vertex of a notional regular tetrahedron outwardly along an optical axis that is substantially collinear with a notional line between a centroid of the notional tetrahedron and the respective vertex;

the field of view of each image being at least substantially coterminous with the field of view of each adjacent image;

assigning, to each pixel in each image, a spherical coordinate on a notional sphere according to a cylindrical projection aligned with the image plane for that image; and

using the spherical coordinates to assign colours derived from the pixels to pixel positions in a spherical image according to a spherical image template;

wherein;

the notional sphere intersects the vertices of the notional tetrahedron;

the notional sphere is centred on the centroid of the notional tetrahedron; and

the image plane of each image is substantially tangential to the notional sphere; and

and wherein the cylindrical projection is aligned with the image plane for that image by a notional cylinder of the cylindrical projection having its cylinder wall substantially tangential to the image plane and its longitudinal axis intersecting the centroid of the notional tetrahedron.

8. The method of claim 7 wherein the field of view of each image overlaps the field of view of each adjacent image.

9. The method of claim 7, wherein the spherical image template is an equirectangular image template.

10. The method of claim 7, wherein assigning the spherical coordinates to the respective pixels according to the cylindrical projection aligned with the image plane for that image comprises assigning the spherical coordinates to the respective pixels according to a precalculated lookup table derived from the cylindrical projection.

11. The method of claim 10, wherein the spherical coordinates in the precalculated lookup table include position adjustments for distortion correction in addition to being derived from the cylindrical projection.

12. The method of claim 8, wherein assigning the spherical coordinates to each pixel in each image results in:

unique pixels each having a unique spherical coordinate; and

pixel groups, each pixel group comprising a plurality of pixels whose spherical coordinates are identical;

and wherein using the spherical coordinates to assign colours to the pixel positions in the spherical image according to the spherical image template comprises;

for each pixel position in the spherical image that maps to a spherical coordinate assigned to a unique pixel, assigning the colour of that unique pixel to that pixel position in the spherical image; and

for each pixel position in the spherical image that maps to a spherical coordinate assigned to the plurality of pixels in a pixel group, assigning to that pixel position in the spherical image a colour blended from the plurality of pixels in the pixel group.

13. The method of claim 12, further comprising:

for each pixel position in the spherical image that maps to a spherical coordinate remaining unassigned to any pixel, assigning to that pixel position in the spherical image a colour determined by oversampling nearby pixel positions in the spherical image.

14. The method of claim 7, further comprising correcting each image for distortion.

15. The method of claim 7, wherein each image defines an image plane representing a field of view of at least 129.5 degrees.

16. The method of claim 7, wherein each image defines an image plane representing a field of view of at least 135 degrees.

17. The method of claim 7, wherein each image defines an image plane representing a field of view of at least 165 degrees.

18. The method of claim 7, wherein each image defines an image plane representing a field of view of between about 165 degrees and about 170 degrees.

19. The method of claim 7, wherein each image is one image in a video stream comprising a plurality of images.

20. The method of claim 7, wherein:

the images when received are multiplexed into a single composite image;

the method further comprising;

isolating each image from the composite image before finding, for each pixel in each image, the spherical coordinate representing the projection of that pixel onto the surface of the notional sphere.

21. A method for generating a spherical image, comprising:

receiving a set of images;

each image defining an image plane representing a field of view from a unique surrounding point of a notional Platonic solid outwardly along an optical axis substantially collinear with a notional line between a centroid of the notional Platonic solid and the respective surrounding point;

each surrounding point being a member of a set of surrounding points selected from the group consisting of;

(a) the set of all vertices of the notional Platonic solid; and

(b) the set of all centroids of faces of the notional Platonic solid;

the number of images in the set being equal to the number of surrounding points in the selected set of surrounding points for the notional Platonic solid;

the field of view of each image being at least substantially coterminous with the field of view of each adjacent image;

assigning, to each pixel in each image, a spherical coordinate on a notional sphere according to a cylindrical projection aligned with the image plane for that image; and

using the spherical coordinates to assign colours derived from the pixels to pixel positions in the spherical image according to a spherical image template;

wherein;

the notional sphere intersects the vertices of the notional Platonic solid;

the notional sphere is centred on the centroid of the notional Platonic solid; and

the image plane of each image is substantially tangential to the notional sphere;

and wherein the cylindrical projection is aligned with the image plane for that image by a notional cylinder of the cylindrical projection having its cylinder wall substantially tangential to the image plane and its longitudinal axis intersecting the centroid of the notional Platonic solid.

22. The method of claim 21 wherein the field of view of each image overlaps the field of view of each adjacent image.

23. The method of claim 21, wherein the spherical image template is an equirectangular image template.

24. The method of claim 21, wherein assigning the spherical coordinates to the respective pixels according to the cylindrical projection aligned with the image plane for that image comprises assigning the spherical coordinates to the respective pixels according to a precalculated lookup table derived from the cylindrical projection.

25. The method of claim 24, wherein the spherical coordinates in the precalculated lookup table include position adjustments for distortion correction in addition to being derived from the cylindrical projection.

26. The method of claim 22, wherein assigning the spherical coordinates to each pixel in each image results in:

unique pixels each having a unique spherical coordinate; and

pixel groups, each pixel group comprising a plurality of pixels whose spherical coordinates are identical;

and wherein using the spherical coordinates to assign colours to the pixel positions in the spherical image according to the spherical image template comprises;

for each pixel position in the spherical image that maps to a spherical coordinate assigned to a unique pixel, assigning the colour of that unique pixel to that pixel position in the spherical image; and

for each pixel position in the spherical image that maps to a spherical coordinate assigned to the plurality of pixels in a pixel group, assigning to that pixel position in the spherical image a colour blended from the plurality of pixels in the pixel group.

27. The method of claim 26, further comprising:

for each pixel position in the spherical image that maps to a spherical coordinate remaining unassigned to any pixel, assigning to that pixel position in the spherical image a colour determined by oversampling nearby pixel positions in the spherical images.

28. The method of claim 21, wherein each image is one image in a video stream comprising a plurality of images.

29. The method of claim 21, wherein the set of surrounding points is the set of all vertices of the notional Platonic solid.

30. The method of claim 21, wherein the set of surrounding points is the set of all centroids of faces of the notional Platonic solid.