Real-time capture and transformation of hemispherical video images to images in rectilinear coordinates

• US 8,160,394 B2
• Filed: 05/11/2006
• Issued: 04/17/2012
• Est. Priority Date: 05/11/2006
• Status: Expired due to Fees

• Current Assignee: Intergraph Software Technologies Company (Hexagon AB)
• Original Assignee: Intergraph Software Technologies Company (Hexagon AB)
• Inventors: Whitaker, Sheila G., Grindstaff, Gene A., Acree, Elaine S.
• Primary Examiner(s): Werner, Brian
• Assistant Examiner(s): Patel, Nirav G
• Application Number:
  US11/432,774
• Publication Number:
  US 20070263093A1
• Time in Patent Office: 2,168 Days
• Field of Search: 382/293
• US Class Current: 382/293
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method of transforming an input image from a hemispherical source to an output image in rectilinear coordinates is disclosed. The method includes receiving data representative of an input image originating from a hemispherical camera and having a size defined by an input image height and an input image width. Once the data is received, an output image width and output image height representative of a size of a rectilinear output image is calculated based on the size of the input image. A rectilinear output image is then generated by mapping input image pixels to locations within the width and height of the output image, without reference to the optical characteristics of the hemispherical camera.

Claims

1. A method of transforming an input image from a hemispherical source to an output image in rectilinear coordinates, the method comprising:

receiving data representative of an input image originating from a hemispherical camera, and having a size defined by an input image height and an input image width;

calculating an output image width and an output image height representative of a size of a rectilinear output image based on the size of the input image;

generating a look-up table containing designated locations of pixels located within the input image and corresponding locations within the output image, wherein generating the look-up table includes;

calculating an inverse mapping for each input image pixel by determining a location of each input image pixel, wherein the input image pixel location is represented by a column and a row,

calculating a look-up table index for each input image pixel, the look-up table index containing an association of a column and row within the output image with an input image pixel, and

adjusting the look-up table index column for each input image pixel such that the look-up table index column is equal to that input image pixel'"'"'s column; and

adjusting the look-up table index row for each input image pixel such that the look-up table index row is equal to that input image pixel'"'"'s row; and

generating a rectilinear output image by mapping pixels to locations within the width and height of the output image based upon the input image data, using the look-up table, and without reference to the optical characteristics of the hemispherical camera.

2. The method of claim 1, further comprising:

converting the location of each input image pixel into a normalized x,y coordinate system.

3. The method of claim 1, wherein generating a look-up table further comprises:

adjusting the column of each input image pixel such that the column lies within the input image width; and

adjusting the row of each input image pixel such that the row lies within the input image height.

4. The method of claim 3, wherein adjusting the column of each input image pixel such that each input pixel lies within the initial input image width further comprises:

adjusting a value for the column of the input image pixel that is less than zero such that the value for the column is equal to zero; and

adjusting a value for the column of the input image pixel that is greater than the input image width such that the value for the column is equal to the input image width.

5. The method of claim 3, wherein adjusting the row of each input image pixel such that each input pixel lies within the initial input image height further comprises:

adjusting a value for the row of the input image pixel that is less than zero such that the value for the row is equal to zero; and

adjusting a value for the row of the input image pixel that is greater than the input image height such that the value for the row is equal to the input image height.

6. The method of claim 1, wherein generating the rectilinear output image further comprises:

using the look-up table to determine the location on the rectilinear output image to which to map the input pixel.

7. A method of generating a reusable look-up table for transforming input images from a hemispherical source to output images in rectilinear coordinates, the method of comprising:

receiving data representative of a first input image into a processor, the data originating from a hemispherical camera, and having a size defined by an input image height and an input image width;

determining, using the processor, a position within the look-up table for the input image pixels, wherein the position within the look-up table represents an output row and column; and

calculating, using the processor, an inverse mapping for each input image pixel by determining a location of pixels within the input image, wherein the location is defined by a column and a row, the reusable look-up table being reusable to transform the first input image and additional input images having defined input image sizes without reference to the optical characteristics of the hemispherical source.

8. The method of claim 7, further comprising:

adjusting, using the processor, the column position in the look-up table for the input image pixels such that it is equal to that pixel'"'"'s input column; and

adjusting, using the processor, the row position in the look-up table for the input image pixels such that it is equal to that pixel'"'"'s input row.

9. The method of claim 7, further comprising:

calculating, using the processor, a width and height defining a size of an output image based on the size of the input image.

10. The method of claim 7, further comprising:

converting, using the processor, the location of each input image pixel into a normalized x,y coordinate system.

11. The method of claim 7, further comprising:

adjusting, using the processor, the column of the input image pixels such that the column lies within the initial input image width; and

adjusting, using the processor, the row of the input image pixels such that the row lies within the initial input image height.

12. A method according to claim 7 further comprising:

storing the look-up table in memory for use in converting later received image data originating from a hemispherical camera to a rectilinear output image.

13. A method of transforming an input image from a hemispherical source to an output image in rectilinear coordinates using a reusable look-up table, wherein the input image from the hemispherical source is represented by a size and contains a number of pixels and the rectilinear output image is represented by a size calculated based on the input image size, the method comprising:

determining a location within the output image to transfer each input image pixel, wherein the output image location is represented by a column and a row;

determining an input image pixel associated with the output image location based on the row and column of the output image location using the reusable look-up table, the look-up table including input image pixel data locations; and

transferring the associated input image pixel to the output image location, the reusable look-up table being created based upon image data from a prior hemispherical image, and configured to be used to transform additional input images having defined input image sizes without reference to the optical characteristics of the hemispherical source.

14. A method according to claim 13 further comprising:

rendering, using an graphics renderer, the rectilinear output image; and

displaying the rectilinear output image on a screen.

15. A computer program product for use on a computer system for transforming an input image from a hemispherical source to an output image in rectilinear coordinates, the computer program product comprising a non-transitory computer readable medium having computer readable code thereon, the computer readable program code comprising:

computer code for receiving data representative of an input image from a hemispherical camera, and having a size defined by an input image height and input image width;

computer code for calculating an output image width and an output image height representative of a size of a rectilinear output image based on the size of the input image;

computer code for generating a look-up table containing designated locations of the pixels located within the input image and corresponding locations within the output image, wherein the computer code for generating the look-up table includes;

computer code for calculating an inverse mapping for each input image pixel by determining a location of each input image pixel wherein the input image pixel location is represented by a column and a row,

computer code for calculating a look-up table index for each input image pixel, the look-up table index containing an association of a column and row within the output image with an input image pixel,

computer code for adjusting the look-up table index column for each input image pixel such that the look-up table index column is equal to that input image pixel'"'"'s column, and

computer code adjusting the look-up table index row for each input image pixel such that the look-up table index row is equal to that input image pixel'"'"'s row; and

computer code for generating a rectilinear output image by mapping pixels to locations within the width and height of the output image based upon the input image data, using the look-up table, and without reference to the optical characteristics of the hemispherical camera.

16. The computer program product of claim 15, wherein the computer code for generating the look-up table further comprises:

computer code for converting the location of each input image pixel into a normalized x,y coordinate system.

17. The computer program product of claim 15, wherein the computer code for generating the look-up table further comprises:

computer code for adjusting the column of each input image pixel such that the column lies within the input image width; and

computer code for adjusting the row of each input image pixel such that the row lies within the input image height.

18. The computer program product of claim 17, wherein the computer code for adjusting the column of each input image pixel such that each input image pixel lies within the initial input image width further comprises:

computer code for adjusting a value for the column of the input image pixel that is less than zero such that the value for the column is equal to zero; and

computer code for adjusting a value for the column of the input image pixel that is greater than the input image width such that the value for the column is equal to the input image width.

19. The computer program product of claim 17, wherein the computer code for adjusting the row of each input image pixel such that each input image pixel lies within the initial input image height further comprises:

computer code adjusting a value for the row of the input image pixel that is less than zero such that the value for the row is equal to zero; and

computer code for adjusting a value for the row of the input image pixel that is greater than the input image height such that the value for the row is equal to the input image height.

20. The computer program product of claim 15, wherein computer code for generating the rectilinear output image further comprises:

computer code for using the look-up table to determine the location on the rectilinear output image to which to map the input image pixel.

21. A computer program product for use on a computer system for generating a reusable look-up table for transforming input images from a hemispherical source to output images in rectilinear coordinates, the computer program product comprising a non-transitory computer readable medium having computer readable code thereon, the computer readable program code comprising:

computer code for receiving data representative of a first input image originating from a hemispherical camera, and having a size defined by an input image height and an input image width;

computer code for determining a position within the look-up table for the input image pixels, wherein the position within the look-up table represents an output row and column; and

computer code for calculating an inverse mapping for each input image pixel by determining a location of pixels within the input image, wherein the location is defined by a column and a row, the reusable look-up table being created based upon image data from a prior hemispherical image, and configured to be used to transform additional input images having defined input image sizes without reference to the optical characteristics of the hemispherical source.

22. The computer program product of claim 21, further comprising:

computer code for adjusting the column position in the look-up table for the input image pixels such that it is equal to that pixel'"'"'s input column; and

computer code for adjusting the row position in the look-up table for the input image pixels such that it is equal to that pixel'"'"'s input row.

23. The computer program product of claim 21, further comprising:

computer code for calculating a width and height defining a size of an output image based on the size of the input image.

24. The computer program product of claim 21, further comprising:

computer code for converting the location of each input image pixel into a normalized x,y coordinate system.

25. The computer program product of claim 21, further comprising:

computer code for adjusting the column of the input images pixel such that the column lies within the initial input image width; and

computer code for adjusting the row of the input image pixels such that the row lies within the initial input image height.

26. A computer program product for use on a computer system for transforming an input image from a hemispherical source to an output image in rectilinear coordinates using a reusable look-up table, wherein the hemispherical input image is represented by a size and contains a number of pixels and the rectilinear output image is represented by a size calculated based on the input image size, the computer program product comprising a non-transitory computer readable medium having computer readable code thereon, the computer readable program code comprising:

computer code for determining a location within the output image to transfer each input image pixel, wherein the output image location is represented by a column and a row;

computer code for determining an input image pixel associated with the output image location based on the row and column of the output image location using the reusable look-up table, the look-up table including input image pixel data locations; and

computer code for transferring the associated input image pixel to the output image location, the reusable look-up table being created based upon image data from a prior hemispherical image, and configured to be used to transform additional input images having defined input image sizes without reference to the optical characteristics of the hemispherical source.

27. A method of transforming an input image from a hemispherical source to an output image in rectilinear coordinates, the method comprising:

receiving data representative of an input image into a processor, the data originating from a hemispherical camera, wherein the data is arranged as a number of pixels defining a size including a height and a width;

calculating, using the processor, a width and height representative of a size of a rectilinear output image based on the size of the input image;

generating a look-up table containing designated locations of the input image pixels and corresponding locations within the output image, wherein generating the look-up table includes;

calculating an inverse mapping for each input image pixel by determining a location of each input image pixel wherein the input image pixel location is represented by a column and a row,

calculating a look-up table index for each input image pixel, the look-up table index containing an association of a column and row within the output image with an input image pixel, and

adjusting the look-up table index column for each input image pixel such that the look-up table index column is equal to that input image pixel'"'"'s column; and

adjusting the look-up table index row for each input image pixel such that the look-up table index row is equal to that input image pixel'"'"'s row; and

generating, using the processor, a rectilinear output image by mapping input image pixels to a location within the width and height of the output image based upon the input image data, the look-up table, and without reference to the optical characteristics of the camera.

28. The method of claim 27, wherein generating a rectilinear output image further comprises:

mapping, using the processor, input image pixels to a location within the width and height of the output image such that only a single input image pixel is mapped to an output image location and the output image does not contain Moire effects.

29. The method of claim 27, wherein generating a rectilinear output image further comprises mapping a single input image pixel to a plurality of locations on the output image.

30. A computer program product for use on a computer system for transforming an input image from a hemispherical source to an output image in rectilinear coordinates, the computer program product comprising a non-transitory computer readable medium having computer readable code thereon, the computer readable program code comprising:

computer code for receiving data representative of an input image originating from a hemispherical camera, wherein the data is arranged as a number of pixels defining a size including a height and a width;

computer code for calculating a width and height representative of a size of a rectilinear output image based on the size of the input image;

computer code for generating a look-up table containing designated locations of the input image pixels and corresponding locations within the output image, wherein the computer code for generating the look-up table includes;

computer code for calculating an inverse mapping for each input image pixel by determining a location of each input image pixel, wherein the input image pixel location is represented by a column and a row,

computer code for calculating a look-up table index for each input image pixel, the look-up table index containing an association of a column and row within the output image with an input image pixel,

computer code for adjusting the look-up table index column for each input image pixel such that the look-up table index column is equal to that input image pixel'"'"'s column, and

computer code for adjusting the look-up table index row for each input image pixel such that the look-up table index row is equal to that input image pixel'"'"'s row; and

computer code for generating a rectilinear output image by mapping input image pixels to a location within the width and height of the output image based upon the input image data, the look-up table, and without reference to the optical characteristics of the camera.

31. The computer program product of claim 30, wherein the computer code for generating a rectilinear output image further comprises:

computer code for mapping input image pixels to a location within the width and height of the output image such that only a single input image pixel is mapped to an output image location and the output image does not contain Moire effects.

32. The computer program product of claim 30, wherein the computer code for generating a rectilinear out image further comprises mapping a single input image pixel to a plurality of locations on the output image.