Non-flat image processing apparatus, image processing method, recording medium, and computer program

• US 20040247173A1
• Filed: 07/16/2004
• Published: 12/09/2004
• Est. Priority Date: 10/29/2001
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Nielsen, Frank, Suzuki, Kosuke
• Granted Patent: US 7,483,590 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Such processes as compression, storage and reproduction of an image mapped on a three-dimensional coordinate system such as a spherical surface or a cylindrical surface are performed suitably by mapping omnidirectional map information to a two-dimensional image such that the redundancy may be low while the amounts of information is kept equal as far as possible with minimized errors. Preferably, the image is converted into a two-dimensional image in a form determined taking the performance of the omnidirectional video image reproduction side into consideration. The technique can be utilized when mapping information which describes a corresponding relationship between the three-dimensional coordinate system of the original omnidirectional image and the mapped two-dimensional image coordinate system is produced to synthesize an omnidirectional video image from an arbitrary viewpoint direction.

Claims

1. An image processing apparatus for a non-planar image for processing a non-planer image represented on a three-dimensional coordinate system, characterized in that it comprises:

two-dimensional planar image mapping means for mapping a non-planar image represented on a three-dimensional coordinate system to a two-dimensional plane; and

mapping information production means for producing mapping information which describes a corresponding relationship between the three-dimensional coordinate system representing the original non-planar image and the mapped two-dimensional image coordinate system.

2. An image processing apparatus for a non-planar image according to claim 1, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and

said two-dimensional planar image mapping means projects a spherical surface to a cylinder and expands the cylinder to a plane to map the cylinder to the two-dimensional planar image.

3. An image processing apparatus for a non-planar image according to claim 2, characterized in that

said two-dimensional planar image mapping means allocates an angle in a horizontal direction of the spherical surface to a horizontal direction of the two-dimensional plane and allocates an angle in a vertical direction of the spherical surface to a vertical direction of the two-dimensional plane both in an equally spaced relationship from each other.

4. An image processing apparatus for a non-planar image according to claim 2, characterized in that

said two-dimensional planar image mapping means allocates an angle in a horizontal direction and/or a vertical direction of the spherical surface in an arbitrarily spaced relationship to a horizontal direction and/or a vertical direction of the two-dimensional plane.

5. An image processing apparatus for a non-planar image according to claim 2, characterized in that

said mapping information production means produces mapping information described in angles in a horizontal direction and a vertical direction on the spherical surface allocated to rows and columns of the two-dimensional planar image.

6. An image processing apparatus for a non-planar image according to claim 1, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and

said two-dimensional planar image mapping means projects a spherical surface to a cylinder such that an area ratio may be correct and expand the cylinder to a plane to map the same to the two-dimensional planar image.

7. An image processing apparatus for a non-planar image according to claim 6, characterized in that

said mapping information production means produces mapping information representing a corresponding relationship between the three-dimensional coordinate system (Θ

, Φ

, r) of the original omnidirectional image and the mapped two-dimensional image coordinate system (TX, TY) in the following expressions;

TX=r×

θ

×

cos φ

TY=r×

φ

8. An image processing apparatus for a non-planar image according to claim 1, characterized in that

said two-dimensional planar image mapping means maps the non-planar image represented on the three-dimensional coordinate system to the two-dimensional planar image such that the amounts of information included in the pixels may be kept equal to each other in omnidirections.

9. An image processing apparatus for a non-planar image according to claim 1, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and

said two-dimensional planar image mapping means utilizes a theorem that, when the z axis of a sphere is cut at equal intervals with parallel planes, the areas on the sphere defined by adjacent ones of the planes becomes constant, to map regions on the sphere sectioned within equally spaced ranges in the Z axis direction on the three-dimensional polar coordinate system randomly to belt-like planar regions sectioned within equally spaced ranges on a two-dimensional orthogonal coordinate system.

10. An image processing apparatus for a non-planar image according to claim 1, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and

said two-dimensional planar image mapping means maps, to a point at the ith row of the jth column of the two-dimensional planar image wherein one row includes w pixels, a point extracted at the iw+jth from the omnidirectional image using a Hanmersley Sequence.

11. An image processing apparatus for a non-planar image according to claim 1, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a cylindrical omnidirectional image, and

said two-dimensional planar image mapping means cuts the cylinder vertically, expands the cylinder into rectangles, equally divides the rectangular image into divisional pieces and disposes the divisional pieces in order from above so as to adapt a prescribed format size.

12. An image processing apparatus for a non-planar image according to claim 1, characterized in that

it further comprises image compression means for compressing the mapped two-dimensional planar image information in a predetermined compression format.

13. An image processing apparatus for a non-planar image according to claim 12, characterized in that

said image compression means adopts, for a moving picture, a compression format of the MPEG-1, MPEG-2, MPEG-4, H.263, H.261, AVI, Motion JPEG or the like and adopts, for a still picture, a compression format of the JPEG, GIF, PNG, BMP, TIFF or the like.

14. An image processing apparatus for a non-planar image according to claim 12, characterized in that

said image compression means allocates, when an unused region in which no pixel is mapped is included in the two-dimensional planar image information, the same pixel value to the unused region.

15. An image processing apparatus for a non-planar image according to claim 1, characterized in that

it further comprises data format conversion means for converting the data format into a data format which includes a set of the two-dimensional planar image mapped by said two-dimensional planar image mapping means or a compressed image of the two-dimensional planar image and the mapping information produced by said mapping information production means.

16. An image processing apparatus for a non-planar image according to claim 15, characterized in that

said data format conversion means inserts audio information in an interleaved relationship in such a form that the audio information is synchronized with the video image information.

17. An image processing apparatus for a non-planar image according to claim 15, characterized in that

said data format conversion means inserts the mapping information into the top of a frame every time a mapping method adopted by said two-dimensional planar image mapping means changes over.

18. An image processing method for a non-planar image for processing a non-planer image represented on a three-dimensional coordinate system, characterized in that it comprises:

a two-dimensional planar image mapping step of mapping a non-planar image represented on a three-dimensional coordinate system to a two-dimensional plane; and

a mapping information production step of producing mapping information which describes a corresponding relationship between the three-dimensional coordinate system representing the original non-planar image and the mapped two-dimensional image coordinate system.

19. An image processing method for a non-planar image according to claim 18, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and,

at the two-dimensional planar image mapping step, a spherical surface is projected to a cylinder and the cylinder is expanded to a plane to map the cylinder to the two-dimensional planar image.

20. An image processing method for a non-planar image according to claim 19, characterized in that,

at the two-dimensional planar image mapping step, an angle in a horizontal direction of the spherical surface is allocated to a horizontal direction of the two-dimensional plane and an angle in a vertical direction of the spherical surface is allocated to a vertical direction of the two-dimensional plane both in an equally spaced relationship from each other.

21. An image processing method for a non-planar image according to claim 19, characterized in that,

at the two-dimensional planar image mapping step, an angle in a horizontal direction and/or a vertical direction of the spherical surface is allocated in an arbitrarily spaced relationship to a horizontal direction and/or a vertical direction of the two-dimensional plane.

22. An image processing method for a non-planar image according to claim 19, characterized in that,

at the mapping information production step, mapping information described in angles in a horizontal direction and a vertical direction on the spherical surface allocated to rows and columns of the two-dimensional planar image is produced.

23. An image processing method for a non-planar image according to claim 18, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and,

at the two-dimensional planar image mapping step, a spherical surface is projected to a cylinder such that an area ratio may be correct and expand the cylinder to a plane to map the same to the two-dimensional planar image.

24. An image processing method for a non-planar image according to claim 23, characterized in that,

at the mapping information production step, mapping information representing a corresponding relationship between the three-dimensional coordinate system (Θ

, Φ

, r) of the original omnidirectional image and the mapped two-dimensional image coordinate system (TX, TY) in the following expressions;

TX=r×

θ

×

cos φ

TY=r×

φ

are produced.

25. An image processing method for a non-planar image according to claim 18, characterized in that,

at the two-dimensional planar image mapping step, the non-planar image represented on the three-dimensional coordinate system is mapped to the two-dimensional planar image such that the amounts of information included in the pixels may be kept equal to each other in omnidirections.

26. An image processing method for a non-planar image according to claim 18, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and

at the two-dimensional planar image mapping step, a theorem that, when the z axis of a sphere is cut at equal intervals with parallel planes, the areas on the sphere defined by adjacent ones of the planes are equal, is utilized to map regions on the sphere sectioned within equally spaced ranges in the Z axis direction on the three-dimensional polar coordinate system randomly to belt-like planar regions sectioned within equally spaced ranges on a two-dimensional orthogonal coordinate system.

27. An image processing method for a non-planar image according to claim 18, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a spherical omnidirectional image, and

at the two-dimensional planar image mapping step, to a point at the ith row of the jth column of the two-dimensional planar image wherein one row includes w pixels, a point extracted at the iw+jth from the omnidirectional image using a Hammersley Sequence is mapped.

28. An image processing method for a non-planar image according to claim 18, characterized in that

the non-planar image represented on the three-dimensional coordinate system is a cylindrical omnidirectional image, and

at the two-dimensional planar image mapping step, the cylinder is cut vertically and expanded into rectangles, and the rectangular image is equally divided into divisional pieces and the divisional pieces are disposed in order from above so as to adapt a prescribed format size.

29. An image processing method for a non-planar image according to claim 18, characterized in that

it further comprises an image compression step of compressing the mapped two-dimensional planar image information in a predetermined compression format.

30. An image processing method for a non-planar image according to claim 29, characterized in that,

at the image compression step, for a moving picture, a compression format of the MPEG-1, MPEG-2, MPEG-4, H.263, H.261, AVI, Motion JPEG or the like is adopted and, for a still picture, a compression format of the JPEG, GIF, PNG, BMP, TIFF or the like is adopted.

31. An image processing method for a non-planar image according to claim 18, characterized in that,

at the image compression step, when an unused region in which no pixel is mapped is included in the two-dimensional planar image information, the same pixel value is allocated to the unused region.

32. An image processing method for a non-planar image according to claim 18, characterized in that

it further comprises a data format conversion step of converting the data format into a data format which includes a set of the two-dimensional planar image mapped by the two-dimensional planar image mapping step or a compressed image of the two-dimensional planar image and the mapping information produced by the mapping information production step.

33. An image processing method for a non-planar image according to claim 32, characterized in that,

at the data format conversion step, audio information is inserted in an interleaved relationship in such a form that the audio information is synchronized with the video image information.

34. An image processing method for a non-planar image according to claim 18, characterized in that,

at the data format conversion step, the mapping information is inserted into the top of a frame every time a mapping method adopted by the two-dimensional planar image mapping means changes over.

35. A storage medium on which computer software described such that a process of a non-planer image represented on a three-dimensional coordinate system is executed on a computer system is physically stored in a computer-readable form, characterized in that the computer software comprises:

a two-dimensional planar image mapping step of mapping a non-planar image represented on a three-dimensional coordinate system to a two-dimensional plane; and

a mapping information production step of producing mapping information which describes a corresponding relationship between the three-dimensional coordinate system representing the original non-planar image and the mapped two-dimensional image coordinate system.

36. A computer program described in a computer-readable form such that a process of a non-planer image represented on a three-dimensional coordinate system is executed on a computer system, characterized in that the computer program comprises:

a two-dimensional planar image mapping step of mapping a non-planar image represented on a three-dimensional coordinate system to a two-dimensional plane; and

mapping information production step of producing mapping information which describes a corresponding relationship between the three-dimensional coordinate system representing the original non-planar image and the mapped two-dimensional image coordinate system.

37. An image processing apparatus for displaying a non-planar image of a three-dimensional coordinate system mapped on a two-dimensional planar image as a video image, characterized in that:

indication means for designating a viewpoint direction and/or a zoom value, and

video image production means for producing a three-dimensional shape video image in an arbitrary viewpoint direction in accordance with the designated viewpoint direction and/or zoom value using mapping information when the non-planar image is mapped to the two-dimensional plane image.

38. An image processing apparatus according to claim 37, characterized in that

the video image production means performs re-mapping from the non-planar video image mapped on the two-dimensional plane to a two-dimensional display screen of a predetermined video image display apparatus.

39. An image processing apparatus according to claim 37, characterized in that the video image production means includes means for calculating polar coordinates from the viewpoint direction and/or zoom value of each of the pixels of the viewpoint direction video image, and means for referring to the two-dimensional image mapping information to convert the polar coordinates into omnidirectional image coordinates.

40. An image processing apparatus according to claim 37, characterized in that

the video image production means includes means for converting omnidirectional image coordinates of each of apexes of primitives included in the omnidirectional video image mapping information, means for converting the polar coordinates into viewpoint direction image coordinates from the viewpoint direction and/or zoom value, and means for producing a viewpoint direction image in a unit of a primitive using the omnidirectional video image as a texture.

41. An image processing apparatus according to claim 37, characterized in that

the video image production means further includes means for interpolating a pixel based on adjacent pixels of the omnidirectional video image.

42. An image processing apparatus according to claim 37, characterized in that

the video image production means includes means for disposing, upon pixel interpolation, in the proximity of each of the opposite left and right ends of the omnidirectional image, a pixel region duplicated by a predetermined width from the other end.

43. An image processing apparatus according to claim 37, characterized in that

the video image production means includes means for producing a lookup table for searching for adjacent pixels with regard to each of pixels of the two-dimensionally mapped omnidirectional image, means for calculating polar coordinates from the viewpoint direction and zoom value of each of the pixels of the viewpoint direction video image, means for referring to the lookup table to search for pixels neighboring with the coordinates corresponding to the pixel of the viewpoint direction video image, means for interpolating the polar coordinates of the pixel using the adjacent pixels, and means for referring to the two-dimensional image mapping information to convert the polar coordinates into the omnidirectional image coordinates.

44. An image processing apparatus according to claim 37, characterized in that

the video image production means falsely realizes video images picked up by cameras of different types of lenses when the viewpoint is on the inner side upon synthesis of an arbitrary viewpoint direction video image.

45. An image processing apparatus according to claim 37, characterized in that

the video image production means produces an omnidirectional video image of an arbitrary three-dimensional shape when the viewpoint is on the outer side upon synthesis of an arbitrary viewpoint direction video image.

46. An image processing method for displaying a non-planar image of a three-dimensional coordinate system mapped on a two-dimensional planar image as a video image, characterized in that:

indication step of designating a viewpoint direction and/or a zoom value, and

video image production step of producing a three-dimensional shape video image in an arbitrary viewpoint direction in accordance with the designated viewpoint direction and/or zoom value using mapping information when the non-planar image is mapped to the two-dimensional plane image.

47. An image processing method according to claim 46, characterized in that

the video image production step performs re-mapping from the non-planar video image mapped on the two-dimensional plane to a two-dimensional display screen of a predetermined video image display apparatus.

48. An image processing method according to claim 46, characterized in that the video image production step includes step of calculating polar coordinates from the viewpoint direction and/or zoom value of each of the pixels of the viewpoint direction video image, and step of referring to the two-dimensional image mapping information to convert the polar coordinates into omnidirectional image coordinates.

49. An image processing method according to claim 46, characterized in that

the video image production step includes step of converting omnidirectional image coordinates of each of apexes of primitives included in the omnidirectional video image mapping information, step of converting the polar coordinates into viewpoint direction image coordinates from the viewpoint direction and/or zoom value, and step of producing a viewpoint direction image in a unit of a primitive using the omnidirectional video image as a texture.

50. An image processing method according to claim 46, characterized in that

the video image production step further includes step of interpolating a pixel based on adjacent pixels of the omnidirectional video image.

51. An image processing method according to claim 46, characterized in that

the video image production step includes step of disposing, upon pixel interpolation, in the proximity of each of the opposite left and right ends of the omnidirectional image, a pixel region duplicated by a predetermined width from the other end.

52. An image processing method according to claim 46, characterized in that

the video image production step includes step of producing a lookup table for searching for adjacent pixels with regard to each of pixels of the two-dimensionally mapped omnidirectional image, step of calculating polar coordinates from the viewpoint direction and zoom value of each of the pixels of the viewpoint direction video image, step of referring to the lookup table to search for pixels neighboring with the coordinates corresponding to the pixel of the viewpoint direction video image, step of interpolating the polar coordinates of the pixel using the adjacent pixels, and step of referring to the two-dimensional image mapping information to convert the polar coordinates into the omnidirectional image coordinates.

53. An image processing method according to claim 46, characterized in that

the video image production step falsely realizes video images picked up by cameras of different types of lenses when the viewpoint is on the inner side upon synthesis of an arbitrary viewpoint direction video image.

54. An image processing method according to claim 46, characterized in that

the video image production step produces an omnidirectional video image of an arbitrary three-dimensional shape when the viewpoint is on the outer side upon synthesis of an arbitrary viewpoint direction video image.

55. A storage medium on which computer software described such that a process for displaying a non-planer image of a three-dimensional coordinate system mapped on a two-dimensional planar image as a video image is executed on a computer system is physically stored in a computer-readable form, characterized in that the computer software includes:

an indication step of designating a viewpoint direction and/or a zoom value, and

a video image production step of producing a three-dimensional shape video image in an arbitrary viewpoint direction in accordance with the designated visual point direction and/or zoom value using mapping information when the non-planar image is mapped to the two-dimensional plane image.

56. A computer program described in a computer-readable form such that a process for displaying a non-planar image of a three-dimensional coordinated system mapped on a two-dimensional planar image as a video image is executed on a computer system, characterized in that it includes:

an indication step of designating a viewpoint direction and/or a zoom value, and

a video image production step of producing a three-dimensional shape video image in an arbitrary viewpoint direction in accordance with the designated visual point direction and/or zoom value using mapping information when the non-planar image is mapped to the two-dimensional plane image.