Image processing device and method, program, program recording medium, data structure, and data recording medium

• US 7,831,086 B2
• Filed: 06/03/2003
• Issued: 11/09/2010
• Est. Priority Date: 06/03/2002
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Tetsujiro, Tanaka, Kenji, Haga, Tsugihiko
• Primary Examiner(s): Bella; Matthew C
• Assistant Examiner(s): Newman; Michael A
• Application Number:
  US10/485,577
• Publication Number:
  US 20050012745A1
• Time in Patent Office: 2,716 Days
• Field of Search: 345/429, 345/427, 345/419, 382/154, 382/300, 348/39
• US Class Current: 382/154
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An image processing apparatus for obtaining an image in accordance with a user'"'"'s viewpoint. Original image data of an object, captured at arbitrary points on a curved surface, is stored. Considering a straight line connecting a viewpoint and a point on the object, an intersection of the straight line and the curved surface will be identified. When the object is viewed from the viewpoint, a pixel value corresponding to a light ray directed from the point on the object to the viewpoint coincides with a pixel value corresponding to a light ray directed from the point on the object to the intersection on the curved surface Ω, and this pixel value of the intersection exists in the original image data. That is, the pixel value for the point on the object as viewed from the viewpoint can be obtained from original image data captured at the intersection on the curved surface.

Claims

1. An image processing apparatus for converting first image data into second image data comprising:

setting means for setting a user'"'"'s viewpoint at a designated position;

converting means for converting intermediate data into second image data as viewed from the user'"'"'s viewpoint, the intermediate data being image data based on pixel values corresponding to light rays incident on pixels of the first image data, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays; and

interpolating means for interpolating a missing pixel value of the intermediate data using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

2. An image processing apparatus according to claim 1,

wherein the setting means detects a position of a user and sets the user'"'"'s viewpoint at the detected position.

3. An image processing apparatus according to claim 1,

wherein the first image data is image data of an image representing a designated object as imaged from a plurality of view points.

4. An image processing apparatus according to claim 1,

wherein the first image data is captured by an imaging apparatus for capturing the first image data while moving the imaging apparatus linearly.

5. An image processing apparatus according to claim 1,

wherein the converting means converts the first image data into the second image data using light-ray information that passes through a predetermined curved surface when the first image data is captured and light-ray information that coincides with light-ray information directed from the user'"'"'s viewpoint to the predetermined curved surface.

6. An image processing apparatus according to claim 5,

wherein the predetermined curved surface is either an open curved surface, or a closed curved surface surrounding a designated object.

7. An image processing apparatus according to claim 1,

wherein the converting means comprises intermediate-data converting means for converting intermediate data into the second image data, and

wherein the intermediate data includes image data in a predetermined format and is composed of pixel values corresponding to light rays incident on pixels constituting the first image data.

8. An image processing apparatus according to claim 7,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

9. An image processing apparatus according to claim 7,

wherein the intermediate data is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

10. An image processing apparatus according to claim 7,

wherein the intermediate data is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

11. An image processing apparatus according to claim 7,

wherein the intermediate-data converting means uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the user'"'"'s viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as the second image data.

12. An image processing apparatus according to claim 7,

wherein the converting means further comprises first image-data converting means for converting the first image data into the intermediate data, and

wherein the intermediate-data converting means converts the intermediate data output from the first image-data converting means into the second image data.

13. An image processing apparatus according to claim 12,

wherein the first image-data converting means converts the first image data into the intermediate data that is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

14. An image processing apparatus according to claim 12,

wherein the first image-data converting means converts the first image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

15. An image processing apparatus according to claim 12,

wherein the first image-data converting means converts the first image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

16. An image processing apparatus according to claim 12,

wherein the intermediate-data converting means uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the first image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

17. An image processing apparatus according to claim 7,

wherein the intermediate data is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point, and wherein the image processing apparatus further comprises interpolating means for interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

18. An image processing apparatus according to claim 17,

wherein the interpolating means interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

19. An image processing apparatus according to claim 1, further comprising display means for displaying the second image data.

20. An image processing method utilized by an image processing apparatus that converts first image data into second image data, the image processing method comprising:

a setting step of setting a user'"'"'s viewpoint at a designated position;

a converting step of converting intermediate data into second image data as viewed from the user'"'"'s viewpoint, the intermediate data being image data based on pixel values corresponding to light rays incident on pixels of the first image data, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays; and

an interpolating step of interpolating a missing pixel value of the intermediate data using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

21. An image processing method according to claim 20,

wherein the setting step detects a position of a user and sets the user'"'"'s viewpoint at the detected position.

22. An image processing method according to claim 20,

wherein the first image data is image data of an image representing a designated object as imaged from a plurality of view points.

23. An image processing method according to claim 20,

wherein the first image data is captured by an imaging apparatus for capturing the first image data while moving the imaging apparatus linearly.

24. An image processing method according to claim 20,

wherein the first converting step converts the first image data into the second image data using light-ray information that passes through a predetermined curved surface when the first image data is captured and light-ray information that coincides with light-ray information directed from the user'"'"'s viewpoint to the predetermined curved surface.

25. An image processing method according to claim 24,

wherein the predetermined curved surface is either an open curved surface, or a closed curved surface surrounding a designated object.

26. An image processing method according to claim 20,

wherein the first converting step comprises an intermediate-data converting step of converting intermediate data into the second image data, and

wherein the intermediate data includes image data in a predetermined format and is composed of pixel values corresponding to light rays incident on pixels constituting the first image data.

27. An image processing method according to claim 26,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

28. An image processing method according to claim 26,

wherein the intermediate data is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

29. An image processing method according to claim 26,

wherein the intermediate data is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

30. An image processing method according to claim 26,

wherein the intermediate-data converting step uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the user'"'"'s viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as the second image data.

31. An image processing method according to claim 26,

wherein the first converting step further comprises a first image-data converting step of converting the first image data into the intermediate data, and wherein the intermediate-data converting step converts the intermediate data into the second image data.

32. An image processing method according to claim 31,

wherein the first image-data converting step converts the first image data into the intermediate data that is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

33. An image processing method according to claim 31,

wherein the first image-data converting step converts the first image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

34. An image processing method according to claim 31,

wherein the first image-data converting step converts the first image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

35. An image processing method according to claim 31,

wherein the intermediate-data converting step uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the first image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

36. An image processing method according to claim 26,

wherein the intermediate data is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point, and wherein the image processing method further comprises an interpolating step of interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

37. An image processing method according to claim 36,

wherein the interpolating step interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

38. An image processing method according to claim 20, further comprising a display step of displaying the second image data.

39. A non-transitory program recording medium having recorded thereon a program for allowing a computer to execute image processing for converting first image data into second image data, the program comprising:

a setting step of setting a user'"'"'s viewpoint at a designated position;

a converting step of converting intermediate data into second image data as viewed from the user'"'"'s viewpoint, the intermediate data being image data based on pixel values corresponding to light rays incident on pixels of the first image data, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays; and

an interpolating step of interpolating a missing pixel value of the intermediate data using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

40. An image processing apparatus for converting image data comprising:

obtaining means for obtaining the image data captured from a plurality of viewpoints;

converting means for converting the image data into intermediate data; and

interpolating means for interpolating a missing pixel value of the intermediate data using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value,

wherein the intermediate data includes image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

41. An image processing apparatus according to claim 40,

wherein the converting means converts the image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

42. An image processing apparatus according to claim 40,

wherein the converting means converts the image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

43. An image processing apparatus according to claim 40,

wherein the converting means uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

44. An image processing apparatus according to claim 40,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point, and

wherein the image processing apparatus further comprises interpolating means for interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

45. An image processing apparatus according to claim 44,

wherein the interpolating means interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

46. An image processing method utilized by an image processing apparatus that converts image data, the image processing method comprising:

an obtaining step of obtaining the image data captured from a plurality of viewpoints;

a converting step of converting the image data into intermediate data, the intermediate data being image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point; and

an interpolating step of interpolating a missing pixel value of the intermediate data using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

47. An image processing method according to claim 46,

wherein the converting step converts the image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to the direction of the light rays, for respective points on the plane.

48. An image processing method according to claim 46,

wherein the converting step converts the image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

49. An image processing method according to claim 46,

wherein the converting step uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

50. An image processing method according to claim 46,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point, and

wherein the image processing method further comprises an interpolating step of interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

51. An image processing method according to claim 50,

wherein the interpolating step interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

52. A non-transitory program recording medium having recorded thereon a program for allowing a computer to execute image processing for converting image data, the program comprising:

an obtaining step of obtaining the image data captured from a plurality of viewpoints;

a converting step of converting the image data into intermediate data; and

an interpolating step of interpolating a missing pixel value of the intermediate data using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value,

wherein the intermediate data includes image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point, and

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

53. A non-transitory computer-readable medium having recorded thereon a computer executable program, when executed by a computer, controlling the computer to create a data structure of a first image data to be converted to a second image data:

wherein intersections of light rays incident on a plane perpendicular to the light rays and passing through a predetermined point serve as pixels of the second image data,

wherein pixel values of the intersections are used as pixel values of intersections of the light rays incident on predetermined pixels of the first image data,

wherein a missing pixel in the second image data is interpolated by using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value, and

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

54. A non-transitory computer-readable medium according to claim 53,

wherein, for each direction of light rays incident on pixels constituting the original image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to the direction of the light rays, for respective points on the plane, are arranged.

55. A non-transitory computer-readable medium according to claim 53,

wherein, for respective points on a plane perpendicular to a direction of light rays incident on pixels constituting the original image data, pixel values corresponding to the light rays in each direction perpendicular to the plane are arranged.

56. A non-transitory computer-readable medium according to claim 53,

wherein pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point are interpolated.

57. A non-transitory computer-readable medium according to claim 56,

wherein the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point are interpolated by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

58. An image processing apparatus comprising:

intermediate-data output means for outputting intermediate data in a predetermined format,

wherein the intermediate data is converted from a first image data and includes pixel values corresponding to light rays from an object and a missing pixel value of the intermediate data is interpolated by using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value;

viewpoint output means for outputting position information of a viewpoint for the object; and

intermediate-data converting means for generating an image of the object as viewed from the viewpoint, using the intermediate data,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

59. An image processing apparatus according to claim 58,

wherein the intermediate data is data in a predetermined format, in which pixel values corresponding to the light rays from the object and a direction of the light rays are associated with each other.

60. An image processing apparatus according to claim 58,

wherein the intermediate data is data in which a direction of the light rays from the object, positions of respective points on a plane that is perpendicular to the direction of the light rays and passing through a predetermined point, and pixel values corresponding to the light rays incident on the respective points in a direction perpendicular to the plane are associated with each other.

61. An image processing apparatus according to claim 60,

wherein the intermediate data is image data representing the pixel values for the positions of the respective points with regard to the direction of the light rays.

62. An image processing apparatus according to claim 60,

wherein the intermediate data is image data representing the pixel values for the respective points on the plane with regard to each direction of the light rays.

63. An image processing apparatus according to claim 59,

wherein the intermediate-data converting means uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as an image of the object as viewed from the viewpoint.

64. An image processing apparatus according to claim 60, further comprising interpolating means for interpolating missing pixel values on the plane passing through the predetermined point.

65. An image processing apparatus according to claim 58, further comprising display means for displaying an image of the object as viewed from the viewpoint.

66. An image processing method utilized by an image processing apparatus comprising:

an intermediate-data output step of outputting intermediate data in a predetermined format,

wherein the intermediate data is converted from a first image data and includes pixel values corresponding to light rays from an object and a missing pixel value of the intermediate data is interpolated by using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value;

a viewpoint output step of outputting position information of a viewpoint for the object; and

an intermediate-data converting step of generating an image of the object as viewed from the viewpoint, using the intermediate data,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.

67. An image processing method according to claim 66,

wherein the intermediate data is data in a predetermined format, in which pixel values corresponding to the light rays from the object and a direction of the light rays are associated with each other.

68. An image processing method according to claim 66,

wherein the intermediate data is data in which a direction of the light rays from the object, positions of respective points on a plane that is perpendicular to the direction of the light rays and passing through a predetermined point, and pixel values corresponding to the light rays incident on the respective points in a direction perpendicular to the plane are associated with each other.

69. An image processing method according to claim 68,

wherein the intermediate data is image data representing the pixel values for the positions of the respective points with regard to the direction of the light rays.

70. An image processing method according to claim 68,

wherein the intermediate data is image data representing the pixel values for the respective points on the plane with regard to each direction of the light rays.

71. An image processing method according to claim 67,

wherein the intermediate-data converting step uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as an image of the object as viewed from the viewpoint.

72. An image processing method according to claim 68, further comprising an interpolating step of interpolating missing pixel values on the plane passing through the predetermined point.

73. An image processing method according to claim 66, further comprising a display step of displaying an image of the object as viewed from the viewpoint.

74. A non-transitory program recording medium having recorded thereon a program that is executed by a computer, the program comprising:

an intermediate-data output step of outputting intermediate data in a predetermined format,

wherein the intermediate data is converted from a first image data and includes pixel values corresponding to light rays from an object and a missing pixel value of the intermediate data is interpolated by using at least one pixel value that is on adjacent planes, an adjacent plane associated with a different view point from the same designated position, the at least one pixel value having same coordinates as the missing pixel value;

a viewpoint output step of outputting position information of a viewpoint for the object; and

an intermediate-data converting step of generating an image of the object as viewed from the viewpoint, using the intermediate data,

wherein each missing pixel that has an interpolated pixel value is excluded from being used in an interpolation process of another missing pixel.