Image processing device and method, program, program recording medium, data structure, and data recording medium

• US 20050012745A1
• Filed: 06/03/2003
• Published: 01/20/2005
• Est. Priority Date: 06/03/2002
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Tetsujiro, Tanaka, Kenji, Haga, Tsugihiko
• Granted Patent: US 7,831,086 B2
• Time in Patent Office: Days
• US Class Current: 345/427
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An image is readily changed in accordance with a user'"'"'s viewpoint. Original image data of an object, captured with arbitrary points on a curved surface Ω as viewpoints, is stored. Considering a straight line L_i connecting a viewpoint P and a point R_i on the object, an intersection of the straight line L_i and the curved surface Ω will be denoted as Q_i. When the object is viewed from the viewpoint P, a pixel value corresponding to a light ray directed from the point R_i on the object to the viewpoint P coincides with a pixel value corresponding to a light ray directed from the point R_i on the object to the point Q_i on the curved surface Ω, and this pixel value corresponding to the light ray directed from the point R_i to the point Q_i exists in the original image data. That is, the pixel value for the point R_i on the object as viewed from the viewpoint P can be obtained from original image data captured with the point Q_i on the curved surface Ω as a viewpoint. The present invention can be applied, for example, to an image processing apparatus for obtaining an image in accordance with a user'"'"'s viewpoint.

Claims

1. An image processing apparatus for converting first image data into second image data, the image processing apparatus comprising:

setting means for setting a user'"'"'s viewpoint at a designated position; and

converting means for converting the first image data into the second image data as viewed from the user'"'"'s viewpoint, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays.

2. An image processing apparatus according to claim 1,

wherein the setting means detects a position of a user and sets the user'"'"'s viewpoint at the position detected.

3. An image processing apparatus according to claim 1,

wherein the first image data is image data representing a designated object as imaged from a plurality of viewpoints.

4. An image processing apparatus according to claim 1,

wherein the first image data is captured by an imaging apparatus for capturing the first image data while moving the imaging apparatus linearly.

5. An image processing apparatus according to claim 1,

wherein the converting means converts the first image data into the second image data using, of light-ray information that passes through a predetermined curved surface when the first image data is captured, light-ray information that coincides with light-ray information directed from the user'"'"'s viewpoint to the predetermined curved surface.

6. An image processing apparatus according to claim 5,

wherein the predetermined curved surface is either an open curved surface, or a closed curved surface surrounding a designated object.

7. An image processing apparatus according to claim 1,

wherein the converting means comprises intermediate-data converting means for converting intermediate data into the second image data, the intermediate data being image data in a predetermined format, composed of pixel values corresponding to light rays incident on pixels constituting the first image data.

8. An image processing apparatus according to claim 7,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

9. An image processing apparatus according to claim 7,

wherein the intermediate data is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

10. An image processing apparatus according to claim 7,

wherein the intermediate data is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

11. An image processing apparatus according to claim 7,

wherein the intermediate-data converting means uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the user'"'"'s viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as the second image data.

12. An image processing apparatus according to claim 7,

wherein the converting means further comprises first image-data converting means for converting the first image data into the intermediate data,

and wherein the intermediate-data converting means converts the intermediate data output from the first image-data converting means into the second image data.

13. An image processing apparatus according to claim 12,

wherein the first image-data converting means converts the first image data into the intermediate data that is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

14. An image processing apparatus according to claim 12,

wherein the first image-data converting means converts the first image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

15. An image processing apparatus according to claim 12,

wherein the first image-data converting means converts the first image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

16. An image processing apparatus according to claim 12,

wherein the intermediate-data converting means uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the first image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

17. An image processing apparatus according to claim 7,

wherein the intermediate data is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point,

and wherein the image processing apparatus further comprises interpolating means for interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

18. An image processing apparatus according to claim 17,

wherein the interpolating means interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

19. An image processing apparatus according to claim 7,

wherein the intermediate data includes image data of an object as viewed from a position so distant that light rays from the object are substantially parallel, and also includes a direction of the light rays.

20. An image processing apparatus according to claim 1,

further comprising display means for displaying the second image data.

21. An image processing method for converting first image data into second image data, the image processing method comprising:

a setting step of setting a user'"'"'s viewpoint at a designated position; and

a converting step of converting the first image data into the second image data as viewed from the user'"'"'s viewpoint, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays.

22. An image processing method according to claim 21,

wherein the setting step detects a position of a user and sets the user'"'"'s viewpoint at the position detected.

23. An image processing method according to claim 21,

wherein the first image data is image data representing a designated object as imaged from a plurality of viewpoints.

24. An image processing method according to claim 21,

wherein the first image data is captured by an imaging apparatus for capturing the first image data while moving the imaging apparatus linearly.

25. An image processing method according to claim 21,

wherein the converting step converts the first image data into the second image data using, of light-ray information that passes through a predetermined curved surface when the first image data is captured, light-ray information that coincides with light-ray information directed from the user'"'"'s viewpoint to the predetermined curved surface.

26. An image processing method according to claim 25,

wherein the predetermined curved surface is either an open curved surface, or a closed curved surface surrounding a designated object.

27. An image processing method according to claim 21,

wherein the converting step comprises an intermediate-data converting step of converting intermediate data into the second image data, the intermediate data being image data in a predetermined format, composed of pixel values corresponding to light rays incident on pixels constituting the first image data.

28. An image processing method according to claim 27,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

29. An image processing method according to claim 27,

wherein the intermediate data is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

30. An image processing method according to claim 27,

wherein the intermediate data is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

31. An image processing method according to claim 27,

wherein the intermediate-data converting step uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the user'"'"'s viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as the second image data.

32. An image processing method according to claim 27,

wherein the converting step further comprises a first image-data converting step of converting the first image data into the intermediate data,

and wherein the intermediate-data converting step converts the intermediate data output from the first image-data converting means into the second image data.

33. An image processing method according to claim 32,

wherein the first image-data converting step converts the first image data into the intermediate data that is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

34. An image processing method according to claim 32,

wherein the first image-data converting step converts the first image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

35. An image processing method according to claim 32,

wherein the first image-data converting step converts the first image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

36. An image processing method according to claim 32,

wherein the intermediate-data converting step uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the first image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

37. An image processing method according to claim 27,

wherein the intermediate data is image data in which pixel values of designated pixels constituting the first image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point,

and wherein the image processing method further comprises an interpolating step of interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

38. An image processing method according to claim 37,

wherein the interpolating step interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

39. An image processing method according to claim 27,

wherein the intermediate data includes image data of an object as viewed from a position so distant that light rays from the object are substantially parallel, and also includes a direction of the light rays.

40. An image processing method according to claim 21,

further comprising a display step of displaying the second image data.

41. A program for allowing a computer to execute image processing for converting first image data into second image data, the program comprising:

a setting step of setting a user'"'"'s viewpoint at a designated position; and

a converting step of converting the first image data into the second image data as viewed from the user'"'"'s viewpoint, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays.

42. A program recording medium having recorded thereon a program for allowing a computer to execute image processing for converting first image data into second image data, the program comprising:

a setting step of setting a user'"'"'s viewpoint at a designated position; and

a converting step of converting the first image data into the second image data as viewed from the user'"'"'s viewpoint, using light-ray information representing trajectories of light rays for the capture of the first image data and representing pixel values corresponding to the light rays.

43. An image processing apparatus for converting image data, the image processing apparatus comprising:

obtaining means for obtaining the image data captured from a plurality of viewpoints; and

converting means for converting the image data into intermediate data, the intermediate data being image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

44. An image processing apparatus according to claim 43,

wherein the converting means converts the image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to a direction of the light rays, for respective points on the plane.

45. An image processing apparatus according to claim 43,

wherein the converting means converts the image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

46. An image processing apparatus according to claim 43,

wherein the converting means uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

47. An image processing apparatus according to claim 43,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point,

and wherein the image processing apparatus further comprises interpolating means for interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

48. An image processing apparatus according to claim 47,

wherein the interpolating means interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

49. An image processing apparatus according to claim 43,

wherein the intermediate data includes image data of an object as viewed from a position so distant that light rays from the object are substantially parallel, and also includes a direction of the light rays.

50. An image processing method for converting image data, the image processing method comprising:

an obtaining step of obtaining the image data captured from a plurality of viewpoints; and

a converting step of converting the image data into intermediate data, the intermediate data being image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

51. An image processing method according to claim 50,

wherein the converting step converts the image data into the intermediate data that is image data representing, for each direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to the direction of the light rays, for respective points on the plane.

52. An image processing method according to claim 50,

wherein the converting step converts the image data into the intermediate data that is image data representing, for respective points on a plane perpendicular to a direction of the light rays incident on the pixels constituting the first image data, pixel values corresponding to the light rays in each direction perpendicular to the plane.

53. An image processing method according to claim 50,

wherein the converting step uses, as pixel values at intersections of the light rays and a plane perpendicular to the light rays and passing through a predetermined point, pixel values corresponding to the light rays that coincide with straight lines connecting a viewpoint for the capture of the image data and pixels of the first image data on a virtual screen that is a predetermined virtual screen, and uses image data composed of the pixel values at the intersections on the plane as the intermediate data.

54. An image processing method according to claim 50,

wherein the intermediate data is image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point,

and wherein the image processing method further comprises an interpolating step of interpolating pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point.

55. An image processing method according to claim 54,

wherein the interpolating step interpolates the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point, by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

56. An image processing method according to claim 50,

wherein the intermediate data includes image data of an object as viewed from a position so distant that light rays from the object are substantially parallel, and also includes a direction of the light rays.

57. A program for allowing a computer to execute image processing for converting image data, the program comprising:

an obtaining step of obtaining the image data captured from a plurality of viewpoints; and

a converting step of converting the image data into intermediate data, the intermediate data being image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

58. A program recording medium having recorded thereon a program for allowing a computer to execute image processing for converting image data, the program comprising:

an obtaining step of obtaining the image data captured from a plurality of viewpoints; and

a converting step of converting the image data into intermediate data, the intermediate data being image data in which pixel values of predetermined pixels constituting the image data are pixel values at intersections of light rays incident on the predetermined pixels and a plane perpendicular to the light rays and passing through a predetermined point.

59. A data structure of image data,

wherein intersections of light rays incident on predetermined pixels constituting original image data and a plane perpendicular to the light rays and passing through a predetermined point serve as pixels,

and wherein pixel values of the pixels are used as pixel values of the predetermined pixels.

60. A data structure according to claim 59,

wherein, for each direction of light rays incident on pixels constituting the original image data, pixel values corresponding to the light rays incident in a direction perpendicular to a plane that is perpendicular to the direction of the light rays, for respective points on the plane, are arranged.

61. A data structure according to claim 59,

wherein, for respective points on a plane perpendicular to a direction of light rays incident on pixels constituting the original image data, pixel values corresponding to the light rays in each direction perpendicular to the plane are arranged.

62. A data structure according to claim 59,

wherein pixel values at points other than the intersections with the light rays on the plane passing through the predetermined point are interpolated.

63. A data structure according to claim 62,

wherein the pixel values at the points other than the intersections with the light rays on the plane passing through the predetermined point are interpolated by pixel values at intersections of another plane passing through the predetermined point and light rays perpendicular to the another plane.

64. A data recording medium having recorded image data thereon,

wherein intersections of light rays incident on predetermined pixels constituting original image data and a plane perpendicular to the light rays and passing through a predetermined point serve as pixels,

and wherein pixel values of the pixels are used as pixel values of the predetermined pixels.

65. An image processing apparatus comprising:

intermediate-data output means for outputting intermediate data in a predetermined format, the intermediate data including pixel values corresponding to light rays from an object;

viewpoint output means for outputting position information of a viewpoint for the object; and

intermediate-data converting means for generating an image of the object as viewed from the viewpoint, using the intermediate data.

66. An image processing apparatus according to claim 65,

wherein the intermediate data is data in a predetermined format, in which pixel values corresponding to the light rays from the object and a direction of the light rays are associated with each other.

67. An image processing apparatus according to claim 65,

wherein the intermediate data is data in which a direction of the light rays from the object, positions of respective points on a plane that is perpendicular to the direction of the light rays and passing through a predetermined point, and pixel values corresponding to the light rays incident on the respective points in a direction perpendicular to the plane are associated with each other.

68. An image processing apparatus according to claim 67,

wherein the intermediate data is image data representing the pixel values for the positions of the respective points with regard to the direction of the light rays.

69. An image processing apparatus according to claim 67,

wherein the intermediate data is image data representing the pixel values for the respective points on the plane with regard to each direction of the light rays.

70. An image processing apparatus according to claim 65,

wherein the intermediate data includes image data of the object as viewed from a position so distant that light rays from the object are substantially parallel, and also includes a direction of the light rays.

71. An image processing apparatus according to claim 66,

wherein the intermediate-data converting means uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as an image of the object as viewed from the viewpoint.

72. An image processing apparatus according to claim 67,

further comprising interpolating means for interpolating missing pixel values on the plane passing through the predetermined point.

73. An image processing apparatus according to claim 65,

further comprising display means for displaying an image of the object as viewed from the viewpoint.

74. An image processing method comprising:

an intermediate-data output step of outputting intermediate data in a predetermined format, the intermediate data including pixel values corresponding to light rays from an object;

a viewpoint output step of outputting position information of a viewpoint for the object; and

an intermediate-data converting step of generating an image of the object as viewed from the viewpoint, using the intermediate data.

75. An image processing method according to claim 74,

wherein the intermediate data is data in a predetermined format, in which pixel values corresponding to the light rays from the object and a direction of the light rays are associated with each other.

76. An image processing method according to claim 74,

wherein the intermediate data is data in which a direction of the light rays from the object, positions of respective points on a plane that is perpendicular to the direction of the light rays and passing through a predetermined point, and pixel values corresponding to the light rays incident on the respective points in a direction perpendicular to the plane are associated with each other.

77. An image processing method according to claim 76,

wherein the intermediate data is image data representing the pixel values for the positions of the respective points with regard to the direction of the light rays.

78. An image processing method according to claim 76,

wherein the intermediate data is image data representing the pixel values for the respective points on the plane with regard to each direction of the light rays.

79. An image processing method according to claim 74,

wherein the intermediate data includes image data of the object as viewed from a position so distant that light rays from the object are substantially parallel, and also includes a direction of the light rays.

80. An image processing method according to claim 75,

wherein the intermediate-data converting step uses, as pixel values of pixels on a virtual screen that is a predetermined virtual screen, pixel values corresponding to the light rays that coincide with straight lines connecting the viewpoint and the pixels on the virtual screen in the intermediate data, and uses image data composed of the pixel values on the virtual screen as an image of the object as viewed from the viewpoint.

81. An image processing method according to claim 76,

further comprising an interpolating step of interpolating missing pixel values on the plane passing through the predetermined point.

82. An image processing method according to claim 74,

further comprising a display step of displaying an image of the object as viewed from the viewpoint.

83. A program that is executed by a computer, the program comprising:

an intermediate-data output step of outputting intermediate data in a predetermined format, the intermediate data including pixel values corresponding to light rays from an object;

a viewpoint output step of outputting position information of a viewpoint for the object; and

an intermediate-data converting step of generating an image of the object as viewed from the viewpoint, using the intermediate data.

84. A program recording medium having recorded thereon a program that is executed by a computer, the program comprising:

an intermediate-data output step of outputting intermediate data in a predetermined format, the intermediate data including pixel values corresponding to light rays from an object;

a viewpoint output step of outputting position information of a viewpoint for the object; and

an intermediate-data converting step of generating an image of the object as viewed from the viewpoint, using the intermediate data.