Image processing apparatus for interpolating and generating images from an arbitrary view point

• US 6,636,234 B2
• Filed: 08/18/1999
• Issued: 10/21/2003
• Est. Priority Date: 02/19/1999
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Katayama, Akihiro, Endo, Takaaki
• Primary Examiner(s): BRIER, JEFFERY A
• Application Number:
  US09/376,329
• Publication Number:
  US 20020171666A1
• Time in Patent Office: 1,525 Days
• Field of Search: 345/437, 345/438, 345/427, 345/646, 345/649, 345/656, 345/672, 345/682
• US Class Current: 345/646
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images

Abstract

There is provided a method of generating an interpolated image without any constraint even in a wide moving range. When two real images have an inclusive relationship, the two images are rotated to make their optical axes coincide with each other. Nonlinear interpolation processing is performed for the two images after the rotations, and the interpolated image is rotated in the line of sight direction of a user.

Claims

1. A method of generating an interpolated image at a predetermined viewpoint position on the basis of first and second images obtained at first and second viewpoint positions different from the predetermined viewpoint position, comprising:

an inputting step of inputting the first and second images;

a determination step of determining whether a field of view of the first image includes a lens center of the second image; and

an image generation step of generating an interpolated image by performing steps (A) to (C) when the field of view of the first image includes the lens center of the second image in said determination step and performing steps (D) to (F) when the field of view of the first image does not include the lens center of the second image in said determination step;

(A) a first correcting step of correcting the first and second images, respectively, such that optical axes of cameras which respectively sensed the first and second images coincide with a first direction substantially coinciding with a direction of a base line connecting the first and second viewpoint positions;

(B) a first interpolation step of calculating coordinates of a corresponding point of an interpolated image having the first direction as a line of sight direction at the predetermined viewpoint position by performing nonlinear proportional distribution for coordinates of a first corresponding point in the first image with respect to a predetermined feature point and coordinates of a second corresponding point in the second image with respect to the feature point;

(C) a second correcting step of correcting the interpolated image from the first direction to the line of sight direction at the predetermined viewpoint position;

(D) a third correcting step of correcting the first and second images, respectively, such that optical axes of the image become substantially perpendicular to the direction of the base line;

(E) a second interpolation step of calculating coordinates of a corresponding point of an interpolated image having the second direction perpendicular to the direction of the base line at the predetermined viewpoint position as a line of sight direction by performing linear proportional distribution for coordinates of a first corresponding point of the first image and coordinates of a second corresponding point of the second image;

(F) a fourth correcting step of correcting the interpolated image from the second direction to the line of sight direction at the predetermined viewpoint position, and outputting the image after the fourth correction as an interpolated image.

2. The method according to claim 1, wherein when the predetermined viewpoint position internally divides an interval between the first and second viewpoint positions at t:

1−

t, the nonlinear proportional distribution processing in the first interpolation step is executed by making a reciprocal value 1/x_e of a coordinate of the interpolated image divide an interval between a reciprocal value 1/x_A of the coordinate of the first corresponding point and a reciprocal value 1/x_B of the coordinate of the second corresponding point at t;

1−

t.

3. The method according to claim 1, wherein when the predetermined viewpoint position internally divides an interval between the first and second viewpoint positions at t:

1−

t, the linear proportional distribution processing in the second interpolation processing is executed by making a coordinate value x_e of the interpolated image divide an interval between a coordinate value x_A of the first corresponding point and a coordinate value x_B of the second corresponding point at t;

1−

t.

4. The method according to claim 1, further comprising a step of determining whether the first and second images have at least corresponding points derived from the same object.

5. The method according to claim 4, further comprising a step of searching an image database for another image having at least a corresponding point when the first and second images do not have at least corresponding points derived from the same object.

6. The method according to claim 1, wherein the first and second viewpoint positions are defined by viewpoint positions and optical axis directions of the cameras in image sensing operation, and the predetermined viewpoint position is defined by a viewpoint position and line of sight direction of a user.

7. The method according to claim 1, wherein the interpolated image output in said outputting step is presented as a panoramic image to a user.

8. A computer program storage medium that stores a program executed by a computer, storing a program code that implements the interpolation method defined in claim 1.

9. A method of generating an interpolated image at a predetermined viewpoint position on the basis of first, second, and third images respectively obtained at first, second, and third viewpoint positions different from the predetermined viewpoint position, comprising:

an inputting step of inputting the first, second and third images;

an internal ratio calculation step of obtaining first and second internal ratios on the basis of a relationship between the predetermined viewpoint position and the first to third viewpoint positions;

a first interpolation step of performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

a second interpolation step of performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

a first image interpolation step of generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of the first interpolation step and the second interpolation step for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship determined by determining if a field of view of one of the first image and the second image includes a lens center of the other;

a second image interpolation step of generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first interpolation step and the second interpolation step for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship determined by determining if a field of view of one of the temporary virtual image and the third image includes a lens center of the other;

a rotation step of rotating the interpolated image, obtained in said second image interpolation step, in a line of sight direction at the predetermined viewpoint position; and

an outputting step of outputting the image after the rotation as an interpolated image.

10. The method according to claim 9, wherein said first interpolation step comprises a step of rotating the two arbitrary images at viewpoint positions in the base line direction to make optical axes of the images coincide with each other, and the interpolation processing is performed for the two rotated images.

11. The method according to claim 9, wherein said second interpolation step comprises a step of rotating the two arbitrary images at viewpoint positions in a direction perpendicular to the base line direction to make the two arbitrary images become parallel to each other, and the interpolation processing is performed for the two rotated images.

12. The method according to claim 9, wherein a viewpoint position which divides, at a second internal ratio, an interval between the third viewpoint position and a temporary virtual position P_T which divides an interval between the first and second viewpoint positions at a first internal ratio coincides with the predetermined viewpoint position.

13. The method according to claim 9, wherein said first interpolation step comprises obtaining a reciprocal value 1/x_e of a coordinate of the temporary virtual image by internally dividing an interval between a reciprocal value 1/x_A of a coordinate of a first corresponding point of one of the two images and a reciprocal value 1/x_B of a coordinate of a second corresponding point of the other image at the first internal ratio.

14. The method according to claim 9, wherein said second interpolation step comprises obtaining a coordinate value x_e of the temporary virtual image by internally dividing an interval between a coordinate value x_A of a first corresponding point of one of the two images and a coordinate value x_B of a second corresponding point of the other image at the second internal ratio.

15. The method according to claim 9, further comprising a step of determining whether the first to third images have at least corresponding points derived from the same object.

16. The method according to claim 15, further comprising a step of searching an image database for another image having at least a corresponding point when the first to third images do not have at least corresponding points derived from the same object.

17. The method according to claim 9, wherein the first to third viewpoint positions are defined by viewpoint positions and optical axis directions of the cameras in image sensing operation, and the predetermined viewpoint position is defined by a viewpoint position and line of sight direction of a user.

18. The method according to claim 9, wherein the interpolated image output in said outputting step is presented as a panoramic image to the user.

19. An image processing apparatus comprising:

image storing means for storing a plurality of images together with viewpoint position information corresponding thereto;

detecting means for detecting both a line of sight direction and viewpoint position of a user;

searching means for searching, in said image storing means for the first to third images including corresponding points in the line of sight direction of the user;

calculation means for obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;

first interpolation means for performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

second interpolation means for performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

first image interpolation means for generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of the nonlinear proportional distribution processing and the linear proportional distribution processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship determined by determining if a field of view of one of the first image and the second image includes a lens center of the other;

second image interpolation means for generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the nonlinear proportional distribution processing and the linear proportional distribution processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship determined by determining if a field of view of one of the temporary virtual image and the third image includes a lens center of the other,

image rotation means for rotating the interpolated image, obtained in the second image interpolation means, in a line of sight direction at the user viewpoint position; and

outputting means for outputting the image after the rotation as an interpolated image.

20. The apparatus according to claim 19, wherein said first interpolation means comprises means for rotating the two arbitrary images at viewpoint positions in the base line direction to make optical axes of the images coincide with each other, and the interpolation processing is performed for the two rotated images.

21. The apparatus according to claim 19, wherein said second interpolation means comprises means for rotating the two arbitrary images at viewpoint positions in a direction perpendicular to the base line direction to make the two arbitrary images become parallel to each other, and the interpolation processing is performed for the two rotated images.

22. The apparatus according to claim 19, wherein a viewpoint position which divides an interval between the third viewpoint position and a temporary virtual position P_T which divides, at a second internal ratio, an interval between the first and second viewpoint positions at a first internal ratio coincides with the user viewpoint position.

23. The apparatus according to claim 19, wherein said first interpolation means obtains a reciprocal value 1/x_e of a coordinate of the temporary virtual image by internally dividing an interval between a reciprocal value 1/x_A of a coordinate of a first corresponding point of one of the two images and a reciprocal value 1/x_B of a coordinate of a second corresponding point of the other image at the first internal ratio.

24. The apparatus according to claim 19, wherein said second interpolation means obtains a coordinate value x_e of the temporary virtual image by internally dividing an interval between a coordinate value x_A of a first corresponding point of one of the two images and a coordinate value x_B of a second corresponding point of the other image at the second internal ratio.

25. The apparatus according to claim 19, further comprising second determination means for determining whether the first to third images have at least corresponding points derived from the same object.

26. The apparatus according to claim 25, further comprising second searching means for searching an image database for another image having at least a corresponding point when the first to third images do not have at least corresponding points derived from the same object.

27. The apparatus according to claim 19, wherein the first to third viewpoint positions are defined by viewpoint positions and optical axis directions of the cameras in image sensing operation, and the user viewpoint position is defined by a viewpoint position and line of sight direction of the user.

28. The apparatus according to claim 19, wherein the interpolated image output in said outputting means is presented as a panoramic image to a user.

29. The apparatus according to claim 19, further comprising second image storing means for storing a temporary virtual image generated by said first image interpolation means in said image storing means, together with corresponding viewpoint position information.

30. The apparatus according to claim 19, further comprising second image storing means for storing a temporary virtual image generated by said second image interpolation means in said image storing means, together with corresponding viewpoint position information.

31. An image processing apparatus for generating a morphed image by processing first and second images respectively obtained by sensing an object at first and second viewpoint positions, comprising:

segmentation means for segmenting the first image into a plurality of first partial images in accordance with a plurality of depths from the first viewpoint position, and also segmenting the second image into a plurality of second partial images in accordance with a plurality of depths from the second viewpoint position; and

processing means for performing a morphing process on the first and second images on the basis of the depths and a change of a viewpoint position.

32. The apparatus according to claim 31, wherein said first and second images are two-dimensional images;

and said segmentation means further comprises;

display means for displaying the first and second images; and

first designation means for designating partial areas in the displayed first and second images as a pair of two-dimensional partial images by adding hierarchical information corresponding to a depth to each of the partial images.

33. The apparatus according to claim 32, wherein said first designation means comprises:

second designation means for designating, on said display means, an occluded partial area in which a two-dimensional partial image belonging to a layer farther from the arbitrary viewpoint position is occluded by a two-dimensional partial image belonging to a layer nearer to the arbitrary viewpoint position; and

retouch means for allowing a user to retouch a pixel value of the occluded partial area.

34. The apparatus according to claim 32, wherein said first designation means comprises rewrite means for rewriting pixel values of one or a plurality of pairs of two-dimensional partial images having a second hierarchical information value other than a first hierarchical information value into a predetermined fixed value with respect to one pair of two-dimensional partial images having the first hierarchical information value.

35. The apparatus according to claim 34, wherein the predetermined fixed value is a pixel value representing black.

36. The apparatus according to claim 32, wherein said segmentation means comprises third designation means for designating a plurality of partial areas in one pair of first and second two-dimensional partial images corresponding to a given layer.

37. The apparatus according to claim 32, wherein said segmentation means stores the first and second original two-dimensional images and a plurality of pairs of two-dimensional partial images, together with hierarchical information.

38. An image display apparatus for reading out a plurality of pairs of two-dimensional partial images generated and stored by said image processing apparatus defined in claim 35, and performing morphing processing to generate an image from an arbitrary viewpoint position comprising:

first morphing processing means for generating, for pairs of two-dimensional partial images corresponding to all depths,

a partial morphed image from an arbitrary viewpoint position on the basis of one pair of first and second two-dimensional partial images corresponding to each depth; and

display means for synthesizing partial morphed images corresponding to the respective depths and generated by said first morphing processing means, and displaying a synthesized image on a predetermined display unit.

39. The apparatus according to claim 38, wherein said apparatus is applied to a walkthrough application.

40. The apparatus according to claim 39, further comprising input means for inputting information about the arbitrary viewpoint position.

41. The apparatus according to claim 40, wherein said input means virtually changes a viewpoint position value.

42. The apparatus according to claim 38, further comprising:

second morphing processing means for generating a morphed image directly based on the first and second two-dimensional images; and

switching means for selectively switching said first morphing processing means and said second morphing processing means.

43. The apparatus according to claim 42, wherein said apparatus further comprises:

input means for inputting information about the arbitrary viewpoint position; and

a memory for determining that one of the first morphing processing and the second morphing processing is to be applied to an arbitrary walkthrough route in a walkthrough virtual space when a viewpoint position of the user exists in the walkthrough route, and storing information indicating a determination result together with a position of the walkthrough route, and

said switching means extracts determination result information corresponding to viewpoint position information input by said input means from said memory, and selects and operates one of said first morphing processing means and said second morphing processing means on the basis of the information.

44. The apparatus according to claim 38, wherein said first morphing means comprises:

an image storing means for storing a plurality of images together with viewpoint position information corresponding thereto;

detecting means for detecting both a line of sight direction and viewpoint position of a user;

searching means for searching in said image storing means for the first to third images including corresponding points in the line of sight direction of the user;

determination means for determining whether any two of the first to third images have an inclusive relationship;

calculation means for obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;

first interpolation means for performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

second interpolation means for performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

first image interpolation means for generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;

second image interpolation means for generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship;

image rotation means for rotating the interpolated image, obtained in the second image interpolation means, in a line of sight direction at the user viewpoint position; and

outputting means for outputting the image after the rotation as an interpolated image.

45. An image processing method of generating a morphed image by processing first and second images respectively obtained by sensing an object at first and second viewpoint positions to obtain a morphed image comprising:

a segmentation step of segmenting the first image into a plurality of first partial images in accordance with a plurality of depths from the first viewpoint position, and also segmenting the second image into a plurality of second partial images in accordance with a plurality of depths from the second viewpoint position; and

a processing step of performing a morphing process on the first and second images on the basis of the depths and a change of a viewpoint position.

46. The method according to claim 45, wherein said first and second images are two-dimensional images;

and said segmentation step further comprises;

a display step of displaying the first and second images; and

a first designation step of designating partial areas in the displayed first and second images as a pair of two-dimensional partial images by adding hierarchical information corresponding to a depth to each of the partial images.

47. The method according to claim 46, wherein the first designation step comprises:

a second designation step of designating, in the display step, an occluded partial area in which a two-dimensional partial image belonging to a layer farther from the arbitrary viewpoint position is occluded by a two-dimensional partial image belonging to a layer nearer to the arbitrary viewpoint position; and

a retouch step of allowing a user to retouch a pixel value of the occluded partial area.

48. The method according to claim 46, wherein said first designation step comprises a rewrite step of rewriting pixel values of one or a plurality of pairs of two-dimensional partial images having a second hierarchical information value other than a first hierarchical information value into a predetermined fixed value with respect to one pair of two-dimensional partial images having the first hierarchical information value.

49. The method according to claim 48, wherein the predetermined fixed value is a pixel value representing black.

50. The method according to claim 46, wherein said segmentation step comprises a third designation step of designating a plurality of partial areas in one pair of first and second two-dimensional partial images corresponding to a given layer.

51. The method according to claim 46, wherein said segmentation step comprises storing the first and second original images and a plurality of pairs of two-dimensional partial images, together with hierarchical information.

52. An image display method of reading out a plurality of pairs of two-dimensional partial images generated and stored by the image processing method defined in claim 49, and performing morphing processing to generate an image from an arbitrary viewpoint position comprising:

a first morphing processing step of performing, for pairs of two-dimensional partial images corresponding to all depths, processing of generating a partial morphed image from an arbitrary viewpoint position on the basis of one pair of first and second two-dimensional partial images corresponding to each depth; and

a display step of synthesizing partial morphed images corresponding to the respective depths and generated in the first morphing processing step, and displaying a synthesized image on a predetermined display unit.

53. The method according to claim 52, wherein the method is applied to a walkthrough application.

54. The method according to claim 53, further comprising an input step of inputting information about the arbitrary viewpoint position.

55. The method according to claim 54, wherein said input step comprises virtually changing a viewpoint position value.

56. The method according to claim 52, further comprising:

a second morphing processing step of generating a morphed image directly based on the first and second two-dimensional images; and

a switching step of selectively switching the first morphing processing step and the second morphing processing step.

57. The method according to claim 56, wherein the method further comprises:

an input step of inputting information about the arbitrary viewpoint position; and

a step of preparing a memory for determining that one of the first morphing processing and the second morphing processing is to be applied to an arbitrary walkthrough route in a walkthrough virtual space when a viewpoint position of the user exists in the walkthrough route, and storing information indicating a determination result together with a position of the walkthrough route; and

said switching step comprises extracting determination result information corresponding to viewpoint position information input in the input step from the memory, and selecting and operating one of the first morphing processing step and the second morphing processing step on the basis of the information.

58. The method according to claim 52, wherein the first morphing step comprises:

a step of preparing an image storing means storing a plurality of images together with viewpoint position information corresponding thereto;

a step of detecting both a line of sight direction and viewpoint position of a user;

a step of searching in the image storing means for the first to third images including corresponding points in the line of sight direction of the user;

a determination step of determining whether any two of the first to third images have an inclusive relationship;

a calculation step of obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;

a first interpolation step of performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

a second interpolation step of performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

a first image interpolation step of generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;

a second image interpolation step of generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship,

an image rotation step of rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position; and

an outputting step of outputting the image after the rotation as an interpolated image.

59. A computer program storage medium that stores a program executed by a computer, storing a program code that implements the method defined in claim 45.

60. An image processing apparatus for interpolating a plurality of images sensed at a plurality of different positions to generate an image at a viewpoint position different from the plurality of positions, comprising:

discrimination means for discriminating presence/absence of an inclusive relationship, in which one image is included in the other image, between the plurality of images; and

image processing means for interpolating the plurality of images on the basis of a discrimination result obtained by said discrimination means,

wherein said image processing means is designed to use different interpolation methods depending on whether said discrimination means discriminates the presence or absence of an inclusive relationship between the plurality of images.

61. The apparatus according to claim 60, wherein said image processing means linearly interpolates coordinates of corresponding points of the plurality of images when said discrimination means discriminates the absence of the inclusive relationship between the images, and nonlinearly interpolates the coordinates of the corresponding points of the plurality of images when said discrimination means discriminate the presence of the inclusive relationship between the images.

62. The apparatus according to claim 61, wherein said discrimination means discriminates whether lens centers of cameras are included in another image when the plurality of images are sensed at the plurality of positions.

63. The apparatus according to claim 60, wherein said image processing means performs morphing processing for the plurality of images.

64. An image processing method of interpolating a plurality of images sensed at a plurality of different positions to generate an image at a viewpoint position different from the plurality of positions, comprising:

a discrimination step of discriminating presence/absence of an inclusive relationship, in which one image is included in the other image, between the plurality of images; and

a image processing step of interpolating the plurality of images on the basis of a discrimination result obtained in said discrimination step,

wherein said image processing step comprises using different interpolation methods depending on whether the presence or absence of an inclusive relationship between the plurality of images is discriminated in said discrimination step.

65. The method according to claim 64, wherein said image processing step comprises linearly interpolating coordinates of corresponding points of the plurality of images when the absence of the inclusive relationship between the images is discriminated in said discrimination step, and nonlinearly interpolating the coordinates of the corresponding points of the plurality of images when the presence of the inclusive relationship between the images is discriminated in said discrimination step.

66. The method according to claim 65, wherein said discrimination step comprises discriminating whether lens centers of cameras are included in another image when the plurality of images are sensed at the plurality of positions.

67. The method according to claim 64, wherein said image processing step comprises performing morphing processing for the plurality of images.