IMAGE PROCESSING APPARATUS FOR INTERPOLATING AND GENERATING IMAGES FROM AN ARBITRARY VIEW POINT

• US 20020171666A1
• Filed: 08/18/1999
• Published: 11/21/2002
• Est. Priority Date: 02/19/1999
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: ENDO, TAKAAKI, KATAYAMA, AKIHIRO
• Granted Patent: US 6,636,234 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images

Abstract

There is provided a method of generating an interpolated image without any constraint even in a wide moving range. When two real images have an inclusive relationship, the two images are rotated to make their optical axes coincide with each other. Nonlinear interpolation processing is performed for the two images after the rotations, and the interpolated image is rotated in the line of sight direction of a user.

Claims

1. A method of generating an interpolated image at a user viewpoint position different from first and second viewpoint positions on the basis of first and second images obtained at the first and second viewpoint positions, comprising:

the first correcting step of correcting the first and second images about the first and second viewpoint positions, respectively, such that optical axes of cameras which respectively sensed the first and second images coincide with a first direction substantially coinciding with a direction of a base line connecting the first and second viewpoint positions;

the first coordinate interpolation step of calculating coordinates of a corresponding point of an interpolated image having the first direction as a line of sight direction at the user viewpoint position by performing nonlinear proportional distribution for coordinates of a first corresponding point in the first image with respect to a predetermined feature point and coordinates of a second corresponding point in the second image with respect to the feature point; and

the second rotation step of rotating the interpolated image from the first direction to the line of sight direction at the user viewpoint position, and outputting the image after correction as an interpolated image.

2. The method according to claim 1, wherein the method further comprises the determination step of determining whether a field of view of the first image includes a lens center of the second image or a field of view of the second image includes a lens center of the first image, and

the first correction step is executed when YES is obtained in the determination step.

3. The method according to claim 1, further comprising:

the determination step of determining whether a field of view of the first image includes a lens center of the second image or a field of view of the second image includes a lens center of the first image;

the third correction step of rotating the first and second images about the first and second viewpoint positions, respectively, such that optical axes of the images become substantially perpendicular to the direction of the base line, when NO is obtained in the determination step;

the second coordinate interpolation step of calculating coordinates of a corresponding point of an interpolated image having the second direction perpendicular to the direction of the base line at the user viewpoint position as a line of sight direction by performing linear proportional distribution for coordinates of a first corresponding point of the first image and coordinates of a second corresponding point of the second image; and

the fourth rotation step of rotating the interpolated image from the second direction to the line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.

4. The method according to claim 1, wherein when the user viewpoint position internally divides an interval between the first and second viewpoint positions at t:

1−

t, a nonlinear proportional distribution processing in the first coordinate interpolation step is executed by making a reciprocal value 1/x_e of a coordinate of the interpolated image divide an interval between a reciprocal value 1/x_A of the coordinate of the first corresponding point and a reciprocal value 1/x_B of the coordinate of the second corresponding point at t;

1−

t.

5. The method according to claim 3, wherein when the user viewpoint position internally divides an interval between the first and second viewpoint positions at t:

1−

t, the linear proportional distribution processing in the second coordinate interpolation processing is executed by making a coordinate value x_e of the interpolated image divide an interval between a coordinate value x_A of the first corresponding point and a coordinate value x_B of the second corresponding point at t;

1−

t.

6. The method according to claim 1, further comprising the step of determining whether the first and second images have at least corresponding points derived from the same object.

7. The method according to claim 1, further comprising the step of searching an image database for another image having at least a corresponding point when the first and second images do not have at least corresponding points derived from the same object.

8. The method according to claim 1, wherein the first and second viewpoint positions are defined by viewpoint positions and optical axis directions of the cameras in image sensing operation, and the user viewpoint position is defined by a viewpoint position and line of sight direction of the user.

9. The method according to claim 1, wherein the interpolated image output in the first or fourth rotation step is presented as a panoramic image to the user.

34. A computer program storage medium that stores a program executed by a computer, storing a program code that implements the interpolation method defined in claim 1.

10. A method of generating an interpolated image at a user viewpoint position different from first, second, and third viewpoint positions on the basis of first, second, and third images respectively obtained at the first, second, and third viewpoint positions, comprising:

the determination step of determining whether any two of the first to third images have an inclusive relationship;

internal ratio calculation step of obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;

the first coordinate interpolation step of performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

the second coordinate interpolation step of performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

the first image interpolation step of generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;

the second image interpolation step of generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship; and

the rotation step of rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.

11. The method according to claim 10, wherein the first coordinate interpolation step comprises the step of rotating the two arbitrary images at viewpoint positions in the base line direction to make optical axes of the images coincide with each other, and the interpolation processing is performed for the two rotated images.

12. The method according to claim 10, wherein the second coordinate interpolation step comprises the step of rotating the two arbitrary images at viewpoint positions in a direction perpendicular to the base line direction to make the two arbitrary images become parallel to each other, and the interpolation processing is performed for the two rotated images.

13. The method according to claim 10, wherein the determination step comprises determining whether a field of view of one image includes a lens center of the other image.

14. The method according to claim 10, wherein a viewpoint position which divides, at a second internal ratio, an interval between the third viewpoint position and a temporary virtual position P_T which divides an interval between the first and second viewpoint positions at a first internal ratio coincides with the user viewpoint position.

15. The method according to claim 10, wherein the first coordinate interpolation step comprises obtaining a reciprocal value 1/x_e of a coordinate of the temporary virtual image by internally dividing an interval between a reciprocal value 1/x_A of a coordinate of a first corresponding point of one of the two images and a reciprocal value 1/x_B of a coordinate of a second corresponding point of the other image at the first internal ratio.

16. The method according to claim 10, wherein the second coordinate interpolation step comprises obtaining a coordinate value x_e of the temporary virtual image by internally dividing an interval between a coordinate value x_A of a first corresponding point of one of the two images and a coordinate value x_B of a second corresponding point of the other image at the second internal ratio.

17. The method according to claim 10, further comprising the step of determining whether the first to third images have at least corresponding points derived from the same object.

18. The method according to claim 10, further comprising the step of searching an image database for another image having at least a corresponding point when the first to third images do not have at least corresponding points derived from the same object.

19. The method according to claim 10, wherein the first to third viewpoint positions are defined by viewpoint positions and optical axis directions of the cameras in image sensing operation, and the user viewpoint position is defined by a viewpoint position and line of sight direction of the user.

20. The method according to claim 10, wherein the interpolated image output in the rotation step is presented as a panoramic image to the user.

21. An image processing apparatus comprising:

an image database storing a plurality of images together with pieces of viewpoint position information corresponding thereto in advance;

means for detecting both a line of sight direction and viewpoint position of a user;

means for searching said image database for the first to third images including corresponding points in the line of sight direction of the user;

determination means for determining whether any two of the first to third images have an inclusive relationship;

internal ratio calculation means for obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;

first coordinate interpolation means for performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

second coordinate interpolation means for performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

first image interpolation means for generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;

second image interpolation means for generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship; and

rotation means for rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.

22. The apparatus according to claim 21, wherein said first coordinate interpolation means comprises means for rotating the two arbitrary images at viewpoint positions in the base line direction to make optical axes of the images coincide with each other, and the interpolation processing is performed for the two rotated images.

23. The apparatus according to claim 21, wherein said second coordinate interpolation means comprises means for rotating the two arbitrary images at viewpoint positions in a direction perpendicular to the base line direction to make the two arbitrary images become parallel to each other, and the interpolation processing is performed for the two rotated images.

24. The apparatus according to claim 21, wherein the determination means determines whether a field of view of one image includes a lens center of the other image.

25. The apparatus according to claim 21, wherein a viewpoint position which divides an interval between the third viewpoint position and a temporary virtual position P_T which divides, at a second internal ratio, an interval between the first and second viewpoint positions at a first internal ratio coincides with the user viewpoint position.

26. The apparatus according to claim 21, wherein the first coordinate interpolation means obtains a reciprocal value 1/x_e of a coordinate of the temporary virtual image by internally dividing an interval between a reciprocal value 1/x_A of a coordinate of a first corresponding point of one of the two images and a reciprocal value 1/x_B of a coordinate of a second corresponding point of the other image at the first internal ratio.

27. The apparatus according to claim 21, wherein the second coordinate interpolation means obtains a coordinate value x_e of the temporary virtual image by internally dividing an interval between a coordinate value x_A of a first corresponding point of one of the two images and a coordinate value x_B of a second corresponding point of the other image at the second internal ratio.

28. The apparatus according to claim 21, further comprising means for determining whether the first to third images have at least corresponding points derived from the same object.

29. The apparatus according to claim 21, further comprising means for searching an image database for another image having at least a corresponding point when the first to third images do not have at least corresponding points derived from the same object.

30. The apparatus according to claim 21, wherein the first to third viewpoint positions are defined by viewpoint positions and optical axis directions of the cameras in image sensing operation, and the user viewpoint position is defined by a viewpoint position and line of sight direction of the user.

31. The apparatus according to claim 21, wherein the interpolated image output in said rotation means is presented as a panoramic image to the user.

32. The apparatus according to claim 21, further comprising means for storing a temporary virtual image generated by said first image interpolation means in said image database, together with corresponding viewpoint position information.

33. The apparatus according to claim 21, further comprising means for storing a temporary virtual image generated by said second image interpolation means in said image database, together with corresponding viewpoint position information.

36. The apparatus according to claim 35, wherein said first and second images are two-dimensional images;

and said segmentation means further comprises;

display means for displaying the first and second two-dimensional images; and

first designation means for designating partial areas in the displayed first and second two-dimensional images as a pair of two-dimensional partial images by adding hierarchical information corresponding to a depth to each of the partial images.

37. The apparatus according to claim 36, wherein said first designation means comprises:

second designation means for designating, on said display means, a partial area in which a two-dimensional partial image belonging to a layer farther from the arbitrary viewpoint position is occluded by a two-dimensional partial image belonging to a layer nearer to the arbitrary viewpoint position; and

retouch means for allowing a user to retouch a pixel value of the occluded partial area.

38. The apparatus according to claim 36, wherein said first designation means comprises rewrite means for rewriting pixel values of one or a plurality of pairs of two-dimensional partial images having a second hierarchical information value other than a first hierarchical information value into a predetermined fixed value with respect to one pair of two-dimensional partial images having the first hierarchical information value.

39. The apparatus according to claim 38, wherein the predetermined fixed value is a pixel value representing black.

40. The apparatus according to claim 35, wherein said segmentation means comprises third designation means for designating a plurality of partial areas in one pair of first and second two-dimensional partial images corresponding to a given layer.

41. The apparatus according to claim 35, wherein said segmentation means stores first and second original two-dimensional images and a plurality of pairs of two-dimensional partial images, together with hierarchical information.

42. An image display apparatus for reading out a plurality of pairs of two-dimensional partial images generated and stored by said image processing apparatus defined in claim 35, and performing morphing processing to generate an image from an arbitrary viewpoint position comprising:

first morphing processing means for performing, for pairs of two-dimensional partial images corresponding to all depths, processing of generating a partial morphed image from an arbitrary viewpoint position on the basis of one pair of first and second two-dimensional partial images corresponding to each depth; and

display means for synthesizing partial morphed images corresponding to the respective depths and generated by said first morphing processing means, and displaying a synthesized image on a predetermined display unit.

43. The apparatus according to claim 42, wherein said apparatus is applied to a walkthrough application.

44. The apparatus according to claim 43, further comprising input means for inputting information about the arbitrary viewpoint position.

45. The apparatus according to claim 44, wherein said input means virtually changes a viewpoint position value.

46. The apparatus according to claim 42, further comprising:

second morphing processing means for generating a morphed image directly based on the first and second two-dimensional images; and

switching means for selectively switching said first morphing processing means and said second morphing processing means.

47. The apparatus according to claim 46, wherein said apparatus further comprises:

input means for inputting information about the arbitrary viewpoint position; and

a memory for determining that one of the first morphing processing and the second morphing processing is to be applied to an arbitrary walkthrough route in a walkthrough virtual space when a viewpoint position of the user exists in the walkthrough route, and storing information indicating a determination result together with a position of the walkthrough route, and

said switching means extracts determination result information corresponding to viewpoint position information input by said input means from said memory, and selects and operates one of said first morphing processing means and said second morphing processing means on the basis of the information.

48. The apparatus according to claim 42, wherein said first morphing means comprises:

an image database storing a plurality of images together with pieces of viewpoint position information corresponding thereto in advance;

means for detecting both a line of sight direction and viewpoint position of a user;

means for searching said image database for the first to third images including corresponding points in the line of sight direction of the user;

determination means for determining whether any two of the first to third images have an inclusive relationship;

internal ratio calculation means for obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;

first coordinate interpolation means for performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

second coordinate interpolation means for performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

first image interpolation means for generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;

second image interpolation means for generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship; and

rotation means for rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.

50. The method according to claim 49, wherein said first and second images are two-dimensional images;

and said segmentation step further comprises;

the display step of displaying the first and second two-dimensional images; and

the first designation step of designating partial areas in the displayed first and second two-dimensional images as a pair of two-dimensional partial images by adding hierarchical information corresponding to a depth to each of the partial images.

51. The method according to claim 50, wherein the first designation step comprises:

the second designation step of designating, in the display step, a partial area in which a two-dimensional partial image belonging to a layer farther from the arbitrary viewpoint position is occluded by a two-dimensional partial image belonging to a layer nearer to the arbitrary viewpoint position; and

the retouch step of allowing a user to retouch a pixel value of the occluded partial area.

52. The method according to claim 50, wherein the first designation step comprises the rewrite step of rewriting pixel values of one or a plurality of pairs of two-dimensional partial images having a second hierarchical information value other than a first hierarchical information value into a predetermined fixed value with respect to one pair of two-dimensional partial images having the first hierarchical information value.

53. The method according to claim 52, wherein the predetermined fixed value is a pixel value representing black.

54. The method according to claim 49, wherein the segmentation step comprises the third designation step of designating a plurality of partial areas in one pair of first and second two-dimensional partial images corresponding to a given layer.

55. The method according to claim 49, wherein the segmentation step comprises storing first and second original two-dimensional images and a plurality of pairs of two-dimensional partial images, together with hierarchical information.

56. An image display method of reading out a plurality of pairs of two-dimensional partial images generated and stored by the image processing method defined in claim 49, and performing morphing processing to generate an image from an arbitrary viewpoint position comprising:

the first morphing processing step of performing, for pairs of two-dimensional partial images corresponding to all depths, processing of generating a partial morphed image from an arbitrary viewpoint position on the basis of one pair of first and second two-dimensional partial images corresponding to each depth; and

the display step of synthesizing partial morphed images corresponding to the respective depths and generated in the first morphing processing step, and displaying a synthesized image on a predetermined display unit.

57. The method according to claim 56, wherein the method is applied to a walkthrough application.

58. The method according to claim 57, further comprising the input step of inputting information about the arbitrary viewpoint position.

59. The method according to claim 44, wherein the input step comprises virtually changing a viewpoint position value.

60. The method according to claim 56, further comprising:

the second morphing processing step of generating a morphed image directly based on the first and second two-dimensional images; and

the switching step of selectively switching the first morphing processing step and the second morphing processing step.

61. The method according to claim 60, wherein the method further comprises:

the input step of inputting information about the arbitrary viewpoint position; and

the step of preparing a memory for determining that one of the first morphing processing and the second morphing processing is to be applied to an arbitrary walkthrough route in a walkthrough virtual space when a viewpoint position of the user exists in the walkthrough route, and storing information indicating a determination result together with a position of the walkthrough route, and

the switching step comprises extracting determination result information corresponding to viewpoint position information input in the input step from the memory, and selecting and operating one of the first morphing processing step and the second morphing processing step on the basis of the information.

62. The method according to claim 56, wherein the first morphing step comprises:

the step of preparing an image database storing a plurality of images together with pieces of viewpoint position information corresponding thereto in advance;

the step of detecting both a line of sight direction and viewpoint position of a user;

the step of searching the image database for the first to third images including corresponding points in the line of sight direction of the user;

the determination step of determining whether any two of the first to third images have an inclusive relationship;

the internal ratio calculation step of obtaining first and second internal ratios on the basis of a relationship between the user viewpoint position and the first to third viewpoint positions;

the first coordinate interpolation step of performing nonlinear proportional distribution processing for two arbitrary images having viewpoint positions having an inclusive relationship to interpolate/generate a temporary virtual image having a base line direction as a line of sight direction at an internal dividing point position between the two viewpoint positions;

the second coordinate interpolation step of performing linear proportional distribution processing for two arbitrary images having viewpoint positions having no inclusive relationship to interpolate/generate a temporary virtual image having a line of sight direction perpendicular to the base line direction at an internal dividing point position between the two viewpoint positions;

the first image interpolation step of generating a temporary virtual image at an internal dividing point position corresponding to the first internal ratio by performing one of first morphing processing and second morphing processing for the first and second images by using the first internal ratio depending on whether the first and second images have an inclusive relationship;

the second image interpolation step of generating an interpolated image at the user viewpoint position corresponding to the second internal ratio by performing one of the first morphing processing and second morphing processing for the temporary virtual image and third image by using the second internal ratio depending on whether the temporary virtual image and third image have an inclusive relationship; and

the rotation step of rotating the interpolated image, obtained in the second image interpolation step, in a line of sight direction at the user viewpoint position, and outputting the image after rotation as an interpolated image.

63. A computer program storage medium that stores a program executed by a computer, storing a program code that implements the method defined in claim 49.

35. An image processing apparatus for generating a morphed image by processing first and second images respectively obtained by sensing an object at first and second viewpoint positions, comprising:

segmentation means for segmenting the first image into a plurality of first partial images in accordance with a plurality of depths from the first viewpoint position, and also segmenting the second image into a plurality of second partial images in accordance with a plurality of depths from the second viewpoint position.

49. An image processing method of generating a morphed image by processing first and second images respectively obtained by sensing an object at first and second viewpoint positions to obtain a morphed image comprising:

the segmentation step of segmenting the first image into a plurality of first partial images in accordance with a plurality of depths from the first viewpoint position, and also segmenting the second image into a plurality of second partial images in accordance with a plurality of depths from the second viewpoint position.

64. An image processing apparatus for interpolating a plurality of images sensed at a plurality of different positions to generate an image at a viewpoint position different from the plurality of positions comprising:

discrimination means for discriminating presence/absence of an inclusive relationship, in which one image is included in the other image, between the plurality of images; and

image processing means for interpolating the plurality of images on the basis of a discrimination result obtained by said discrimination means,

wherein said image processing means is designed to use different interpolation methods depending on whether said discrimination means discriminates the presence or absence of an inclusive relationship between the plurality of images.

65. The apparatus according to claim 64, wherein said image processing means linearly interpolates coordinates of corresponding points of the plurality of images when said discrimination means discriminates the absence of the inclusive relationship between the images, and nonlinearly interpolates the coordinates of the corresponding points of the plurality of images when said discrimination means discriminates the presence of the inclusive relationship between the images.

66. The apparatus according to claim 65, wherein said discrimination means discriminates whether lens centers of cameras are included in another image when the plurality of images are sensed at the plurality of positions.

67. The apparatus according to claim 64, wherein said image processing means performs morphing processing for the plurality of images.

69. The method according to claim 68, wherein the image processing step comprises linearly interpolating coordinates of corresponding points of the plurality of images when the absence of the inclusive relationship between the images is discriminated in the discrimination step, and nonlinearly interpolating the coordinates of the corresponding points of the plurality of images when the presence of the inclusive relationship between the images is discriminated in the discrimination step.

70. The method according to claim 69, wherein the discrimination step comprises discriminating whether lens centers of cameras are included in another image when the plurality of images are sensed at the plurality of positions.

71. The method according to claim 68, wherein the image processing step comprises performing morphing processing for the plurality of images.

68. An image processing method of interpolating a plurality of images sensed at a plurality of different positions to generate an image at a viewpoint position different from the plurality of positions, comprising:

the discrimination step of discriminating presence/absence of an inclusive relationship, in which one image is included in the other image, between the plurality of images; and

the image processing step of interpolating the plurality of images on the basis of a discrimination result obtained in the discrimination step,

wherein the image processing step comprises using different interpolation methods depending on whether the presence or absence of an inclusive relationship between the plurality of images is discriminated in the discrimination step.