Method and apparatus for displaying a synthesized image viewed from a virtual point of view

• US 7,307,655 B1
• Filed: 07/29/1999
• Issued: 12/11/2007
• Est. Priority Date: 07/31/1998
• Status: Expired due to Term

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Okamoto, Shusaku, Nakagawa, Masamichi, Nobori, Kunio, Morimura, Atsushi
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): Selby, Gevell
• Application Number:
  US09/744,787
• Time in Patent Office: 3,057 Days
• Field of Search: 348/207.99, 348/216, 348/148, 348/149, 348/113, 348/116, 348 42- 50, 348/51, 348139-141, 348/373, 348/333.06, 348/118, 340/929, 340/903, 358/88, 358/229, 358/3, 358/209, 352/57, 382/106, 382/154, 356/3, 356/3.14, 356/12
• US Class Current: 348/222.1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

An image generating apparatus comprising: a camera or a plurality of cameras; space reconstructing means for mapping the input image from said camera, into a predetermined space model of a predetermined three-dimensional space; point-of-view converting means of synthesizing an image viewed from an arbitrary virtual point of view in said predetermined three-dimensional space by referencing to said space data mapped by said space reconstructing means; and displaying means of displaying the image converted by said point-of-view converting means.

Claims

1. An image generating apparatus comprising:

at least one camera provided on a vehicle;

a space reconstructing unit operable to generate a space data by mapping an input image from said at least one camera into a predetermined space model of a predetermined three-dimensional space;

a converting unit operable to generate a synthesized image by referencing to said space data, said synthesized image including a shape of the vehicle and a surrounding space of the vehicle, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle, said virtual viewpoint being different from a viewpoint of the camera; and

a displaying unit operable to display the synthesized image.

2. An image generating apparatus of claim 1 comprising a camera parameter table for storing a camera parameter indicating characteristics of said camera, wherein said space reconstructing unit performs the mapping depending on said camera parameter.

3. An image generating apparatus of claim 1 wherein said space model is a predetermined plane in said three-dimensional space.

4. An image generating apparatus of claim 3 wherein said camera is attached on said vehicle and said predetermined plane is the road surface under the vehicle.

5. An image generating apparatus of claim 1 wherein, during the constitution of a predetermined point P of said space model as a pixel from said virtual viewpoint, in case that said point P corresponds to an image shot by one camera, said converting unit determines a color of said pixel using a color of pixels of the image from the camera.

6. An image generating apparatus of claim 1 wherein, during the constitution of a predetermined point P of said space model as a pixel from said virtual viewpoint, in case that said point P corresponds to an image shot by a plurality of cameras, said converting unit determines a color of said pixel using a color of pixels of the images from said plurality of cameras.

7. An image generating apparatus of claim 1 wherein, during the constitution of a predetermined point P of said space model as a pixel from said virtual viewpoint, in case that said point P does not correspond to an image shot by any camera, said converting unit determines a color of said pixel using a color interpolated by the color of the periphery of said point P or using a color that is identifiable as a part where there is no object.

8. An image generating apparatus of claim 2 comprising calibrating means of obtaining the camera parameter indicating said camera characteristics by input or calculation.

9. An image generating apparatus of claim 8 comprising characteristic point extracting means of acquiring, by manual input or automatic extraction, a plurality of characteristic points necessary to obtain the camera parameter by said calculation, wherein

said calibrating means calculates at least any of an attachment position of the camera, an attachment angle of the camera, a lens distortion correction value of the camera, and a focal length of the camera using the characteristic point extracted by said characteristic point extracting means.

10. An image generating apparatus of claim 9 wherein said characteristic point extracting means includes the function of extracting a plurality of characteristic points a coordinate value of which is known in said three-dimensional space, and wherein

said calibrating means calculates said camera parameter using said plurality of characteristic points.

11. An image generating apparatus of claim 10 wherein said camera is attached on said vehicle and wherein a marker provided in a part of the vehicle is used as the plurality of characteristic points, the coordinate value of which is known in said three-dimensional space.

12. An image generating apparatus of claim 10 comprising characteristic point generating means of generating, within a field of view of said camera, a plurality of characteristic points the coordinate value of which is known in said three-dimensional space.

13. An image generating apparatus of claim 12 wherein said characteristic point generating means has the function of irradiating a pattern light as said characteristic point within the field of view of said camera.

14. An image generating apparatus of any of claims 8 to 13 comprising a temperature sensor for measuring a temperature and a temperature correction table for storing an amount of variation in said camera parameter which changes in response to the temperature, wherein

said calibrating means has the function of monitoring the temperature of said temperature sensor successively and of correcting said camera parameter depending on said temperature correction table in response to a temperature change.

15. An image generating apparatus of claim 14 wherein said temperature sensor is attached in the vicinity of or in association with each of said camera.

16. An image generating apparatus of any of claims 4 or 11, wherein said space data generated by said space reconstructing unit is temporarily stored in a space data buffer, and wherein said apparatus comprises:

movement direction detecting means for detecting a movement direction of said vehicle;

movement distance detecting means for detecting a movement distance of the vehicle in a unit time; and

space data converting means for converting the space data stored in said space data buffer using at least either the movement direction of the vehicle detected by said movement direction detecting means or the movement distance of the vehicle detected by said movement distance detecting means.

17. An image generating apparatus of claim 1 wherein said camera is attached on said vehicle, and wherein said apparatus comprises:

camera correction indicating means of detecting a situation requiring the calibration of the camera thereby indicating a camera calibration when the camera calibration is necessary; and

correction history recording means of recording a date and time or a running distance when said camera calibration is performed.

18. An image generating apparatus of claim 17 wherein said camera correction indicating means has the function of referencing to said correction history recording means thereby indicating the camera calibration to the driver when a predetermined time has elapsed since a time of the previous calibration or when the running distance since the time of the previous calibration has reached a predetermined distance.

19. The image generating apparatus of claim 1 wherein a relative positional relationship of the space model with respect to the vehicle is fixed.

29. An image generating apparatus of claim 1 wherein said camera is attached on said vehicle and said space reconstructing unit comprises space model generating means of generating said space model.

30. An image generating apparatus of claim 1 wherein said space model is a model of one or more planes, a model of one or more curved surfaces, or a model of the combination of one or more planes and one or more curved surfaces.

31. An image generating apparatus of claim 29 wherein said space model generating means generates, as said space model, at least a first plane of a road surface and at least one second plane intersecting said first plane.

32. An image generating apparatus of claim 31 comprising:

a distance sensor for measuring a distance; and

obstacle detecting means of measuring, as a situation of surroundings of said vehicle, at least a distance to an obstacle existing in the surroundings of the vehicle by said distance sensor, and wherein

said second plane is generated by said space model generating means when the distance from the vehicle to the obstacle detected by said obstacle detecting means is smaller than a predetermined value.

33. An image generating apparatus of claim 32 wherein said second plane generated by said space model generating means is removed from said space model when said distance from the vehicle to the obstacle is larger than said predetermined value.

34. An image generating apparatus of claim 29 wherein said space model generating means generates said space model in response to a situation of surroundings of said vehicle.

35. An image generating apparatus of claim 34 comprising:

a distance sensor for measuring a distance; and

obstacle detecting means for measuring, as the situation of the surroundings of said vehicle, at least a distance to an obstacle existing in the surroundings of the vehicle by said distance sensor.

36. An image generating apparatus of claim 34 wherein said vehicle situation is a condition of a road surface in the surroundings of said vehicle, and wherein said apparatus comprises road-surface characteristics detecting means of detecting characteristics of the condition of the road surface.

37. An image generating apparatus of claim 36 wherein a white line indicating a parking space, an end point of the white line, or a corner of intersecting white lines is used as said characteristics of the condition of the road surface.

38. An image generating apparatus of claim 34 further comprising:

movement direction detecting means of detecting a movement direction of said vehicle; and

movement distance detecting means of detecting a movement distance of the vehicle in a unit time; and

wherein said apparatus successively corrects said space model depending on a present position of characteristics on a road surface obtained from process results of said movement direction detecting means and said movement distance detecting means.

39. An image generating apparatus of claim 36 comprising characteristics correcting means of correcting a process result while displaying the process result of said road-surface characteristics detecting means on said displaying unit.

20. An image generating method comprising:

generating a space data by mapping an input image from at least one camera provided on a vehicle, located in a predetermined three-dimensional space, into a predetermined space model of said predetermined three-dimensional space; and

generating a synthesized image by referencing to the space data, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle, said virtual viewpoint being different from a viewpoint of the camera;

wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

21. An image generating method of claim 20 wherein said space model is a plane constituting a part of said three-dimensional space.

22. An image generating method of claim 20 wherein said mapping is performed depending on a camera parameter indicating camera characteristics.

23. An image generating method of claim 22 comprising a calibrating step of obtaining the camera parameter indicating said camera characteristics by input or calculation.

24. An image generating method of claim 23 wherein said calibrating step is a step of obtaining the camera parameter by calculation, and wherein said method comprises a characteristic point extracting step of acquiring, by manual input or automatic extraction, a plurality of characteristic points necessary for the calculation of said camera parameter.

25. An image generating method of claim 24 comprising a characteristic point generating step of generating, within a field of view of said camera, a plurality of characteristic points a coordinate value of which is known in said three-dimensional space.

26. An image generating method of claim 25 wherein said characteristic point generating step includes at least the step of irradiating a light pattern as said characteristic point within the field of view of said camera.

27. An image generating method of claim 25 wherein said camera is attached on said vehicle, and wherein said method comprises:

a movement direction detecting step of detecting a movement direction of said vehicle;

a movement distance detecting step of detecting a movement distance of said vehicle in a unit time; and

a space data converting step of converting the space data using both the movement direction of the vehicle detected by said movement direction detecting step and the movement distance of the vehicle detected by said movement distance detecting step.

28. An image generating method of claim 25 wherein said camera is attached on said vehicle, and wherein

said method comprises:

a camera correction indicating step of detecting a situation requiring a calibration of said camera thereby indicating a camera calibration when the camera calibration is necessary; and

a correction history recording step of recording a date and time or a running distance when the camera calibration is performed.

47. The image generating method of claim 20 wherein a relative positional relationship of the space model with respect to the vehicle is fixed.

40. An image generating apparatus comprising:

at least one camera provided on a vehicle;

a camera parameter table for storing a camera parameter indicating camera characteristics;

mapping means for mapping the input image from said camera, into a predetermined space model of the three-dimensional space using said camera parameter;

a converting means for synthesizing an image by referencing to said space data mapped by said mapping means, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle; and

camera parameter correcting means for correcting the parameter of said camera, wherein the virtual viewpoint is different from a stereoscopic image formed from input images, and the synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

41. An image generating apparatus of claim 40 wherein said converting means switches said virtual viewpoint according to a time of process or a time of normal operation of said camera parameter correcting means.

42. An image generating apparatus of claim 41 wherein said converting means determines said virtual viewpoint depending on the camera parameter of any of said cameras in the time of process of said camera parameter correcting means.

43. An image generating apparatus of claim 42 wherein said converting means determines said virtual viewpoint depending on a camera parameter of the camera being under the correction, in the time of process of said camera parameter correcting means.

44. An image generating apparatus of any of claims 40 to 43 wherein a direction of an operation of changing the direction of the viewpoint is set to be reverse to the direction of said viewpoint, in said converting means.

45. An image generating apparatus of any of claims 40 to 43, further comprising displaying means, operable to display the synthesized image, wherein said displaying means displays a mark indicating a boundary, in superposition on the synthesized image in a boundary part where respective images contact when displaying the image from each camera.

46. An image generating apparatus of claim 40 comprising:

guide data recording means of storing guide data serving as a guide for the correction of said camera parameter; and

displaying means of displaying the input image from said camera and said guide data in superposition.

48. An image generating apparatus of claim 46 wherein said guide data is characteristics of a body of said vehicle represented by a point and/or a line segment.

49. An image generating apparatus of claim 46 wherein said guide data is obtained using:

characteristic point generating means for generating, within a field of view of said camera, a plurality of characteristic points, a coordinate value of which is known in the three-dimensional space;

characteristic point extracting means of extracting said plurality of characteristic points; and

said plurality of extracted characteristic points.

50. An image generating apparatus comprising:

at least one camera provided on a vehicle;

image inputting means of inputting an image from said camera;

a camera parameter table for storing a camera parameter indicating camera characteristics;

mapping means for mapping, using said camera parameter, each pixel constituting the input image into a predetermined space model of a predetermined three-dimensional space generated by a modeling of a situation of surroundings of the vehicle;

a point-of-view parameter table for storing a virtual-point-of-view parameter including at least the data of a position and an orientation for each of a plurality of virtual viewpoint;

converting means of synthesizing an image viewed from each of said plurality of virtual viewpoints of said point-of-view parameter table in said predetermined three-dimensional space by referencing to the space data mapped by said mapping means, said synthesized image arranged in a manner as looked down from the plurality of virtual viewpoints located outside the vehicle;

point-of-view parameter correcting means of correcting the contents of said point-of-view parameter table; and

displaying means of displaying the image converted by said converting means, wherein each of the virtual viewpoints is different from a viewpoint of a stereoscopic image formed from input images, and wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

51. An image generating apparatus of claim 50 wherein said plurality of virtual viewpoints correspond to said cameras, respectively.

52. An image generating apparatus of claim 50 or 51 wherein a direction of operation is set to be reverse to an actual alteration of the point-of-view parameter at least for the operations in orientation, position, and rotation among the operations of changing said virtual-point-of-view parameter in said point-of-view parameter correcting means.

53. An image generating apparatus of any of claims 50 or 51 wherein, when said point-of-view parameter correcting means corrects said virtual-point-of-view parameter, a fixed tentative virtual viewpoint is provided, and a progress of the correction of the virtual viewpoint being under the correction is successively displayed as the synthesized image viewed from said tentative virtual viewpoint.

54. An image generating apparatus of any of claims 50 or 51 wherein said displaying means displays a mark indicating a boundary, in superposition on the synthesized image in the boundary part where respective images contact when displaying the image from each camera.

55. An image generating apparatus comprising:

at least one camera provided on a vehicle;

space reconstructing means of mapping input images from said plurality of cameras, into a space model representing a predetermined three-dimensional space;

converting means of synthesizing the input images of said cameras into a single virtual image, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle, the virtual viewpoint being different from a viewpoint of each input image and different from a viewpoint of a stereoscopic image formed from input images; and

displaying means of displaying the image converted by said converting means, wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

56. An image generating apparatus comprising:

at least one camera provided on a vehicle;

space reconstructing means of mapping an input image from said camera, into a predetermined space model of a predetermined three-dimensional space;

converting means of synthesizing an image by referencing to space data mapped by said space reconstructing means, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle; and

displaying means of displaying the image converted by said converting means, wherein the virtual viewpoint is different from a point on a line segment which connects respective viewpoints of input images from which a stereoscopic image is formed, wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

57. An image generating apparatus comprising:

at least one camera provided on a vehicle;

space reconstructing means of mapping input images from said at least one camera, into a space model representing a predetermined three-dimensional space;

converting means of synthesizing the input images of said cameras into a single virtual image, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle, the virtual viewpoint being different from a viewpoint of each input image, and different from a point on a line segment which connects respective viewpoints of the input images from which a stereoscopic image is formed; and

displaying means of displaying the image converted by said converting means, wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

58. An image generating method comprising:

generating a space data by mapping an input image from at least one camera provided on an outer surface side of a vehicle, located in a predetermined three-dimensional space, into a predetermined space model of said predetermined three-dimensional space; and

generating a synthesized image by referencing to the space data, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle, said virtual viewpoint being different from a viewpoint of the camera, wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

59. An image generating apparatus comprising:

at least one camera provided on a vehicle;

a camera parameter table for storing a camera parameter indicating camera characteristics;

mapping means for mapping an input image from said camera, into a predetermined space model of a three-dimensional space using said camera parameter;

converting means for synthesizing an image by referencing to space data mapped by said mapping means, said synthesized image arranged in a manner as looked down from a plurality of virtual viewpoints located outside the vehicle; and

camera parameter correcting means for correcting the parameter of said camera, wherein each of the virtual viewpoints is different from a point on a line segment which connects respective viewpoints of input images from which a stereoscopic image is formed, wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

60. An image generating apparatus comprising:

at least one camera provided on a vehicle;

image inputting means of inputting an image from said camera;

a camera parameter table for storing a camera parameter indicating camera characteristics;

mapping means for mapping, using said camera parameter, each pixel constituting the input image into a predetermined space model of a predetermined three-dimensional space generated by modeling of a situation of surroundings of the vehicle;

a point-of-view parameter table for storing a virtual-point-of-view parameter including at least data of a position and an orientation for each of a plurality of viewpoints;

converting means of synthesizing an image by referencing to the space data mapped by said mapping means, said synthesized image arranged in a manner as looked down from the plurality of virtual viewpoints located outside the vehicle;

point-of-view parameter correcting means of correcting contents of said point-of-view parameter table; and

displaying means of displaying the image converted by said converting means, wherein each of the virtual viewpoints is different from a point on a line segment which connects respective viewpoints of input images from which a stereoscopic image is formed, wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.

61. An image generating apparatus comprising:

at least one camera provided on a vehicle;

space reconstructing means of mapping an input image from said camera, into a predetermined space model of a predetermined three-dimensional space, said space model having a fixed relative positional relationship with respect to said vehicle;

converting means of synthesizing an image by referencing to space data mapped by said space reconstructing means, said synthesized image arranged in a manner as looked down from a virtual viewpoint located outside the vehicle, said virtual viewpoint being different from a viewpoint of the camera; and

display means of displaying the image converted by said converting means,

wherein said synthesized image includes a shape of the vehicle and a surrounding space of the vehicle.