Picture synthesizing apparatus

• US 7,058,207 B2
• Filed: 02/08/2002
• Issued: 06/06/2006
• Est. Priority Date: 02/09/2001
• Status: Expired due to Term

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Mizusawa, Kazufumi, Miwa, Michio, Iida, Ryosuke
• Primary Examiner(s): Werner, Brian
• Assistant Examiner(s): LAVIN, CHRISTOPHER L
• Application Number:
  US10/067,745
• Publication Number:
  US 20020110262A1
• Time in Patent Office: 1,579 Days
• Field of Search: 382/104, 382/154, 382/293, 382/295, 348/113, 348/116, 348/118, 348/119, 345/419, 345/420, 340/988, 340/436, 340/437, 340/932.2, 340/461, 340/431
• US Class Current: 382/104
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/602 with an adjustable viewpoint
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: B62D 15/0295 by overlaying a vehicle path based on present steering angle over an image without processing that image
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

There is disclosed a picture synthesizing apparatus for superimposing an auxiliary image upon an image converted as if the image were photographed with a virtual viewpoint, displaying the image in a monitor, and providing the image such that an approach to a surrounding solid object can easily be predicted from the monitor image. The picture synthesizing apparatus includes: an image pickup unit 101 disposed in a car; a viewpoint change image synthesizing unit 102 for changing a viewpoint of the image obtained by the image pickup unit 101 and synthesizing the image; a car straight locus line generation unit 103 for generating a locus line at an arbitrary height of the car, when the car linearly advances; and a car locus line drawing unit 104 for drawing the locus line generated by the car straight locus line generation unit 103 on the image synthesized by the viewpoint change image synthesizing unit 102.

Claims

1. A picture synthesizing apparatus comprising:

an image pickup means that is in operative association with a vehicle disposed on a road surface and that is adapted to obtain an original image of an object outside said vehicle as viewed from a viewpoint of the image pickup means;

a viewpoint change image synthesizing means that is adapted to produce a synthesized image, in which the outside object is viewed from a virtual viewpoint different from the viewpoint of the image pickup means and different, from the original image obtained by said image pickup means;

wherein said synthesizing means is adapted to draw the synthesized image on a plane corresponding to the road surface;

a vehicle locus line generation means that is adapted to generate a locus line of said vehicle placed at an arbitrary height above the road surface;

a vehicle locus line drawing means that is adapted to draw the locus line, generated by said vehicle locus line generation means, on the synthesized image produced by said viewpoint change image synthesizing means such that the drawn locus line is adapted to be viewed from the viewpoint of the image pick up means, wherein when the locus line as viewed from the viewpoint of the image pickup means is projected onto the road surface to form a first projected line, the first projected line is viewed from a virtual viewpoint, and the first projected line viewed from the virtual viewpoint is again projected onto the plane corresponding to the road surface to obtain a second projected line as the locus line drawn on the synthesized image;

a storage means that is adapted to store predetermined auxiliary data; and

a drawing means that is adapted to superimpose stored auxiliary data upon a synthesized image produced by said viewpoint change image synthesizing means based on auxiliary data read from said storage means;

wherein said image pick up means is adapted to obtain an original image in which a rear part of said vehicle is positioned in a view field; and

wherein said picture synthesizing apparatus is adapted to represent said vehicle by an illustration of a skeleton or a wire frame, and is adapted to provide an image including a portion of said vehicle comprising a tire that is likely to come into contact with said outside object.

2. The picture synthesizing apparatus according to claim 1 wherein said vehicle locus line generation means comprises three-dimensional locus line generation means, road surface projection means, and synthesized image projection means.

3. The picture synthesizing apparatus according to claim 1 wherein said vehicle locus line generation means is adapted to generate a locus line that represents a vehicle that is moving linearly.

4. The picture synthesizing apparatus according to claim 1, further comprising steering wheel angle information output means that is adapted to output an angle of a a steering wheel of said vehicle, wherein said vehicle locus line generation means is adapted to generate a the locus line that is in accordance with said steering wheel angle information.

5. The picture synthesizing apparatus according to claim 4 further comprising a function for drawing a section of said vehicle moved apart from a rear end of said vehicle along said locus line with an elapse of time on said synthesized image.

6. The picture synthesizing apparatus according to claim 4 further comprising a function for drawing a solid diagram of said vehicle moved apart from a rear end of said vehicle along said locus line as a function of an elapse of time on said synthesized image.

7. The picture synthesizing apparatus according to claim 4 wherein said vehicle locus line generation means comprises three-dimensional shape storage means, three-dimensional locus region generation means, road surface projection means, and synthesized image projection means.

8. The picture synthesizing apparatus according to claim 7 wherein said three-dimensional shape storage means is adapted to store a shape of said vehicle.

9. The picture synthesizing apparatus according to claim 7 wherein said three-dimensional shape storage means is adapted to store a shape of a rectangular parallelepiped inscribing said vehicle.

10. The picture synthesizing apparatus according to claim 7 wherein said three-dimensional shape storage means is adapted to store a shape of a bumper of said vehicle.

11. The picture synthesizing apparatus according to claim 4, further comprising obstacle collision prediction means for detecting an obstacle present outside and proximate to said vehicle, and means adapted to predict a possibility of collision of said vehicle with said obstacle upon movement of said vehicle in the direction of said obstacle.

12. The picture synthesizing apparatus according to claim 11 wherein said vehicle locus line drawing means is adapted not to draw the locus line of said vehicle ahead of a collision place, when said obstacle collision prediction means predicts the collision of said vehicle with said obstacle.

13. The picture synthesizing apparatus according to claim 11 wherein said vehicle locus line drawing means is adapted to emphasize and display a place, when and where said obstacle collision prediction means predicts the collision of said vehicle with said obstacle.

14. The picture synthesizing apparatus according to claim 4, further comprising multi screen generation means for displaying the image synthesized by said viewpoint change image synthesizing means in a multiplicity of divided screens.

15. The picture synthesizing apparatus according to claim 14 wherein said vehicle locus line drawing means is adapted to draw a locus of the vehicle in each screen generated by said multi-screen generation means.

16. The picture synthesizing apparatus according to claim 15 wherein said vehicle locus line drawing means is adapted to draw the locus line or a vehicle frame indicating the same position in the same color in different screens, when said locus line is drawn in a plurality of screens.

17. The picture synthesizing apparatus according to claim 14 wherein said image pickup means includes at least one of means for picking up an image behind said vehicle, and means for picking up an image beside said vehicle.

18. The picture synthesizing apparatus according to claim 14 wherein said vehicle locus line drawing means is adapted to draw a locus line of a rear end of said vehicle on an image beside said vehicle, or an image obtained by converting said image beside the vehicle.

19. The picture synthesizing apparatus according to claim 1 further comprising a function that is adapted to interpolate a locus line on a road surface associated with said vehicle;

wherein the locus line is adapted to be at an arbitrary height with respect to said road surface, and is adapted to draw a line vertical to said road surface on said synthesized image.

20. The picture synthesizing apparatus according to claim 1 further comprising means that are adapted to draw a locus line of a bumper of said vehicle and/or a locus line of a vehicle height on said synthesized image.

21. The picture synthesizing apparatus according to claim 1 that has a function of changing a color or a thickness of said locus line in accordance with a distance from said vehicle and drawing the locus line in said color or thickness to identify said distance.

22. The picture synthesizing apparatus according to claim 1 comprising:

an auxiliary line generation means that is adapted to draw an auxiliary line adapted to indicate an arbitrary position apart from said vehicle; and

an auxiliary line drawing means that is adapted to draw an auxiliary line generated by said auxiliary line generation means on a synthesized image produced by said viewpoint change image synthesizing means.

23. The picture synthesizing apparatus according to claim 22, further comprising multi-screen generation means for displaying the image synthesized by said viewpoint change image synthesizing means in a multiplicity of divided screens, wherein said auxiliary line drawing means is adapted to draw the generated auxiliary line in each screen generated by said multi-screen generation means.

24. The picture synthesizing apparatus according to claim 22 wherein said auxiliary line generation means is adapted to generate an auxiliary line indicating a position of a rear end of said vehicle.

25. The picture synthesizing apparatus according to claim 22 wherein said auxiliary line generation means is adapted to generate an auxiliary line indicating a constant distance from a rear end of said vehicle.

26. The picture synthesizing apparatus according to claim 22 wherein said auxiliary line generation means is adapted to generate an auxiliary line indicating a width of a general vehicle.

27. The picture synthesizing apparatus according to claim 1 that has a function that is adapted to superimpose an auxiliary line upon a rear edge of said vehicle, and that is adapted to provide an image emphasizing/indicating the corresponding position.

28. The picture synthesizing apparatus according to claim 1 that has a function of providing an image showing a three dimensional illustration prepared as if the image of said vehicle were picked up by an actually disposed image pick up unit and converted/synthesized.

29. The picture synthesizing apparatus according to claim 1 wherein said portion of said vehicle that is adapted to come into contact with said outside object comprises a tire or a bumper.

30. The picture synthesizing apparatus according to claim 29 that has a function that is adapted to provide an image obtained by transforming/synthesizing an image actually obtained by said image pickup means in a region corresponding to a bumper of the illustration.

31. The picture synthesizing apparatus according to claim 1 that has a function that is adapted to provide an image showing a mirror confirmation line behind a rear end of a bumper of said vehicle by a constant distance and is adapted to be disposed horizontally with said bumper.

32. The picture synthesizing apparatus according to claim 1 that has a function that is adapted to superimpose an illustration of two wall surfaces disposed vertically to a road surface in a rear end position of said vehicle, and on an inner side of the rear end position of said vehicle, and having the same width as a width of said vehicle upon a displayed image, and providing an image metaphorically representing said vehicle as a solid object.

33. The picture synthesizing apparatus according to claim 1 that has a function that is adapted to provide an image including a road surface passage locus indicating a position obtained by projecting a position passed by a body end of said vehicle onto a road surface, and a bumper end passage locus indicating a position passed by a bumper end of said vehicle, when said vehicle moves backwards, and a solid auxiliary line for connecting the loci to produce a solid sense.

34. The picture synthesizing apparatus according to claim 33 that has a function that is adapted to provide an image showing said road surface locus like a tire trace, and is adapted to indicate the bumper end passage locus connected to a bumper end of an actual image or an illustration of said vehicle.

35. The picture synthesizing apparatus according to claim 33 that has a function that is adapted to provide an image showing said road surface passage locus drawn from a perspective of a tire explicitly indicated in an illustration.

36. The picture synthesizing apparatus according to claim 35 that has a function that is adapted to provide an image showing a mirror confirmation line that is a constant distance behind a rear end of a bumper of said vehicle and is horizontal with said bumper.

37. The picture synthesizing apparatus according to claim 33 that further comprises a locus calculation unit that is adapted to calculate a predicted locus from a steering angle signal inputted from the outside, and that has a function that is adapted to provide an image including a road surface passage locus corresponding to a steering angle of said vehicle, a bumper end passage locus corresponding to the steering angle of said vehicle, and a solid auxiliary line for connecting the loci to produce a solid sense.

38. The picture synthesizing apparatus according to claim 1 that has a function that is adapted to provide an image simultaneously showing a road surface passage locus, a bumper end passage locus, and a passage locus indicating a position passed by an appropriate height portion of a body of a vehicle, when said vehicle moves backwards.

39. The picture synthesizing apparatus according to claim 38 that has a function that is adapted to provide an image showing said road surface passage locus like a tire trace, and indicating said bumper end passage locus connected to a bumper end of an actual image or an illustration.

40. The picture synthesizing apparatus according to claim 38 that has a function that is adapted to provide an image showing said road surface passage locus drawn from a tire explicitly indicated in an illustration.

41. The picture synthesizing apparatus according to claim 40 that has a function that is adapted to provide an image showing a mirror confirmation line behind a rear end of a bumper of said vehicle by a constant distance and horizontally with said bumper.

42. The picture synthesizing apparatus according to claim 38 that further comprises a locus calculation unit that is adapted to calculate a predicted locus from a steering angle signal inputted from the outside, and that has a function that is adapted to provide an image that is adapted to simultaneously show road surface passage locus corresponding to a steering angle of said vehicle, said bumper end passage locus corresponding to the steering angle of said vehicle, a passage locus indicating a position passed by an appropriate height portion of a body of said vehicle, and an illustration representing a rear part of said vehicle.

43. The picture synthesizing apparatus according to claim 1 that has a function that is adapted to provide an image simultaneously showing a road surface passage locus, a bumper upper surface locus indicating a position passed by an end of a bumper upper surface of said vehicle, a bumper lower surface passage locus indicating a position passed by an end of a bumper lower surface of said vehicle, and an illustration imitating a rear part of said vehicle, when said vehicle moves backwards.

44. The picture synthesizing apparatus according to claim 43 that has a function that is adapted to provide an image showing said road surface passage locus like a tire trace, and is adapted to indicate the bumper end passage locus connected to a bumper end of an actual image or an illustration of the vehicle.

45. The picture synthesizing apparatus according to claim 43 that has a function that is adapted to provide an image showing said road surface passage locus drawn from a tire explicitly indicated in an illustration.

46. The picture synthesizing apparatus according to claim 45 that has a function that is adapted to provide an image showing a mirror confirmation line that is a constant distance behind a rear end of a bumper of said vehicle and is disposed horizontal to the bumper.

47. The picture synthesizing apparatus according to claim 43 that further comprises a locus calculation unit adapted to calculate a predicted locus from a steering angle signal inputted from outside, and that has a function that is adapted to provide an image simultaneously showing said road surface passage locus corresponding to the steering angle of said vehicle, said bumper lower surface passage locus corresponding to the steering angle of said vehicle, and an illustration representing a bumper of said vehicle.

48. The picture synthesizing apparatus as claimed in claim 1 wherein said vehicle is a car.

49. An image synthesis/display apparatus comprising:

a picture synthesizing apparatus comprising;

an image pickup means that is adapted to be attached to a vehicle, that is adapted to be disposed on a road surface, and is enabled to obtain an original image of an object that is outside said vehicle and is adapted to be viewed from a viewpoint of the image pickup means;

a viewpoint change image synthesizing means that is adapted to produce a synthesized image, in which the outside object is viewed from a virtual viewpoint that is different from the viewpoint of the image pickup means, from the original image obtained by said image pickup means and is adapted to draw the synthesized image on a plane corresponding to the road surface;

a vehicle locus line generation means that is adapted to generate a locus line of said vehicle placed at an arbitrary height from the road surface; and

a vehicle locus line drawing means that is adapted to draw the locus line generated by said vehicle locus line generation means on the synthesized image produced by said viewpoint change image synthesizing means such that the locus line is viewed from the viewpoint of the image pickup means, wherein the locus line viewed from the viewpoint of the image pickup means is adapted to be projected onto the road surface to obtain a first projected line on the road surface, the first projected line is adapted to be viewed from the virtual viewpoint, and the first projected line as viewed from the virtual viewpoint is adapted to again be projected onto the plane corresponding to the road surface to obtain a second projected line as the locus line drawn on the synthesized image;

a display means that is adapted to display a combined image of the synthesized image and an image of the locus line drawn on the plane by said picture synthesizing apparatus;

a display data conversion means that is adapted to convert said combined image to be displayed into data suitable for said display means

storage means that is adapted to store auxiliary data; and

a drawing means that is adapted to superimpose stored auxiliary data upon a synthesized image produced by said viewpoint change image synthesizing means based on auxiliary data read from said storage means;

wherein said image pick up means is adapted to obtain an original image in which a rear part of said vehicle is positioned in a view field, wherein said viewpoint change image is adapted to produce a synthesized image including an image of at least a rear portion of said vehicle, and

wherein said picture synthesizing apparatuis is adapted to represent said vehicle by an illustration of a skeleton or wire frame, and is adapted to provide an image that is adapted to explicitly indicate a portion of said vehicle comprising a tire that is adapted to come into contact with said outside object.

50. The image synthesis/display apparatus according to claim 49, further comprising:

an auxiliary line generation means that is adapted to generate an auxiliary line indicating an arbitrary position apart from said vehicle; and

an auxiliary line drawing means that is adapted to draw the auxiliary line generated by said auxiliary line generation means on the synthesized image produced by said viewpoint change image synthesizing means;

wherein said display means is adapted to display the synthesized image including the auxiliary line; and

wherein the display data conversion means is adapted to convert said synthesized image including the auxiliary line to be displayed into data suitable for said display means.

51. The image synthesis/display apparatus as claimed in claim 49 wherein said vehicle is a car.

52. An image acquirement warning apparatus comprising:

detection means for detecting an approaching state of an object that is outside a vehicle and is immanent to collide with said vehicle, wherein the direction of the approaching state indicates that the object is adapted to collide with the vehicle;

a picture synthesizing apparatus comprising;

an image pickup means that is attached to said vehicle and/or said object and is adapted to obtain an original image of an object outside said vehicle viewed from a viewpoint of the image pickup means;

a viewpoint change image synthesizing means that is adapted to produce a synthesized image, in which the outside object is adapted to be viewed from a virtual viewpoint different from the viewpoint of the image pickup means, and different from the original image obtained by said image pickup means and is adapted to draw the synthesized image on a plane corresponding to the road surface;

a vehicle locus line generation means adapted to generate a locus line of said vehicle placed at an arbitrary height above the road surface;

a vehicle locus line drawing means adapted to draw the locus line generated by said vehicle locus line generation means on the synthesized image produced by said viewpoint change image synthesizing means such that when the locus line is viewed from the viewpoint of the image pickup means, the locus line viewed from the viewpoint of the image pickup means is adapted to be projected onto the road surface to obtain a first projected line on the road surface, the first projected line is adapted to be viewed from a vertical viewpoint, and the first projected line viewed from the vertical viewpoint is again projected onto the plane corresponding to the road surface to obtain a second projected line as the locus line drawn on the synthesized image;

warning means for generating a warning signal as a result of said approaching state determined by said detection means and/or a relation between said vehicle and said outside object in the image synthesized by said picture synthesizing apparatus a storage means that is adapted to store predetermined data beforehand;

a drawing means that is adapted to superimpose predetermined auxiliary data upon the synthesized image produced by said viewpoint change image synthesizing means based on the data read from said storage means;

wherein said image pickup means is adapted to obtain an original image in which a rear part of said vehicle is positioned in a view field; and

wherein said picture synthesizing apparatus is adapted to represent said vehicle by an illustration of a skeleton or a wire frame, and is adapted to provide an image including a portion of said vehicle, comprising a tire, that is likely to come into contact with said outside object.

53. The image acquirement warning apparatus according to claim 52, wherein said picture synthesizing apparatus further comprises:

an auxiliary line generation means that is adapted to generate an auxiliary line indicating an arbitrary position apart from said vehicle; and

an auxiliary line drawing means that is adapted to draw the auxiliary line generated by said auxiliary line generation means on the synthesized image produced by said viewpoint change image synthesizing means; and

wherein said warning means is adapted to cause a warning signal to be generated in response to the approaching state identified by said detection means and/or a spatial relation between said vehicle and said outside object in the synthesized image including the auxiliary line.

54. The image acquirement warning apparatus according to claim 52, wherein the picture synthesizing apparatus further comprises:

a storage means that is adapted to store predetermined data beforehand; and

a drawing means that is adapted to superimpose predetermined auxiliary data upon the synthesized image produced by said viewpoint change image synthesizing means based on the data read from said storage means.

55. The image acquirement warning apparatus according to claim 52, further comprising warning signal generation condition setting means for a user to arbitrarily set a condition for generating the warning signal by said warning means.

56. The image acquirement warning apparatus as claimed in claim 52 wherein said vehicle is a car.

57. A vehicle position recognition apparatus comprising:

a picture synthesizing apparatus comprising;

a plurality of image pickup means that are adapted to be attached to a vehicle that is adapted to be disposed on a road surface, and includes a rear image pickup means for picking up an original image of an outside object disposed behind said vehicle;

a viewpoint change image synthesizing means that is adapted to produce a synthesized image, in which the outside object is viewed from a virtual viewpoint different from the viewpoints of the image pickup means, and from the original image obtained by said image pickup means, and is adapted to draw the synthesized image on a plane corresponding to the road surface;

a vehicle locus line generation means that is adapted to generate a locus line of said vehicle placed at an arbitrary height from the road surface;

a vehicle locus line drawing means that is adapted to draw the locus line generated by said vehicle locus line generation means on the synthesized image produced by said viewpoint change image synthesizing means such that the locus line is adapted to be viewed from the viewpoint of the image pickup means, the locus line viewed from the viewpoint of the image pickup means is adapted to be projected onto the road surface to obtain a first projected line on the road surface, the first projected line viewed from the vertical viewpoint is adapted to again be projected onto the plane corresponding to the road surface to obtain a second projected line as the locus line drawn on the synthesized image;

an image detection means that is adapted to detect an image of an arbitrary object from the original image obtained by said rear image pickup means or the synthesized image produced by said picture synthesizing apparatus;

a recognition means that is adapted to recognize a spatial relation between the image of the arbitrary object detected by said image detection means and the image of said vehicle;

a comparison means that is adapted to compare said spatial relation recognized by said recognition means with a predetermined position relation, and is adapted to detect an amount of deviation between the recognized spatial relations from the predetermined position relations;

a storage means that is adapted to store predetermined data beforehand;

a drawing means that is adapted to superimpose predetermined auxiliary data upon the synthesized image produced by said viewpoint change image synthesizing means based on the data read from said storage means;

wherein said image pickup means is adapted to obtain an original image in which a rear part of said vehicle is positioned in a view field; and

wherein said picture synthesizing apparatus is adapted to represent said vehicle by an illustration of a skeleton or a wire frame, and is adapted to provide an image including a portion of said vehicle, comprising a tire, that is likely to come into contact with said outside object.

58. The vehicle position recognition apparatus according to claim 57:

wherein said picture synthesizing apparatus further comprises;

an auxiliary line generation means that is adapted to draw an auxiliary line indicating an arbitrary position apart from said vehicle; and

an auxiliary line drawing means that is adapted to draw the auxiliary line generated by said auxiliary line generation means on the synthesized image produced by said viewpoint change image synthesizing means.

59. A vehicle position recognition apparatus as claimed in claim 57 wherein said vehicle is a car.

60. The image vehicle position recognition apparatus as claimed in claim 57 wherein said vehicle is a car.