Picture composing apparatus and method

• US 7,034,861 B2
• Filed: 07/06/2001
• Issued: 04/25/2006
• Est. Priority Date: 07/07/2000
• Status: Active Grant

• Current Assignee: Panasonic Automotive Systems Company of America (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Okada, Tsuyoshi, Miwa, Michio, Nishizawa, Masato, Iida, Ryosuke, Onda, Katsumasa
• Primary Examiner(s): Le, Vu
• Application Number:
  US09/899,229
• Publication Number:
  US 20020018047A1
• Time in Patent Office: 1,754 Days
• Field of Search: 348/36, 348/38, 348/113, 348/148, 348/149, 348/159, 701/200, 701/211, 701/300, 340/539.25, 340/937, 352/69
• US Class Current: 348/36
• CPC Class Codes: B60Q 9/005 using a video camera
• CPC Class Codes: B60Q 9/007 providing information about the distance to an obstacle, e.g. varying sound
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A picture composing apparatus designed to combine a plurality of images taken by a plurality of image pickup devices. In the apparatus, a first projecting unit projects the plurality of images taken by the image pickup devices onto a projection section in accordance with an image pickup situation of the image pickup devices to generate a plurality of first projected images, and a second projecting unit projects the plurality of first projected images to a three-dimensional projection model to generate a second projected image. Also included in the apparatus are a virtual image pickup device for virtually picking up the second projected image and an image pickup situation determining unit for determining an image pickup situation of the virtual image pickup device, whereby the second projected image is picked up by the virtual image pickup device in the image pickup situation determined by the pickup situation determining unit to combine the plurality of images taken by the plurality of image pickup devices, thus producing a high-quality composite picture. This apparatus can offer a natural composite picture in which joints among the images taken by the image pickup devices do not stand out. In addition, when mounted on a motor vehicle, this apparatus allows a driver to easily seize the surrounding situations and the positional relationship of the motor vehicle with respect to other objects.

Claims

1. A picture composing apparatus comprising:

a plurality of image pickup means;

image pickup situation detecting means for detecting an image pickup situation of said image pickup means;

first projecting means for projecting a plurality of images taken by said image pickup means onto a projection section in accordance with the image pickup situation detected by said image pickup situation detecting means to generate a plurality of first projected images;

second projecting means for projecting said plurality of first projected images to a three-dimensional projection model to generate a second projected image;

virtual image pickup means for virtually picking up said second projected image;

three-dimensional projection model determining means for determining a position of said three-dimensional projection model and a shape thereof; and

pickup situation determining means for determining an image pickup situation of said virtual image pickup means,

said second projected image being picked up by said virtual image pickup means in the image pickup situation determined by said pickup situation determining means to combine the plurality of images taken by said plurality of image pickup means.

2. The picture composing apparatus according to claim 1, wherein said image pickup situation detecting means previously detects at least one of a position, posture and image pickup characteristic of said image pickup means with respect to said projection section.

3. The picture composing apparatus according to claim 1, wherein said first projecting means projects points on the images taken by said image pickup means to said projection section through the use of a transformation matrix obtained by associating a plurality of points on the taken images with a plurality of points on said projection section.

4. The picture composing apparatus according to claim 1, wherein said projection section is substantially a plane.

5. The picture composing apparatus according to claim 1, wherein said three-dimensional projection model is a spherical surface.

6. The picture composing apparatus according to claim 1, wherein said three-dimensional projection model is a combination of a plurality of surfaces different in shape from each other.

7. The picture composing apparatus according to claim 6, wherein said three-dimensional projection model is a combination of a plane and a cylinder, and when said second projecting means conducts the projection to said three-dimensional projection model, a distant place is projected to said cylinder and the other place is projected to said plane and said virtual image pickup means picks up the projected image.

8. The picture composing apparatus according to claim 6, wherein said plurality of surfaces different in shape are combined so that a joint therebetween does not stand out.

9. The picture composing apparatus according to claim 6, wherein said three-dimensional projection model is a combination of a sphere and a cylinder, and when said second projecting means conducts the projection to said three-dimensional projection model, a distant place is projected to said cylinder and the other place is projected to said sphere and said virtual image pickup means picks up the projected image.

10. The picture composing apparatus according to claim 8, wherein said three-dimensional projection model is a combination of three types of surfaces of a cylinder, a plane and a curved surface establishing smooth boundary between said cylinder and said plane.

11. The picture composing apparatus according to claim 1, wherein a line segment on the image taken by said image pickup means is converted into a straight line on a composite picture.

12. The picture composing apparatus according to claim 1, wherein said plurality of image pickup means are mounted on a motor vehicle, and motor vehicle running direction detecting means is further provided to detect a running direction of said motor vehicle.

13. The picture composing apparatus according to claim 12, wherein a position of a viewing point of said virtual image pickup means is set on a straight line passing through a projection center of said second projecting means in parallel with the running direction of said motor vehicle.

14. The picture composing apparatus according to claim 12, wherein a line segment representative of the running direction of said motor vehicle is converted into a straight line on a composite picture.

15. The picture composing apparatus according to claim 1, wherein said plurality of image pickup means are mounted on a motor vehicle, and motor vehicle body position detecting means is further provided to detect a body position of said motor vehicle.

16. The picture composing apparatus according to claim 15, wherein a position of a viewing point of said virtual image pickup means is set on a plane passing through an image obtained by plane-projecting an end portion of said body of said motor vehicle and a projection center of said second projecting means.

17. The picture composing apparatus according to claim 11, wherein said image pickup means is mounted on a rear end portion of a motor vehicle, and rear end portion of said motor vehicle is converted into a straight line on a composite picture.

18. The picture composing apparatus according to claim 11, wherein said image pickup means is mounted on a side portion of a motor vehicle, and said side portion of said motor vehicle is converted into a straight line on a composite picture.

19. The picture composing apparatus according to claim 11, wherein said image pickup means is mounted on a side portion of a motor vehicle, and a center line of a lane adjacent to a lane said motor vehicle is running on is converted into a straight line on a composite picture.

20. The picture composing apparatus according to claim 11, wherein said image pickup means is mounted on a rear portion of a motor vehicle, and a center line of a road perpendicular to a road said motor vehicle is running on is converted into a straight line on a composite picture.

21. The picture composing apparatus according to claim 11, wherein said image pickup means is mounted on a front portion of a motor vehicle, and a center line of a road perpendicular to a road said motor vehicle is running on is converted into a straight line on a composite picture.

22. A picture composing apparatus comprising:

a plurality of image pickup means;

image pickup situation detecting means for detecting an image pickup situation of said image pickup means;

first projecting means for projecting a plurality of images taken by said image pickup means onto a projection surface in accordance with the image pickup situation detected by said image pickup situation detecting means to generate a plurality of first projected images;

second projecting means for projecting the plurality of first projected images to a spherical surface to generate a second projected image; and

converting means for spreading the second projected image on a plane round one point on said spherical surface.

23. A picture composing method comprising:

an image pickup step of picking up a plurality of images;

an image pickup situation detecting step of detecting an image pickup situation in said image pickup step;

a first projecting step of projecting a plurality of images picked up to a projection section in accordance with the image pickup situation detected in said image pickup situation detecting step to generate a plurality of first projected images;

a second projecting step of projecting the plurality of first projected images to a three-dimensional projection model to generate a second projected image;

a virtual image pickup step of virtually picking up the second projected image;

a three-dimensional projection model determining step of determining a position of said three-dimensional projection model and a shape thereof; and

an image pickup situation determining step of determining the image pickup situation in said virtual image pickup step,

the second projected image being picked up in said virtual image pickup step in the projection situation determined in said image pickup situation determining step to combine the plurality of images taken in said image pickup step.

24. A picture composing method according to claim 23, further comprising a step of sending signal information from motor vehicle running direction detecting means, which detects a running direction of a motor vehicle, to image pickup situation determining means used for said image pickup situation determining step.

25. A picture composing method according to claim 23, further comprising a step of sending signal information from motor vehicle body position detecting means, which detects a body position of a motor vehicle, to image pickup situation determining means used for said image pickup situation determining step.

26. A picture composing method comprising:

an image pickup step of picking up a plurality of images;

an image pickup situation detecting step of detecting an image pickup situation in said image pickup step;

a first projecting step of projecting a plurality of images, picked up in said image pickup step, to a projection section in accordance with the image pickup situation detected in the image pickup situation detecting step, to generate a plurality of first projected images;

a second projecting step of projecting the plurality of first projected images to a spherical surface to generate a second projected image;

a plane spreading step of spreading the second projected image on a plane round one point on said spherical surface; and

a spherical surface position determining step of determining a position of said spherical surface,

a plurality of images obtained in said plane spreading step are combined to produce a composite picture.