Method and device for observing the environment of a vehicle

• US 9,898,670 B2
• Filed: 12/05/2014
• Issued: 02/20/2018
• Est. Priority Date: 12/13/2013
• Status: Active Grant

• Current Assignee: TTTech Auto AG (TTTech Computertechnik AG)
• Original Assignee: FTS Computertechnik GmbH
• Inventors: Poledna, Stefan
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US15/103,952
• Publication Number:
  US 20160314365A1
• Time in Patent Office: 1,173 Days
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

The invention relates to monitoring the area in front of a vehicle by means of an apparatus that comprises at least two imaging devices (110, 120). Provided are a first imaging device (110), which covers a first imaging angle, and a second imaging device (120), which covers a second, greater imaging angle. The first imaging device (110) covers a first zone (111) of the area in front of the vehicle, while at the same time, the second imaging device (120) covers a second zone (121) of the area in front of the vehicle. The two imaging devices (110, 120) are positioned spaced, in particular spaced laterally, from one another such that a central area (140) is covered by both the first and the second imaging devices (110, 120). By fusing the data acquired by the imaging devices (110, 120), a stereo image of the central area is generated, while monoscopic images are generated of those zones that are each covered by only a first or a second imaging device.

Claims

1. A method of monitoring the area in front of a vehicle, the method comprising:

providing an apparatus, which comprises at least two imaging devices (110, 120), wherein at least one first, preferably precisely one first imaging device (110) covers a first imaging angle, and at least one second, preferably precisely one second imaging device (120) covers a second imaging angle, the second imaging angle being greater than the first imaging angle and wherein the at least one first imaging device (110) and the at least one second imaging device (120) are arranged spaced, in particular spaced laterally, from one another;

covering, with the at least one, preferably precisely one first imaging device (110), a first zone (111) of the area in front of the vehicle, particularly the area in close proximity to the front of the vehicle;

preferably simultaneously, covering, with the at least one, preferably precisely one second imaging device (120), a second zone (121) of the area in front of the vehicle, in particular the distant area in front of the vehicle, wherein the at least one first imaging device (110) and the at least one second imaging device (120) are arranged in relation to one another such that a central area (140), which is an area of overlap between the first zone and the second zone, is covered by both the at least one first and the at least one second imaging device (110, 120);

fusing data acquired by the imaging devices (110, 120) to generate a stereo image of the central area, using a fusion computer (202); and

generating a monoscopic image of those zones that are each covered by only a first or a second imaging device, using the fusion computer (202).

2. The method of claim 1, wherein the imaging devices (110, 120) each comprise a sensor, in particular an optical sensor, and a lens.

3. The method of claim 2, wherein the at least one first imaging device (110) has a telephoto lens.

4. The method of claim 2, wherein the at least one second imaging device (120) has a wide angle lens.

5. The method of claim 1, wherein the optical axes (112, 122) of the imaging devices (110, 120), in particular the optical axes of the sensors, are aligned parallel to one another.

6. The method claim 5, wherein the optical axes (112, 122) of the imaging devices (110, 120), in particular the optical axes of the sensors, extend parallel to a vehicle longitudinal axis (130) of the vehicle.

7. The method of claim 1, wherein the normal distance between the optical axes (112, 122) of the first and second imaging devices, in particular the optical axes of the optical sensors, is greater than 4 cm.

8. The method of claim 1, wherein the imaging devices have access to a global time base, and the times at which data are acquired by the imaging devices are determined by the progression of global time.

9. The method of claim 1, wherein the imaging devices have access to a global time base, and the times at which data is acquired by the imaging devices are each defined in a time stamp of the global time assigned to an image.

10. The method of claim 1, wherein the imaging devices monitor the area surrounding the vehicle cyclically.

11. The method of claim 1, wherein if the first or second imaging device should fail, the area surrounding the vehicle will be monitored by the remaining functional at least one second or at least one first imaging device, and the vehicle will be guided to a safe state, preferably by the fusion computer.

12. The method of claim 2, wherein the focal lengths of the lenses of the imaging devices are adjusted, preferably by the fusion computer, such that objects recognized as relevant in the area surrounding the vehicle are shifted to the center of the field of view of the imaging devices in subsequent frames captured by the imaging devices.

13. The method of claim 1, wherein the orientation of the optical axes of the imaging devices, particularly of the sensors, is adjusted, preferably by the fusion computer, such that the objects recognized as relevant in the area surrounding the vehicle are shifted to the center of the field of view of the sensors in subsequent frames.

14. An apparatus for monitoring the area in front of a vehicle, said apparatus comprising:

at least two imaging devices, wherein at least one first, preferably precisely one first imaging device has a first imaging angle which is such that a first zone of the area in front of the vehicle, in particular the area in close proximity to the front of the vehicle, can be covered by the at least one first imaging device, and wherein at least one second, preferably precisely one second imaging device has a second imaging angle which is such that, preferably simultaneously, a second zone of the area in front of the vehicle, in particular the distant area in front of the vehicle, can be covered by the at least one second imaging device, the second imaging angle being greater than the first imaging angle; and

a fusion computer,

wherein the at least one first imaging device and the at least one second imaging device are arranged spaced, in particular spaced laterally, from one another,

wherein the at least one first imaging device and the at least one second imaging device are arranged in relation to one another such that a central area, which is an area of overlap between the first zone and the second zone, is covered by both the at least one first and the at least one second imaging device,

wherein the fusion computer is configured to generate a stereo image of the central area by fusing data relating thereto, acquired by the imaging devices, and

wherein the apparatus is configured to generate a monoscopic image of those areas that are each covered by only a first or a second imaging device, preferably by the fusion computer.

15. The apparatus of claim 14, wherein the imaging devices each comprise a sensor, in particular an optical sensor and a lens.

16. The apparatus of claim 15, wherein the at least one first imaging device has a telephoto lens.

17. The apparatus of claim 15, wherein the at least one second imaging device has a wide angle lens.

18. The apparatus of claim 14, wherein the optical axes of the imaging devices, in particular the optical axes of the sensors, are aligned parallel to one another.

19. The apparatus of claim 18, wherein the optical axes of the imaging devices, in particular the optical axes of the sensors, extend parallel to a vehicle longitudinal axis of the vehicle.

20. The apparatus of claim 14, wherein the normal distance between the optical axes of the first and second imaging devices, in particular the optical axes of the optical sensors, is greater than 4 cm.

21. The apparatus of claim 14, wherein the imaging devices have access to a global time base, and the times at which data are acquired by the imaging devices are determined by the progression of global time.

22. The apparatus of claim 14, wherein the imaging devices have access to a global time base, and the times at which data are acquired by the imaging devices are each defined in a time stamp of the global time assigned to an image.

23. The apparatus of claim 14, wherein the imaging devices monitor the area surrounding the vehicle cyclically.

24. The apparatus of claim 14, wherein if the first or second imaging device should fail, the area surrounding the vehicle will be monitored by the remaining functional at least one second or at least one first imaging device, and the vehicle will be guided to a safe state, preferably by the fusion computer.

25. The apparatus of claim 14, wherein the focal lengths of the lenses of the imaging devices are adjusted, preferably by the fusion computer, such that objects recognized as relevant in the area surrounding the vehicle are shifted to the center of the field of view of the imaging devices in subsequent frames captured by the imaging devices.

26. The apparatus of claim 14, wherein the orientation of the optical axes of the imaging devices, particularly of the sensors, is adjusted, preferably by the fusion computer, such that the objects recognized as relevant in the area surrounding the vehicle are shifted to the center of the field of view of the sensors in subsequent frames.

27. The apparatus of claim 14, wherein the imaging devices are mounted on or behind the windshield of the vehicle.

28. The apparatus of claim 14, wherein one, a plurality, or all of the imaging devices are designed for the wavelength range of visible light.

29. The apparatus of claim 14, wherein one, a plurality, or all of the imaging devices are designed for a wavelength range beyond the wavelength range of visible light.