DYMANIC FUSION METHOD AND DEVICE OF IMAGES

• US 20140341434A1
• Filed: 01/17/2014
• Published: 11/20/2014
• Est. Priority Date: 05/17/2013
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: LIN, CHE-TSUNG, LIN, YU-CHEN
• Granted Patent: US 9,311,818 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/50 characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2210/62 Semi-transparency
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/161 Decentralised systems, e.g. inter-vehicle communication
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection

Abstract

A dynamic fusion method of images includes: receiving broadcast information from surrounding vehicles of a host vehicle; determining whether at least one of the surrounding vehicles travels in the same lane as the host vehicle to become a neighboring vehicle of the host vehicle according to the broadcast information; determining whether the neighboring vehicle is too close to the host vehicle and blocks the view of the host vehicle; and performing a transparentization or translucentization process on the neighboring vehicle in an image captured by the host vehicle when the neighboring vehicle blocks the view of the host vehicle.

Claims

1. A dynamic fusion method of images, comprising the steps of:

receiving broadcast information transmitted from surrounding vehicles of a host vehicle;

determining according to the broadcast information whether at least one of the surrounding vehicles travels longitudinally with the host vehicle in the same lane, and defining the at least one of the surrounding vehicles individually and correspondingly as at least one neighboring vehicle of the host vehicle;

determining whether a preceding vehicle from the at least one neighboring vehicle exists, wherein the preceding vehicle travels along just in front of the host vehicle and blocks a view of the host vehicle; and

performing a transparentization process on the preceding vehicle when the preceding vehicle is spaced from the host vehicle at a distance less than a predetermined minimum spacing.

2. The dynamic fusion method of claim 1, wherein the transparentization process includes the steps of:

obtaining feature correspondences from the images captured by the preceding vehicle and the host vehicle so as to determine the two respective epipoles in the corresponding images and a scale factor defining a ratio relation of pixels near the two epipoles;

extracting an image near the epipole of the image of the preceding vehicle, and scaling down the extracted image according to the scale factor; and

blending the scaled image with pixels within a boundary near the epipole of the image of the host vehicle.

3. The dynamic fusion method of claim 2, further including the step of:

while in obtaining the feature points from the respective images of the preceding vehicle and the host vehicle, locating a correspondence from the images, and calculating a corresponding relation between the feature correspondences with respect to the correspondence in the images of the two vehicles according to an equation of p′

^t Fp=0 wherein the p is coordinate of the correspondence in the image of the preceding vehicle, the p′

is coordinate of the correspondence in the image of the host vehicle, and the F is a fundamental matrix.

4. The dynamic fusion method of claim 3, wherein the equation is one of a scale-invariant feature transform (SIFT) algorithm and a speeded-up robust feature (SURF) algorithm.

5. The dynamic fusion method of claim 2, further including the steps of:

estimating a fundamental matrix from the feature correspondences so as thereby to obtain the respective epipoles of the images of the preceding vehicle and the host vehicle and the ratio relation of the pixels near the two epipoles; and

extracting pixels around the epipole of the image of the preceding vehicle and pixels around the epipole inside the preceding vehicle body in the image of the host vehicle according to the ratio relation, and blending the extracted pixels in the image of the host vehicle with the ones in the image of the preceding vehicle, wherein a boundary defining the vehicle body is determined by a vehicle detection algorithm.

6. The dynamic fusion method of claim 2, wherein the images of the preceding vehicle and the host vehicle are obtained by a plurality of image sensors.

7. The dynamic fusion method of claim 6, further comprising:

obtaining a geometric relation between the images based on the feature correspondences of the images; and

blending the images of the preceding vehicle and the host vehicle according to the geometric relation.

8. The dynamic fusion method of claim 6, wherein one of the image sensors is one of an infrared image capturing device, an optoelectronic coupling element and a metal-oxide complementary (MOS) photosensitive element.

9. The dynamic fusion method of claim 6, wherein, when the image sensors are installed in the preceding vehicle and the host vehicle, the images are mutually transmitted via a wireless communication.

10. The dynamic fusion method of claim 9, wherein the broadcast information or the images are transmitted via the wireless communication, and the wireless communication is one of Dedicated Short Range Communications/Wireless Access in a Vehicular Environment (DSRC/WAVE), a Wi-H, a WiMax and an LTE.

11. The dynamic fusion method of claim 5, wherein the ratio relation is expressed by one of a linear formula and a non-linear formula.

12. The dynamic fusion method of claim 5, wherein an origin of a coordinate system for elucidating the ratio relation is one of the epipole of the image of the host vehicle and an arbitrary point in the image of the host vehicle.

13. The dynamic fusion method of claim 1, wherein the broadcast information includes image information and positioning information of the surrounding vehicles.

14. The dynamic fusion method of claim 1, wherein a resulted level of the transparentization process is one of a transparent state, a semitransparent state, and a middle state between the transparent state and an opaque state.

15. A dynamic fusion device of images, comprising:

a capturing unit, for capturing surrounding images of a host vehicle;

a receiving unit, for receiving broadcast information transmitted from surrounding vehicles of the host vehicle;

a processing unit, coupled to the receiving unit and the capturing unit, for determining whether at least one of the surrounding vehicles travels longitudinally with the host vehicle in the same lane, for defining at least one of the surrounding vehicles individually and correspondingly as at least one neighboring vehicle of the host vehicle, for determining whether a preceding vehicle from the at least one neighboring vehicle exists, wherein the preceding vehicle travels along just in front of the host vehicle and blocks a view of the host vehicle, and for performing a transparentization process on the preceding vehicle when the preceding vehicle is spaced from the host vehicle at a distance less than a predetermined minimum spacing.

16. The dynamic fusion device of claim 15, wherein the transparentization process performed by the processing unit includes the steps of:

obtaining feature correspondences from images captured by the preceding vehicle and the host vehicle so as to determine the two respective epipoles of the images and a scale factor defining a ratio relation of pixels near the two epipoles;

extracting an image near the epipole of the image of the preceding vehicle, and scaling down the extracted image according to the ratio factor; and

blending the scaled image with pixels within a boundary near the epipole of the image of the host vehicle.

17. The dynamic fusion device of claim 16, wherein the processing unit further obtains the feature points from the respective images of the preceding vehicle and the host vehicle, matches the feature points in the images of the two vehicles to form feature correspondences according to a predetermined algorithm.

18. The dynamic fusion device of claim 16, wherein the processing unit further includes

estimating a fundamental matrix based on the feature correspondences so as thereby to obtain the respective epipoles of the images of the preceding vehicle and the host vehicle and further extracting pixels defining the preceding vehicle body in the image of the host vehicle based on the ratio relation;

wherein the extracted pixels defining the preceding vehicle body in the image of the host vehicle are blended with the corresponding pixels in the image of the preceding vehicle around the epipole thereof as a center, wherein a boundary for the blending is determined by a vehicle detection algorithm.

19. The dynamic fusion device of claim 18, wherein the images of the preceding vehicle and the host vehicle are captured by the images captured by a plurality of image sensors.

20. The dynamic fusion device of claim 18, wherein the processing unit further obtains a geometric relation between the images based on the feature correspondences of the images, and blends the images of the preceding vehicle and the host vehicle according to the geometric relation.

21. The dynamic fusion device of claim 15, further including:

a display unit, coupled to the processing unit, for displaying the transparentized preceding vehicle.

22. The dynamic fusion device of claim 21, wherein the display device is one of a head-on display, a monitor and another image display device.

23. The dynamic fusion device of claim 15, wherein the capturing unit is an image sensing device having one of a video camera, a camera and other image capturing device.

24. The dynamic fusion device of claim 23, wherein the image sensing device is one of an infrared image capturing device, an optoelectronic coupling element, and a MOS photosensitive element.

25. The dynamic fusion device of claim 15, wherein the receiving unit of the host vehicle receives an image of the preceding vehicle captured by the capturing unit, the preceding vehicle captures an image of a further-preceding vehicle that travels in front of the preceding vehicle, and the images are transmitted to the receiving unit of the host vehicle via a wireless communication.

26. The dynamic fusion device of claim 15, wherein the receiving unit of the host vehicle receives a warning generated by one of a brake of the further-preceding vehicle and a collision between the preceding vehicle and the further-preceding vehicle, and the warning is transmitted to the receiving unit of the host vehicle via the wireless communication.

27. A dynamic fusion and transparentization method of images, comprising the steps of:

basing on the feature correspondences in images captured by a host vehicle and a preceding vehicle located in front of the host vehicle to obtain respective epipoles and a scale factor describing a ratio relation of pixels near the two epipoles;

extracting an image near the epipole of the image of the preceding vehicle, and scaling down the extracted image according to the ratio factor; and

blending the scaled image with pixels within a boundary near the epipole of the image of the host vehicle.