ENHANCED TOP-DOWN VIEW GENERATION IN A FRONT CURB VIEWING SYSTEM

• US 20140347470A1
• Filed: 05/16/2014
• Published: 11/27/2014
• Est. Priority Date: 05/23/2013
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: ZHANG, WENDE, WANG, JINSONG, LYBECKER, KENT S., PIASECKI, JEFFREY S., LITKOUHI, BAKHTIAR BRIAN, FRAKES, RYAN M.
• Granted Patent: US 9,886,636 B2
• Time in Patent Office: Days
• US Class Current: 348/118
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/002 specially adapted for covering the peripheral part of the vehicle, e.g. for viewing tyres, bumpers or the like integrated in the windows B60R1/001; combined with rear-view mirrors B60R1/08
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2011/004 outside the vehicle
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A system and method for creating an enhanced virtual top-down view of an area in front of a vehicle, using images from left-front and right-front cameras. The enhanced virtual top-down view not only provides the driver with a top-down view perspective which is not directly available from raw camera images, but also removes the distortion and exaggerated perspective effects which are inherent in wide-angle lens images. The enhanced virtual top-down view also includes corrections for three types of problems which are typically present in de-warped images—including artificial protrusion of vehicle body parts into the image, low resolution and noise around the edges of the image, and a “double vision” effect for objects above ground level.

Claims

1. A system for providing enhanced top-down viewing of an area in front of a vehicle, said system comprising:

a first camera mounted at the left front of the vehicle;

a second camera mounted at the right front of the vehicle;

an image acquisition module in communication with the first and second camera, said image acquisition module receiving raw images from the cameras, and the raw images from the cameras have a generally forward view perspective;

a processor including a memory module, said processor being configured to provide a virtual top-down view of the area in front of the vehicle, where the virtual top-down view is synthesized from the raw images from the cameras, and the virtual top-down view includes corrections for artificial protrusion of vehicle body parts into the view, low resolution and noise around edges of the view, and a double vision effect for objects above ground level; and

a display unit in a cockpit area of the vehicle for displaying the virtual top-down view from the processor for viewing by a driver of the vehicle.

2. The system of claim 1 wherein the correction for artificial protrusion of vehicle body parts into the view includes identifying an occlusion region in the virtual top-down view, where the occlusion region is an area in the virtual top-down view where the vehicle body parts artificially appear to extend too far forward, and the occlusion region is identified using calculations based on known vehicle geometry and camera location information.

3. The system of claim 2 wherein the occlusion region is synthetically re-textured to appear as ground in the virtual top-down view, including using spatial filling techniques, where the occlusion region is re-textured based on surrounding texture and structure from the raw images, and using temporal filling techniques, where actual pixels from just outside the occlusion region in the raw images are transposed into the occlusion region in the virtual top-down view based on vehicle motion.

4. The system of claim 1 wherein the correction for low resolution and noise around edges of the view includes identifying low-resolution regions near left, right and upper periphery of the virtual top-down view, where the low-resolution regions are areas in the virtual top-down view where image resolution is low due to de-warping, and further includes correcting the low-resolution regions using spatial resolution enhancement, where spatial resolution enhancement includes combining surface texture from outside the low-resolution regions with color and intensity inside the low-resolution regions.

5. The system of claim 4 wherein correcting the low-resolution regions further includes using temporal resolution enhancement, where temporal resolution enhancement includes transposing actual pixels from just outside the low-resolution region in the raw images into the low-resolution region in the virtual top-down view based on vehicle motion.

6. The system of claim 1 wherein the correction for a double vision effect for objects above ground level includes using structure-from-motion techniques which use a sequence of the raw images and object data from a vehicle radar or lidar system to build three-dimensional models of objects in front of the vehicle, and the three-dimensional models are used to eliminate double-imaging in the virtual top-down view.

7. The system of claim 1 wherein the correction for a double vision effect for objects above ground level includes analyzing the virtual top-down view to detect the double vision effect, assuming a new height above ground level for a feature which appears in duplicate, and re-running a de-warping calculation based on the new height.

8. The system of claim 1 wherein the correction for a double vision effect for objects above ground level includes analyzing the virtual top-down view to detect the double vision effect, finding common feature points in the raw images from the first and second cameras before merging, and transforming the raw images using image registration techniques so that the common feature points align in the virtual top-down view.

9. The system of claim 1 wherein the virtual top-down view is displayed in a first window on the display unit and a bird'"'"'s-eye view is displayed in a second window on the display unit, where the first window is larger than the second window, and the bird'"'"'s-eye view includes a depiction of the vehicle in the center with a forward-view image in front of the vehicle depiction and a rear-view image behind the vehicle depiction.

10. The system of claim 1 wherein the virtual top-down view includes an indication of whether a curb in front of the vehicle will make contact with a low-hanging body component of the vehicle, and a warning is provided if it is determined that the vehicle will contact the curb.

11. A method for providing enhanced top-down viewing of an area in front of a vehicle, said method comprising:

providing images from front-mounted cameras on the vehicle, where the images have a generally forward view perspective;

synthesizing a virtual top-down view from the images from the cameras;

performing, using a microprocessor, a de-warping calculation on the virtual top-down view to produce a de-warped virtual top-down view;

enhancing the de-warped virtual top-down view to produce an enhanced virtual top-down view, including correcting for artificial protrusion of vehicle body parts into the view, correcting low resolution and noise around edges of the view, and correcting a double vision effect for objects above ground level; and

displaying the enhanced virtual top-down view on a display device for viewing by a driver.

12. The method of claim 11 wherein correcting for artificial protrusion of vehicle body parts into the view includes identifying an occlusion region in the virtual top-down view, where the occlusion region is an area in the virtual top-down view where the vehicle body parts artificially appear to extend too far forward, and the occlusion region is identified using calculations based on known vehicle geometry and camera location information.

13. The method of claim 12 wherein the occlusion region is synthetically re-textured to appear as ground in the virtual top-down view, including using spatial filling techniques, where the occlusion region is re-textured based on surrounding texture and structure from the images, and using temporal filling techniques, where actual pixels from just outside the occlusion region in the images are transposed into the occlusion region in the virtual top-down view based on vehicle motion.

14. The method of claim 11 wherein correcting low resolution and noise around edges of the view includes identifying low-resolution regions near left, right and upper periphery of the virtual top-down view, where the low-resolution regions are areas in the virtual top-down view where image resolution is low due to de-warping, and further includes correcting the low-resolution regions using temporal resolution enhancement, where temporal resolution enhancement includes transposing actual pixels from just outside the low-resolution region in the images into the low-resolution region in the virtual top-down view based on vehicle motion.

15. The method of claim 14 wherein correcting the low-resolution regions further includes using spatial resolution enhancement, where spatial resolution enhancement includes combining surface texture from outside the low-resolution regions with color and intensity inside the low-resolution regions.

16. The method of claim 11 wherein correcting a double vision effect for objects above ground level includes using structure-from-motion techniques which use a sequence of the images to build three-dimensional models of objects in front of the vehicle, and the three-dimensional models are used to eliminate double-imaging in the virtual top-down view.

17. The method of claim 11 wherein correcting a double vision effect for objects above ground level includes analyzing the virtual top-down view to detect the double vision effect, assuming a new height above ground level for a feature which appears in duplicate, and re-running a de-warping calculation based on the new height.

18. The method of claim 11 wherein correcting a double vision effect for objects above ground level includes analyzing the virtual top-down view to detect the double vision effect, finding common feature points in the images from the cameras before merging, and transforming the images using image registration techniques so that the common feature points align in the virtual top-down view.

19. A method for providing enhanced top-down viewing of an area in front of a vehicle, said method comprising:

providing images from front-mounted cameras on the vehicle, where the images have a generally forward view perspective;

synthesizing a virtual top-down view from the images from the cameras;

performing, using a microprocessor, a de-warping calculation on the virtual top-down view to produce a de-warped virtual top-down view;

enhancing the de-warped virtual top-down view to produce an enhanced virtual top-down view by correcting for artificial protrusion of vehicle body parts into the view, including defining an occlusion region by calculating which pixels of the enhanced virtual top-down view are blocked by vehicle body parts, performing a spatial filling of the occlusion region based on image texture and structure for areas surrounding the occlusion region, performing a temporal filling of the occlusion region based on inter-frame motion to fill pixels into the occlusion region, and combining the spatial filling and the temporal filling of the occlusion region;

correcting low resolution and noise around edges of the enhanced virtual top-down view, including identifying low resolution regions around edges of the enhanced virtual top-down view, performing a spatial resolution enhancement of the low resolution regions based on image texture for higher resolution regions surrounding the low resolution regions, performing a temporal resolution enhancement of the low resolution regions based on inter-frame motion to fill pixels into the low resolution regions, and combining the spatial resolution enhancement and the temporal resolution enhancement of the low resolution regions;

correcting a double vision effect for objects above ground level in the enhanced virtual top-down view, including creating a three-dimensional model of the above-ground objects including curbs detected in the images from the cameras, above-ground objects detected using stereo vision and structure from motion techniques applied to the images from the cameras, and above-ground objects detected using other sensors onboard the vehicle, and rendering images of the above-ground objects based on the three-dimensional model; and

displaying the enhanced virtual top-down view on a display device for viewing by a driver.

20. The method of claim 19 wherein the virtual top-down view includes an indication of whether a curb in front of the vehicle will make contact with a low-hanging body component of the vehicle, and a warning is provided if it is determined that the vehicle will contact the curb.