ENHANCED FRONT CURB VIEWING SYSTEM

• US 20140347485A1
• Filed: 05/16/2014
• Published: 11/27/2014
• Est. Priority Date: 05/23/2013
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhang, Wende, Wang, Jinsong, Lybecker, Kent S., Piasecki, Jeffrey S., Litkouhi, Bakhtiar Brian, Frakes, Ryan M.
• Granted Patent: US 10,093,247 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/605 the adjustment being automatic
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/70 characterised by an event-triggered choice to display a specific image among a selection of captured images

Abstract

A system and method for determining when to display frontal curb view images to a driver of a vehicle, and what types of images to display. A variety of factors—such as vehicle speed, GPS/location data, the existence of a curb in forward-view images, and vehicle driving history—are evaluated as potential triggers for the curb view display, which is intended for situations where the driver is pulling the vehicle into a parking spot which is bounded in front by a curb or other structure. When forward curb-view display is triggered, a second evaluation is performed to determine what image or images to display which will provide the best view of the vehicle'"'"'s position relative to the curb. The selected images are digitally synthesized or enhanced, and displayed on a console-mounted or in-dash display device.

Claims

1. A system for providing enhanced images of an area in front of a vehicle, said system comprising:

a first camera mounted at the left front of the vehicle;

a second camera mounted at the right front of the vehicle;

an image acquisition module in communication with the first and second cameras, said image acquisition module receiving raw images from the cameras;

a processor including a memory module, said processor being configured to determine whether to activate a front curb view display based on a detected frontal parking maneuver, said processor being further configured to determine what images to include in the front curb view display, and to provide the images including enhanced images synthesized from the raw images from the cameras; and

a display unit in a cockpit area of the vehicle for displaying the front curb view display from the processor for viewing by a driver of the vehicle.

2. The system of claim 1 further comprising a user input device for allowing the driver of the vehicle to signal that he or she wants to activate a front curb view display.

3. The system of claim 2 wherein the user input device is a microphone and the driver utters a spoken command to activate the front curb view display.

4. The system of claim 1 wherein the processor includes a heuristics module for detecting a frontal parking maneuver, said heuristics module including a vehicle state sensing module for evaluating vehicle dynamic conditions, an object detection module for detecting a curb or parking space ahead of the vehicle, and a map module for determining from Global Positioning System data and digital map data that the vehicle is in a parking lot.

5. The system of claim 4 wherein the heuristics module is configured to activate the front curb view display when a vehicle speed is below a predetermined threshold and a curb or parking space is detected ahead of the vehicle.

6. The system of claim 4 wherein the vehicle state sensing module detects vehicle speed, steering angle, yaw rate, longitudinal acceleration and lateral acceleration, and the vehicle state sensing module determines when a parking maneuver is underway or possible.

7. The system of claim 4 wherein the object detection module uses image processing techniques to identify one or more parking spaces or structures ahead of the vehicle, and the parking spaces or structures are included in the front curb view display.

8. The system of claim 7 wherein the object detection module also uses signals from an object detection sensor to detect a curb or structure ahead of the vehicle, where the heuristics module is configured to activate the front curb view display when a vehicle speed is below a predetermined threshold and the curb or structure is detected ahead of the vehicle.

9. The system of claim 8 wherein the object detection sensor is selected from the group consisting of an ultrasonic sensor, a short-range radar sensor and a LIDAR sensor.

10. The system of claim 1 wherein the processor includes a mode selection module which is configured to select an optimal display mode in the front curb view display based on data from the heuristics module, where the optimal display mode includes one or more views in front of the vehicle which depict a parking space or structure toward which the vehicle is moving.

11. The system of claim 1 wherein the processor includes a learning module configured to determine when to activate the front curb view display and what views to include in the front curb view display based on past driver behavior, including preprogrammed learning based on parking view display preferences of a general population of drivers, and adaptive learning based on parking view display preferences of the driver of the vehicle.

12. The system of claim 11 wherein the adaptive learning includes using driving history and Global Positioning System data to identify locations where the front curb view display was activated in the past.

13. The system of claim 12 wherein the adaptive learning also includes recognition of frequent destinations of the vehicle, where the frequent destinations include a home location, a work location and a school location.

14. The system of claim 1 wherein the front curb view display includes a top-down display which is synthesized from the raw images from the first and second cameras.

15. A system for providing enhanced images of an area in front of a vehicle, said system comprising:

a first camera mounted at the left front of the vehicle;

a second camera mounted at the right front of the vehicle;

an image acquisition module in communication with the first and second camera, said image acquisition module receiving raw images from the cameras;

a user input device for allowing a driver of the vehicle to signal that he or she wants to activate a front curb view display;

a heuristics module for detecting a frontal parking maneuver, said heuristics module including a vehicle state sensing module for evaluating vehicle speed and steering angle, an image analysis module for detecting a curb or parking space ahead of the vehicle, and a map module for determining from Global Positioning System and digital map data that the vehicle is in a parking lot;

a learning module in communication with the heuristics module and the user input device for adaptively learning when to activate the front curb view display based on past vehicle and driver behavior;

a processor including a memory module, said processor being configured to determine whether to activate a front curb view display based on signals from the user input device, the heuristics module and the learning module, said processor being further configured to determine what images to include in the front curb view display, and to provide the images including enhanced images synthesized from the raw images from the cameras; and

a display unit in a cockpit area of the vehicle for displaying the front curb view display from the processor for viewing by a driver of the vehicle.

16. The system of claim 15 wherein the processor is configured to activate the front curb view display when the vehicle speed is below a predetermined threshold, a curb or parking space is detected ahead of the vehicle, and the vehicle is determined to be located in a parking lot.

17. The system of claim 15 wherein the processor includes a mode selection module which is configured to select an optimal display mode in the front curb view display based on data from the heuristics module, where the optimal display mode includes one or more views in front of the vehicle which depict a parking space or structure toward which the vehicle is moving.

18. A method for providing enhanced front curb viewing in a vehicle, said method comprising:

determining a vehicle speed by reading a vehicle speed signal from a vehicle speed sensor;

determining whether the vehicle speed is below a speed threshold;

determining whether a vehicle driver has selected front curb viewing;

evaluating a vehicle operating environment to determine whether any factors indicate that front curb viewing should be activated;

evaluating past behavior of the vehicle and the driver to determine whether front curb viewing should be activated in a current situation based on whether front curb viewing was activated in a similar situation previously;

activating front curb viewing if indicated based on vehicle speed, driver selection, vehicle operating environment and past behavior;

processing, using a microprocessor, signals from front-mounted cameras on the vehicle to produce enhanced front curb viewing images; and

displaying the enhanced front curb viewing images on a display device for viewing by the driver.

19. The method of claim 18 further comprising determining whether the vehicle is located in a parking lot based on Global Positioning System data and digital map data, and only activating front curb viewing if the vehicle is located in a parking lot.

20. The method of claim 18 wherein evaluating a vehicle operating environment includes determining whether a curb or parking space is detected ahead of the vehicle.

21. The method of claim 18 wherein determining whether a vehicle driver has selected front curb viewing includes monitoring signals from a microphone to determine whether the driver has uttered a spoken command to select front curb viewing.

22. The method of claim 18 wherein activating front curb viewing if indicated based on past behavior includes evaluating preprogrammed learning which is based on parking view display preferences of a general population of drivers, and adaptive learning which is based on parking view display preferences of the driver of the vehicle.

23. The method of claim 22 wherein the preprogrammed learning and the adaptive learning indicate which views to display when front curb viewing is activated, where the views include top-down and perspective orientations, and the views include straight-ahead, front-left and front-right directions.

24. The method of claim 18 wherein displaying the enhanced front curb viewing images includes synthesizing a top-down display from the signals from the front-mounted cameras, and further includes synthesizing a bird'"'"'s-eye view from the signals from the front-mounted cameras and signals from rear-mounted and side-mounted cameras.