Method for using street level images to enhance automated driving mode for vehicle

• US 9,335,178 B2
• Filed: 01/28/2014
• Issued: 05/10/2016
• Est. Priority Date: 01/28/2014
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Nickolaou, James N.
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US14/166,083
• Publication Number:
  US 20150211870A1
• Time in Patent Office: 833 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/34 Route searching; Route guidance
• CPC Class Codes: G08G 1/164 Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A method that utilizes high-definition street level images provided by a network of stationary traffic cameras to identify potential hazards or concerns located beyond the range of vehicle mounted devices, and to provide an advanced warning or to take some other remedial action in response thereto. In one embodiment, the method uses multiple items taken from the street level images to corroborate a potential concern before saving that concern to a concern profile where the concern is linked or otherwise associated with a particular geographic zone. By taking remedial actions well in advance of a potential concern, the method provides more opportunity to adjust or otherwise address the potential concern, which can be particularly advantageous when the host vehicle is being operated in an automated driving mode.

Claims

1. A method of enhancing an automated driving mode of a host vehicle, comprising the steps of:

comparing a host vehicle location to a geographic zone saved in a concern profile, and the host vehicle location corresponds to a current location of the host vehicle or an anticipated future location of the host vehicle and the concern profile is based on street level images of road segments gathered from a plurality of stationary traffic cameras;

identifying a potential concern in response to the comparison of the host vehicle location to the saved geographic zone, and the potential concern is associated with the geographic zone in the concern profile; and

performing a remedial action in response to the identification of the potential concern, wherein the remedial action is performed before the host vehicle encounters the potential concern and the remedial action affects the automated driving mode of the host vehicle.

2. The method of claim 1, wherein the method further comprises the steps of:

gathering street level images from the plurality of stationary traffic cameras, and the street level images are of road segments beyond the range of sensors mounted on the host vehicle;

identifying first and second items from the street level images that pertain to a potential concern with a particular road segment;

evaluating the first and second items from the street level images and corroborating the potential concern by determining if both the first and second items verify the presence of the potential concern; and

associating the potential concern with a geographic zone that corresponds to the particular road segment and is saved in the concern profile.

3. The method of claim 2, wherein the gathering step further comprises gathering high-definition street level images of road segments from the plurality of stationary traffic cameras.

4. The method of claim 3, wherein the high-definition street level images are received from the plurality of stationary traffic cameras at a back-end facility and are accompanied by at least one piece of data that puts the images into context and is selected from the group consisting of:

a camera identifier, a time stamp, or a camera position.

5. The method of claim 2, wherein the identifying step further comprises monitoring video from one or more stationary traffic camera(s), extracting a high-definition still image from the video, and identifying at least one of the first and second items from the high-definition still image.

6. The method of claim 5, wherein the first item is identified from a first high-definition still image taken from a traffic camera at a first time and the second item is identified from a second high-definition still image taken from the same traffic camera at a second time, and the evaluating step further comprises evaluating the first and second items in conjunction with one another and using same camera corroboration to verify the presence of the potential concern.

7. The method of claim 5, wherein the first item is identified from a first high-definition still image taken from a first traffic camera and the second item is identified from a second high-definition still image taken from a nearby second traffic camera, and the evaluating step further comprises evaluating the first and second items in conjunction with one another and using different camera corroboration to verify the presence of the potential concern.

8. The method of claim 2, wherein the evaluating step further comprises classifying the potential concern into one or more predetermined category(ies), and at least one of the predetermined category(ies) is selected from the group consisting of:

construction concerns, traffic concerns, or weather concerns.

9. The method of claim 8, wherein the identifying step further comprises identifying first and second items from the street level images where at least one of the first and second items is selected from the group consisting of:

construction barrels, barricades, lane closures, lane shifts, lane marking occlusions, temporary or permanent signs, construction equipment, or work crews; and

the evaluating step further comprises evaluating the first and second items from the street level images in conjunction with one another in order to corroborate the potential concern and classifying the potential concern as a construction concern.

10. The method of claim 8, wherein the identifying step further comprises identifying first and second items from the street level images where at least one of the first and second items is selected from the group consisting of:

traffic jams or backups, traffic patterns, stationary or slow moving objects in the road, emergency vehicles, tow trucks, debris in the road, or emergency personnel directing traffic; and

the evaluating step further comprises evaluating the first and second items from the street level images in conjunction with one another in order to corroborate the potential concern and classifying the potential concern as a traffic concern.

11. The method of claim 8, wherein the identifying step further comprises identifying first and second items from the street level images where at least one of the first and second items is selected from the group consisting of:

glare on the road, snow on passing vehicles, clouded or blurred images indicating the presence of fog, smoke or high winds, active window wipers of passing vehicles, the presence of salt trucks or snow plows, lane markings that are obscured or occluded, or greater than average vehicle spacing or slower average vehicle speeds; and

the evaluating step further comprises evaluating the first and second items from the street level images in conjunction with one another in order to corroborate the potential concern and classifying the potential concern as a weather concern.

12. The method of claim 2, wherein the evaluating step further comprises rating the potential concern according to its possible severity or impact on vehicles that are operating in an automated driving mode and are traveling on the particular road segment.

13. The method of claim 2, wherein at least one of the first and second items from the street level images relates to active wipers of passing vehicles, and the evaluating step further comprises corroborating the potential concern by determining if the active wipers of passing vehicles and the other of the first and second items verify the presence of a potential weather concern.

14. The method of claim 1, wherein the comparing step is performed in response to the automated driving mode for the host vehicle being activated or enabled, and the performing step further comprises making changes to the automated driving mode or disabling the automated driving mode before the host vehicle encounters the potential concern.

15. The method of claim 1, wherein the comparing step is performed in response to a navigational route being requested or generated, and the performing step further comprises making changes to the automated driving mode or disabling the automated driving mode before the host vehicle encounters the potential concern.

16. The method of claim 1, wherein the comparing step is performed in response to the host vehicle being driven into a new geographic zone, and the performing step further comprises making changes to the automated driving mode or disabling the automated driving mode before the host vehicle encounters the potential concern.

17. A method of enhancing an automated driving mode of a host vehicle, comprising the steps of:

comparing a host vehicle location to a geographic zone saved in a concern profile, and the host vehicle location corresponds to a current location of the host vehicle or an anticipated future location of the host vehicle and the concern profile is based on street level images gathered from a plurality of image sources;

identifying a potential concern in response to the comparison of the host vehicle location to the saved geographic zone, and the potential concern is associated with the geographic zone in the concern profile; and

performing at least one remedial action in response to the identification of the potential concern, wherein the remedial action is performed before the host vehicle encounters the potential concern and the remedial action is selected from the group consisting of;

requiring a driver of the host vehicle to acknowledge a warning while the host vehicle is operating in the automated driving mode, making changes to the automated driving mode, or disabling the automated driving mode before the host vehicle encounters the potential concern.

18. A method of enhancing an automated driving mode of a host vehicle, comprising the steps of:

gathering street level images from one or more image source(s), and the street level images are of a particular road segment that is beyond the range of sensors mounted on the host vehicle;

identifying first and second items from the street level images that pertain to a potential concern with the particular road segment;

evaluating the first and second items from the street level images and corroborating the potential concern by determining if both the first and second items verify the presence of the potential concern; and

saving the potential concern in a data collection that is stored in electronic memory so that the host vehicle can later access the data collection and determine if there are any potential concerns that may affect the host vehicle when it is being driven in the automated driving mode.