Vehicle and mobile device traffic hazard warning techniques

• US 8,493,198 B1
• Filed: 07/11/2012
• Issued: 07/23/2013
• Est. Priority Date: 07/11/2012
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Vasquez, Juan, Mok, Vinson
• Primary Examiner(s): HUNNINGS, TRAVIS R
• Application Number:
  US13/546,563
• Time in Patent Office: 377 Days
• Field of Search: 340/904, 340/438, 340/436, 340/425.5
• US Class Current: 340/436
• CPC Class Codes: B60Q 2900/30 Lamps commanded by wireless transmissions
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/16 Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/167 Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A computer-implemented method includes monitoring an environment external to a vehicle via a sensor of the vehicle or a mobile device. The monitoring includes recording audio or video signals based on an output of the sensor. Audio or image recognition is performed via the mobile device based on the audio or video signals. Based on results of the audio or image recognition, objects in an area through which the vehicle is to pass are detected. The method includes determining which ones of the detected objects satisfy a predetermined criteria. The predetermined criteria includes object features that are indiscernible or marginally discernible to a vehicle operator with a perception deficiency. Selected ones of the detected objects that satisfy the predetermined criteria are monitored. The vehicle operator is alerted of the selected ones of the detected objects with an alert predetermined to be discernible to the vehicle operator.

Claims

1. A computer-implemented method, comprising:

monitoring, via a sensor associated with at least one of a vehicle and a mobile device, an environment external to the vehicle,

wherein the monitoring of the environment includes recording audio signals of the environment based on an output of the sensor;

performing audio recognition based on the audio signals via a recognition module of the mobile device;

detecting, at the mobile device and based on results of the audio recognition, objects in an area through which the vehicle is to pass;

determining, at the mobile device, which ones of the detected objects satisfy one or more predetermined criteria, the one or more predetermined criteria including object features that are indiscernible or marginally discernible to a vehicle operator with a hearing deficiency, the object features including audible signatures predetermined to be marginally discernible or indiscernible by the vehicle operator;

identifying, at the mobile device, selected ones of the detected objects to monitor when the selected ones of the detected objects satisfy the one or more predetermined criteria; and

alerting the vehicle operator of the selected ones of the detected objects with an alert predetermined to be discernible to the vehicle operator, the selected ones identified as having one or more of the audible signatures.

2. The computer-implemented method of claim 1, further comprising monitoring the environment via a sensing network including:

monitoring the area via the sensor, wherein the sensing network comprises the sensor and a second sensor; and

monitoring a second area via a second sensor,

wherein the audio signals or the video signals are generated based on an output of the second sensor.

3. The computer-implemented method of claim 1, further comprising:

capturing, by the mobile device, images of the environment while performing the monitoring of the environment;

generating, at the mobile device, an image signal that represents the captured images;

performing image recognition based on the image signal via the recognition module of the mobile device; and

detecting, at the mobile device, the detected objects in the area based on the image recognition.

4. The computer-implemented method of claim 1, wherein:

the identifying of the selected ones of the detected objects comprises identifying the selected ones of the detected objects as traffic indicators; and

the alert predetermined to be discernible to the vehicle operator identifies the detected objects as traffic indicators.

5. The computer-implemented method of claim 1, wherein:

the identifying of the selected ones of the detected objects comprises identifying the selected ones of the detected objects as traffic indicators; and

the alert predetermined to be marginally discernible or indiscernible to the vehicle operator identifies the detected objects as traffic indicators.

6. The computer-implemented method of claim 1, wherein the alert predetermined to be discernible to the vehicle operator is at least one of an audio alert and a video alert when the vehicle operator has a visual deficiency.

7. The computer-implemented method of claim 1, wherein the alert predetermined to be discernible to the vehicle operator is a video alert when the vehicle operator has a hearing deficiency.

28. The computer-implemented method of claim 1, wherein the alerting the vehicle operator of the selected ones comprises altering an image of the detected objects displayed on the mobile device.

29. The computer-implemented method of claim 28, wherein the altering the image of the detected objects displayed on the mobile device comprises altering the image of the detected objects in an augmented reality of the environment displayed on the mobile device.

8. A computer-implemented method, comprising:

receiving, via a criteria setting module within a mobile device, input parameters defining at least one object to monitor;

storing the input parameters in memory of the mobile device;

determining objects to monitor and objects not to monitor based on the input parameters, wherein the input parameters are received from a vehicle operator;

monitoring, via a sensing network mounted within the mobile device, an environment external to a vehicle, the monitoring of the environment including capturing, by the mobile device and from within the vehicle, images of the environment;

generating, at the mobile device, an image signal that represents the captured images;

performing image recognition based on the image signal and the input parameters via a recognition module of the mobile device;

detecting, at the mobile device, objects in a path of the vehicle based on results of the image recognition;

identifying, at the mobile device, selected ones of the detected objects as traffic indicators to monitor based on one or more predetermined criteria, the one or more predetermined criteria including patterns of the traffic indicators; and

alerting, via the mobile device, a vehicle operator of the detected objects identified as having one or more of the patterns.

9. The computer-implemented method of claim 8, wherein:

the image recognition is activated to perform during a predetermined event; and

the predetermined event includes the vehicle arriving within a predetermined distance of an intersection.

10. The computer-implemented method of claim 9, further comprising:

determining that the vehicle is within the predetermined distance of the intersection based on at least one of global positioning system data, information from a gyroscope, and map information; and

activating image recognition when the vehicle is within the predetermined distance.

11. The computer-implemented method of claim 8, wherein the sensing network monitors the environment via image sensors mounted at least one of in and on the mobile device.

12. The computer-implemented method of claim 8, further comprising:

generating, at the mobile device, a recognition request signal based on the image signal;

transmitting the recognition request signal from the mobile device to at least one of a vehicle network in the vehicle and a server external to the vehicle;

receiving, at the mobile device, image recognition results from the at least one of the vehicle network and the server; and

identifying, at the mobile device, the selected ones of the detected objects based on the image recognition results from the at least one of the vehicle network and the server.

13. The computer-implemented method of claim 8, wherein alerting the vehicle operator comprises altering an image of the detected objects having the one or more of the patterns in an augmented reality of the environment displayed on the mobile device.

14. The computer-implemented method of claim 8, further comprising:

determining, at the mobile device, a direction of travel of the vehicle;

based on the direction of travel, monitoring, at the mobile device, one of (i) an area forward of the vehicle and (ii) an area at least one of on a side of the vehicle and rearward of the vehicle; and

generating, at the mobile device, the image signal based on the monitoring of the one of (i) the area forward of the vehicle and (ii) the area at least one of on the side of the vehicle and rearward of the vehicle.

15. The computer-implemented method of claim 8, further comprising:

determining, at the mobile device, a geographical position of the vehicle via a global positioning system;

detecting, at the mobile device, an upcoming intersection based on the geographical position and navigation data;

activating, at the mobile device, at least a portion of the sensing network based on detection of the intersection; and

performing image recognition to detect objects in the intersection,

wherein alerting the vehicle operator comprises indicating types of objects in the intersection.

30. The computer-implemented method of claim 8, wherein the alerting the vehicle operator of the detected objects comprises altering an image of the detected objects displayed on the mobile device.

16. A mobile device, comprising:

a criteria setting module configured to receive input parameters defining at least one object to monitor;

a memory configured to store the input parameters;

a sensor configured to (i) monitor an environment external to a vehicle, (ii) capture images of the environment from within the vehicle via sensors of the mobile device, and (ii) generate an image signal representing the captured images;

a recognition module configured to;

(i) at least one of perform image recognition and request performance of the image recognition based on the image signal, and (ii) detect objects in a path of the vehicle based on results of the image recognition, wherein the image recognition is based on the input parameters;

an object indicator module configured to identify selected ones of the detected objects as traffic indicators to monitor based on one or more predetermined criteria, the one or more predetermined criteria include patterns of the traffic indicators; and

an alert network configured to alert a vehicle operator of the detected objects identified as having one or more of the patterns.

17. The mobile device of claim 16, wherein:

the recognition module is configured to (i) generate a recognition request signal based on the image signal, and (ii) transmit the recognition request signal from the mobile device within the vehicle to at least one of a vehicle network in the vehicle and a server external to the vehicle;

the mobile device further comprises a transceiver configured to receive image recognition results from the at least one of the vehicle network and the server; and

the object indicator module is configured to identify the selected ones of the detected objects based on the image recognition results from the at least one of the vehicle network and the server.

18. The mobile device of claim 16, wherein the alert network is configured to alert the vehicle operator by altering an image of the detected objects having the one or more of the patterns displayed on the mobile device.

19. The mobile device of claim 16, wherein the alert network is configured to alert the vehicle operator by altering an image of the detected objects having the one or more of the patterns in an augmented reality of the environment displayed on the mobile device.

20. The mobile device of claim 16, further comprising:

a sensing network comprising the sensor; and

a telematics module configured to determine a direction of travel of the vehicle,

wherein, based on the direction of travel, the sensing network is configured to monitor one of (i) an area forward of the vehicle and (ii) an area at least one of on a side of the vehicle and rearward of the vehicle, and

wherein the sensing network is configured to generate the image signal based on the monitoring of the one of (i) the area forward of the vehicle and (ii) the area at least one of on the side of the vehicle and rearward of the vehicle.

21. The mobile device of claim 16, further comprising:

a global positioning system configured to determine a geographical position of the vehicle;

a navigation system configured to detect an upcoming intersection based on the geographical position and navigation data; and

a control module configured to activate at least a portion of the sensor based on detection of the intersection,

wherein the recognition module is configured to perform image recognition to detect objects in the intersection, and

wherein the alerting of the vehicle operator comprises indicating types of objects in the intersection.

22. A computer-implemented method, comprising:

monitoring, via a sensing network of a mobile device, an environment external to the vehicle;

generating, at the mobile device, an image signal;

performing image recognition based on the image signal via a recognition module of the mobile device;

detecting, at the mobile device, objects in a path of the vehicle based on results of the image recognition;

identifying, at the mobile device, selected ones of the detected objects as traffic indicators to monitor based on one or more predetermined criteria, the one or more predetermined criteria including object features representative of patterns of the traffic indicators; and

alerting, via an alert network, a vehicle operator of the detected objects identified as having one or more of the patterns by altering an image of the detected objects having the one or more of the patterns displayed on the mobile device.

23. The computer-implemented method of claim 22, wherein the sensing network monitors the environment via image sensors mounted at least one of in and on the vehicle.

24. The computer-implemented method of claim 22, wherein the sensing network monitors the environment via image sensors mounted at least one of in and on the mobile device.

25. The computer-implemented method of claim 22, further comprising:

generating, at the mobile device, a recognition request signal based on the image signal;

transmitting the recognition request signal from the mobile device within the vehicle to at least one of a vehicle network in the vehicle and a server external to the vehicle;

receiving, at the mobile device, image recognition results from at least one of the vehicle network and the server; and

identifying, at the mobile device, the selected ones of the detected objects based on the image recognition results from the at least one of the vehicle network and the server.

26. The computer-implemented method of claim 22, further comprising:

signaling, via the mobile device, one of a vehicle control module and a countermeasure network of the detected objects having the one or more of the patterns;

determining, at the mobile device, distances between the vehicle and the detected objects having the one or more of the patterns;

monitoring, at the mobile device, vehicle operator behavior performed in response to the alerting; and

performing a countermeasure based on the detected objects having the one or more of the patterns, the distances, and the vehicle operator behavior.

27. The computer-implemented method of claim 22, wherein the altering the image of the detected objects having the one or more of the patterns displayed on the mobile device comprises altering the image of the detected objects having the one or more of the patterns in an augmented reality of the environment displayed on the mobile device.