Autonomous vehicle detection of and response to emergency vehicles

• US 9,278,689 B1
• Filed: 11/13/2014
• Issued: 03/08/2016
• Est. Priority Date: 11/13/2014
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Delp, Michael J.
• Primary Examiner(s): Edwards, Jerrah
• Assistant Examiner(s): Malhotra, Sanjeev
• Application Number:
  US14/541,056
• Time in Patent Office: 481 Days
• Field of Search: 701/23, 701/25, 701/41, 701/301, 348/148, 340/903, 382/104
• US Class Current: 1/1
• CPC Class Codes: B60Q 1/346 with automatic actuation
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 5/24 Position of single direction-finder fixed by determining direction of a plurality of spaced sources of known location
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G08G 1/0965 responding to signals from another vehicle, e.g. emergency vehicle
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

An automated driving system and methods are disclosed. The automated driving system includes a perception system disposed on an autonomous vehicle. The automated driving system can detect, based on an audio recording captured by the perception system, an emergency vehicle operating in an emergency mode. After detection, an image of the environment surrounding the autonomous vehicle can be captured by the perception system. Based on at least one of the audio recording and the image of the environment surrounding the autonomous vehicle, a location of the emergency vehicle in respect to the autonomous vehicle can be determined. If the location of the emergency vehicle is such that the autonomous vehicle is required to comply with a traffic regulation, the automated driving system can send a command to one or more vehicle systems of the autonomous vehicle to implement one or more maneuvers to comply with the traffic regulation.

Claims

1. An automated driving system, comprising:

a perception system disposed on an autonomous vehicle; and

a computing device in communication with the perception system, comprising;

one or more processors for controlling operations of the computing device; and

a memory for storing data and program instructions used by the one or more processors, wherein the one or more processors are configured to execute instructions stored in the memory to;

detect, based on at least two audio recordings captured by the perception system, an emergency vehicle operating in an emergency mode, wherein each audio recording is captured at a different location on the autonomous vehicle;

after detection, capture, using the perception system, an image of an environment surrounding the autonomous vehicle;

determine a location of the emergency vehicle in respect to the autonomous vehicle based on a comparison of a time of arrival of sound within each of the at least two audio recordings; and

if the location of the emergency vehicle is such that the autonomous vehicle is required to comply with a traffic regulation, send a command to one or more vehicle systems of the autonomous vehicle to implement one or more maneuvers to comply with the traffic regulation.

2. The automated driving system of claim 1, wherein detection of the emergency vehicle operating in the emergency mode is based on at least sound matching of a siren associated with the emergency vehicle within the audio recording captured by the perception system.

3. The automated driving system of claim 1, wherein determination of the location of the emergency vehicle in respect to the autonomous vehicle is based on at least one of sensor fusion from LIDAR returns and sensor fusion from radar returns and depth estimation from stereo camera images and depth estimation from multiple monocular images captured by the perception system.

4. The automated driving system of claim 1, wherein the processor is further configured to:

if the location of the emergency vehicle is such that the autonomous vehicle is not required to comply with the traffic regulation, continue a planned vehicle path.

5. The automated driving system of claim 1, wherein the one or more processors are further configured to:

determine a distance between the autonomous vehicle and the emergency vehicle.

6. The automated driving system of claim 5, wherein sending the command to the one or more vehicle systems of the autonomous vehicle to implement the one or more maneuvers to comply with the traffic regulation is further based on the distance between the autonomous vehicle and the emergency vehicle being below a predetermined threshold.

7. The automated driving system of claim 5, wherein determination of the distance between the emergency vehicle and the autonomous vehicle is based on at least one of a decibel level of a siren associated with the emergency vehicle within the audio recording captured by the perception system and shape matching within the image captured by the perception system.

8. The automated driving system of claim 5, wherein the one or more processors are further configured to:

if the distance between the autonomous vehicle and the emergency vehicle is above a predetermined threshold, continue a planned vehicle path.

9. A computer-implemented method of automated driving, comprising:

detecting, based on an image captured by a perception system disposed on an autonomous vehicle, an emergency vehicle;

after detecting the emergency vehicle, determining, based on at least two audio recordings captured by the perception system, whether the emergency vehicle is operating in an emergency mode, wherein each audio recording is captured at a different location on the autonomous vehicle;

if the emergency vehicle is operating in the emergency mode, determining a location of the emergency vehicle in respect to the autonomous vehicle based on a comparison of a time of arrival of sound within each of the at least two audio recordings; and

if the location of the emergency vehicle is such that the autonomous vehicle is required to comply with a traffic regulation, sending a command to one or more vehicle systems of the autonomous vehicle to implement one or more maneuvers to comply with the traffic regulation.

10. The method of claim 9, wherein detecting the emergency vehicle is based on at least one of shape matching, text matching, and light pattern matching within the image captured by the perception system.

11. The method of claim 9, wherein determining whether the emergency vehicle is operating in the emergency mode is based on detecting a siren associated with the emergency vehicle in the audio recording captured by the perception system.

12. The method of claim 9, wherein determining the location of the emergency vehicle in respect to the autonomous vehicle is based on at least one of sensor fusion from LIDAR returns and sensor fusion from radar returns and depth estimation from stereo camera images and depth estimation from multiple monocular images captured by the perception system.

13. The method of claim 9, further comprising:

if the location of the emergency vehicle is such that the autonomous vehicle is not required to comply with the traffic regulation, continuing a planned vehicle path.

14. The method of claim 9, further comprising:

determining a distance between the autonomous vehicle and the emergency vehicle.

15. The method of claim 14, wherein sending the command to the one or more vehicle systems of the autonomous vehicle to implement the one or more maneuvers to comply with the traffic regulation is further based on the distance between the autonomous vehicle and the emergency vehicle being below a predetermined threshold.

16. The method of claim 14, wherein determining the distance between the emergency vehicle and the autonomous vehicle is based on at least one of a decibel level of a siren associated with the emergency vehicle within the audio recording captured by the perception system and shape matching within the image captured by the perception system.

17. The method of claim 14, further comprising:

if the distance between the autonomous vehicle and the emergency vehicle is above a predetermined threshold, continuing a planned vehicle path.

18. A computing device, comprising:

one or more processors for controlling operations of the computing device; and

a memory for storing data and program instructions used by the one or more processors, wherein the one or more processors are configured to execute instructions stored in the memory to;

determine, using a perception system disposed on an autonomous vehicle, a location of an emergency vehicle operating in an emergency mode based on at least two audio recordings captured by the perception system with each audio recording captured at a different location on the autonomous vehicle, where the location of the emergency vehicle is determined in respect to the autonomous vehicle based on a comparison of a time of arrival of sound within each of the at least two audio recordings; and

if the location of the emergency vehicle is such that the autonomous vehicle is required to comply with a traffic regulation, send a command to one or more vehicle systems of the autonomous vehicle to implement one or more maneuvers to comply with the traffic regulation.