Autonomous vehicle operation feature evaluation

• US 10,223,479 B1
• Filed: 05/15/2015
• Issued: 03/05/2019
• Est. Priority Date: 05/20/2014
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Konrardy, Blake, Christensen, Scott T., Hayward, Gregory, Farris, Scott
• Primary Examiner(s): Chad, Aniss
• Assistant Examiner(s): Mikowski, Justin C
• Application Number:
  US14/713,244
• Time in Patent Office: 1,390 Days
• CPC Class Codes: B60Q 11/005 for lighting devices, e.g. indicating if lamps are burning or not
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60R 2021/0027 Post collision measures, e.g. notifying emergency services
• CPC Class Codes: B60R 21/00 Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks safety belts or body harnesses in vehicles B60R22/00; devices, apparatus or methods for life-saving in general A62B; safety devices for propulsion unit control specially adapted for, or arranged in, vehicles B60K28/00; seats constructed to protect the occupant from the effect of abnormal g-forces, e.g. crash or safety seats, B60N2/42; energy-absorbing arrangements for hand wheels for steering vehicles B62D1/11; energy-absorbing arrangements for vehicle steering columns B62D1/19; harnessing in aircraft B64D25/00
• CPC Class Codes: B60W 2040/0818 Inactivity or incapacity of driver
• CPC Class Codes: B60W 2050/0095 Automatic control mode change
• CPC Class Codes: B60W 2050/0096 Control during transition between modes
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 15/931 of land vehicles
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G01S 19/48 by combining or switching between position solutions derived from the satellite radio beacon positioning system and position solutions derived from a further system
• CPC Class Codes: G01S 2013/9318 Controlling the steering
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G06F 30/15 Vehicle, aircraft or watercraft design
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06Q 10/0635 Risk analysis of enterprise or organisation activities
• CPC Class Codes: G06Q 20/0855 involving a third party
• CPC Class Codes: G06Q 40/04 : Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06Q 50/40 : Business processes related to the transportation industry shipping G06Q10/083
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G07C 5/008 : communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/08 : Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G07C 5/0808 : Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0816 : Indicating performance data, e.g. occurrence of a malfunction
• CPC Class Codes: G07C 5/085 : using electronic data carriers
• CPC Class Codes: G08B 21/06 : indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: G08B 25/08 : using communication transmission lines G08B13/19658, G08B21/0286, G08B25/016 take precedence
• CPC Class Codes: G08G 1/005 : including pedestrian guidance indicator
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096741 : where the source of the transmitted information selects which information to transmit to each vehicle
• CPC Class Codes: G08G 1/09675 : where a selection from the received information takes place in the vehicle
• CPC Class Codes: G08G 1/096758 : where no selection takes place on the transmitted or the received information
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G08G 1/096783 : where the origin of the information is a roadside individual element
• CPC Class Codes: G08G 1/096791 : where the origin of the information is another vehicle
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/147 : where the parking area is within an open public zone, e.g. city centre
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 : Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: G08G 1/205 : Indicating the location of the monitored vehicles as destination, e.g. accidents, stolen, rental
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04W 4/44 : for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]
• CPC Class Codes: H04W 4/46 : for vehicle-to-vehicle communication [V2V]
• CPC Class Codes: H04W 4/90 : Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]

Abstract

Methods and systems for testing the operation of autonomous or semi-autonomous operation features in a virtual test environment are provided. Computer-executable instructions for implementing the features may be received and executed on a test device. Test input signals may be presented to software routines associated with the features, which may generate output signals including control commands. The output signals may be used to predict the response of a vehicle in the virtual test environment, which may include a simulation of vehicle responses to the output signals. Measures of the effectiveness of the features may be determined based upon the predicted responses of the vehicle, which may then be used to determine risk levels associated with the features.

Claims

1. A computer-implemented method for testing the effectiveness of one or more autonomous operation features for controlling vehicles by simulating operation of the one or more autonomous operation features in a virtual vehicle in a plurality of virtual test environments associated with a plurality of environmental conditions, comprising:

receiving, at one or more processors, a set of computer-readable instructions for implementing the one or more autonomous operation features, wherein the set of computer-readable instructions comprises one or more software routines configured to receive one or more input signals from at least one sensor and generate one or more output signals for controlling a vehicle;

receiving, at one or more processors of a sensor input simulator, the plurality of virtual test environments;

for each virtual test environment of the plurality of virtual test environments, automatically testing the one or more autonomous operation features without input from a user by;

generating, by one or more processors of the sensor input simulator, a fully virtual vehicle simulation including the virtual vehicle within the virtual test environment;

executing, by the one or more processors, the one or more software routines to respond to test input signals for the virtual vehicle within the fully virtual vehicle simulation;

receiving, at the one or more processors from the sensor input simulator, a plurality of test input signals for the virtual test environment, wherein the test input signals simulate the one or more signals from at least one sensor corresponding to a test scenario associated with the virtual test environment;

generating, by the one or more software routines executing on the one or more processors, one or more test output signals for the virtual vehicle in response to the received test input signals for the virtual test environment;

predicting, by the one or more processors of the sensor input simulator, one or more responses of the virtual vehicle in the virtual test environment to the one or more test output signals;

determining, by one or more processors of a server, a measure of the effectiveness of the one or more autonomous operation features based at least in part upon the one or more predicted responses of the virtual vehicle to the one or more test output signals associated with the plurality of virtual test environments; and

adjusting, by the one or more processors of the server, a risk profile associated with a vehicle having the one or more autonomous operation features based upon the determined measure of effectiveness by updating the risk profile to include one or more risk levels associated with autonomous operation of the vehicle by the one or more autonomous operation features.

2. The computer-implemented method of claim 1, wherein the sensor input simulator accesses the one or more test input signals from a database containing a plurality of test signals.

3. The computer-implemented method of claim 1, wherein generating the fully virtual vehicle simulation includes:

generating, by the one or more processors of the sensor input simulator, a simulation of the virtual vehicle in the virtual test environment;

determining, by the one or more processors of the sensor input simulator, simulated sensor data associated with the virtual vehicle in the virtual test environment; and

determining, by the one or more processors of the sensor input simulator, the one or more test input signals based at least in part upon the simulated sensor data.

4. The computer-implemented method of claim 1, wherein the measure of the effectiveness of the one or more autonomous operation features includes one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features.

5. The computer-implemented method of claim 4, wherein determining the one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features includes predicting, by the one or more processors of the server, the one or more risk levels based at least in part upon a comparison of (i) the one or more test output signals generated by the one or more software routines, (ii) one or more other test output signals generated by one or more other software routines of one or more other autonomous operation features in response to one or more other test input signals, and (iii) observed operating data regarding the one or more other autonomous operation features disposed within a plurality of other vehicles operating outside the plurality of virtual test environments.

6. The computer-implemented method of claim 5, wherein the observed operating data regarding the one or more other autonomous operation features includes data regarding actual losses associated with insurance policies covering the plurality of other vehicles having the one or more other autonomous operation features.

7. The computer-implemented method of claim 1, wherein each of the plurality of virtual test environments is based at least in part upon observed data regarding actual environments recorded by one or more sensors communicatively connected to a plurality of vehicles operating outside the plurality of virtual test environments.

8. A computer system for testing the effectiveness of one or more autonomous operation features for controlling vehicles by simulating operation of the one or more autonomous operation features in a virtual vehicle in a plurality of virtual test environments associated with a plurality of environmental conditions, comprising:

one or more processors;

a sensor input simulator configured to receive the plurality of virtual test environments and, for each virtual test environment;

(i) generate a fully virtual vehicle simulation including the virtual vehicle within the virtual test environment and (ii) predict responses of the virtual vehicle within the virtual test environment; and

a non-transitory program memory coupled to the one or more processors and storing executable instructions that when executed by the one or more processors cause the computer system to;

receive a set of computer-readable instructions for implementing the one or more autonomous operation features, wherein the set of computer-readable instructions comprises one or more software routines configured to receive one or more input signals from at least one sensor and generate one or more output signals for controlling a vehicle;

for each virtual test environment of the plurality of virtual test environments, automatically testing the one or more autonomous operation features without input from a user by;

execute the one or more software routines to respond to test input signals for the virtual vehicle within the fully virtual vehicle simulation;

receive a plurality of test input signals for the virtual test environment from the sensor input simulator, wherein the test input signals simulate the one or more signals from at least one sensor corresponding to a test scenario associated with the virtual test environment;

generate one or more test output signals for the virtual vehicle using the one or more software routines in response to the received test input signals for the virtual test environment;

cause the sensor input simulator to predict one or more responses of the virtual vehicle in the virtual test environment to the one or more test output signals;

determine a measure of the effectiveness of the one or more autonomous operation features based at least in part upon the one or more predicted responses of the virtual vehicle to the one or more test output signals associated with the plurality of virtual test environments; and

adjust a risk profile associated with a vehicle having the one or more autonomous operation features based upon the determined measure of effectiveness by updating the risk profile to include one or more risk levels associated with autonomous operation of the vehicle by the one or more autonomous operation features.

9. The computer system of claim 8, wherein the the sensor input simulator is further configured to access the one or more test input signals from a database containing a plurality of test signals.

10. The computer system of claim 8, wherein the sensor input simulator is further configured to, for each virtual test environment:

generate a simulation of the virtual vehicle in the virtual test environment;

determine simulated sensor data associated with the virtual vehicle in the virtual test environment; and

determine the one or more test input signals based at least in part upon the simulated sensor data.

11. The computer system of claim 8, wherein the measure of the effectiveness of the one or more autonomous operation features includes one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features.

12. The computer system of claim 11, wherein the one or more risk levels associated with operation of the virtual vehicle by the one or more autonomous operation features are determined based at least in part upon a comparison of (i) the one or more test output signals generated by the one or more software routines, (ii) one or more other test output signals generated by one or more other software routines of one or more other autonomous operation features in response to one or more other test input signals, and (iii) observed operating data regarding the one or more other autonomous operation features disposed within a plurality of other vehicles operating outside the plurality of virtual test environments.

13. The computer system of claim 12, wherein the observed operating data regarding the one or more other autonomous operation features includes data regarding actual losses associated with insurance policies covering the plurality of other vehicles having the one or more other autonomous operation features.

14. The computer system of claim 8, wherein each of the plurality of virtual test environments is based at least in part upon observed data regarding actual environments recorded by one or more sensors communicatively connected to a plurality of vehicles operating outside the plurality of virtual test environments.

15. A tangible, non-transitory computer-readable medium storing executable instructions for testing the effectiveness of one or more autonomous operation features for controlling vehicles by simulating operation of the one or more autonomous operation features in a virtual vehicle in a plurality of virtual test environments associated with a plurality of environmental conditions that, when executed by at least one processor of a computer system, cause the computer system to:

receive a set of computer-readable instructions for implementing the one or more autonomous operation features, wherein the set of computer-readable instructions comprises one or more software routines configured to receive one or more input signals from at least one sensor and generate one or more output signals for controlling a vehicle;

cause a sensor input simulator to receive the plurality of virtual test environments;

for each virtual test environment of the plurality of virtual test environments, automatically testing the one or more autonomous operation features without input from a user by;

cause the sensor input simulator to generate a fully virtual vehicle simulation including the virtual vehicle within the virtual test environment;

execute the one or more software routines to respond to test input signals for the virtual vehicle within the fully virtual vehicle simulation;

receive a plurality of test input signals for the virtual test environment from the sensor input simulator, wherein the test input signals simulate the one or more signals from at least one sensor corresponding to a test scenario associated with the virtual test environment;

generate one or more test output signals for the virtual vehicle using the one or more software routines in response to the received test input signals for the virtual test environment;

cause the sensor input simulator to predict one or more responses of the virtual vehicle in the virtual test environment to the one or more test output signals;

determine a measure of the effectiveness of the one or more autonomous operation features based at least in part upon the one or more predicted responses of the virtual vehicle to the one or more test output signals associated with the plurality of virtual test environments; and

adjust a risk profile associated with a vehicle having the one or more autonomous operation features based upon the determined measure of effectiveness by updating the risk profile to include one or more risk levels associated with autonomous operation of the vehicle by the one or more autonomous operation features.

16. The tangible, non-transitory computer-readable medium of claim 15, wherein the executable instructions that cause the computer system to cause the sensor input simulator to generate the fully virtual vehicle simulation further cause the sensor input simulator to access the one or more test input signals from a database containing a plurality of test signals.

17. The tangible, non-transitory computer-readable medium of claim 15, wherein the executable instructions that cause the sensor input simulator to generate the fully virtual vehicle simulation further cause the sensor input simulator to:

generate a simulation of the virtual vehicle in the virtual test environment;

determine simulated sensor data associated with the virtual vehicle in the virtual test environment; and

determine the one or more test input signals based at least in part upon the simulated sensor data.

18. The tangible, non-transitory computer-readable medium of claim 15, wherein the measure of the effectiveness of the one or more autonomous operation features includes one or more risk levels associated with autonomous operation of the virtual vehicle by the one or more autonomous operation features.

19. The tangible, non-transitory computer-readable medium of claim 18, wherein the one or more risk levels associated with operation of the virtual vehicle by the one or more autonomous operation features are determined based at least in part upon a comparison of (i) the one or more test output signals generated by the one or more software routines, (ii) one or more other test output signals generated by one or more other software routines of one or more other autonomous operation features in response to one or more other test input signals, and (iii) observed operating data regarding the one or more other autonomous operation features disposed within a plurality of other vehicles operating outside the plurality of virtual test environments.

20. The tangible, non-transitory computer-readable medium of claim 19, wherein the observed operating data regarding the one or more other autonomous operation features includes data regarding actual losses associated with insurance policies covering the plurality of other vehicles having the one or more other autonomous operation features.