Synchronization of vehicle sensor information

• US 9,959,764 B1
• Filed: 05/04/2016
• Issued: 05/01/2018
• Est. Priority Date: 10/18/2013
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Binion, Todd, Harr, Joe, Fields, Brian, Cielocha, Steven, Balbach, Steve J.
• Primary Examiner(s): Mahne, Kevin P
• Application Number:
  US15/145,993
• Time in Patent Office: 727 Days
• Field of Search: None
• CPC Class Codes: B60R 2021/0027 Post collision measures, e.g. notifying emergency services
• CPC Class Codes: B60R 21/00 Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks safety belts or body harnesses in vehicles B60R22/00; devices, apparatus or methods for life-saving in general A62B; safety devices for propulsion unit control specially adapted for, or arranged in, vehicles B60K28/00; seats constructed to protect the occupant from the effect of abnormal g-forces, e.g. crash or safety seats, B60N2/42; energy-absorbing arrangements for hand wheels for steering vehicles B62D1/11; energy-absorbing arrangements for vehicle steering columns B62D1/19; harnessing in aircraft B64D25/00
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 40/10 related to vehicle motion
• CPC Class Codes: G01S 13/02 Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
• CPC Class Codes: G01S 15/02 using reflection of acoustic waves G01S15/66 takes precedence
• CPC Class Codes: G01S 17/02 Systems using the reflection of electromagnetic waves other than radio waves G01S17/66 takes precedence
• CPC Class Codes: G06F 30/15 Vehicle, aircraft or watercraft design
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/085 using electronic data carriers
• CPC Class Codes: G07C 5/0858 wherein the data carrier is removable
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G07C 5/0891 Video recorder in combination with video camera
• CPC Class Codes: G07C 5/12 in graphical form
• CPC Class Codes: G08G 1/01 Detecting movement of traffic to be counted or controlled G08G1/07 - G08G1/14 take precedence
• CPC Class Codes: G08G 1/162 event-triggered
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A method includes receiving and storing operational data including a first plurality of data points indicative of a plurality of respective states of an operational parameter of the vehicle at a plurality of respective times, external data including a second plurality of data points indicative of a plurality of respective states of an environment external to the vehicle at a plurality of respective times, and synchronization data. The method also includes generating a virtual model of an event involving the vehicle using the stored data, at least by generating a first visual representation of the plurality of respective states of the operational parameter, generating a second visual representation of the plurality of respective states of the external environment, and using the synchronization data to cause the first visual representation to be displayed simultaneously with, and in a time-aligned manner with, the second visual representation.

Claims

1. A method in an electronic processing system comprising a memory and one or more processors, the method comprising:

receiving, at the electronic processing system, first operational data (i) generated by a first unit arranged within a vehicle and (ii) indicative of a first operational parameter of the vehicle, wherein receiving first operational data includes receiving a first plurality of data points indicative of a plurality of respective states of the first operational parameter at a plurality of respective times;

receiving, at the electronic processing system, external data that (i) was received by the vehicle from a mobile communication device of a pedestrian external to the vehicle via a wireless link and (ii) is indicative of objects sensed by the mobile communication device, wherein receiving external data includes receiving a second plurality of data points indicative of a plurality of respective states of an environment external to the vehicle at a plurality of respective times;

receiving, at the electronic processing system, synchronization data indicating a temporal relationship between the first plurality of data points and the second plurality of data points;

storing the received first operational data, the received external data, and the received synchronization data in the memory; and

generating, via the one or more processors, a virtual model of an event involving the vehicle using the stored first operational data, the stored external data, and the stored synchronization data, wherein generating the virtual model includes

generating, using the first plurality of data points, a first visual representation of the plurality of respective states of the first operational parameter,

generating, using the second plurality of data points, a second visual representation of the plurality of respective states of the environment external to the vehicle, and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation such that the first visual representation is time-aligned with the second visual representation.

2. The method of claim 1, wherein receiving synchronization data includes receiving data indicating which of the first plurality of data points corresponds to which of the second plurality of data points.

3. The method of claim 1, wherein receiving synchronization data includes receiving:

a first plurality of time stamps each corresponding to a respective one of the first plurality of data points; and

a second plurality of time stamps each corresponding to a respective one of the second plurality of data points.

4. The method of claim 1, wherein receiving first operational data indicative of the first operational parameter of the vehicle includes receiving first operational data indicative of:

velocity of the vehicle;

acceleration of the vehicle;

braking of the vehicle;

or

steering of the vehicle.

5. The method of claim 1, wherein receiving first operational data indicative of the first operational parameter of the vehicle includes receiving first operational data indicative of motion of the vehicle with respect to six degrees of freedom.

6. The method of claim 1, wherein receiving first operational data indicative of the first operational parameter of the vehicle includes receiving first operational data indicative of status of an automated driving system of the vehicle.

7. The method of claim 1, wherein receiving first operational data indicative of the first operational parameter of the vehicle includes receiving diagnostics information associated with the vehicle.

8. The method of claim 1, wherein generating the virtual model of an event involving the vehicle includes generating a virtual model of an accident involving the vehicle.

9. The method of claim 8, further comprising at least one of:

determining fault for the accident based on the virtual model of the accident involving the vehicle;

or

determining damages resulting from the accident based on the virtual model of the accident involving the vehicle.

10. The method of claim 1, wherein:

the method further comprises

receiving, at the electronic processing system, second operational data (i) generated by a second unit arranged within the vehicle and (ii) indicative of a second operational parameter of the vehicle, wherein receiving second operational data includes receiving a third plurality of data points indicative of a plurality of respective states of the second operational parameter at a plurality of respective times, and

storing the received second operational data in the memory;

receiving synchronization data includes receiving synchronization data indicating a temporal relationship between the first plurality of data points, the second plurality of data points, and the third plurality of data points;

generating the virtual model of an event involving the vehicle further includes generating a third visual representation of the plurality of respective states of the second operational parameter; and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation includes using the synchronization data to cause the first visual representation to be displayed simultaneously with both the second visual representation and the third visual representation, such that the first visual representation is time-aligned with both the second visual representation and the third visual representation.

11. The method of claim 1, wherein receiving first operational data includes receiving first operational data at a server remote from the vehicle via one of (i) a wireless communication network, or (ii) a portable memory device.

12. An electronic processing system comprising:

a memory;

a data collection unit configured to

receive first operational data (i) generated by a first unit arranged within a vehicle and (ii) indicative of a first operational parameter of the vehicle, wherein receiving first operational data includes receiving a first plurality of data points indicative of a plurality of respective states of the first operational parameter at a plurality of respective times,

receive external data that (i) was received by the vehicle from a mobile communication device of a pedestrian external to the vehicle via a wireless link and (ii) is indicative of objects sensed by the mobile communication device, wherein receiving external data includes receiving a second plurality of data points indicative of a plurality of respective states of an environment external to the vehicle at a plurality of respective times,

receive synchronization data indicating a temporal relationship between the first plurality of data points and the second plurality of data points, and

store the received first operational data, the received external data, and the received synchronization data in the memory; and

a model generation unit coupled to the data collection unit and configured to generate a virtual model of an event involving the vehicle using the stored first operational data, the stored external data, and the stored synchronization data, wherein generating the virtual model includes;

generating, using the first plurality of data points, a first visual representation of the plurality of respective states of the first operational parameter,

generating, using the second plurality of data points, a second visual representation of the plurality of respective states of the environment external to the vehicle, and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation such that the first visual representation is time-aligned with the second visual representation.

13. The electronic processing system of claim 12, wherein the synchronization data includes data indicating which of the first plurality of data points corresponds to which of the second plurality of data points.

14. The electronic processing system of claim 12, wherein the synchronization data includes both (i) a first plurality of time stamps each corresponding to a respective one of the first plurality of data points, and (ii) a second plurality of time stamps each corresponding to a respective one of the second plurality of data points.

15. The electronic processing system of claim 12, wherein the data collection unit is configured to receive first operational data indicative of:

velocity of the vehicle;

acceleration of the vehicle;

braking of the vehicle;

steering of the vehicle;

diagnostics information associated with the vehicle;

or

status of an automated driving system of the vehicle.

16. A tangible, non-transitory, computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:

store operational data, external data, and synchronization data in a memory, wherein

the operational data (i) was generated by a first unit arranged within a vehicle and (ii) is indicative of a first operational parameter of the vehicle, the operational data including a first plurality of data points indicative of a plurality of respective states of the first operational parameter at a plurality of respective times,

the external data (i) was received by the vehicle from a mobile communication device of a pedestrian external to the vehicle via a wireless link and (ii) is indicative of objects sensed by the mobile communication device, the external data including a second plurality of data points indicative of a plurality of respective states of an environment external to the vehicle at a plurality of respective times, and

the synchronization data indicates a temporal relationship between the first plurality of data points and the second plurality of data points, and

generate a virtual model of an event involving the vehicle using the stored operational data, the stored external data, and the stored synchronization data, wherein generating the virtual model includes;

generating, using the first plurality of data points, a first visual representation of the plurality of respective states of the first operational parameter,

generating, using the second plurality of data points, a second visual representation of the plurality of respective states of the environment external to the vehicle, and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation such that the first visual representation is time-aligned with the second visual representation.

17. The tangible, non-transitory, computer-readable medium of claim 16, wherein the synchronization data includes data indicating which of the first plurality of data points corresponds to which of the second plurality of data points.

18. The tangible, non-transitory, computer-readable medium of claim 16, wherein the synchronization data includes (i) a first plurality of time stamps each corresponding to a respective one of the first plurality of data points, and (ii) a second plurality of time stamps each corresponding to a respective one of the second plurality of data points.

19. The tangible, non-transitory, computer-readable medium of claim 16, wherein the operational data includes data indicative of:

velocity of the vehicle;

acceleration of the vehicle;

braking of the vehicle;

steering of the vehicle;

diagnostics information associated with the vehicle;

or

status of an automated driving system of the vehicle.

20. The tangible, non-transitory, computer-readable medium of claim 16, wherein the operational data includes data received at a server remote from the vehicle via one of (i) a wireless communication network, or (ii) a portable memory device.