Method, apparatus, and computer readable medium for verifying a safe vehicle operation via a portable device

• US 10,633,003 B1
• Filed: 12/05/2018
• Issued: 04/28/2020
• Est. Priority Date: 12/05/2018
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Abundis Vargas, Lidia Alejandra
• Primary Examiner(s): Blount, Eric
• Application Number:
  US16/210,780
• Time in Patent Office: 510 Days
• Field of Search: 3404255
• CPC Class Codes: B60W 2050/0064 using a remote, e.g. cordless, transmitter or receiver unit, e.g. remote keypad or mobile phone
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0015 specially adapted for safety
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G07C 5/08 Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G08G 1/012 from other sources than vehicle or roadside beacons, e.g. mobile networks
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 for classifying traffic situation
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: H04W 4/027 using movement velocity, acceleration information
• CPC Class Codes: H04W 4/40 for vehicles, e.g. vehicle-to-pedestrians [V2P]

Abstract

An approach is provided for verifying a safe operation of a vehicle (e.g., an autonomous vehicle) via a portable device. The approach involves initiating a capture of environmental sensor data by a user device, wherein the sensor data indicates at least a partial view of an environment around a vehicle, and wherein the user device is independent of the vehicle. The approach also involves processing the environmental sensor data to generate a digitized representation of the environment. The approach also involves calculating a predicted motion of the vehicle based on the digitized representation. The approach further involves initiating a capture of motion sensor data by the user device, wherein the motion sensor data indicates an observed motion of the vehicle. The approach further involves identifying a safe operation of the vehicle based on a comparison by the user device of the observed motion to the predicted motion of the vehicle.

Claims

1. A method comprising:

initiating a capture of environmental sensor data by a user device, wherein the sensor data indicates at least a partial view of an environment around a vehicle, and wherein the user device is independent of the vehicle;

processing the environmental sensor data to generate a digitized representation of the environment;

calculating a predicted motion of the vehicle based on the digitized representation;

initiating a capture of motion sensor data by the user device, wherein the motion sensor data indicates an observed motion of the vehicle; and

identifying a safe operation of the vehicle based on a comparison by the user device of the observed motion to the predicted motion of the vehicle.

2. The method of claim 1, further comprising:

flagging a safety concern event associated with the vehicle based on the user device determining that the observed motion deviates from the predicted motion by more than a threshold margin.

3. The method of claim 2, wherein the safety concern event includes at least one of:

the vehicle ignoring a traffic signal;

the vehicle driving over a road margin;

the vehicle failing to keep in a lane; and

the vehicle failing to react to a behavior of another road user.

4. The method of claim 2, further comprising:

initiating a transmission of a signal from the user device to the vehicle based on the safety concern event,

wherein the signal indicates that the vehicle should re-assess the environment, override a vehicle planned motion, come to a stop, or a combination thereof.

5. The method of claim 2, further comprising:

calculating a safety metric for the vehicle based on the safety concern event, one or more previously flagged safety concern events, or a combination thereof; and

providing data to present a representation of the safety metric in a user interface to a user.

6. The method of claim 5, further comprising:

aggregating the safety concern event, the one or more previously flagged safety concern events, or a combination thereof from a plurality of users to calculate the safety metric.

7. The method of claim 5, wherein the safety metric is calculated with respect to the vehicle, a make of the vehicle, an operator of the vehicle, or a combination thereof.

8. The method of claim 1, wherein the environmental sensor data includes image data captured from an image sensor of the user device, an entity external to the vehicle, or a combination thereof;

the method further comprising;

processing the image data using a computer vision system to generate the digitized representation.

9. The method of claim 1, wherein the user device accesses a geographic database that is not available to the vehicle to generate the digitized representation.

10. The method of claim 1, wherein the vehicle is an autonomous vehicle, and wherein the user device is carried by a passenger of the autonomous vehicle and is mounted to the autonomous vehicle during a trip by the passenger.

11. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs,

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,

initiate a capture of environmental sensor data by the apparatus, wherein the sensor data indicates at least a partial view of an environment around a vehicle, and wherein the apparatus is independent of the vehicle;

process the environmental sensor data to generate a digitized representation of the environment;

calculate a predicted motion of the vehicle based on the digitized representation;

initiate a capture of motion sensor data by the apparatus, wherein the motion sensor data indicates an observed motion of the vehicle; and

identify a safe operation of the vehicle based on a comparison of the observed motion to the predicted motion of the vehicle.

12. The apparatus of claim 11, wherein the apparatus is further caused to:

flag a safety concern event associated with the vehicle based on determining that the observed motion deviates from the predicted motion by more than a threshold margin.

13. The apparatus of claim 12, wherein the safety concern event includes at least one of:

the vehicle ignoring a traffic signal;

the vehicle driving over a road margin;

the vehicle failing to keep in a lane; and

the vehicle failing to react to a behavior of another road user.

14. The apparatus of claim 12, wherein the apparatus is further caused to:

initiate a transmission of a signal from the apparatus to the vehicle based on the safety concern event,

wherein the signal indicates that the vehicle should re-assess the environment, override a vehicle planned motion, come to a stop, or a combination thereof.

15. The apparatus of claim 12, wherein the apparatus is further caused to:

calculate a safety metric for the vehicle based on the safety concern event, one or more previously flagged safety concern events, or a combination thereof; and

provide data to present a representation of the safety metric in a user interface to a user.

16. A non-transitory computer-readable storage medium carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to perform:

initiating a capture of environmental sensor data by the apparatus, wherein the sensor data indicates at least a partial view of an environment around a vehicle, and wherein the apparatus is independent of the vehicle;

processing the environmental sensor data to generate a digitized representation of the environment;

calculating a predicted motion of the vehicle based on the digitized representation;

initiating a capture of motion sensor data by the apparatus, wherein the motion sensor data indicates an observed motion of the vehicle; and

identifying a safe operation of the vehicle based on a comparison of the observed motion to the predicted motion of the vehicle.

17. The non-transitory computer-readable storage medium of claim 16, wherein the apparatus is caused to further perform:

flagging a safety concern event associated with the vehicle based on determining that the observed motion deviates from the predicted motion by more than a threshold margin.

18. The non-transitory computer-readable storage medium of claim 17, wherein the safety concern event includes at least one of:

the vehicle ignoring a traffic signal;

the vehicle driving over a road margin;

the vehicle failing to keep in a lane; and

the vehicle failing to react to a behavior of another road user.

19. The non-transitory computer-readable storage medium of claim 17, wherein the apparatus is caused to further perform:

initiating a transmission of a signal from the apparatus to the vehicle based on the safety concern event,

wherein the signal indicates that the vehicle should re-assess the environment, override a vehicle planned motion, come to a stop, or a combination thereof.

20. The non-transitory computer-readable storage medium of claim 17, wherein the apparatus is caused to further perform:

calculating a safety metric for the vehicle based on the safety concern event, one or more previously flagged safety concern events, or a combination thereof; and

providing data to present a representation of the safety metric in a user interface to a user.