Vehicular threat detection based on image analysis

• US 9,064,152 B2
• Filed: 02/28/2012
• Issued: 06/23/2015
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Lord, Richard T., Lord, Robert W., Myhrvold, Nathan P., Tegreene, Clarence T., Hyde, Roderick A., Wood, Lowell L. Jr., Ishikawa, Muriel Y., Wood, Victoria Y. H., Whitmer, Charles, Bahl, Paramvir, Burger, Douglas C., Chandra, Ranveer, Gates, William H. III, Holman, Paul, Kare, Jordin T., Mundie, Craig J., Paek, Tim, Tan, Desney S., Zhong, Lin, Dyor, Matthew G.
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US13/407,570
• Publication Number:
  US 20130142393A1
• Time in Patent Office: 1,211 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

Techniques for ability enhancement are described. Some embodiments provide an ability enhancement facilitator system (“AEFS”) configured to enhance a user'"'"'s ability to operate or function in a transportation-related context as a pedestrian or a vehicle operator. In one embodiment, the AEFS is configured perform vehicular threat detection based at least in part on analyzing image data. An example AEFS receives data that represents an image of a vehicle. The AEFS analyzes the received data to determine vehicular threat information, such as that the vehicle may collide with the user. The AEFS then informs the user of the determined vehicular threat information, such as by transmitting a warning to a wearable device configured to present the warning to the user.

Claims

1. A method for enhancing ability in a transportation-related context, the method comprising:

receiving image data, at least some of which represents an image of a first vehicle;

determining vehicular threat information based at least in part on the image data, by;

identifying multiple threats to a user; and

identifying a first threat of the multiple threats that is more significant than at least one other of the multiple threats, by;

modeling multiple potential accidents that each correspond to one of the multiple threats to determine a severity associated with each potential accident based on a collision force associated with each potential accident, the collision force based at least on mass and acceleration; and

selecting the first threat based at least in part on which of the multiple potential accidents has the highest collision force; and

presenting the vehicular threat information via a wearable device of the user by instructing the user to avoid the first one of the multiple threats.

2. The method of claim 1, wherein the receiving image data includes:

receiving image data from a camera of a vehicle that is occupied by the user.

3. The method of claim 2, wherein the vehicle is operated by the user.

4. The method of claim 2, wherein the vehicle is operating autonomously.

5. The method of claim 1, wherein the receiving image data includes:

receiving image data from a camera of the wearable device.

6. The method of claim 1, wherein the receiving image data includes:

receiving image data from a camera of the first vehicle.

7. The method of claim 1, wherein the receiving image data includes:

receiving image data from a camera of a vehicle that is not the first vehicle and that is not occupied by the user.

8. The method of claim 1, wherein the receiving image data includes:

receiving image data from a road-side camera.

9. The method of claim 1, wherein the receiving image data includes:

receiving video data that includes multiple images of the first vehicle taken at different times.

10. The method of claim 9, wherein the receiving video data that includes multiple images of the first vehicle taken at different times includes:

receiving a first image of the first vehicle taken at a first time; and

receiving a second image of the first vehicle taken at a second time, wherein the first and second times are sufficiently different such that velocity and/or direction of travel of the first vehicle is determined with respect to positions of the first vehicle shown in the first and second images.

11. The method of claim 1, wherein the determining vehicular threat information includes:

determining a threat posed by the first vehicle to the user.

12. The method of claim 1, wherein the determining vehicular threat information includes:

determining a threat posed by the first vehicle to some other entity besides the user.

13. The method of claim 1, wherein the determining vehicular threat information includes:

determining a threat posed by a vehicle occupied by the user to the first vehicle.

14. The method of claim 1, wherein the determining vehicular threat information includes:

determining a threat posed by a vehicle occupied by the user to some other entity besides the first vehicle.

15. The method of claim 1, wherein the determining vehicular threat information includes:

identifying the first vehicle in the image data.

16. The method of claim 1, wherein the determining vehicular threat information includes:

determining that the first vehicle is moving towards the user when the first vehicle appears to be becoming larger over a sequence of multiple images represented by the image data.

17. The method of claim 1, wherein the determining vehicular threat information includes:

determining motion-related information about the first vehicle, based on one or more images of the first vehicle.

18. The method of claim 17, wherein the determining motion-related information about the first vehicle includes:

determining the motion-related information with respect to timestamps associated with the one or more images.

19. The method of claim 17, wherein the determining motion-related information about the first vehicle includes:

determining a position of the first vehicle.

20. The method of claim 17, wherein the determining motion-related information about the first vehicle includes:

determining a velocity of the first vehicle.

21. The method of claim 20, wherein the determining a velocity of the first vehicle includes:

determining the velocity with respect to a fixed frame of reference.

22. The method of claim 20, wherein the determining a velocity of the first vehicle includes:

determining the velocity with respect to a frame of reference of the user.

23. The method of claim 17, wherein the determining motion-related information about the first vehicle includes determining mass of the first vehicle.

24. The method of claim 1, wherein the determining vehicular threat information includes:

determining that the first vehicle is driving erratically.

25. The method of claim 1, wherein the determining vehicular threat information includes:

determining that the first vehicle is driving with excessive speed.

26. The method of claim 1, wherein the determining vehicular threat information includes:

identifying objects other than the first vehicle in the image data.

27. The method of claim 1, wherein the determining vehicular threat information includes:

determining driving conditions based on the image data, the driving conditions including road surface conditions and/or lighting conditions.

28. The method of claim 1, further comprising:

determining vehicular threat information that is not related to the first vehicle, including by receiving and processing image data that includes images of objects and/or conditions aside from the first vehicle.

29. The method of claim 1, wherein the determining vehicular threat information includes:

determining the vehicular threat information based on motion-related information that is not based on images of the first vehicle, the motion-related information including at least one of;

information about position, velocity, and/or acceleration of the user obtained from sensors in the wearable device;

information about position, velocity, and/or acceleration of the user obtained from devices in a vehicle of the user; and

/or information about position, velocity, and/or acceleration of the first vehicle obtained from devices of the first vehicle.

30. The method of claim 1, further comprising:

determining a likelihood of an accident associated with each of the multiple threats; and

selecting the first threat based at least in part on which of the multiple threats has the highest associated likelihood.

31. The method of claim 1, further comprising:

determining a mass of an object associated with each of the multiple threats; and

selecting the first threat based at least in part on which of the objects has the highest mass.

32. The method of claim 1, wherein the identifying a first one of the multiple threats that is more significant than at least one other of the multiple threats includes:

selecting the most significant threat from the multiple threats.

33. The method of claim 1, further comprising:

determining that an evasive action with respect to the first vehicle poses a threat to some other object; and

instructing the user to take some other evasive action that poses a lesser threat to the some other object.

34. The method of claim 1, further comprising:

identifying multiple threats that each have an associated likelihood and cost; and

determining a course of action that minimizes an expected cost with respect to the multiple threats.

35. The method of claim 34, wherein the cost is based on one or more of a cost of damage to a vehicle, a cost of injury or death of a human, a cost of injury or death of an animal, a cost of damage to a structure, a cost of emotional distress, and/or cost to a business or person based on negative publicity associated with an accident.

36. The method of claim 34, wherein the identifying multiple threats includes:

identifying multiple threats that are each related to different persons or things.

37. The method of claim 34, wherein the identifying multiple threats includes:

identifying multiple threats that are each related to the user.

38. The method of claim 34, wherein the determining a course of action that minimizes an expected cost includes:

minimizing expected costs to the user posed by the multiple threats.

39. The method of claim 34, wherein the determining a course of action that minimizes an expected cost includes:

minimizing overall expected costs posed by the multiple threats, the overall expected costs being a sum of expected costs borne by the user and other parties to an accident;

recommending to the user a course of action that has a first cost to the user, thereby avoiding an accident having a second cost to another party, wherein the first cost is lower than the second cost.

40. The method of claim 1, further comprising:

receiving data representing an audio signal emitted by the first vehicle; and

determining the vehicular threat information based further on the data representing the audio signal.

41. The method of claim 40, wherein the determining the vehicular threat information based further on the data representing the audio signal includes:

performing acoustic source localization to determine a position of the first vehicle based on multiple audio signals received via multiple microphones.

42. The method of claim 1, further comprising:

receiving data representing the first vehicle obtained at a road-based device; and

determining the vehicular threat information based further on the data representing the first vehicle.

43. The method of claim 42, wherein the receiving data representing the first vehicle obtained at a road-based device includes:

receiving the data from a sensor deployed at an intersection.

44. The method of claim 1, further comprising:

transmitting the vehicular threat information to a law enforcement entity.

45. The method of claim 44, further comprising:

determining a license place identifier, a vehicle description, a location, and/or a direction of travel of the first vehicle based on the image data; and

transmitting an indicator of the license plate identifier, the vehicle description, the location, and/or the direction of travel to the law enforcement entity.

46. The method of claim 1, wherein, for each of the multiple threats, the mass and acceleration is determined based on multiple images of the threat.

47. The method of claim 34, wherein determining the course of action that minimizes the expected cost with respect to the multiple threats includes calculating, for each of the multiple threats, a product of the associated likelihood and the associated cost.