Driver assist using smart mobile devices

• US 9,975,483 B1
• Filed: 02/08/2013
• Issued: 05/22/2018
• Est. Priority Date: 02/08/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ramaswamy, Sharadh
• Primary Examiner(s): Wu, Zhen Y
• Application Number:
  US13/763,330
• Time in Patent Office: 1,929 Days
• Field of Search: 340435
• CPC Class Codes: B60Q 11/00 Arrangement of monitoring devices for devices provided for in groups B60Q1/00 - B60Q9/00
• CPC Class Codes: B60W 2540/00 Input parameters relating to occupants
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2554/801 Lateral distance
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

Approaches are described for utilizing a combination of location determining and tracking techniques to attempt to provide driver assistance under a variety of different circumstances, such as location assistance to locate or get back to a particular object or location, and visual and/or audible assistance to alert a driver of potential danger. For example, a driver assistance system can be provided that can monitor traffic conditions as well as attentiveness of a driver using a portable computing device (such as a mobile phone) to provide visual and/or audible notifications to alert the driver to potential dangers and/or take defensive action. The driver assistance system can further be used to automatically determine the location of a user'"'"'s parked automobile and provide assistance, such as directions, to get back to the location where the user parked his or her automobile.

Claims

1. A computer implemented method for providing operator assistance, the method comprising:

capturing, using a first camera on a first side of a computing device, a first image including at least a portion of a face of a user, the computing device positioned proximate to a dashboard of a first vehicle;

capturing, using a second camera on a second side of the computing device, a second image including at least a portion of an environment outside the first vehicle;

determining, using at least one processor of the computing device and based at least in part on the first image, a gaze direction of the user;

based at least in part on analysis of the second image, determining a presence of an object or event in the at least a portion of the environment that presents potential harm in relation to the first vehicle;

determining that the object or event satisfies a threshold condition sufficient to trigger an audio and vibration alarm to alert the user of the potential harm, wherein the threshold condition includes flashing brake or warning lights of a second vehicle located within a threshold distance of a driving path of the first vehicle;

determining, using the at least one processor of the computing device and based at least in part on the gaze direction, that the gaze direction of the user is substantially away from the object or event in the environment; and

providing the audio and vibration alarm to alert the user and providing steering instructions to guide the first vehicle around the potential harm in response to determining the object or event satisfies the threshold condition and the gaze direction being substantially away from the object or event in the environment.

2. The computer implemented method of claim 1, further comprising:

determining that the object or event satisfies a danger criterion; and

providing a notification when the gaze direction is substantially towards the object.

3. The computer implemented method of claim 1, wherein satisfying the threshold condition further includes detecting at least one of the second vehicle stopping within a threshold distance of the driving path of the first vehicle, the second vehicle moving in a direction towards the first vehicle, flashing a turn signal of the second vehicle located within a threshold distance of the driving path of the first vehicle, or the second vehicle moving within a threshold distance of the driving path of the first vehicle.

4. A computer implemented method, comprising:

determining an orientation of a user based at least in part on sensor information obtained from a mobile computing device, the mobile computing device positioned proximate to the user and within an interior of a first vehicle;

determining, by the mobile computing device, at least one characteristic of at least one object or event in an environment with respect to the first vehicle based at least on the sensor information, wherein the at least one characteristic includes flashing brake or warning lights of a second vehicle located within a threshold distance of a driving path of the first vehicle;

determining, based on the orientation of the user, that a gaze direction of the user is substantially away from the at least one object or event;

determining, based at least on the gaze direction of the user and the at least one characteristic, that a notification criterion is met; and

providing an audio and vibration notification within the interior of the first vehicle and providing steering instructions to guide the first vehicle around the second vehicle in response to the notification criterion being met.

5. The computer implemented method of claim 4, further comprising:

capturing, using a first camera on a first side of the mobile computing device, a first image including at least a portion of a face of the user;

capturing, using a second camera on a second side of the mobile computing device, a second image including at least a portion of the environment outside the first vehicle;

determining, based at least in part on the first image, the gaze direction of the user; and

determining an object or event represented in the second image, the object or event being associated with the at least one characteristic.

6. The computer implemented method of claim 4, further comprising:

capturing, using a first camera on a first side of the mobile computing device, a set of images including a view of at least a portion of a face of the user;

determining, based at least in part on the set of images, a blinking frequency of the user;

determining, based at least in part on comparing the blinking frequency to at least one blinking frequency threshold, that the user is likely falling asleep; and

providing the audio notification based at least in part on the blinking frequency being below the at least one blinking frequency threshold.

7. The computer implemented method of claim 4, further comprising:

capturing, using a camera of the mobile computing device, image data that includes at least a portion of an environment outside the first vehicle;

analyzing the image data to determine an amount of motion of the environment with respect to the first vehicle;

determining that the first vehicle has parked based at least in part on the determined amount of motion;

determining a location of the mobile computing device using at least one location-determination component of the mobile computing device, the location corresponding to a parked location of the first vehicle; and

in response to a request from the user of the mobile computing device to obtain directions to the parked location of the first vehicle, providing the user with directions from a current location of the mobile computing device to the parked location of the first vehicle.

8. The computer implemented method of claim 4, further comprising:

monitoring a rate of change of position of the mobile computing device over a period of time;

determining a transition from a first rate of change of position to a second rate of change of position for the period of time, the first rate of change of position above a maximum walking speed and the second rate of change of position below the maximum walking speed;

determining a location of the mobile computing device using at least one location-determination component of the mobile computing device based at least in part on the transition, the location corresponding to a parked location of the first vehicle; and

storing the location in a non-transitory computer readable storage medium of the mobile computing device.

9. The computer implemented method of claim 8, further comprising:

receiving a request from the user of the mobile computing device to obtain directions to the parked location of the first vehicle;

retrieving the location from the non-transitory computer readable storage medium; and

providing the user with directions from a current location of the mobile computing device to the parked location of the first vehicle.

10. The computer implemented method of claim 4, further comprising, providing a visual alert, an audiovisual alert, or a tactile alert.

11. The computer implemented method of claim 4, wherein determining the orientation of the user further includes determining at least one of a blinking frequency of the user, a head location of the user, or a body position of the user relative to the mobile computing device.

12. The computer implemented method of claim 4, wherein determining at least one characteristic further includes detecting at least one of the second vehicle stopping within a threshold distance of a driving path of the first vehicle, the second vehicle moving in a direction towards the first vehicle, flashing a turn signal of the second vehicle located within a threshold distance of the driving path of the first vehicle, or the second vehicle moving within a threshold distance of the driving path of the first vehicle.

13. The computer implemented method of claim 4, wherein determining the gaze direction of the user includes:

capturing video data using at least one camera, the video data including at least a portion of the face of the user;

locating position information for the user'"'"'s eyes in the video data; and

determining the gaze direction based at least in part on the position information for the user'"'"'s eyes.

14. A computing system, comprising:

at least one processor;

at least one camera; and

memory including instructions that, when executed by the processor, cause the computing system to;

determine an orientation of a user based at least in part on sensor information obtained from a mobile computing device, the mobile computing device positioned proximate to the user and within an interior of a first vehicle;

determine, by the mobile computing device, at least one characteristic of at least one object or event in an environment with respect to the first vehicle based at least in part on the sensor information, wherein the at least one characteristic includes flashing brake or warning lights of a second vehicle located within a threshold distance of a driving path of the first vehicle;

determine, based on the orientation of the user, that a gaze direction of the user is substantially away from the at least one object or event;

determine, based at least in part on the gaze direction of the user and the at least one characteristic, that a notification criterion is met; and

provide an audio and vibration notification within the interior of the first vehicle and provide steering instructions to guide the first vehicle around the second vehicle in response to the notification criterion being met.

15. The computing system of claim 14, wherein the instructions, when executed, further cause the computing system to:

capture, using a first camera on a first side of the computing system, a first image including at least a portion of a face of the user;

capture, using a second camera on a second side of the computing system, a second image including at least a portion of an environment outside the first vehicle;

determine, based at least in part on the first image, a gaze direction of the user; and

determine an object or event represented in the second image, the object or event being associated with the at least one characteristic.

16. The computing system of claim 14, wherein the instructions, when executed, further cause the computing system to:

capture, using a first camera on a first side of the computing system, a set of images including a view of at least a portion of a face of the user;

determine, based at least in part on the set of images, a blinking frequency of the user;

determine, based at least in part on comparing the blinking frequency to at least one blinking frequency threshold, that the user is likely falling asleep; and

provide the audio notification based at least in part on the blinking frequency being below the at least one blinking frequency threshold.

17. The computing system of claim 14, wherein the instructions, when executed, further cause the computing system to:

capture, using a camera of the mobile computing device, image data that includes at least a portion of an environment outside the first vehicle;

analyze the image data to determine an amount of motion of the environment with respect to the first vehicle;

determine that the first vehicle has parked based at least in part on the determined amount of motion;

determine a location of the mobile computing device using at least one location-determination component of the mobile computing device, the location corresponding to a parked location of the first vehicle; and

in response to a request from the user of the mobile computing device to obtain directions to the parked location of the first vehicle, provide the user with directions from a current location of the mobile computing device to the parked location of the first vehicle.

18. The computing system of claim 14, wherein the instructions, when executed, further cause the computing system to:

monitor a rate of change of position of the computing system over a period of time;

determine a transition from a first rate of change of position to a second rate of change of position for the period of time, the first rate of change of position above a maximum walking speed and the second rate of change of position below the maximum walking speed;

determine a location of the computing system using at least one location-determination component of the computing system based at least in part on the transition, the location corresponding to a parked location of the first vehicle; and

store the location in a non-transitory computer readable storage medium of the computing system.

19. The computing system of claim 14, wherein a location of the object or event and the gaze direction of the user are based at least in part on a relative orientation of the computing system to the user and the object or event.

20. A non-transitory computer readable storage medium storing one or more sequences of instructions executable by one or more processors to perform a set of operations comprising:

determining an orientation of a user based at least in part on sensor information obtained from a mobile computing device, the mobile computing device positioned proximate to the user and within an interior of a first vehicle;

determining, by the mobile computing device, at least one characteristic of at least one object or event in an environment with respect to the first vehicle based at least on the sensor information, wherein the at least one characteristic includes flashing brake or warning lights of a second vehicle located within a threshold distance of a driving path of the first vehicle;

determining, based on the orientation of the user, that a gaze direction of the user is substantially away from the at least one object or event;

determining, based at least on the gaze direction of the user and the at least one characteristic, that a notification criterion is met; and

providing an audio and vibration notification within the interior of the first vehicle and providing steering instructions to guide the first vehicle around the second vehicle in response to the notification criterion being met.

21. The non-transitory computer readable storage medium of claim 20, further comprising instructions executed by the one or more processors to perform the operations of:

capturing, using a first camera on a first side of the mobile computing device, a first image including at least a portion of a face of the user;

capturing, using a second camera on a second side of the mobile computing device, a second image including at least a portion of an environment outside the first vehicle;

determining, based at least in part on the first image, the gaze direction of the user; and

determining an object or event represented in the second image, the object or event being associated with the at least one characteristic.

22. The non-transitory computer readable storage medium of claim 20,

monitoring a rate of change of position of the mobile computing device over a period of time;

determining a transition from a first rate of change of position to a second rate of change of position for the period of time, the first rate of change of position above a maximum walking speed and the second rate of change of position below the maximum walking speed;

determining a location of the mobile computing device using at least one location-determination component of the mobile computing device based at least in part on the transition, the location corresponding to a parked location of the first vehicle; and

storing the location in a non-transitory computer readable storage medium of the mobile computing device.

23. The non-transitory computer readable storage medium of claim 20, further comprising instructions executed by the one or more processors to perform the operations of:

capturing, using a camera of the mobile computing device, image data that includes a representation of at least a portion of an environment outside the first vehicle;

analyzing the image data to determine a first velocity of the first vehicle and a second velocity of the first vehicle;

determining that the first vehicle has parked based at least in part on a difference between the first velocity and the second velocity, the first velocity being above a maximum walking speed and the second velocity being below the maximum walking speed; and

determining a location of the mobile computing device using at least one location-determination component of the mobile computing device based at least in part on the difference being observed for at least a predetermined period of time.

24. The non-transitory computer readable storage medium of claim 20, wherein determining the gaze direction of the user includes estimating the gaze direction based at least in part on a direction of a head of the user.