Real-time warning for distracted pedestrians with smartphones

• US 10,036,809 B2
• Filed: 09/25/2015
• Issued: 07/31/2018
• Est. Priority Date: 09/26/2014
• Status: Active Grant

• Current Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Original Assignee: Board of Regents of the University of Michigan (University of Michigan)
• Inventors: Shin, Kang G., Tung, Yu-Chih
• Primary Examiner(s): Pihulic, Daniel
• Application Number:
  US14/865,262
• Publication Number:
  US 20160091606A1
• Time in Patent Office: 1,040 Days
• CPC Class Codes: G01S 15/08 Systems for measuring distance only indirect measurement G01S15/46
• CPC Class Codes: G01S 15/526 by comparing echos in different sonar periods
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/93 for anti-collision purposes

Abstract

Studies have shown that using smartphones while walking increases the probability of pedestrians colliding with obstacles. Techniques are presented for providing real-time warnings to distracted users of smartphones. The techniques are implemented by an application which makes use of sensors commonly found in most smartphone. By estimating distances to nearby objects using acoustic signals, a generalized solution is achieved without requiring any prior knowledge of the user'"'"'s environment. The process can be enhanced by using images acquired from the phone'"'"'s rear camera.

Claims

1. A computer-implemented system for identifying obstacles in path of a pedestrian using a handheld computing device, comprising:

a motion estimator configured to receive an input from an accelerometer residing in the handheld computing device and operates to determine speed at which the handheld computing device is moving;

an acoustic detector interfaced with a speaker in the handheld computing device and operates to emit acoustic signals via the speaker, the acoustic detector interfaced with a microphone in the handheld computing device and, via the microphone, operates to detect the acoustic signals reflected by objects proximate to the handheld computing device and, in response to the detected acoustic signals, determines distance to a given object using timing between the emitted acoustic signals and the detected acoustic signals; and

a motion filter interfaced with the motion estimator and the acoustic detector, the motion filter determines rate at which distance to the given object is changing from the distance measures received from the acoustic detector and, in response to a determination that the rate at which the distance to the given object changes is substantially same to the speed at which the handheld computing device is moving, identifies the given object as an obstacle in the path of the pedestrian, wherein the motion estimator, the acoustic detector and motion filter reside in the handheld computing device and are implemented by instructions executed by a processor residing in the handheld computing device.

2. The system of claim 1 wherein the motion filter estimates an expected distance to the given object using the distance to the given object, compares the expected distance for the given object to other distances determined by the acoustic detector, and increases probability that the given object is an obstacle when difference between the expected distance for the given object and at least one other distance determined by the acoustic detector is less than a threshold.

3. The system of claim 1 wherein the motion filter identifies the given object as an obstacle in response to a determination that the rate at which the distance changes is within one meter per second of the speed at which the handheld computing device is moving.

4. The system of claim 1 wherein the motion filter, in response to a determination that the rate at which the distance to the given object changes differs from the speed at which the handheld computing device is moving, ignores the given object.

5. The system of claim 1 wherein the acoustic detector and the motion filter are operational when the speed at which the pedestrian is moving is greater than zero.

6. The system of claim 1 further comprises an alert mechanism interfaced with the motion filter, where the motion filter determines a distance to the given object and the alert mechanism compares the distance to the given obstacle to a threshold and, in response to a determination that the distance to the given obstacle is less than the threshold, generates an alert for the pedestrian.

7. The system of claim 1 further comprises a visual detector configured to receive image data from a camera in the handheld computing device and determines presence of objects in the path of the pedestrian from the image data.

8. The system of claim 7 wherein the visual detector determines presence of object from the image data by determining texture of the ground being traversed by the pedestrian and distinguishing an object as being different from the determined texture of the ground using a back-projection method.

9. The system of claim 7 wherein the motion filter determines a number of objects proximate to the handheld computing device and, in response to a determination that the number of object exceeds a threshold, activates the visual detector.

10. The system of claim 7 wherein the visual detector further determines tilt of the handheld computing device with respect to the ground using the input from the accelerometer, determines height of the handheld computing device from the ground using the acoustic signals, and determines a distance to a given object from the image data based in part on the tilt of the handheld computing device and the height of the handheld computing device.

11. The system of claim 10 wherein the motion filter identifies the given object as an obstacle in response to a determination that the rate at which the distance to the given object changes is substantially same to the speed at which the handheld computing device is moving and a corresponding object occurs in the image data as determined by the visual detector.

12. A computer-implemented method for warning a pedestrian of obstacles in the path of the pedestrian carrying a handheld computing device, comprising:

determining, by a processor residing in the computing device, speed at which the computing device is moving from input of a motion sensor residing in the computing device;

determining, by the processor, a present distance to a given object proximate to the computing device, where the distance is computed from timing between acoustic signals emitted from a speaker of the computing device and acoustic signals detected by a microphone of the computing device;

determining, by the processor, rate at which distance to the given object is changing;

identifying, by the processor, the given object as an obstacle in the path of the pedestrian, the identification occurring in response to a determination that the rate at which the distance to the given object changes is substantially same to the speed at which the handheld computing device is moving; and

generating, by the processor, an alert for the obstacle in response to a determination that the distance to the given object is less than a threshold.

13. The method of claim 12 further comprises identifying signals reflected from objects proximate to the computing device using a bandpass filter and a matched filter corresponding to acoustic signals emitted from the speaker.

14. The method of claim 12 further comprises determining distance to the given object by multiplying speed of sound by time difference between the signals reflect from the given object and corresponding acoustic signals emitted from the speaker.

15. The method of claim 12 wherein identifying the given object as an obstacle further comprises

determining distance to objects proximate to the computing device during a window of time;

estimating an expected distance to the given object from the present distance to the given object and the speed at which the computing device is moving;

comparing the expected distance for the given object to the distances of objects proximate to the computing device in the window of time;

increasing probability that the given object is an obstacle when difference between the expected distance for the given object and at least one of the distances of objects proximate to the computing device is less than a threshold.

16. The method of claim 12 further comprises filtering the given object in response to a determination that the rate at which the distance to the given object changes differs from the speed at which the computing device is moving.

17. The method of claim 12 further comprises

determining a number of objects proximate to the computing device;

comparing the number of proximate objects to a threshold; and

activating a camera in the computing device in response to a determination that the number of proximate objects exceeds the threshold.

18. The method of claim 17 further comprises

determining presence of objects in the path of the pedestrian from image data captured by the camera;

and

identifies the given object as an obstacle in response to a determination that the rate at which the distance to the given object changes is substantially same to the speed at which the handheld computing device is moving and a corresponding object occurs in the image data captured by the camera.