ASSOCIATING A USER IDENTITY WITH A MOBILE DEVICE IDENTITY

• US 20170308757A1
• Filed: 09/11/2015
• Published: 10/26/2017
• Est. Priority Date: 09/11/2014
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Nguyen, Thanh Le, Kim, Yu Seung, Tague, Patrick, Zhang, Joy
• Granted Patent: US 10,354,145 B2
• Time in Patent Office: Days
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04L 12/4625 Single bridge functionality, e.g. connection of two networks over a single bridge
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A system includes, in one aspect, one or more processing devices that perform operations comprising: detecting one or more human objects in images captured by a visual image recording device; obtaining a motion timeseries for each of the detected one or more human objects using the captured images; obtaining a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on received RF signals from the one or more mobile devices; and generating an association between (i) identifying data for a first mobile device of the one or more mobile devices, and (ii) identifying data for one of the one or more human objects representing a first human, wherein the first mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the one of the one or more human objects representing the first human.

Claims

1. A system comprising:

a visual image recording device configured to capture images;

a radio frequency (RF) receiver configured to receive RF signals from one or more mobile devices;

one or more processing devices; and

one or more computer-readable media storing instructions that are executable by the one or more processing devices to perform operations comprising;

detecting one or more human objects in the captured images;

obtaining a motion timeseries for each of the detected one or more human objects using the captured images;

obtaining a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on the received RF signals from the one or more mobile devices; and

generating an association between (i) identifying data for a first mobile device of the one or more mobile devices, and (ii) identifying data for one of the one or more human objects representing a first human, wherein the first mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the one of the one or more human objects representing the first human.

2. The system of claim 1, wherein the visual image recording device comprises a camera.

3. The system of claim 1, wherein a human object is a visual representation of a portion of a human.

4. The system of claim 1, wherein the portion of the human comprises one or more of a face of the human, shoulders of the human, or upper body of the human.

5. The system of claim 1, wherein the RF receiver that is configured to receive the RF signals from the one or more mobile devices comprises a Wi-Fi receiver that is configured to receive Wi-Fi data packets from the one or more mobile devices, and wherein each of the Wi-Fi data packets comprises a MAC address and an IP address.

6. The system of claim 1, wherein detecting the one or more human objects comprises detecting a portion of a human in the captured images.

7. The system of claim 1, wherein detecting the one or more human objects comprises detecting the one or more human objects that appear in multiple captured images using an appearance model.

8. The system of claim 1, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises tracking a three-dimensional trajectory of the one or more human objects using the captured images.

9. The system of claim 1, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining speed of movement for each of the one or more human objects using the captured images.

10. The system of claim 9, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining that the speed of movement exceeds a threshold speed.

11. The system of claim 1, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises determining RSS variance, minimum, maximum, ranges, and coefficient of variation extracted from a two second sliding window.

12. The system of claim 1, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises computing spectral density of the received RF signals for each of the one or more mobile devices and averaging the computed spectral density over bands of interest.

13. The system of claim 1, wherein generating the association comprises computing a similarity score between the motion time series for each of the detected one or more human objects and the RSS timeseries for each of the one or more mobile devices.

14. The system of claim 13, wherein generating the association comprises generating the association based on the computed similarity score between the motion timeseries for the human object representing the first human and the RSS timeseries for the first mobile device exceeding a threshold score.

15. The system of claim 1, wherein the operations further comprise:

detecting an inverse proportionality between computed distances of the human object representing the first human from the visual image recording device and received RSS of the first mobile device, and

generating the association based on the detected inverse proportionality.

16. A system comprising:

one or more processing devices; and

one or more computer-readable media storing instructions that are executable by the one or more processing devices to perform operations comprising;

detecting one or more human objects in images captured by a visual image recording device;

obtaining a motion timeseries for each of the detected one or more human objects using the captured images;

receiving RF signals from one or more mobile devices;

obtaining a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on the received RF signals from the one or more mobile devices; and

generating an association between (i) identifying data for a first mobile device of the one or more mobile devices, and (ii) identifying data for one of the one or more human objects representing a first human, wherein the first mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the one of the one or more human objects representing the first human.

17. A method comprising:

detecting, by one or more processors, one or more human objects in images captured by a visual image recording device;

obtaining, by one or more processors, a motion timeseries for each of the detected one or more human objects using the captured images;

receiving RF signals from one or more mobile devices;

obtaining, by one or more processors, a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on the received RF signals from the one or more mobile devices; and

generating, by one or more processors, an association between (i) identifying data for a first mobile device of the one or more mobile devices, and (ii) identifying data for one of the one or more human objects representing a first human, wherein the first mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the one of the one or more human objects representing the first human.

18. The method of claim 17, wherein detecting the one or more human objects comprises detecting a portion of a human in the captured images.

19. The method of claim 17, wherein detecting the one or more human objects comprises detecting the one or more human objects that appear in multiple captured images using an appearance model.

20. The method of claim 17, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises tracking a three-dimensional trajectory of the one or more human objects using the captured images.

21. The method of claim 17, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining speed of movement for each of the one or more human objects using the captured images.

22. The method of claim 21, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining that the speed of movement exceeds a threshold speed.

23. The method of claim 17, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises determining RSS variance, minimum, maximum, ranges, and coefficient of variation extracted from a two second sliding window.

24. The method of claim 17, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises computing spectral density of the received RF signals for each of the one or more mobile devices and averaging the computed spectral density over bands of interest.

25. The method of claim 17, wherein generating the association comprises computing a similarity score between the motion time series for each of the detected one or more human objects and the RSS timeseries for each of the one or more mobile devices.

26. The method of claim 25, wherein generating the association comprises generating the association based on the computed similarity score between the motion timeseries for the human object representing the first human and the RSS timeseries for the first mobile device exceeding a threshold score.

27. The method of claim 17, further comprising:

detecting an inverse proportionality between computed distances of the human object representing the first human from the visual image recording device and received RSS of the first mobile device; and

generating the association based on the detected inverse proportionality.