Associating a user identity with a mobile device identity

• US 10,354,145 B2
• Filed: 09/11/2015
• Issued: 07/16/2019
• Est. Priority Date: 09/11/2014
• Status: Active Grant

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Nguyen, Thanh Le, Kim, Yu Seung, Tague, Patrick, Zhang, Joy
• Primary Examiner(s): Misleh, Justin P.
• Application Number:
  US15/510,451
• Publication Number:
  US 20170308757A1
• Time in Patent Office: 1,404 Days
• Field of Search: 382118
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04L 12/4625 Single bridge functionality, e.g. connection of two networks over a single bridge
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A system includes, in one aspect, one or more processing devices that perform operations comprising: detecting one or more human objects in images captured by a visual image recording device; obtaining a motion timeseries for each of the detected one or more human objects using the captured images; obtaining a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on received RF signals from the one or more mobile devices; and generating an association between (i) identifying data for a first mobile device of the one or more mobile devices, and (ii) identifying data for one of the one or more human objects representing a first human, wherein the first mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the one of the one or more human objects representing the first human.

Claims

1. A system comprising:

a visual image recording device configured to capture images;

a radio frequency (RF) receiver configured to receive RF signals from one or more mobile devices;

one or more processing devices; and

one or more computer-readable media storing instructions that are executable by the one or more processing devices to perform operations comprising;

detecting one or more human objects in the captured images;

obtaining a motion timeseries for each of the detected one or more human objects using the captured images;

processing the motion timeseries for each of the detected one or more human objects into a visual motion timeseries;

obtaining a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on the received RF signals from the one or more mobile devices;

processing the RSS timeseries, using a machine learning model, for each of the one or more mobile devices into a RF motion timeseries, wherein the machine learning model determines at least an RSS variance and a coefficient of variation extracted from a sliding time window of measurement;

determining an association between a first feature in the visual motion timeseries and a second feature in the RF motion timeseries; and

generating, in response to determining the association, an association between (i) a first identifier representing a particular mobile device of the one or more mobile devices corresponding to the second feature in the RF motion timeseries, and (ii) a second identifier representing a particular human object of the one or more human objects corresponding to the first feature in the visual motion timeseries, wherein the particular mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the particular human object of the one or more human objects.

2. The system of claim 1, wherein the visual image recording device comprises a camera.

3. The system of claim 1, wherein a human object is a visual representation of a portion of a human.

4. The system of claim 3, wherein the portion of the human comprises one or more of a face of the human, shoulders of the human, or upper body of the human.

5. The system of claim 1, wherein the RF receiver that is configured to receive the RF signals from the one or more mobile devices comprises a Wi-Fi receiver that is configured to receive Wi-Fi data packets from the one or more mobile devices, and wherein each of the Wi-Fi data packets comprises a MAC address and an IP address.

6. The system of claim 1, wherein detecting the one or more human objects comprises detecting a portion of a human in the captured images.

7. The system of claim 1, wherein detecting the one or more human objects comprises detecting the one or more human objects that appear in multiple captured images using an appearance model.

8. The system of claim 1, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises tracking a three-dimensional trajectory of the one or more human objects using the captured images.

9. The system of claim 1, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining speed of movement for each of the one or more human objects using the captured images.

10. The system of claim 9, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining that the speed of movement exceeds a threshold speed.

11. The system of claim 1, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises determining the RSS variance, minimum, maximum, ranges, and the coefficient of variation extracted from a two second sliding time window.

12. The system of claim 1, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises computing spectral density of the received RF signals for each of the one or more mobile devices and averaging the computed spectral density over bands of interest.

13. The system of claim 1, wherein generating the association comprises computing a similarity score between the motion time series for each of the detected one or more human objects and the RSS timeseries for each of the one or more mobile devices.

14. The system of claim 13, wherein generating the association comprises generating the association based on the computed similarity score between the motion timeseries for the particular human object and the RSS timeseries for the particular mobile device exceeding a threshold score.

15. The system of claim 1, wherein the operations further comprise:

detecting an inverse proportionality between computed distances of the particular human object from the visual image recording device and received RSS of the particular mobile device, and

generating the association based on the detected inverse proportionality.

16. A system comprising:

one or more processing devices; and

one or more computer-readable media storing instructions that are executable by the one or more processing devices to perform operations comprising;

detecting one or more human objects in images captured by a visual image recording device;

obtaining a motion timeseries for each of the detected one or more human objects using the captured images;

processing the motion timeseries for each of the detected one or more human objects into a visual motion timeseries;

receiving RF signals from one or more mobile devices;

obtaining a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on the received RF signals from the one or more mobile devices;

processing the RSS timeseries, using a machine learning model, for each of the one or more mobile devices into a RF motion timeseries, wherein the machine learning model determines at least an RSS variance and a coefficient of variation extracted from a sliding time window of measurement;

determining an association between a first feature in the visual motion timeseries and a second feature in the RF motion timeseries; and

generating, in response to determining the association, an association between (i) a first identifier representing a particular mobile device of the one or more mobile devices corresponding to the second feature in the RF motion timeseries, and (ii) a second identifier representing a particular human object of the one or more human objects corresponding to the first feature in the visual motion timeseries, wherein the particular mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the particular human object of the one or more human objects.

17. A method comprising:

detecting, by one or more processors, one or more human objects in images captured by a visual image recording device;

obtaining, by one or more processors, a motion timeseries for each of the detected one or more human objects using the captured images;

processing the motion timeseries for each of the detected one or more human objects into a visual motion timeseries;

receiving RF signals from one or more mobile devices;

obtaining, by one or more processors, a received signal strength (RSS) timeseries for each of the one or more mobile devices, based on the received RF signals from the one or more mobile devices;

processing the RSS timeseries, using a machine learning model, for each of the one or more mobile devices into a RF motion timeseries, wherein the machine learning model determines at least an RSS variance and a coefficient of variation extracted from a sliding time window of measurement;

determining an association between a first feature in the visual motion timeseries and a second feature in the RF motion timeseries; and

generating, by one or more processors, in response to determining the association, an association between (i) a first identifier representing a particular mobile device of the one or more mobile devices corresponding to the second feature in the RF motion timeseries, and (ii) a second identifier representing a particular human object of the one or more human objects corresponding to the first feature in the visual motion timeseries, wherein the particular mobile device has an RSS timeseries that fluctuates at a time period corresponding to movement in the obtained motion timeseries for the particular human object of the one or more human objects.

18. The method of claim 17, wherein detecting the one or more human objects comprises detecting a portion of a human in the captured images.

19. The method of claim 17, wherein detecting the one or more human objects comprises detecting the one or more human objects that appear in multiple captured images using an appearance model.

20. The method of claim 17, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises tracking a three-dimensional trajectory of the one or more human objects using the captured images.

21. The method of claim 17, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining speed of movement for each of the one or more human objects using the captured images.

22. The method of claim 21, wherein obtaining the motion timeseries for each of the detected one or more human objects comprises determining that the speed of movement exceeds a threshold speed.

23. The method of claim 17, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises determining the RSS variance, minimum, maximum, ranges, and the coefficient of variation extracted from a two second sliding time window.

24. The method of claim 17, wherein obtaining the RSS timeseries for each of the one or more mobile devices comprises computing spectral density of the received RF signals for each of the one or more mobile devices and averaging the computed spectral density over bands of interest.

25. The method of claim 17, wherein generating the association comprises computing a similarity score between the motion time series for each of the detected one or more human objects and the RSS timeseries for each of the one or more mobile devices.

26. The method of claim 25, wherein generating the association comprises generating the association based on the computed similarity score between the motion timeseries for the particular human object and the RSS timeseries for the particular mobile device exceeding a threshold score.

27. The method of claim 17, further comprising:

detecting an inverse proportionality between computed distances of the particular human object from the visual image recording device and received RSS of the particular mobile device; and

generating the association based on the detected inverse proportionality.