Association of unique person to a mobile device using repeat face image matching

• US 10,198,625 B1
• Filed: 03/26/2016
• Issued: 02/05/2019
• Est. Priority Date: 03/26/2016
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Shin, Joonhwa, Sharma, Rajeev, Kim, Donghun
• Primary Examiner(s): Rao, Anand S
• Application Number:
  US15/081,859
• Time in Patent Office: 1,046 Days
• Field of Search: 348142-160
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04W 4/02 Services making use of location information

Abstract

A method and system for associating a physically identifiable feature of a person with the unique identifier of a mobile device carried by the person as detected across repeat visits to a physical location or multiple locations. Specifically, associating the face of a repeat visitor with a unique identifier of a mobile device, such as the device'"'"'s MAC address, by means of a repeat visit analysis without any explicit or voluntary participation of the person, for example, in a form of providing their information by participation in a survey. For each detection of a particular MAC address associated with a mobile device at a particular physical location, a set of candidate faces can be captured at that physical location. Revisiting faces can be identified and grouped together, and a probability that each group of faces is associated with a particular MAC address can then be calculated. After satisfying several constraints, a maximum probability can be determined, resulting in a match of a particular face to a particular MAC address.

Claims

1. A method for associating a physically identifiable feature of a person with a unique identifier of a mobile device utilizing at least a camera, at least a mobile signal sensor, and at least a processor for the association comprising the following steps, repeated for each visit of the mobile device:

a. detecting the unique identifier of the mobile device using a mobile signal sensor for at least one physical location using a Mobile Device Visit Detection module, wherein a Mobile Device Visit Detection module captures a timestamp indicating the time when the mobile device was detected,

b. capturing a plurality of input images of a face of at least a person at the physical location using at least a camera using a Candidate Face Collection module, wherein the Candidate Face Collection module captures the plurality of input images during a period of time when the mobile device was detected by the mobile signal sensors at the physical location,

c. storing the unique identifier of the mobile device and plurality of input images in a database,

d. clustering the plurality of input images into a face group using a Face Clustering module, and

e. matching a particular face group to a particular mobile device using a Face to Mobile Matching module.

2. The method of claim 1, wherein the unique identifier of the mobile device is the MAC address of the radio of the mobile device, and wherein the Face to Mobile Matching module uses the MAC address of the device to match a particular face group to a particular mobile device.

3. The method of claim 1, wherein the database includes unique identifier data and plurality of input image data captured from at least one physical locations.

4. The method of claim 3, wherein the database further includes timestamp and location data of the mobile device, and timestamp and location data for the plurality of input images.

5. The method of claim 1, wherein the Mobile Device Visit Detection module captures a timestamp and location for the mobile device, and the Candidate Face Collection module records timestamp and location information for the plurality of input images.

6. The method of claim 1, wherein the period of time when the Candidate Face Collection module captures the plurality of input images satisfies t_in−

Δ

t<

t<

t_out+Δ

t, where t_in is the time when the mobile device is first detected by the mobile signal sensor at the physical location, t_out is the time when the mobile device is last detected by the mobile signal sensor at the physical location, t is at least one moment when the Candidate Face Collection module captures the plurality of input images, and Δ

t is a predetermined amount of time used to account for a delay between when the mobile device enters or exits the physical location and when it is detected by the mobile signal sensor.

7. The method of claim 1, wherein the Face to Mobile Matching module further comprises the computation of the probability that a particular face group matches a given mobile device using a Face-to-Mobile Matching Probability Computation module, and wherein the best match is determined by finding the face group with the maximum probability of a match given the mobile device using the Best Matching Discovery module.

8. The method of claim 7, wherein the probability that a particular face group matches a given mobile device is computed based on how frequently the particular face group is captured in the physical space at the times when the given mobile device is detected within the physical space.

9. The method of claim 7, further comprising a Candidate Face Dropout module, wherein the Candidate Face Dropout module drops from consideration a face group with a probability calculated to be less than a defined threshold.

10. The method of claim 1, further incorporating constraints comprising a minimum number of visits of a mobile device, a minimum probability of association of a particular face group to a particular mobile device, and a minimum difference from the most probable face group to the next most probable face group.

11. The method of claim 1, wherein the clustering of the input images into face groups is based on the similarity of the faces in a feature space.

12. The method of claim 11, wherein the clustering of the input images into face groups comprises Hierarchical Agglomerative Clustering or Iterative K-Mean clustering.

13. A system for associating a physically identifiable feature of a person with a unique identifier of a mobile device utilizing at least a camera, at least a mobile signal sensor, and at least a processor for the association comprising the following steps, repeated for each visit of the mobile device:

a. detecting the unique identifier of the mobile device using a mobile signal sensor for at least one physical location using a Mobile Device Visit Detection module, wherein a Mobile Device Visit Detection module captures a timestamp indicating the time when the mobile device was detected,

b. capturing a plurality of input images of a face of at least a person at the physical location using at least a camera using a Candidate Face Collection module, wherein the Candidate Face Collection module captures the plurality of input images during a period of time when the mobile device was detected by the mobile signal sensors at the physical location,

c. storing the unique identifier of the mobile device and plurality of input images in a database,

d. clustering the plurality of input images into a face group using a Face Clustering module, and

e. matching a particular face group to a particular mobile device using a Face to Mobile Matching module.

14. The system of claim 13, wherein the unique identifier of the mobile device is the MAC address of the radio of the mobile device, and wherein the Face to Mobile Matching module uses the MAC address of the device to match a particular face group to a particular mobile device.

15. The system of claim 13, wherein the database includes unique identifier data and plurality of input image data captured from at least one physical locations.

16. The system of claim 15, wherein the database further includes timestamp and location data of the mobile device, and timestamp and location data for the plurality of input images.

17. The system of claim 13, wherein the Mobile Device Visit Detection module captures a timestamp and location for the mobile device, and the Candidate Face Collection module records timestamp and location information for the plurality of input images.

18. The system of claim 13, wherein the period of time when the Candidate Face Collection module captures the plurality of input images satisfies t_in−

Δ

t<

t<

t_out+Δ

t, where t_in is the time when the mobile device is first detected by the mobile signal sensor at the physical location, t_out is the time when the mobile device is last detected by the mobile signal sensor at the physical location, t is at least one moment when the Candidate Face Collection module captures the plurality of input images, and Δ

t is a predetermined amount of time used to account for a delay between when the mobile device enters or exits the physical location and when it is detected by the mobile signal sensor.

19. The system of claim 13, wherein the Face to Mobile Matching module further comprises the computation of the probability that a particular face group matches a given mobile device using a Face-to-Mobile Matching Probability Computation module, and wherein the best match is determined by finding the face group with the maximum probability of a match given the mobile device using the Best Matching Discovery module.

20. The system of claim 19, wherein the probability that a particular face group matches a given mobile device is computed based on how frequently the particular face group is captured in the physical space at the times when the given mobile device is detected within the physical space.

21. The system of claim 19, further comprising a Candidate Face Dropout module, wherein the Candidate Face Dropout module drops from consideration a face group with a probability calculated to be less than a defined threshold.

22. The system of claim 13, further incorporating constraints comprising a minimum number of visits of a mobile device, a minimum probability of association of a particular face group to a particular mobile device, and a minimum difference from the most probable face group to the next most probable face group.

23. The system of claim 13, wherein the clustering of the input images into face groups is based on the similarity of the faces in a feature space.

24. The system of claim 23, wherein the clustering of the input images into face groups comprises Hierarchical Agglomerative Clustering or Iterative K-Mean clustering.