Time-in store estimation using facial recognition

• US 9,792,489 B2
• Filed: 02/14/2017
• Issued: 10/17/2017
• Est. Priority Date: 12/12/2012
• Status: Active Grant

• Current Assignee: Cognyte Technologies Israel Ltd. (Cognyte Software Ltd.)
• Original Assignee: Verint Systems Limited (Verint Systems Incorporated)
• Inventors: Golan, Oren, Zimerman, Gadi, Figov, Zvi, Waisman, Ofer, Falckovitch, Elad, Yankovitch, Doron
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US15/432,298
• Publication Number:
  US 20170220853A1
• Time in Patent Office: 245 Days
• Field of Search: 382103, 382107, 382115, 382116, 382117, 382118, 382236, 348169, 348170, 348171, 348172, 348352
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

A method of monitoring the amount of time spent in a specified area by an individual comprises employing a first camera to automatically create one or more entrance images, each entrance image containing a face of an entering individual that passes a first location, and storing each entrance image in a database along with a corresponding entrance time that the entering individual passed the entrance location. An exit image is compared to the entrance images in the database to identify a matching entrance image containing the same face as the exit image. A stay time is then determined for the exiting individual by determining the difference between the entrance time corresponding to the matching entrance image and the exit time.

Claims

1. A non-transitory computer readable medium having instructions stored thereon for monitoring a time-in-store for an individual, in which the instructions when executed by a computing system cause the computing system to at least:

receive an image of an entrance location from a first camera;

timestamp the image of the entrance location with an associated entrance time;

detect one or more faces in the image of the entrance location using a face detection algorithm;

create an entrance image of the one or more faces detected in the image of the entrance location;

store the entrance image and the entrance time together in an entrance image database;

receive an image of an exit location from a second camera;

timestamp the image of the exit location with an associated exit time; and

detect one or more faces in the image of the exit location using the face detection algorithm;

create an exit image of the one or more faces detected in the image of the exit location;

store the exit image and the exit time together in an exit image database;

detect a matching entrance image by at least using a facial recognition algorithm to compare the exit image stored in the exit image database to one or more entrance images stored in the entrance image database, wherein the matching entrance image contains a facial match to the exit image;

calculate a stay time for a matched exiting individual by at least determining the difference between the exit time stored in the database corresponding to the exit image and the entrance time stored in the database corresponding to the matching entrance image.

2. The non-transitory computer readable medium of claim 1, wherein receiving an image of an entrance location includes receiving video images of the entrance location;

and wherein receiving an image of an exit location includes receiving video images of the exit location.

3. The non-transitory computer readable medium of claim 2, wherein the instructions further cause the computing system to at least process the video images of the entrance location or the exit location to detect changes;

wherein the detecting one or more faces in the image of the entrance location includes processing the video images of the entrance location with the face detection algorithm after detecting a change in the images of the entrance location; and

wherein the detecting one or more faces in the image of the exit location includes processing the video images of the exit location with the face detection algorithm after detecting a change in the images of the exit location.

4. The non-transitory computer readable medium of claim 2, wherein the detecting one or more faces in the image of the entrance location includes continuously processing the video images of the entrance location and the exit location using the face detection algorithm.

5. The non-transitory computer readable medium of claim 1, wherein the instructions further causes the computing system to at least:

identify unmatched entrance images in the entrance image database; and

remove unmatched entrance images from the entrance image database that have entrance times before a predefined time period.

6. The non-transitory computer readable medium of claim 1, wherein the instructions further cause the computing system to at least:

calculate stay times for multiple matched entrance and exit images over a predefined time period; and

average the stay times within the predefined time period to create an average stay time for the predefined time period.

7. The non-transitory computer readable medium of claim 6, wherein the instructions further cause the computing system to at least calculate an estimated arrival time at a check-out location for an individual passing an entrance location based on the average stay time.

8. A system for monitoring time spent in a location by an individual, the system comprising:

an image processing module configured to;

receive an image of an entrance location from a first camera,

process the received image of the entrance location using a face detection algorithm to detect faces within the received image of the entrance location,

create an entrance image of each face detected in the image of the entrance location,

timestamp each entrance image with an entrance time that corresponds to when an individual in the entrance image passed the entrance location,

store each entrance image and the corresponding entrance time in a database,

receive an image of an exit location from a second camera,

process the received image of the exit location using the face detection algorithm to detect faces within the received image of the exit location,

create an exit image of each face detected in the image of the exit location;

timestamp each exit image with an exit time that corresponds to when an individual in the exit image passed the exit location, and

store each entrance image and the corresponding entrance time in the database;

an image matching module configured to compare an exit image stored in the database to one or more of the entrance images stored in the database using a facial recognition algorithm to identify a matching entrance image containing the same face as an exiting individual oft exit image; and

a stay time calculator module configured to calculate a stay time for a matched exiting individual by determining the difference between the entrance time stored in the database corresponding to the matching entrance image and the exit time stored in the database corresponding to the exit image.

9. The system of claim 8, wherein the first camera is positioned to record individuals entering a store and the second camera is positioned to record individuals exiting the store;

and wherein the stay time for the exiting individual is the time that the exiting individual spent in the store.

10. The system of claim 8 further comprising a centralized image processing module that receives data from both the first camera and the second camera, wherein the first image processing module and the second image processing module are encompassed in the centralized image processing module.

11. The system of claim 8, wherein unmatched entrance images are removed from the database of entrance images after a predefined period of time.

12. A method of monitoring the amount of time spent in a specified area by an individual, the method comprising:

receiving a plurality of entrance images, each entrance image from one or more cameras containing a face of an entering individual who entered a location;

timestamping each entrance image with an entrance time corresponding to when the entering individual entered the location and storing each entrance image in a database along with the corresponding entrance time;

receiving an exit image containing a face of an exiting individual exiting the location;

timestamping the exit image with an exit time corresponding to when the exiting individual exited the location and storing the exit image in the database along with the corresponding exit time;

identifying a matching entrance image containing the same face as the exit image by at least comparing the exit image stored in the database to the entrance images stored in the database; and

calculating a stay time for the exiting individual by at least determining the difference between the entrance time stored in the database corresponding to the matching entrance image and the exit time stored in the database corresponding to the exit image.

13. The method of claim 12, further including a first step of employing a first camera to automatically record an image of the location;

and

a step of creating the plurality of entrance images by at least;

processing the image of the location with a face detection algorithm to detect a face;

determining a facial feature descriptor for the detected face using a facial recognition algorithm; and

generating an entrance image to include the facial feature descriptor.

14. The method of claim 12, further including a first step of employing a first camera to automatically record an image of the location;

and

a step of creating the exit image by at least;

processing the image of the location with a face detection algorithm to detect a face;

determining a facial feature descriptor for the detected face using a facial recognition algorithm; and

generating an exit image to include the facial feature descriptor.

15. The method of claim 12, further including a first step of employing a first camera to automatically record an image of the location;

and

a step of creating the plurality of entrance images by at least;

processing the image of the location with a person detection algorithm to detect a person;

using the person detection algorithm to identify a head area of the detected person in the image of the location;

generating an entrance image based on the head area of the detected person.

16. The method of claim 15, further including using a facial feature extractor algorithm to determine a facial feature descriptor for the detected person.

17. The method of claim 12, further comprising determining stay times of multiple exiting individuals over a predefined time period based on a comparison of entrance and exit times of matching entrance and exit images.

18. The method of claim 17, further comprising averaging the stay times within the predefined time period to create an average stay time for the predefined time period.

19. The method of claim 18, further comprising creating a graphical depiction of average stay time over time.

20. The method of claim 12, wherein the step of identifying the matching entrance image includes locating an entrance image in the database if any which meets at least a predefined commonality threshold with the exit image.