Time-in-store estimation using facial recognition

• US 9,245,276 B2
• Filed: 12/12/2013
• Issued: 01/26/2016
• Est. Priority Date: 12/12/2012
• Status: Active Grant

• Current Assignee: Cognyte Technologies Israel Ltd. (Cognyte Software Ltd.)
• Original Assignee: Verint Systems Limited (Verint Systems Incorporated)
• Inventors: Golan, Oren, Zimerman, Gadi, Figov, Zvi, Waisman, Ofer, Yankovitch, Doron, Falckovitch, Elad
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US14/104,099
• Publication Number:
  US 20140161316A1
• Time in Patent Office: 775 Days
• Field of Search: 382/103, 382/107, 382115-118, 382/236, 348169-172, 348/352
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

A method of monitoring the amount of time spent in a specified area by an individual comprises employing a first camera to automatically create one or more entrance images, each entrance image containing a face of an entering individual that passes a first location, and storing each entrance image in a database along with a corresponding entrance time that the entering individual passed the entrance location. A second camera is also employed to automatically create an exit image of a face of an exiting individual that passes a second location, and the exit image is recorded along with the corresponding exit time that the exiting individual passed the exit location. The exit image is then compared to the entrance images in the database to identify a matching entrance image containing the same face as the exit image. A stay time is then determined for the exiting individual by determining the difference between the entrance time corresponding to the matching entrance image and the exit time.

Claims

1. A method of monitoring the amount of time spent in a specified area by an individual, the method comprising:

receiving a plurality of entrance images, each entrance image containing a face of an entering individual that passes a first location;

storing each entrance image in a database along with a corresponding entrance time that the entering individual passes the entrance location;

receiving an exit image containing a face of an existing individual that passes a second location;

storing the exit image along with a corresponding exit time that the exiting individual passes the exit location;

comparing the exit image to the entrance images in the database;

identifying a matching entrance image containing the same face as the exit image; and

determining a stay time for the exiting individual by determining the difference between the entrance time corresponding to the matching entrance image and the exit time;

averaging the stay times within the predefined time period to create an average stay time for the predefined time period; and

determining stay times of multiple exiting individuals over a predefined time period.

2. The method of claim 1, further including a first step of employing a first camera to automatically record an image of a first location;

and

wherein the step of creating the one or more entrance images includes;

processing the image of the first location with a face detection algorithm to detect a face;

determining a facial feature descriptor for the detected face using a facial recognition algorithm; and

generating the entrance image to include the facial feature descriptor.

3. The method of claim 1, further including a first step of employing a first camera to automatically record an image of a second location:

and

wherein the step of creating the one or more exit images includes;

processing the image of the second location with a face detection algorithm to detect a face;

determining a facial feature descriptor for the detected face using a facial recognition algorithm; and

generating the exit image to include the facial feature descriptor.

4. The method of claim 1, further a first step of employing a first camera to automatically record an:

image of a first location; and

wherein the step of creating the one or more entrance images includes;

processing the image of the first location with a person detection algorithm to detect a person;

using the person detection algorithm to identify a head area of the detected person in the image of the first location;

generating an entrance image based on the head area of the detected person.

5. The method of claim 4, further including using a facial feature extractor algorithm to determine a facial feature descriptor for the detected person.

6. The method of claim 1, further comprising creating a graphical depiction of average stay time over time.

7. The method of claim 1 wherein the step of identifying the matching entrance image includes locating an entrance image in the database, if any, that meets at least a predefined commonality threshold with the exit image.

8. A system for monitoring time spent in a location by an individual, the system comprising:

a first camera positioned to record images of individuals passing a first location;

a first image processing module configured to;

process the recorded images from the first camera using a face detection algorithm to detect faces within the recorded image;

create an entrance image of each detected face; and

determine an entrance time for each entrance image, wherein the entrance time is the time that the individual in the entrance image passed the first location;

a database configured to receive and store each entrance image and the corresponding entrance time;

a second camera positioned to record individuals passing a second location;

a second image processing module configured to;

process the recorded images from the second camera using a face detection algorithm to detect faces within the recorded image;

create an exit image of each detected face; and

determine an exit time for each exit image, wherein the exit time is the time that the individual in the exit image passed the second location;

an image matching module configured to compare the exit image to the entrance images in the database using a facial recognition algorithm to identify a matching entrance image containing the same face as the exit image, wherein unmatched entrance images are removed from the database of entrance images after a predefined period of time; and

a stay time calculator module configured to calculate a stay time for the exiting individual by determining the difference between the entrance time corresponding to the matching entrance image and the exit time.

9. The system of claim 8, wherein the first camera is positioned to record individuals entering a store and the second camera is positioned to record individuals exiting the store:

and wherein the stay time for the exiting individual is the time that the exiting individual spent in the store.

10. The system of claim 8, having a centralized image processing module that receives data from both the first camera and the second camera, wherein the first image processing module mid the second image processing module are encompassed in the centralized image processing module.

11. A method of monitoring a time-in-store for an individual, the method comprising:

recording an image of an entrance location with a first camera;

recording an entrance time associated with the image of the entrance location;

detecting one or more faces in the image of the entrance location using a face detection algorithm;

creating an entrance image of the one or more faces detected in the image of the entrance location;

storing the entrance image and the entrance time together in an entrance image database;

recording an image of an exit location with a second camera;

recording an exit time associated with the image of the exit location; and

detecting one or more faces in the image of the exit location using the face detection algorithm;

creating an exit image of the one or more faces detected in the image of the exit location;

storing the exit image and the exit time together;

comparing the exit image to images in the entrance image database using a facial recognition algorithm;

detecting a matching entrance image, wherein the matching entrance image contains a facial match to the exit image;

calculating a stay time by determining the difference between the exit time and the entrance time associated with the matching entrance image;

identifying unmatched entrance images in the entrance image database; and

removing unmatched entrance images from the entrance image database that have entrance times before a predefined time period.

12. The method of claim 11, wherein recording an image of an entrance location includes continuously recording video images of the first location:

and

recording an image of au exit location includes continuously recording video images of the second location.

13. The method of claim 12, further including processing the video images of the first location or the second location to detect changes;

and

wherein the detecting one or more faces in the image of the entrance location includes processing the video images of the first location with the face detection algorithm after detecting a change in the images of the first location, and the detecting one or more faces in the image of the exit location includes processing the video images of the second location with the face detection algorithm after detecting a change in the images of the second location.

14. The method of claim 12, wherein the detecting one or more faces in the image of the entrance location includes continuously processing the video images of the first location and the second location using the face detection algorithm.

15. The method of claim 11, further comprising calculating stay times for multiple matched entrance and exit images over a predefined time period and averaging the stay times within the predefined time period to create an average stay time for the predefined time period.

16. The method of claim 15, further comprising calculating an estimated arrival time at a check-out location for an individual passing an entrance location based on the average stay time.