Association of unique person to point-of-sale transaction data

• US 10,083,358 B1
• Filed: 07/26/2016
• Issued: 09/25/2018
• Est. Priority Date: 07/26/2016
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Shin, Joonhwa, Sharma, Rajeev, Yoon, Youngrock, Kim, Donghun
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US15/220,052
• Time in Patent Office: 791 Days
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

A method and system for associating an image of a face of at least a person with Point-of-Sale (PoS) data by aligning a first event time series with a second event time series based on a dynamic time disparity. An exemplary embodiment can generate an event time series containing facial recognition data for a person or persons during the PoS transaction process. These data can form a vision-based checkout event time series. An embodiment can also collect PoS transaction data from the retail checkout system, using timestamp information to create a second event time series. As there may be a time disparity between the time series, they can then be aligned in order to match events from one time series to the other. Faces identified in checkout events in the first time series can be registered to PoS events and the results stored in a database.

Claims

1. A method for associating at least an image of a face of at least a person with Point-of-Sale (PoS) data by aligning a first event time series with a second event time series based on a dynamic time disparity, and utilizing at least a camera and at least a processor for executing the steps comprising:

a. generating a first event time series using an Event Time Series Generation module,

b. generating a second event time series using the Event Time Series Generation module,

c. aligning the first event time series with the second event time series based on the dynamic time disparity between the time series, using an Event Time Series Clock Alignment module, comprising the steps of;

i. performing a dynamic time warping to find a warping path between corresponding events between the first event time series and the second event time series, wherein the dynamic time warping can be one of a discrete dynamic time warping or a continuous dynamic time warping,

ii. checking for convergence by determining whether the standard deviation of a time difference between corresponding events in the first event time series and the second event time series is less than a predetermined threshold,

iii. adjusting the time index of the second event time series based on the time disparity between the event time series,

iv. testing the alignment of the time series to see if the best alignment has been obtained,

v. sorting the corresponding events based on the time disparity between events, and removing outliers that have a time disparity greater than a threshold, and

vi. repeating the Event Time Series Clock Alignment module steps until the best alignment is obtained,

d. matching first time series events with second time series events based on the alignment of the time series, using an Event Matching module,

e. registering face images from a series of checkout events to a series of PoS events using a Face-to-PoS Registration module, and

f. storing the time series and registration data in a database.

2. The method of claim 1, wherein the first event time series is comprised of a plurality of checkout events and associated timestamps, and wherein the second event time series is comprised of a plurality of PoS events.

3. The method of claim 2, wherein the first event time series is further comprised of a set of face images along with a timestamp for each image, and wherein the face images are of persons identified as participating in a checkout event.

4. The method of claim 3, further comprising:

determining a transaction owner for a checkout event, determining a checkout time, and generating a face group from the face images for each checkout event.

5. The method of claim 4, wherein the checkout time is determined by calculating a maximum likelihood for the checkout event time, wherein the maximum likelihood is calculated by finding the highest probability across one of a Gaussian distribution, asymmetric Gaussian distribution, or logisitic distribution.

6. The method of claim 4, wherein the face group is comprised of face images that are identified as moving through a checkout area at the same time.

7. The method of claim 4, further comprising the generation of a tree illustrating the relationship of the face images in the face group with the transaction owner.

8. The method of claim 1, wherein the Event Matching module further comprises finding the PoS event that minimizes the absolute time difference to each checkout event to form a checkout event association pair.

9. A system for associating at least an image of a face of at least a person with Point-of-Sale (PoS) data by aligning a first event time series with a second event time series based on a dynamic time disparity, and utilizing at least a camera and at least a processor for executing the steps comprising:

a. generating a first event time series using an Event Time Series Generation module,

b. generating a second event time series using the Event Time Series Generation module,

c. aligning the first event time series with the second event time series based on the dynamic time disparity between the time series, using an Event Time Series Clock Alignment module,

d. comprising the steps of;

i. performing a dynamic time warping to find a warping path between corresponding events between the first event time series and the second event time series, wherein the dynamic time warping can be one of a discrete dynamic time warping or a continuous dynamic time warping,

ii. checking for convergence by determining whether the standard deviation of a time difference between corresponding events in the first event time series and the second event time series is less than a predetermined threshold,

iii. adjusting the time index of the second event time series based on the time disparity between the event time series,

iv. testing the alignment of the time series to see if the best alignment has been obtained,

v. sorting the corresponding events based on the time disparity between events, and removing outliers that have a time disparity greater than a threshold, and

vi. repeating the Event Time Series Clock Alignment module steps until the best alignment is obtained,

e. matching first time series events with second time series events based on the alignment of the time series, using an Event Matching module,

f. registering face images from a series of checkout events to a series of PoS events using a Face-to-PoS Registration module, and

g. storing the time series and registration data in a database.

10. The system of claim 9, wherein the first event time series is comprised of a plurality of checkout events and associated timestamps, and wherein the second event time series is comprised of a plurality of PoS events.

11. The system of claim 10, wherein the first event time series is further comprised of a set of face images along with a timestamp for each image, and wherein the face images are of persons identified as participating in a checkout event.

12. The system of claim 11, further comprising:

determining a transaction owner for a checkout event, determining a checkout time, and generating a face group from the face images for each checkout event.

13. The system of claim 12, wherein the checkout time is determined by calculating a maximum likelihood for the checkout event time, wherein the maximum likelihood is calculated by finding the highest probability across one of a Gaussian distribution, asymmetric Gaussian distribution, or logisitic distribution.

14. The system of claim 12, wherein the face group is comprised of face images that are identified as moving through a checkout area at the same time.

15. The system of claim 12, further comprising the generation of a tree illustrating the relationship of the face images in the face group with the transaction owner.

16. The system of claim 9, wherein the Event Matching module further comprises finding the PoS event that minimizes the absolute time difference to each checkout event to form a checkout event association pair.