System and process for detecting, tracking and counting human objects of interest

• US 10,558,890 B2
• Filed: 07/01/2016
• Issued: 02/11/2020
• Est. Priority Date: 07/03/2007
• Status: Active Grant

• Current Assignee: Shoppertrak RCT LLC
• Original Assignee: Shoppertrak RCT Corporation (Johnson Controls International plc)
• Inventors: Wang, Zhiqian, Kastilahn, William C.
• Primary Examiner(s): Nakhjavan, Shervin K
• Application Number:
  US15/200,864
• Publication Number:
  US 20160314597A1
• Time in Patent Office: 1,320 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method of identifying, tracking, and counting human objects of interest based upon at least one pair of stereo image frames taken by at least one image capturing device, comprising the steps of: obtaining said stereo image frames and converting each said stereo image frame to a rectified image frame using calibration data obtained for said at least one image capturing device; generating a disparity map based upon a pair of said rectified image frames; generating a depth map based upon said disparity map and said calibration data; identifying the presence or absence of said objects of interest from said depth map and comparing each of said objects of interest to existing tracks comprising previously identified objects of interest; for each said presence of an object of interest, adding said object of interest to one of said existing tracks if said object of interest matches said one existing track, or creating a new track comprising said object of interest if said object of interest does not match any of said existing tracks; updating each said existing track; and maintaining a count of said objects of interest in a given time period based upon said existing tracks created or modified during said given time period.

Claims

1. A method of identifying and tracking objects of interest, comprising the steps of:

obtaining, by a processor, stereo image frames from at least two image capturing devices, where each image capturing device is located in a different area;

identifying with the processor a presence of one or more objects of interest within the stereo image frames, each stereo image frame including location information indicative of a zone correlating to the area where each of the objects of interest was detected, each of the one or more objects of interest having a plurality of properties;

applying, to the plurality of properties for each of the objects of interest, one or more predicate operators, the predicate operators used to obtain a current status of each object of interest in each stereo image frame based on a determination of whether each of the plurality of properties satisfies a specified condition defined by a corresponding predicate operator;

determining, as a function of the applied one or more predicate operators, the current status of each object of interest;

updating, as a function of the current status of each object of interest, the plurality of properties for each object of interest;

determining, for each of the objects of interest and based on the updated plurality of properties, a predicted position of the object of interest;

comparing, with the processor, each of the objects of interest to previously identified objects of interest;

if an object of interest matches one of the previously identified objects of interest based, at least in part, on the predicted position of the object of interest, using the location information associated with the object of interest to track movement by the object of interest.

2. The method of claim 1, wherein the at least one of the more than one image capturing device includes a stereo camera.

3. The method of claim 1, wherein a direction of the objects of interest can be determined.

4. The system described in claim 1, wherein a speed of the objects of interest can be determined.

5. The method of claim 1, wherein an amount of time spent by the objects of interest within one of the zones can be determined.

16. The method of claim 1, wherein the one or more predicate operators includes one of an operator to determine whether a point in one of the objects of interest intersects boundaries of a height map, an operator to determine whether the one of the objects of interest is bouncing back and forth, and an operator to determine whether the one of the objects of interest corresponds to another one of the objects of interest.

6. A method of detecting and tracking objects of interest, comprising the steps of:

obtaining stereo image frames from a plurality of image capturing device, where each image capturing device is located in a different zone;

identifying, by a processor, a presence of detected objects of interest within one of the stereo image frames, each of the one or more detected objects of interest having a plurality of properties;

applying, to the plurality of properties for each of the detected objects of interest, one or more predicate operators, the predicate operators used to obtain a current status of each detected object of interest in each stereo image based on a determination of whether each of the plurality of properties satisfies a specified condition defined by a corresponding predicate operator;

determining, based on an evaluation of whether one or more of the properties of each detected object of interest satisfies a specified condition, the current status of each detected object;

updating, as a function of the current status of each object of interest, the plurality of properties for each detected object of interest;

determining, for each of the detected objects of interest and based on the updated plurality of properties, a predicted position of the detected object of interest;

comparing with the processor each of the detected objects of interest to existing tracks comprising previously identified objects of interest;

if there is a match between one of the detected objects of interest and one of the existing tracks based, at least in part, on the predicted position of the object of interest, adding to the existing track the detected object of interest and location information indicative of the zone where the detected object of interest was detected;

if there is not a match between the detected object of interest and one of the existing tracks, creating a new track including the newly detected object of interest and location information indicative of the zone where the newly detected object of interest was detected; and

using the location information to track a movement of the objects of interest.

7. The method of claim 6, wherein at least one of the image capturing devices includes a stereo camera.

8. The method of claim 6, wherein a direction of the objects of interest can be determined.

9. The method of claim 6, wherein a speed of the objects of interest can be determined.

10. The method of claim 6, wherein an amount of time spent by the objects of interest within one of the zones can be determined.

11. An object tracking system based upon image frames taken at a facility, the system comprising:

a plurality of image capturing devices adapted to obtain image frames from the facility;

a boot loader module adapted to load programs into memory of the system and to execute the programs in the system;

a processor for detecting objects based upon the image frames,

wherein, the processor is adapted to;

obtain at least one pair of image frames from each image capturing device;

identify a presence of objects of interest, each of the one or more objects of interest having a plurality of properties;

apply, to the plurality of properties for each of the objects of interest, one or more predicate operators, the predicate operators used to obtain a current status of each object of interest in each image frame based on a determination of whether each of the plurality of properties satisfies a specified condition defined by a corresponding predicate operator;

determine, as a function of the applied one or more predicate operators, the current status of each object of interest;

updating, as a function of the current status of each object of interest, the plurality of properties for each object of interest;

determine, for each of the objects of interest and based on the updated plurality of properties, a predicted position of the object of interest;

compare each of the objects of interest to existing tracks comprising previously identified objects of interest;

if an object of interest is present and the object of interest matches a previously identified object of interest, add the object of interest to one of the existing tracks associated with the previously identified object of interest;

if an object of interest is present and the object of interest does not match any of the previously identified object of interest, create a new track including the newly detected object of interest, each existing track and new track including location information indicative of a zone correlating to an area where the object of interest was detected; and

use the location information to track movement of at least one of the objects of interest.

12. The system described in claim 11, wherein the at least one image capturing device includes a stereo camera.

13. The system described in claim 11, wherein a direction of the objects of interest can be determined.

14. The system described in claim 11, wherein a speed of the objects of interest can be determined.

15. The system described in claim 11, wherein the system can determine whether the objects of interest are moving into or out of the area.