SYSTEM AND PROCESS FOR DETECTING, TRACKING AND COUNTING HUMAN OBJECTS OF INTEREST

• US 20090010490A1
• Filed: 07/03/2007
• Published: 01/08/2009
• Est. Priority Date: 07/03/2007
• Status: Active Grant

• Current Assignee: Shoppertrak RCT LLC (Johnson Controls International plc)
• Original Assignee: Shoppertrak RCT Corporation (Johnson Controls International plc)
• Inventors: Kastilahn, William C., Wang, Zhiqian
• Granted Patent: US 7,965,866 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method of identifying, tracking, and counting human objects of interest based upon at least one pair of stereo image frames taken by at least one image capturing device, comprising the steps of: obtaining said stereo image frames and converting each said stereo image frame to a rectified image frame using calibration data obtained for said at least one image capturing device; generating a disparity map based upon a pair of said rectified image frames; generating a depth map based upon said disparity map and said calibration data; identifying the presence or absence of said objects of interest from said depth map and comparing each of said objects of interest to existing tracks comprising previously identified objects of interest; for each said presence of an object of interest, adding said object of interest to one of said existing tracks if said object of interest matches said one existing track, or creating a new track comprising said object of interest if said object of interest does not match any of said existing tracks; updating each said existing track; and maintaining a count of said objects of interest in a given time period based upon said existing tracks created or modified during said given time period.

Claims

1. A method of identifying, tracking, and counting objects of interest based upon at least one pair of stereo image frames taken by at least one image capturing device, comprising the steps of:

(a) obtaining said at least one pair of stereo image frames and converting each said stereo image frame to a rectified image frame using calibration data obtained from said at least one image capturing device;

(b) generating a disparity map based upon said rectified image frame;

(c) generating a depth map based upon said disparity map and said calibration data;

(d) identifying the presence or absence of said objects of interest from said depth map and comparing each of said objects of interest to existing tracks comprising previously identified objects of interest;

(e) for each said presence of an object of interest, adding said object of interest to one of said existing tracks if said object of interest matches said one existing track, or creating a new track comprising said object of interest if said object of interest does not match any of said existing tracks;

(f) updating each said existing track; and

(g) maintaining a count of said objects of interest in a given time period based upon said existing tracks created or modified during said given time period.

2. The method of claim 1 wherein said calibration data comprises parameters of said image capturing device, such parameters including at least one of optical center, radio distortion, lens distortion, focal lengths, and relative sensor position.

3. The method of claim 1 wherein said pair of stereo image frames comprises a first image taken by a first sensor at said image capturing device, and a second image taken by a second sensor at said image capturing device at the same time said first image is taken by said first sensor.

4. The method of claim 3 wherein said rectified image frame comprises a first rectified image converted from said first image and a second rectified image converted from said second image.

5. The method of claim 1 wherein the step of identifying human object of interest comprises the additional steps of:

(a) a first component detection step wherein connected areas in said depth image satisfying a predefined connectivity property are labeled as components;

(b) a second object detection step wherein local maxima detections are computed for each said component;

(c) merging said local maxima detections closer than a specified merging distance to each other; and

(d) labeling all remaining local maxima detections as candidate detects.

6. The method of claim 5 wherein said first component detection step includes calculating properties of said components and applying predicate operators for said components.

7. The method of claim 6 wherein said predicate operators include a noise operator to check whether size of said first component is beneath a predetermined threshold, and a boundary operator to check whether said component is at boundaries of said height map.

8. The method of claim 7 wherein applying predicate operators for said components includes eliminating those first components whose noise operators indicate a noisy state.

9. The method of claim 5 wherein merging said local maxima detections includes eliminating those local maxima detections in background.

10. The method of claim 5 wherein merging said local maxima detections includes eliminating those local maxima detections whose boundary operators indicate a state of being at the boundary or in a dead zone.

11. The method of claim 5 wherein merging said local maxima detections includes eliminating the local maxima detection with less volume if the height difference in two local maxima detections is less than a predefined length threshold.

12. The method of claim 11 wherein merging said local maxima detections includes eliminating the local maxima detection with less height if the height difference in two local maxima detections is larger than or equal to said predefined length threshold and the height of the local maxima detection with the higher height is less than a predetermined maximum height threshold.

13. The method of claim 12 wherein merging said local maxima detections includes eliminating the local maxima detection with greater height if said height difference of two local maxima detections is larger than or equal to said predefined length threshold and said height of the local maxima detection with the higher height is greater than said predetermined maximum height threshold.

14. The method of claim 5 wherein merging said local maxima detections includes eliminating the local maxima detection whose area is less than a predetermined portion of the other local maxima detection.

15. The method of claim 1 wherein each said track comprises attributes corresponding to the position and height of the object of interest comprised in said track, a timestamp, a track identifier, and a set of additional track attributes.

16. The method of claim 15 wherein said set of additional track attributes comprises a status attribute for each said track taking one value from three possible values:

active, inactive, or deleted, wherein said status attribute is set to be active if said track comprises the object of interest in the most recent height map, inactive if said track does not comprise any object of interest in the most recent height map, and deleted if said track is marked for deletion.

17. The method of claim 15 wherein said set of additional track attributes comprises a start point, an end point, a height, and a position for said track.

18. The method of claim 15 wherein said set of additional track attributes comprises a noise operator to determine whether said track bounces back and forth to a certain extent in a predetermined minimum noisy track duration.

19. The method of claim 18 wherein updating said existing track includes the additional steps of:

(a) set the status attribute of said existing track;

(b) update the height and the position attributes of said existing track;

(c) check the noise operator for said existing track and update related attributes of said existing track;

(d) update a span attribute of said existing track;

(e) collect a count carried by said existing track if said existing track becomes noisy or has been in background; and

(f) update zone information.

20. The method of claim 1 wherein the matching between said existing tracks and said objects of interest include the additional steps of:

(a) analyzing said existing tracks;

(b) first pass matching between said existing tracks and said object of interest;

(c) searching for suboptimal locations for unpaired tracks;

(d) second pass matching between said existing tracks and said object of interest; and

(e) updating said existing tracks and creating new tracks.

21. At least one computer readable medium encoded with computer-executable instructions that, when executed on a computer system, perform a method for identifying, tracking, and counting objects of interest based upon at least one pair of stereo image frames taken by at least one image capturing device, the method comprising the steps of:

(a) generating depth maps based upon said at least one pair of stereo image frames and calibration data obtained from said at least one image capturing device;

(b) identifying the presence or absence of said objects of interest from said depth maps and comparing each of said objects of interest to existing tracks comprising previously identified objects of interest;

(c) for each said presence of an object of interest, adding said object of interest to one of said existing tracks if said object of interest matches said one existing track, or creating a new track comprising said object of interest if said object of interest does not match any of said existing tracks;

(d) updating each said existing track; and

(e) maintaining a count of said objects of interest in a given time period based upon said existing tracks created or modified during said given time period.

22. The at least one computer-readable medium of claim 21 wherein said depth map comprises a time attribute and pixels, wherein each of said pixels includes a depth attribute.

23. The at least one computer-readable medium of claim 21 wherein said calibration data comprises parameters of said image capturing device, such parameters including at least one of optical center, lens distortion, focal length, and relative sensor position.

24. The at least one computer-readable medium of claim 21 wherein identifying human object of interest comprises the additional steps of:

(a) a first component detection step wherein connected areas in said depth image satisfying a predefined connectivity property are labeled as components;

(b) a second object detection step wherein local maxima detections are computed for each said component;

(c) merging said local maxima detections closer than a specified merging distance to each other; and

(d) labeling all remaining local maxima detections as candidate detects.

25. The at least one computer-readable medium of claim 24 wherein said first component detection step includes calculating properties of said components and applying predicate operators for said components.

26. The at least one computer-readable medium of claim 25 wherein said predicate operators include a noise operator to check whether size of said first component is beneath a predetermined threshold, and a boundary operator to check whether said component is at boundaries of said height map.

27. The at least one computer-readable medium of claim 26 wherein said first component detection step further includes eliminating those first components whose noise operators indicate a noisy state.

28. The at least one computer-readable medium of claim 24 wherein merging said local maxima detections includes eliminating those local maxima detections in background.

29. The at least one computer-readable medium of claim 24 wherein merging said local maxima detections includes eliminating those local maxima detections whose boundary operators indicate a state of being at the boundary of said height map or in dead zones.

30. The at least one computer-readable medium of claim 24 wherein merging said local maxima detections includes eliminating the local maxima detection with less volume if the height difference in two local maxima detections is less than a predefined length threshold.

31. The at least one computer-readable medium of claim 24 wherein merging said local maxima detections includes eliminating the local maxima detection with less height if the height difference in two local maxima detections is larger than or equal to said predefined length threshold and the height of the local maxima detection with the higher height is less than a predetermined maximum height threshold.

32. The at least one computer-readable medium of claim 24 wherein merging said local maxima detections includes eliminating the local maxima detection with greater height if said height difference in two local maxima detections is larger than or equal to said predefined length threshold and said height of the local maxima detection with the higher height is greater than said predetermined maximum height threshold.

33. The at least one computer-readable medium of claim 24 wherein merging said local maxima detections includes eliminating the local maxima detection whose area is less than a predetermined portion of the other local maxima detection.

34. The at least one computer-readable medium of claim 21 wherein each said track comprises attributes corresponding to the position and height of the object of interest comprised in said track, a timestamp, a track identifier, and a set of additional track attributes.

35. The at least one computer-readable medium of claim 34 wherein said set of additional track attributes comprises a status attribute for each said track taking one value from three possible values:

active, inactive, or deleted, wherein said status attribute is set to be active if said track comprises the object of interest in the most recent height map, inactive if said track does not comprise any object of interest in the most recent height map, and deleted if said track is marked for deletion.

36. The at least one computer-readable medium of claim 35 wherein said set of additional track attributes comprises a start point, an end point, a height, and a position for said track.

37. The at least one computer-readable medium of claim 35 wherein said set of additional track attributes comprises a noise operator to determine whether said track bounces back and forth to a certain extent in a predetermined minimum noisy track duration.

38. The at least one computer-readable medium of claim 37 wherein updating said existing track includes the additional steps of:

(a) set the status attribute of said existing track;

(b) update the height and the position attributes of said existing track;

(c) check the noise operator for said existing track and update related attributes of said existing track;

(d) update a span attribute of said existing track;

(e) collect a count carried by said existing track if said existing track becomes noisy or has been in background longer than a given period of time; and

(f) update zone information.

39. The at least one computer-readable medium of claim 21 wherein the matching between said existing tracks and said objects of interest include the additional steps of:

(a) analyzing said existing tracks;

(b) first pass matching between said existing tracks and said object of interest;

(c) searching for suboptimal locations for unpaired tracks;

(d) second pass matching between said existing tracks and said object of interest; and

(e) updating said existing tracks and creating new tracks.

40. An object identifying, tracking, and counting system based upon image frames taken at a facility, the system comprising:

at least one image capturing device comprising video sensors adapted to obtain said image frames from said facility;

a boot loader module adapted to load programs into memory of said system and execute said programs in said system;

a counting module adapted to compute height maps, object detections, object tracks, and object counts based upon said image frames; and

a system management and communication module adapted to provide communication functions, such as storing data to a database at a remote site, making video recordings, and sending emails.

41. The object identifying, tracking, and counting system of claim 40, wherein the counting module is adapted to:

obtain at least one pair of said image frames from said at least one image capturing device and convert each said image frame to a rectified image frame using calibration data of said at least one image capturing device;

generate a depth map based upon said rectified image frames and said calibration data;

identify the presence or absence of objects of interest from said depth map;

compare each of said objects of interest to existing tracks comprising previously identified objects of interest;

for each said presence of an object of interest, add said object of interest to one of said existing tracks if said object of interest matches said one existing track, or create a new track comprising said object of interest if said object of interest does not match any of said existing tracks;

update each said existing track; and

maintain a count of said objects of interest in a given time period based upon said existing tracks created or modified during said given time period.

42. The object identifying, tracking, and counting system of claim 41, wherein said calibration data comprises parameters of said image capturing device, such parameters including at least one of optical center, radio distortion, lens distortion, focal lengths, and relative sensor position.

43. The object identifying, tracking, and counting system of claim 40, wherein said pair of image frames comprises a first image taken by a first sensor of said video sensors at said image capturing device, and a second image taken by a second sensor of said video sensors at said image capturing device at the same time said first image is taken by said first sensor.

44. The object identifying, tracking, and counting system of claim 43, wherein said rectified image frame comprises a first rectified image converted from said first image and a second rectified image converted from said second image.

45. The object identifying, tracking, and counting system of claim 41, wherein the counting module is further adapted to:

label connected areas in said depth image satisfying a predefined connectivity property as components;

compute local maxima detections for each said component;

merge said local maxima detections closer than a specified merging distance to each other; and

label all remaining local maxima detections as candidate detects.

46. The object identifying, tracking, and counting system of claim 41 wherein the counting module is further adapted to analyze said existing tracks;

match between said existing tracks and said object of interest;

search for suboptimal locations for unpaired tracks;

update said existing tracks;

create new tracks; and

collect a count carried by said existing track.