Method of detecting and tracking groups of people

• US 20030107649A1
• Filed: 12/07/2001
• Published: 06/12/2003
• Est. Priority Date: 12/07/2001
• Status: Active Grant

• Current Assignee: X Corp. (f/k/a Twitter, Inc.) (X Holdings Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Flickner, Myron D., Haritaoglu, R. Ismail
• Granted Patent: US 7,688,349 B2
• Time in Patent Office: Days
• US Class Current: 348/150
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A computer-interfaced camera system identifies and tracks groups of socially interrelated people. The system can be used, for example, to track people as they wait in a checkout line or at a service counter. In a preferred implementation, each recorded camera frame is segmented into foreground regions containing several people. The foreground regions are further segmented into individuals using temporal segmentation analysis. Once an individual person is detected, an appearance model based on color and edge density in conjunction with a mean-shift tracker is used to recover the person'"'"'s trajectory. Groups of people are determined by analyzing inter-person distances over time.

Claims

1. A method of detecting groups of socially-interrelated individuals within a number of people, comprising:

visually recording movement of the people over time by generating a series of images;

detecting the presence of individuals in the images;

tracking movement of the individuals to determine their respective trajectories; and

analyzing the trajectories to determine the existence of the groups of socially-interrelated individuals.

2. The method of claim 1, said analyzing the trajectories including determining relative distances between individuals.

3. The method of claim 1, said detecting the presence of individuals including:

for at least a first image, constructing a modified image of pixels in which pixels corresponding to objects whose positions remain fixed are excluded from the modified image;

temporally segmenting the modified image; and

assigning respective portions of the modified image to said individuals.

4. The method of claim 3, comprising using color to distinguish the individuals from the objects.

5. The method of claim 3, wherein an individual'"'"'s trajectory is determined by analyzing the color of pixels on sequential images in conjunction with a tracking algorithm.

6. The method of claim 5, wherein the tracking algorithm is a mean-shifting tracking algorithm.

7. The method of claim 5, wherein an individual'"'"'s trajectory is determined with the aid of color density analysis of edges of the portions.

8. The method of claim 1, wherein at least some of the groups are waiting in a line.

9. The method of claim 8, further comprising counting the number of groups in the line.

10. The method of claim 9, wherein the groups are shopping groups in a given checkout lane, the method further comprising opening at least one additional checkout lane when the number of groups in said given checkout lane exceeds a predetermined number.

11. The method of claim 1, said recording movement comprising using a plurality of cameras, wherein the cameras are directed at different spatial regions.

12. The method of claim 1, wherein said series of images includes a series of stereo images.

16. A computer program product comprising a machine-readable medium having computer-executable program instructions thereon for carrying out the method of claim 1.

13. A method of identifying shopping groups in a checkout lane, comprising:

camera recording people in the checkout lane, thereby generating a series of images;

resolving individual persons within the images; and

determining relative distances between the persons over time to identify shopping groups in the checkout lane.

14. The method of claim 13, comprising electronically counting the number of shopping groups in the checkout lane.

15. The method of claim 14, further comprising opening at least one additional checkout lane in response to said counting.

17. A computer program product comprising a machine-readable medium having computer-executable program instructions thereon for carrying out the method of claim 13.