Efficient multi-hypothesis multi-human 3D tracking in crowded scenes

• US 8,098,891 B2
• Filed: 11/24/2008
• Issued: 01/17/2012
• Est. Priority Date: 11/29/2007
• Status: Active Grant

• Current Assignee: Cloud BYTE LLC
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Lv, Fengjun, Xu, Wei, Gong, Yihong
• Primary Examiner(s): Azarian, Seyed
• Application Number:
  US12/277,278
• Publication Number:
  US 20090296985A1
• Time in Patent Office: 1,149 Days
• Field of Search: 382/100, 382/103, 382/106, 382/107, 382/154, 382/155, 382/156, 382/162, 382/168, 382/173, 382/181, 382/190, 382/203, 382/209, 382/214, 382/219, 382/220, 382/232, 382/243, 382/254, 382/274, 382/276, 382285-298, 382/305, 382/312, 348/94, 348/143, 340/10.1, 706/14, 703/6, 455/456.1
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking

Abstract

System and methods are disclosed to perform multi-human 3D tracking with a plurality of cameras. At each view, a module receives each camera output and provides 2D human detection candidates. A plurality of 2D tracking modules are connected to the CNNs, each 2D tracking module managing 2D tracking independently. A 3D tracking module is connected to the 2D tracking modules to receive promising 2D tracking hypotheses. The 3D tracking module selects trajectories from the 2D tracking modules to generate 3D tracking hypotheses.

Claims

1. A method to perform multi-human three dimensional (3D) tracking, comprising:

for each single view, providing two dimensional (2D) human detection candidates from a camera to a 2D tracking module wherein a Convolutional Neural Network (CNN) generates the 2D human detection candidates;

a. independently performing 2D tracking in each 2D tracking module and reporting promising 2D tracking hypotheses to a 3D tracking module;

b. selecting trajectories from the 2D tracking modules to generate 3D tracking hypotheses; and

c. determining a difference score between the detection and the trajectory as a weighted sum of appearance, location, blob size, and orientation.

2. The method of claim 1, comprising providing feedback to the 2D tracking modules to update a 2D tracking module status.

3. The method of claim 1, wherein the 2D tracking comprises matching between new detections and existing tracking hypotheses.

4. The method of claim 1, wherein each hypothesis comprises a tracking trajectory.

5. The method of claim 1, comprising determining a difference between the detection and the trajectory for each pair of detection and hypothesis.

6. The method of claim 1, comprising matching between new detections and existing tracking hypotheses based on one or more of:

appearance, location, blob size, and orientation.

7. The method of claim 1, comprising determining a difference score for each pair of detection and hypothesis.

8. The method of claim 1, wherein each 3D tracking hypothesis comprises a correspondence between a pair of 2D hypotheses from each of the two views and each correspondence results in a 3D trajectory.

9. The method of claim 1, comprising generating a pair-wise correspondence matrix among 2D hypotheses from each of the two views.

10. The method of claim 1, comprising sorting 3D hypotheses according to a 3D intersection error.

11. The method of claim 1, comprising pruning a 3D hypothesis based on a hypothesis conflict or expiration.

12. The method of claim 1, wherein the 2D trajectories are synchronously updated.

13. The method of claim 1, comprising updating a 2D trajectory if a corresponding 3D trajectory is added, pruned, or combined with another 3D trajectory.

14. An apparatus to perform multi-human 3D tracking with a plurality of cameras, comprising:

a. at each view, a module coupled to each camera to provide 2D human detection candidates;

b. a plurality of 2D tracking modules each coupled to the CNN and each 2D tracking module managing 2D tracking independently;

3D tracking module coupled to the 2D tracking modules to receive promising 2D tracking hypotheses, the 3D tracking module selecting trajectories from the 2D tracking modules to generate 3D tracking hypotheses wherein a Convolutional Neural Network (CNN) generates the 2D human detection candidates and the module determining a difference score between the detection and the trajectory as a weighted sum of appearance, location, blob size, and orientation.

15. The apparatus of claim 14, wherein the 3D tracking module provides feedback to the 2D tracking modules to update a current status of each 2D tracking module.

16. The apparatus of claim 14, wherein each 3D tracking hypothesis comprises a correspondence between a pair of 2D hypotheses from each of the two views and each correspondence results in a 3D trajectory.

17. The apparatus of claim 14, wherein the 2D trajectories are synchronously updated.