Fast multi-object detection and tracking system

• US 10,475,204 B2
• Filed: 08/29/2017
• Issued: 11/12/2019
• Est. Priority Date: 08/30/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Wang, Shiting, Hu, Qi, Wen, Dongchao
• Primary Examiner(s): Bhatnagar, Anand P
• Application Number:
  US15/689,943
• Publication Number:
  US 20180061076A1
• Time in Patent Office: 805 Days
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

The present invention relates to the fast multi-object detection and tracking system. According to the system of the present invention, only a few frames are to be detected and the other frames are split into different size of steady motion segments in a binary way, and then the system of the present invention can predicted accurately. The system can help to achieve high tracking speed with multi-persons in high-definition videos and gain high accuracy as well.

Claims

1. A multi-object detection and tracking system, comprising:

a) preprocess unit which is configured to select a plurality of first sampled frames to divide a sequence of images into frame cells;

b) global object detector which is configured to perform object detection on the whole image of the first sampled frames, and give out global detection results;

c) frame sampling unit which is configured to select at least one second sampled frame in a frame cell;

d) local object detector which is configured to perform object detection on the region-of-interest of the second sampled frames, and output local detection results;

e) data association unit which is configured to align the global detection results and the local detection results with existed trajectories by object similarity.

2. The system of claim 1, wherein the first sampled frames are selected by uniformly sampling the sequence of images with a predetermined interval N, N≥

1.

3. The system of claim 1, wherein the frame cell is composed of two of the first sampled frames and all the frames between them.

4. The system of claim 1, wherein, the frame cells are sequentially processed for object tracking.

5. The system of claim 1, wherein the global object detector and the local object detector are the same in terms of classifiers.

6. The system of claim 1, wherein the second sampled frames splitting each frame cell into different sizes of steady motion segments are determined according to the motion attributes of the tracking object.

7. The system of claim 1, wherein the second sampled frames split each frame cell into different sizes of steady motion segments according to a binary search method, further comprising:

1) taking unverified-motion segment as input frames, when initialization, take the whole frame cell;

2) selecting the middle frame of the input frames as the second sampled frame, and deciding its region-of-interest;

3) using the local object detector to detect on region-of-interest of the selected middle frame, and outputting the local detection results;

4) changing the unverified-motion segment and repeating search until meeting any predefined stop conditions.

8. The system of claim 7, wherein the unverified-motion segment is updated by the left half of the input frames when the local detection on the second sampled frame fails;

otherwise, take all the right frames of the selected frame in the frame cell;

wherein the left half of input frames are the frames starts from the first frame to the current second sampled frame of the input frames, and the right half of frame cell are the frames starts from the current second sampled frame to the end of the frame cell.

9. The system of claim 7, wherein the predefined stop conditions including:

i) the total number of successfully detected times in the whole frame cell reach predetermined times;

or

ii) the number of selected frames equals a predetermined threshold.

10. The system of claim 7, wherein the middle frame of the input frames is the frame whose timestamp is ceiling or round to half of the sum of the start frame timestamp and the end frame timestamp.

11. The system of claim 7, wherein deciding the region-of-interest of the second sampled frame by expanding the motion-predicted region for existed tracking objects, according to historical tracking results on both former frames and latter frames.

12. The system of claim 1, further comprising a trajectory estimation unit which is configured to fill in trajectories for un-sampled frames and fail-associated frames in the frame cell, by interpolation with given tracking results of both the first and second sampled frames.

13. A multi-object detection and tracking method, comprising:

a) selecting a plurality of first sampled frames to divide a sequence of images into frame cells;

b) performing object detection on the whole image of the first sampled frames, and giving out global detection results;

c) selecting at least one second sampled frame in a frame cell;

d) performing object detection on the region-of-interest of the second sampled frames, and outputting local detection results;

e) aligning the global detection results and the local detection results with existed trajectories by object similarity.

14. The method of claim 13, wherein, the first sampled frames are selected by uniformly sampling the sequence of images with a predetermined interval N, N≥

1.

15. The method of claim 13, wherein the frame cell is composed of two of the first sampled frames and all the frames between them.

16. The method of claim 13, wherein the frame cells are sequentially processed for object tracking.

17. The method of claim 13, wherein the second sampled frames splitting each frame cell into different sizes of steady motion segments are determined according to the motion attributes of the tracking object.

18. The method of claim 13, wherein the second sampled frames split each frame cell into different sizes of steady motion segments according to a binary search method, further comprising:

1) taking unverified-motion segment as input frames, when initialization, take the whole frame cell;

2) selecting the middle frame of the input frames as the second sampled frame, and deciding its region-of-interest;

3) using a local object detector to detect on region-of-interest of the selected middle frame, and outputting the local detection results;

4) changing the unverified-motion segment and repeating search until meeting any predefined stop conditions.

19. The method of claim 18, wherein the unverified-motion segment is updated by the left half of the input frames when the local detection on the second sampled frame fails;

otherwise, take all the right frames of the selected frame in the frame cell;

wherein the left half of input frames are the frames starts from the very first frame to the current second sampled frame of the input frames, and the right half of frame cell are the frames starts from the current second sampled frame to the end of the frame cell.

20. The method of claim 18, wherein the predefined stop conditions including:

i) the total number of successfully detected times in the whole frame cell reach predetermined times;

or

ii) the number of selected frames equals a predetermined threshold.

21. The method of claim 18, wherein the middle frame of the input frames is the frame whose timestamp is ceiling or round to half of the sum of the start frame timestamp and the end frame timestamp.

22. The method of claim 18, wherein deciding the region-of-interest of the second sampled frame by expanding the motion-predicted region for existed tracking objects, according to historical tracking results on both former frames and latter frames.

23. The method of claim 13, further comprising filling in trajectories for un-sampled frames and fail-associated frames in the frame cell, by interpolation with given tracking results of both the first and second sampled frames.