Surveillance system

• US 9,082,278 B2
• Filed: 03/19/2010
• Issued: 07/14/2015
• Est. Priority Date: 03/19/2010
• Status: Active Grant

• Current Assignee: University-Industry Cooperation Group of KyungHee University
• Original Assignee: University-Industry Cooperation Group of KyungHee University
• Inventors: Park, Gwang Hoon
• Primary Examiner(s): Woolcock, Madhu
• Application Number:
  US12/727,961
• Publication Number:
  US 20110228092A1
• Time in Patent Office: 1,943 Days
• Field of Search: 348/77, 348/135, 348/137, 348/139, 348/143, 348152-155, 348/159, 37524016-24017
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/11 among a plurality of spatial predictive coding modes
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/157 Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/192 the adaptation method, adaptation tool or adaptation type being iterative or recursive
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A surveillance system includes an input unit to receive a plurality of video frames that are acquired through the use of a first camera, a video processing unit to perform prediction and encoding processes on the plurality of video frames, a moving-object tracking unit to track a moving object between the video frames based on at least part of the results from the prediction and encoding processes to generate location information of the moving object being tracked, and a control unit to generate a control signal based on the location information and to transmit the control signal to a second camera to control the movement of the second camera.

Claims

1. A surveillance system comprising:

an input unit configured to receive a plurality of video frames that are acquired through a first camera;

a video processor configured to generate prediction information and encoding information on the plurality of video frames that have been encoded;

a moving-object tracking processor configured to;

track a moving object in the plurality of video frames based on at least part of the prediction information and the encoding information;

perform a first motion classification to identify a list of candidate image areas in the plurality of video frames for the moving object based on the prediction information and the encoding information;

weight each candidate image area identified in the list of candidate image areas, wherein a greater weight is given to candidate image areas determined from the prediction information and a lesser weight is given to candidate image areas determined from the encoding information;

perform a second motion classification on each of the candidate image areas in the list of candidate image areas based on the weights of the candidate image areas and on additional prediction information or additional encoding information to eliminate some of the candidate image areas in order to identify final image areas from the list of candidate image areas; and

generate location information of the moving object based on the final image areas; and

a camera control processor configured to generate a control signal based on the location information and to transmit the control signal to a second camera to control movement of the second camera effective to capture a feature part of the moving object.

2. The system of claim 1, wherein the prediction information includes at least one of motion vectors, residual data and macroblock type information on the plurality of video frames;

and

wherein the video processor executes a transform/encoding instruction configured to generate the encoding information which indicates a number of bits allocated to encode a macroblock in the plurality of video frames.

3. The system of claim 2, wherein the moving-object tracking processor performs the first motion classification to determine a list of candidate macroblocks including moving objects based on at least part of the prediction information and the encoding information, wherein the list of candidate image areas includes the list of candidate macroblocks.

4. The system of claim 3, wherein the moving-object tracking processor performs the first motion classification based on a motion vector of a macroblock, wherein if a magnitude of the motion vector is greater than a particular value, the macroblock is determined to be included in the list of candidate macroblocks.

5. The system of claim 3, wherein the moving-object tracking processor performs the first motion classification based on residual data of a macroblock, wherein if a magnitude of the residual data is greater than a particular value, the macroblock is determined to be included in the list of candidate macroblocks.

6. The system of claim 1, wherein the moving-object tracking processor performs the first motion classification to determine a list of candidate macroblocks based on macroblock type information, wherein if the macroblock type information indicates that a macroblock is to be processed in an intra-coding mode, the macroblock is determined to be included in the list of candidate macroblocks, wherein the macroblock type information is included in the prediction information and wherein the list of candidate image areas includes the list of candidate macroblocks.

7. The system of claim 1, wherein the moving-object tracking processor performs the first motion classification to determine a list of candidate macroblocks based on macroblock type information, wherein if the macroblock type information indicates that a macroblock includes M×

M image pixels, M being no greater than a particular number, the macroblock is determined to be included in the list of candidate macroblocks, wherein the macroblock type information is included in the prediction information and wherein the list of candidate image areas includes the list of candidate macroblocks.

8. The system of claim 3, wherein the moving-object tracking processor performs the first motion classification based on the encoding information, wherein if the encoding information indicates that a number of coded bits for a macroblock is greater than a particular number, the macroblock is determined to be included in the list of candidate macroblocks.

9. The system of claim 3, wherein the moving-object tracking processor performs the second motion classification to eliminate a macroblock out of the list of candidate macroblocks, wherein if the additional prediction information indicates that the macroblock includes M×

M image pixels, M being no greater than a particular number, the macroblock is determined to be eliminated out of the list of candidate macroblocks.

10. The system of claim 3, wherein the moving-object tracking processor performs the second motion classification to eliminate a macroblock out of the list of candidate macroblocks based on at least part of the additional prediction information, wherein if the prediction information indicates that the macroblock is surrounded by skipped macroblocks, the macroblock is determined to be eliminated out of the list of candidate macroblocks.

11. The system of claim 1, wherein the second camera includes a pan-tilt-zoom (PTZ) camera, wherein the PTZ camera is panned or tilted to take a video based on the control signal.

12. A surveillance method comprising:

generating prediction information and encoding information on a plurality of video frames that are acquired from a first camera;

tracking a moving object in the plurality of video frames based on at least part of the prediction information and the encoding information, the prediction information including motion vectors, residual data and macroblock type information on the plurality of video frames;

determining candidate image areas in the plurality of video frames for the moving object using the prediction information and the encoding information,

weighting each of the candidate image areas, wherein a greater weight is given to candidate image areas determined from the prediction information and a lesser weight is given to candidate image areas determined from the encoding information;

selecting final image areas based on the weights of the candidate image areas and by applying classification criteria that is based at least on the prediction information to all of the candidate image areas in order to eliminate some of the candidate image areas;

generating location information for the moving object based on the final image areas;

generating a control signal based on the location information of the moving object; and

transmitting the control signal to a second camera to control movement of the second camera effective to capture a feature part of the moving object.

13. The method of claim 12, wherein tracking the moving object includes:

performing a first motion classification to determine a list of candidate macroblocks containing moving objects; and

performing a second motion classification to determine a list of final macroblocks containing moving objects, wherein the candidate image areas includes the list of candidate macroblocks.

14. The method of claim 13, wherein in the first motion classification, if a magnitude of a motion vector of a macroblock is greater than a particular value, the macroblock is determined to be included in the list of candidate macroblocks.

15. The method of claim 13, wherein in the first motion classification, if a magnitude of residual data of a macroblock is greater than a particular value, the macroblock is determined to be included in the list of candidate macroblocks.

16. The method of claim 13, wherein in the first motion classification, if a macroblock type information indicates that a macroblock is to be processed in an intra-coding mode, the macroblock is determined to be included in the list of candidate macroblocks.

17. The method of claim 13, wherein in the first motion classification, if a macroblock type information indicates that a macroblock includes M×

M image pixels, M being no greater than a particular number, the macroblock is determined to be included in the list of candidate macroblocks.

18. The method of claim 13, wherein in the first motion classification, if the encoding information indicates that a number of coded bits for a macroblock is greater than a particular number, the macroblock is determined to be included in the list of candidate macroblocks.

19. The method of claim 13, wherein in the second motion classification, if the prediction information indicates that a macroblock includes M×

M image pixels, M being no greater than a particular number, the macroblock is determined to be eliminated out of the list of candidate macroblocks.

20. The method of claim 13, wherein in the second motion classification, if the prediction information indicates that a macroblock is surrounded by skipped macroblocks, the macroblock is determined to be eliminated out of the list of candidate macroblocks.

21. A surveillance system comprising:

an input unit configured to receive a plurality of video frames that are acquired through a first camera;

a video processor configured to generate prediction information and encoding information on the plurality of video frames;

a moving-object tracking processor configured to;

track a moving object in the plurality of video frames based on at least part of the prediction information and the encoding information to generate location information of the moving object,

perform a first motion classification to determine a list of candidate macroblocks in the plurality of video frames that include moving objects based on the at least part of the prediction information and the encoding information,

weight each candidate macroblock in the list of candidate macroblocks, wherein a greater weight is given to candidate macroblocks determined from the prediction information and a lesser weight is given to candidate macroblocks determined from the encoding information,

perform a second motion classification based on the weights of the candidate macroblocks and the prediction information on each of the candidate macroblocks to eliminate some of the candidate macroblocks from the list of candidate macroblocks and determine final macroblocks from the list of candidate macroblocks;

determine the location information based on the final macroblocks;

a camera control processor configured to generate a control signal based on the location information and to transmit the control signal to a second camera to control movement of the second camera.

22. The surveillance system of claim 21, wherein the moving-object tracking processor performs the first motion classification based on one or more of a motion vector of a macroblock, residual data of the macroblock, or type information of the macroblock, wherein the macroblock is included in the list of candidate macroblocks if at least one of a magnitude of the motion vector is greater than a particular value, a magnitude of the residual data is greater than a particular value, or the type information of the macroblock indicates that the macroblock is to be processed in an intra-coding mode.

23. The system of claim 22, wherein the moving-object tracking processor performs the first motion classification based on at least one of the encoding information or the type information of the macroblock, wherein the macroblock is included in the list of candidate macroblocks if the encoding information indicates that a number of coded bits for the macroblock is greater than a particular number or the type information of the macroblock indicates that a macroblock includes M×

M image pixels, M being no greater than a particular number.

24. The system of claim 22, wherein the moving-object tracking processor performs the second motion classification to eliminate a macroblock out of the list of candidate macroblocks based on at least part of the prediction information, wherein if the prediction information indicates that the macroblock includes M×

M image pixels, M being no greater than a particular number, or if the prediction information indicates that the macroblock is surrounded by skipped macroblocks, the macroblock is determined to be eliminated out of the list of candidate macroblocks.

25. The surveillance system of claim 21, wherein the control signal is effective to control the movement of the second camera to capture a feature part of the moving object.

26. A surveillance system comprising:

an input unit configured to receive a plurality of video frames that are acquired through a first camera;

a video processor configured to generate prediction information and encoding information associated with the plurality of video frames that have been encoded;

a moving-object tracking processor configured to;

perform a first motion classification to determine a list of candidate macroblocks in the plurality of video frames that include moving objects based on the prediction information and the encoding information associated with the encoded plurality of video streams,

wherein a macroblock is included in the list of candidate macroblocks in response to at least one of;

a number of bits used to encode the macroblock is greater than a particular number;

a magnitude of a motion vector for the macroblock is greater than a particular value;

residual information of the macroblock is greater than a specific value;

an intra-coding mode is assigned to the macroblock; and

/or

the macroblock is a sophisticated macroblock;

weight each candidate macroblock in the list of candidate macroblocks, wherein a greater weight is given to candidate macroblocks determined from the prediction information and a lesser weight is given to candidate macroblocks determined from the encoding information;

perform a second motion classification based on the weights of the candidate macroblocks and on the prediction information on each of the candidate macroblocks to eliminate some of the candidate macroblocks from the list of candidate macroblocks and to determine final macroblocks from the list of candidate macroblocks,

wherein at least;

a macroblock with a size no greater than a particular value, if a variable-size block motion estimate method is used, is eliminated from the list of candidate macroblocks; and

a sophisticated macroblock surrounded by skipped macroblocks is eliminated from the list of candidate macroblocks;

determine location information based on the final macroblocks; and

track an object based on the location information; and

a camera control processor configured to generate a control signal based on the location information and to transmit the control signal to a second camera to control movement of the second camera.