Systems and methods for video content analysis

• US 9,609,348 B2
• Filed: 08/28/2014
• Issued: 03/28/2017
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: Intersil Americas LLC (Renesas Electronics Corporation)
• Original Assignee: Intersil Americas LLC (Renesas Electronics Corporation)
• Inventors: Shi, Fang, Ming, Jin, Wu, Qi, You, Fan, Bao, Kai
• Primary Examiner(s): ELAHI, SHAN E
• Application Number:
  US14/472,313
• Publication Number:
  US 20140369417A1
• Time in Patent Office: 943 Days
• Field of Search: 375/240.16
• US Class Current: 1/1
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Video analytics systems and methods are described that typically comprise a video encoder operable to generate macroblock video analytics metadata (VAMD) from a video frame. Functional modules receive the VAMD and an encoded version of the video frame is configured to generate video analytics information related to the frame using the VAMD and the encoded video frame. The downstream decoder can use the VAMD to obtain a global motion vector related to the frame, detect and track motion of an object within the frame and monitor a line provided or found within the frame. Traversals of the line by a moving object can be detected and counted using information in the VAMD and the line may be part of a polygon that delineates an area to be monitored within the encoded frame. The VAMD can comprise macroblock level and video frame level information.

Claims

1. A method comprising a processor, a memory, a video sensor, a video encoder and a transceiver, the memory including instructions stored thereon which, when executed by the processor, perform a method for generating video analytics, the method comprising:

providing, by the processor, information representative of a sequence of images captured by the video sensor to the video encoder that is adapted to encode the information using macroblock-based video encoding to obtain a plurality of video frames, wherein the video sensor and video encoder are co-located in a first apparatus;

generating, by the processor, pixel domain video analytics metadata (VAMD) that includes video content analysis information for each of a plurality of macroblocks while encoding the information in the video encoder;

generating, by the processor, a global video analytics message applicable to a plurality of images in the sequence of images using the VAMD;

generating, by the processor, a local video analytics message applicable to a first video frame in the plurality of video frames using the VAMD; and

transmitting, by the transceiver, the plurality of video frames through a network to a second apparatus with a package comprising the VAMD and the local video analytics message or the global video analytics message,

wherein the second apparatus includes a video analytics processor configured to process the package transmitted by the first apparatus.

2. The method of claim 1, wherein:

the video analytics processor in the second apparatus is configured to generate video analytics information related to the plurality of video frames based on the local video analytics message or the global video analytics message.

3. The method of claim 1, wherein the plurality of video frames is obtained by:

compressing the sequence of images.

4. The method of claim 1, wherein generating the VAMD comprises:

generating motion vectors for the plurality of macroblocks.

5. The method of claim 4, wherein the motion vectors have sub-pixel granularity.

6. The method of claim 4, wherein generating the VAMD comprises:

filtering the motion vectors to obtain one filtered motion vector for each of the plurality of macroblocks; and

providing filtered motion vectors in the VAMD.

7. The method of claim 1, wherein the video encoder is embedded in a communications device.

8. The method of claim 1, wherein the video encoder is provided in a device that functions as a camera.

9. The method of claim 1, wherein:

the global video analytics message includes information related to a background frame, a foreground object segmentation descriptor, a camera parameter, predefined motion alarm regions coordination and index, or a virtual line; and

the local video analytics message includes information related to global motion vectors, motion alarm region alarm status, virtual line counting results, object tracking parameters, or camera moving parameters.

10. A device comprising:

a camera;

a video encoder;

a video analytics engine;

a communication interface; and

a video sensor in the camera configured to capture a sequence of images;

the video encoder configured to;

encode the sequence of images in video frames using macroblock-based video encoding to provide encoded video frames, and

generate video analytics metadata (VAMD) that includes video content analysis information for each of a plurality of macroblocks processed the sequence of images in the video frames;

the video analytics engine configured to process the VAMD, and to generate one or more video analytics messages from results obtained by processing the VAMD; and

the communication interface configured to transmit the encoded video frames to a video decoder of a client device, and to transmit the VAMD and the one or more video analytics messages in a layered package to a video analytics processor in the client device that is configured to generate video analytics information related to the sequence of images based on the VAMD, the one or more video analytics messages, and the encoded video frames.

11. The device of claim 10, wherein the encoded video frames include compressed video frames.

12. The device of claim 10, wherein the VAMD comprises motion vectors generated for the plurality of macroblocks.

13. The device of claim 10,

wherein the one or more video analytics messages comprises at least one global video analytics message applicable to a plurality of images in the sequence of images or at least one local video analytics message applicable to a first video frame in the encoded video frames.

14. The device of claim 10, wherein the video sensor is provided in a camera.

15. An apparatus comprising:

a camera;

a video encoder configured to provide encoded video frames representative of images received from the camera using macroblock-based video encoding, wherein the video encoder is further configured to generate video analytics metadata (VAMD) that includes video content analysis information for each of a plurality of macroblocks processed while encoding the images;

a video analytics engine configured to process the VAMD, and to generate a global video analytics message applicable to a plurality of images in the images received from the camera or a local video analytics message applicable to one of the encoded video frames; and

a communication interface adapted to transmit the encoded video frames to a video decoder of a client device, and to transmit the VAMD, global video analytics message or the local video analytics message in a layered package to a video analytics processor in the client device that is configured to generate video analytics information related to the images received from the camera based on the VAMD, the global video analytics message or the local video analytics message, and the encoded video frames.

16. The apparatus of claim 15, wherein the encoded video frames include compressed video frames.

17. The apparatus of claim 15, wherein the VAMD comprises motion vectors generated for the plurality of macroblocks.

18. The apparatus of claim 15, wherein the global video analytics message includes information related to a background frame, a foreground object segmentation descriptor, a camera parameter, predefined motion alarm regions coordination and index, or a virtual line.

19. The apparatus of claim 15, wherein the local video analytics message includes information related to global motion vectors, motion alarm region alarm status, virtual line counting results, object tracking parameters, or camera moving parameters.

20. The apparatus of claim 15, wherein results obtained by processing the VAMD include information related to motion indexing, background extraction, object segmentation, motion detection, virtual line detection, object counting, motion tracking, speed estimation, a background model, a motion alarm, virtual line detections, or electronic image stabilization parameters.