Multi-resolution video analysis and key feature preserving video reduction strategy for (real-time) vehicle tracking and speed enforcement systems

• US 8,953,044 B2
• Filed: 10/05/2011
• Issued: 02/10/2015
• Est. Priority Date: 10/05/2011
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Wu, Wencheng, Bernal, Edgar A., Loce, Robert P., Hoover, Martin E.
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): ELAHI, SHAN E
• Application Number:
  US13/253,261
• Publication Number:
  US 20130088600A1
• Time in Patent Office: 1,224 Days
• Field of Search: 348/149
• US Class Current: 348/149
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08G 1/054 photographing overspeeding vehicles
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Systems and methods are disclosed that include a video-based analysis system that detects, tracks and archives vehicles in video stream data at multiple resolutions. The system includes an image capturing device that captures video stream data having video at a first high resolution. A vehicle detection module detects at least one vehicle within the video. A vehicle analysis module is configured to analyze the video and to extract one or more key vehicle features from the video to enable identification of a vehicle of interest (VOI) according to a set of predetermined criteria. A subsampling module creates a reduced resolution video stream in a second subsampled resolution that is lower than the first high resolution while maintaining the one or more extracted key features within the reduced resolution video stream in the first high resolution, and archives the reduced resolution video stream into a video database.

Claims

1. A video-based analysis system comprising:

an image capturing device that captures video stream data having video data frames at a first high resolution;

a vehicle detection module that detects at least one vehicle within the video data frames;

a vehicle analysis module that analyzes the video data frames containing the detected vehicle and to extract or tag one or more key vehicle features from the video data frames to enable identification of a vehicle of interest (VOI) according to a set of predetermined criteria, wherein the vehicle analysis module includes a vehicle of interest determination module that filters vehicles detected to determine which vehicles are VOIs according to the set of predetermined criteria that are selected for filtering vehicles detected depending upon a mode of operation, wherein the set of predetermined criteria includes a measured speed exceeding a speed threshold, a red light infraction, a stop sign infraction, wrong way driving, expired registration, and a tailgating violation; and

a subsampling module that creates a reduced resolution video stream in a second subsampled resolution that is lower than the first high resolution while maintaining the one or more extracted key features within the reduced resolution video stream in the first high resolution, and archives the reduced resolution video stream into a video database.

2. The system of claim 1 wherein the vehicle detection module detects and tracks the vehicle in different resolutions respectively that are lower resolutions than the first resolution while keeping a portion of the video data frames in the first high resolution for analysis by the vehicle analysis module.

3. The system of claim 2, wherein the key vehicle features include one or more of vehicle trajectory, measured vehicle speed, vehicle color, vehicle make, vehicle model, vehicle type, license plate, observable defects, geometric measures, and facial data of driver or passengers, and are extracted or tagged by the vehicle analysis module based on portions of data frames within the first high resolution video stream data having the vehicle, the vehicle having been identified as a vehicle of interest according to the predetermined criteria, and vehicle analytic data provided by the vehicle detection module located communicatively upstream.

4. The system of claim 3, wherein the key vehicle features further include a result of a matching test to a provided vehicle alert and wherein the portions include a region of interest within each selected frame or entire frames within the stream data.

5. The system of claim 1, wherein the subsampling module maintains the at least one of the extracted key features by preserving a region of interest within frames having the vehicle detected at a spatial resolution higher than the subsampled resolution.

6. The system of claim 1, wherein the subsampling module maintains the at least one of the extracted key features by embedding metadata into the video data stream archived in the video database, the metadata including vehicle analytic data based on dynamic calculations of the vehicle and objects therein, and vehicle identification data for characteristics of the vehicle and geometric objects therein.

7. The system of claim 1, wherein the subsampling module reduces spatial resolution, temporal resolution or bit depth of the video data stream to create the reduced resolution video stream by splitting full resolution frames having the key features and replacing the split full resolution frames, or regions within the full resolution frames, having the key features with uncompressed frames and designating the replaced high resolution frames as I frames that are compressed with no prediction when being encoded.

8. The system of claim 1, wherein the vehicle analysis module comprises a vehicle identification module that extracts the key features according to the mode of operation in one or more video data frames from the video stream data having the VOI identified, wherein the key vehicle features include vehicle color, vehicle make, vehicle model, vehicle type, license plate data, observable defects, geometric measures, and facial data of driver or passengers.

9. The system of claim 8, wherein when the mode of operation includes a speed enforcement mode the vehicle of interest determination module filters vehicles according to the measured speed exceeding the speed threshold and the predetermined criteria selected include at least the license plate data, when the mode of operation includes an AMBER-Alert mode substantially all vehicles are determined to be VOIs.

10. A video-based analysis and tracking system comprising:

a vehicle analysis module that analyzes video data frames containing a detected vehicle and extracts a plurality of key vehicle features from the video to identify a vehicle of interest (VOI) according to a set of predetermined criteria and vehicle analytic data provided thereto, wherein the vehicle analysis module includes a vehicle of interest determination module that filters vehicles detected to determine which vehicles are VOIs according to the set of predetermined criteria that are selected for filtering vehicles detected depending upon a mode of operation; and

a subsampling module that reduces resolution of the video stream data to a second lower resolution that is at least one of a spatial resolution and a temporal resolution while maintaining at least one of the extracted key features in the first resolution and archives the video at the second resolution into a video database.

11. The system of claim 10, wherein the vehicle analysis module includes a vehicle of interest determination module that identifies a vehicle of interest according to the set of predetermined criteria pertaining to vehicle analytic data collected from potential vehicles of interest, and a vehicle identification module that extracts the key features in one or more video data frames having the first resolution from the video stream data having the VOI identified.

12. The system of claim 11, further comprising:

a vehicle detection module that detects the potential vehicle of interest and determines the vehicle analytic data from the potential vehicle of interest in one or more video data frames, wherein the vehicle detection module further tracks the vehicle in at least one different resolution than for detecting the potential vehicle based upon a different size and resolution of different image frame buffers for detecting and tracking.

13. The system of claim 10, wherein the subsampling module merges and compresses video frames subsampled by an image frame buffer of the subsampling module with image frames of the VOI in the first resolution higher than the second resolution and corresponding extracted key vehicle features, as well as with metadata pertaining to the VOI and having the vehicle analytic data to create the video data stream in the second resolution archived.

14. The system of claim 10, wherein the subsampling module archives the video data stream by reducing the resolution and compressing the video data stream by using predictive video encoding that includes estimating a target frame or a portion of the target frame based on past frames or future frames and compresses only a difference between an estimated frame and the target frame.

15. A method for a video analysis system executed via at least one processor with a memory storing executable instructions to execute the method, comprising:

capturing video stream data having video data frames at a first high resolution in an image capture device;

detecting a vehicle within the video data frames with a detection module;

extracting one or more key vehicle features from the video data frames having the vehicle detected;

identifying a vehicle of interest according to a set of predetermined criteria related to vehicle analytic data from the video frames having the vehicle detected;

filtering vehicles detected to determine which vehicles are VOIs according to the set of predetermined infraction criteria that are selected for filtering vehicles detected depending upon a mode of operation; and

reducing the resolution of the video stream data with a subsampling module to a video data stream having a second resolution while maintaining one or more extracted key features in the first resolution that is a higher resolution than the second resolution.

16. The method of claim 15, further comprising:

archiving the video stream data reduced by the subsampling module and having the key vehicle features extracted in the first resolution in a database together with metadata having at least one of vehicle information and the vehicle analytic data.

17. The method of claim 15, wherein the key features include image data and non-image including one or more of vehicle trajectory, measured vehicle speed, vehicle color, vehicle make, vehicle model, vehicle type, license plate, observable defects, geometric measures, facial data of driver or passengers.

18. The method of claim 15, wherein reducing the resolution of the video data stream includes compressing the video data stream by using predictive video encoding of the video data stream that includes estimating a target frame or a portion of the target frame based on past frames or future frames and compressing only a difference between an estimated frame and the target frame.

19. The method of claim 15, wherein archiving the video data includes encoding non-image data of the key features into metadata in a video format and merging the image data with the video data frames reduced to the second resolution, wherein the vehicle analytic data includes at least one of a vehicle trajectory, a vehicle speed, facial data and one or more vehicle characteristics and wherein the key features include a license plate image and license plate data of the VOI in the first high resolution.

20. The method of claim 15, wherein reducing the resolution of the video data stream includes merging and compressing includes the video data stream with the key features having a higher resolution than the second resolution using a variable bit rate (VBR) technique that includes assigning a target compression rate to one or more video data frames containing the VOI.

21. The method of claim 15, wherein reducing the resolution to the video stream data includes merging video data frames having a subsampled resolution, the vehicle analytic data related to the VOI, and video data frames having the first resolution and the corresponding key feature having a higher resolution than the second resolution by splitting at least a portion of a target frame that is subsampled by a subsampling module into one or more subsampled phase images and replacing at least a portion of subsequent frames to the target frame with the at least a portion of subsampled phase images.