Feature-based video compression

• US 8,964,835 B2
• Filed: 12/30/2011
• Issued: 02/24/2015
• Est. Priority Date: 03/31/2005
• Status: Active Grant

• Current Assignee: Euclid Discoveries LLC
• Original Assignee: Euclid Discoveries LLC
• Inventors: Pace, Charles P.
• Primary Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US13/341,482
• Publication Number:
  US 20120155536A1
• Time in Patent Office: 1,152 Days
• Field of Search: 375/240.08
• US Class Current: 375/240.08
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/543 using regions

Abstract

Systems and methods of processing video data are provided. Video data having a series of video frames is received and processed. One or more instances of a candidate feature are detected in the video frames. The previously decoded video frames are processed to identify potential matches of the candidate feature. When a substantial amount of portions of previously decoded video frames include instances of the candidate feature, the instances of the candidate feature are aggregated into a set. The candidate feature set is used to create a feature-based model. The feature-based model includes a model of deformation variation and a model of appearance variation of instances of the candidate feature. The feature-based model compression efficiency is compared with the conventional video compression efficiency.

Claims

1. A computer implemented method of processing video data formed of a series of video frames comprising:

using a first video encoding process an d feature-based encoding process, processing the video frames by;

encoding the video frames with the first video encoding process;

processing one or more of the frames to detect one or more instances of a feature by searching the one or more frames for a region of pels having coherency and computational complexity as compared to other pels in the one or more frames;

modeling variation of the feature instance relative to other instances of the feature to create a feature-based encoding of the feature instance, wherein modeling variation of the feature instance includes;

identifying other instances of the feature in other frames of the video series;

aggregating two or more of the instances of the feature into a set;

processing the feature instances in the set to create the models for the feature based encoding;

comparing compression efficiency of the feature-based encoding of the feature instance relative to an encoding of the feature instance resulting from the first video encoding process;

determining from the comparison which encoding enables greater compression efficiency; and

using the results of the comparing and determining step, applying feature-based encoding to portions of one or more of the video frames, and applying conventional encoding to other portions of the one or more video frames.

2. A computer implemented method as in claim 1 wherein the first video encoding process includes one or more of the following:

an MPEG encoding process or a motion compensated prediction process.

3. A computer implemented method as in claim 2 wherein the motion compensated prediction process is used to segment the feature instance from one or more non-features in the one or more video frames.

4. A computer implemented method as in claim 1 further including using one or more frames in the series that have been previously encoded using the first encoding process to determine positional information for the feature instance, the positional information including a position and a spatial perimeter of one or more instances of the feature in the one or more previously encoded video frames.

5. A computer implemented method as in claim 1 wherein searching the one or more video frames for a region of pels having coherency further includes filtering the one or more video frames for a spatially continuous region of pels in close spatial proximity to each other.

6. A computer implemented method as in claim 5 wherein the spatially continuous region of pels in close spatial proximity to each other is capable of being tracked in one or more frames in the series.

7. A computer implemented method as in claim 5 wherein filtering the one or more video frames for a spatially continuous region of pels in close spatial proximity to each other further includes:

processing an identified region of pels to evaluate whether it is combinable with one or more other pels in close spatial proximity based on coherency criteria in order to create a larger spatial region;

encoding the larger spatial region using feature-based encoding; and

comparing the feature-based encoding of the larger spatial region with an encoding of the larger spatial region resulting from the first encoding process.

8. A computer implemented method as in claim 7 wherein comparing the feature-based encoding of the larger spatial region with an encoding of the larger spatial region resulting from the first encoding process further includes:

determining which encoding provides compression efficiency; and

if the feature-based encoding of the larger spatial region provides improved compression efficiency, using the larger spatial region to represent the feature instance.

9. A computer implemented method as in claim 8 wherein if the feature-based encoding of the larger spatial region does not provide improved compression efficiency relative to an encoding of the larger spatial region using the first encoding process, discarding the other pels that were used to create the larger spatial region.

10. A computer implemented method as in claim 9 wherein discarding the other pels that were used to create the larger spatial region further includes excluding from the feature instance the other pels that were used to create the larger spatial region.

11. A computer implemented method as in claim 1 wherein the coherency is based on correspondences in appearance of pels in close spatial proximity.

12. A computer implemented method as in claim 1 wherein the feature instance is predicted to provide relatively increased compactness in the feature-based encoding relative to an encoding of the feature instance resulting from the first encoding process.

13. A computer implemented method as in claim 1 wherein modeling variation of the feature instance relative to other instances of the feature results in feature-based models, which are used to create the feature-based encoding of the feature instance.

14. A computer implemented method as in claim 13 wherein the feature instance is transformed using a linear transform to create the feature-based models.

15. A computer implemented method as in claim 1 wherein modeling variation of a feature instance further includes modeling using one or more of the following modeling processes:

modeling structural variation of instances of the feature identified in the one or more video frames;

or

modeling appearance variation of instances of the feature identified in the one or more video frames.

16. A computer implemented method as in claim 15 wherein modeling the appearance variation further includes creating one or more appearance models using a PCA decomposition to model spatial variation between corresponding feature instances.

17. A computer implemented method as in claim 15 wherein modeling the structural variation further includes creating one or more structural models using a PCA decomposition to model spatial variation of corresponding feature instances.

18. A computer implemented method as in claim 1 wherein modeling variation of a feature instance further includes comparing feature based models of instances of the feature to identify an efficiently compressed feature-based encoding.

19. A computer implemented method as in claim 18 wherein comparing compression efficiency further includes based on the results of the comparison, using the feature-based model having greater compression efficiency.

20. A computer implemented method as in claim 1 wherein the region of pels further includes one or more macroblocks or portions of macroblocks.

21. A computer method as in claim 1 further including:

comparing one or more feature-based encodings of a feature instance to determine which feature-based encoding provides greater compression efficiency;

comparing compression efficiency of the resulting more efficiently compressed feature-based encoding with an encoding of the feature instance resulting from the first video encoding process;

determining from the comparison which encoding enables greater compression efficiency; and

using the results of the comparing and determining step, applying feature-based encoding to portions of one or more of the video frames, and applying conventional video encoding to other portions of the one or more video frames.

22. A computer implemented method as in claim 1 further including using spatial positions of feature instances from the one or more video frames to predict another instance of the feature in a current frame in the series.

23. A computer implemented method as in claim 22 wherein predicting another instance of the feature in a current frame in the series further includes using parameters from models of feature instances appearing in temporally recent video frames in the series to create one or more models of the predicted feature instance in the current frame.

24. A computer implemented method as in claim 23 further including using the one or more models of feature instances appearing in temporally recent video frames to interpolate or extrapolate parameters of the predicted feature instance in the current frame.

25. A computer implemented method as in claim 23 wherein the models of feature instances appearing in temporally recent video frames in the series include one or more appearance or structural models.

26. A computer implemented method as in claim 24 further including using an extended Kalman filter to interpolate or extrapolate the parameters for the feature-based model of the predicted feature instance in the current frame.

27. A computer implemented method as in claim 24 wherein there is a reduction in computing resources required to process an encoding of the predicted feature instance in the current frame relative to an encoding of the predicted feature instance in the current frame using the first encoding process.

28. A computer implemented method as in claim 1 wherein the one or more feature instances are free of correspondence to distinct salient entities in the one or more video frames.

29. A computer implemented method as in claim 1 wherein the feature instance includes one or more objects or sub-objects.

30. A computer implemented method as in claim 1 wherein the feature instance includes at least elements of two or more salient entities, background or other parts of the video frames.

31. A computer implemented method as in claim 1 wherein the feature instance is not an object.

32. A computer implemented method as in claim 1 wherein the feature instance is identified in the one or more frames using compressed sensing.

33. A computer implemented method as in claim 1 wherein the step of comparing the compression efficiency further includes comparing the compression efficiency of two or more encoding processes.

34. A computer implemented method as in claim 33 wherein the two or more encoding processes comprises two or more of following encoding processes:

the feature-based encoding process, the first video encoding process, or any other encoding process.

35. A computer system for processing video comprising:

a hybrid encoder configured to encode at least a portion of one or more video frames using a first video encoding process based on motion compensated prediction and encoding one or more other portions of the video frames using another video encoding process based on feature-based compression, the encoder configured to;

identify in a plurality of video frames instances of a feature, where the feature is a spatially continuous region of pels that are in close spatial proximity to each other having coherency and exhibiting complexity;

determine that an encoding of the feature instances using models from a feature-based encoding process provides greater compression efficiency relative to an encoding of the features instances resulting from the first video encoding process;

using the results of the and determining step, applying feature-based encoding to portions of one or more of the video frames, and applying conventional encoding to other portions of the one or more video frames;

wherein applying the feature based encoding to one or more of the feature instances by modeling their variation in the plurality of the video frames, wherein modeling variation of the feature instance includes;

identifying other instances of the feature in other frames of the video series;

aggregating two or more of the instances of the feature into a set;

processing the feature instances in the set to create the models for the feature based encoding.

36. A computer system as in claim 35 wherein the first video encoding process is an MPEG encoding process.

37. A computer system as in claim 35 wherein the motion compensated prediction process is used to segment the feature instance from one or more non-features in the one or more video frames.

38. A computer system as in claim 35 wherein the hybrid encoder is configured to use one or more frames in the series that have been previously encoded using the first encoding process to determine positional information for the feature instance, the positional information including a position and a spatial perimeter of one or more instances of the feature in the one or more previously encoded video frames.

39. A computer system as in claim 35 wherein identifying in a plurality of video frames instances of a feature further includes searching the one or more video frames for a region of pels having coherency by filtering the one or more video frames for a spatially continuous region of pels in close spatial proximity to each other.

40. A computer system as in claim 35 wherein the spatially continuous region of pels in close spatial proximity to each other is capable of being tracked in one or more frames in the series.

41. A computer system as in claim 39 wherein filtering the one or more video frames for a spatially continuous region of pels in close spatial proximity to each other further includes:

processing an identified region of pels to evaluate whether it is combinable with one or more other pels in close spatial proximity based on coherency criteria in order to create a larger spatial region;

encoding the larger spatial region using feature-based encoding; and

comparing the feature-based encoding of the larger spatial region with an encoding of the larger spatial region resulting from the first encoding process.

42. A computer implemented method as in claim 40 wherein comparing the feature-based encoding of the larger spatial region with an encoding of the larger spatial region resulting from the first encoding process further includes:

determining which encoding provides improved compression efficiency; and

if the feature-based encoding of the larger spatial region provides improved compression efficiency, using the larger spatial region to represent the feature instance.

43. A non-transitory computer readable medium storing instructions so as when executed to cause one or more computer processors to encode video frames by:

processing the video frames with a first video encoding process that uses motion compensated prediction;

searching or more of the frames for an instance of a feature that includes a spatially continuous region of pels in close spatial proximity to each other having coherency and exhibiting complexity;

identifying other instances of the feature in other frames of the video series;

aggregating two or more of the instances of the feature into a set;

processing the feature instances in the set to create the models for the feature based encoding;

comparing compression efficiency of a feature based encoding of the feature instance relative to an encoding of the feature instance resulting from the first encoding process; and

determining from the comparison which encoding enables greater compression efficiency; and

using the results of the comparing and determining step, applying feature-based encoding to portions of one or more of the video frames, and applying conventional encoding to other portions of the one or more video frames.

44. A computer implemented method for processing video data comprising:

processing portions of encoded video frames including one or more encoded features, the one or more encoded features including portions of features encoded using a feature-based encoding process and portions of features encoded using a motion compensated encoding process;

separating encoded features from non-feature portions of the video frames;

separating the portions of features encoded using feature-based encoding from portions of features encoded using a motion compensated encoding process;

decoding the non-feature portions of the video and the portions of features encoded using a motion compensated encoding process using a motion compensated decoding process;

decoding the portions of features encoded using feature-based encoding using encoded feature parameters and feature models generated by the decoder; and

compositing the decoded non-feature portions, the decoded motion compensated encoded feature portions and the decoded feature-based encoded feature portions to generate a decoded video signal;

wherein generating feature models by the decoder includes;

filtering on or more of the decoded frames to detect an instance of a feature that is a spatially continuous region of pels that are in close spatial proximity to each other, the regions of pels having coherency and exhibiting complexity; and

identifying other instances of the feature in other decoded frames of the video series;

aggregating two or more of the instances of the feature into a set;

processing the feature instances in the set to create the models for the feature based decoding.

45. A computer system for processing video comprising:

a hybrid decoder configured to decode portions of the video frames including one or more encoded features using a first video encoding process based on motion compensated prediction and to decode other portions of the video frames using another video decoding process based on feature based compression by;

separating encoded features from non-feature portions of the video frames;

separating the portions of features encoded using feature-based encoding from portions of features encoded using the first encoding process;

decoding the non-feature portions of the video and the portions of features encoded using the first encoding process using a motion compensated decoding process;

decoding the portions of features encoded using feature-based encoding using encoded feature parameters and feature models generated by the decoder; and

compositing the decoded non-feature portions, the decoded first encoding process encoded feature portions and the decoded feature-based encoded feature portions to generate a decoded video signal;

wherein generating feature models by the decoder includes;

identifying in a plurality of decoded video frames instances of a feature where the feature is a spatially continuous region of pels that are in close spatial proximity to each other having coherency and exhibiting complexity; and

identifying other instances of the feature in other decoded frames of the video series;

aggregating two or more of the instances of the feature into a set;

processing the feature instances in the set to create the models for the feature based decoding.

46. A non-transitory computer readable medium storing instructions so as when executed to cause one or more computer processors to decode video frames by:

processing the encoded video frames including one or more encoded features, with a first video decoding process based on motion compensated prediction and to decode other portions of the video frames using another video decoding process based on feature based compression by;

separating encoded features from non-feature portions of the video frames;

separating the portions of features encoded using feature-based encoding from portions of features encoded using a motion compensated encoding process;

decoding the non-feature portions of the video and the portions of features encoded using a motion compensated encoding process using a motion compensated decoding process;

decoding the portions of features encoded using feature-based encoding using encoded feature parameters and feature models generated by the decoder; and

compositing the decoded non-feature portions, the decoded motion compensated encoded feature portions and the decoded feature-based encoded feature portions to generate a decoded video signal;

wherein generating feature models by the decoder includes;

identifying, in one or more of the decoded frames, an instance of a feature that is a spatially continuous region of pels in close spatial proximity to each other having coherency and exhibiting complexity; and

identifying other instances of the feature in other decoded frames of the video series;

aggregating two or more of the instances of the feature into a set;

processing the feature instances in the set to create the models for the feature based decoding.