Video compression repository and model reuse

• US 8,902,971 B2
• Filed: 02/20/2013
• Issued: 12/02/2014
• Est. Priority Date: 07/30/2004
• Status: Active Grant

• Current Assignee: Euclid Discoveries LLC
• Original Assignee: Euclid Discoveries LLC
• Inventors: Pace, Charles P., DeForest, Darin, Lee, Nigel, Pizzorni, Renato, Wingard, Richard
• Primary Examiner(s): Torrente, Richard
• Assistant Examiner(s): SUH, JOSEPH JINWOO
• Application Number:
  US13/772,230
• Publication Number:
  US 20130170541A1
• Time in Patent Office: 650 Days
• Field of Search: 375/240.02, 375/240.16, 375/240.01, 725/32, 370/352, 380/202, 709/245, 707/102, 707/204, 386/124, 382/103
• US Class Current: 375/240.02
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model

Abstract

Systems and methods of improving video encoding/decoding efficiency may be provided. A feature-based processing stream is applied to video data having a series of video frames. Computer-vision-based feature and object detection algorithms identify regions of interest throughout the video datacube. The detected features and objects are modeled with a compact set of parameters, and similar feature/object instances are associated across frames. Associated features/objects are formed into tracks, and each track is given a representative, characteristic feature. Similar characteristic features are clustered and then stored in a model library, for reuse in the compression of other videos. A model-based compression framework makes use of the preserved model data by detecting features in a new video to be encoded, relating those features to specific blocks of data, and accessing similar model information from the model library. The formation of model libraries can be specialized to include personal, “smart” model libraries, differential libraries, and predictive libraries. Predictive model libraries can be modified to handle a variety of demand scenarios.

Claims

1. A method of providing video data, comprising:

encoding a subject video stream by a feature-based compression process that utilizes feature models from a global feature model library, said encoding implicitly using the feature models to indicate macroblocks in the subject video to encode by forming tracks using location information from one or more of the feature models in multiple reference frames, the tracks indicating respective locations of macroblocks in a current frame of the subject video stream being encoded, and explicitly using pixels of the feature models to improve prediction of said macroblocks relative to a first prediction of said macroblocks,

where the encoding results in an encoded video data,

where the use of feature models to indicate macroblocks in the subject video avoids the need to segment features from non-features during the feature-based compression process of the subject video, and wherein the feature-based compression process applies feature-based prediction across multiple different video sources based on the feature models, the multiple different video sources being at least the subject video stream and one or more input videos used to generate the global feature model library;

transmitting the encoded video data to a requesting device upon command, said feature models from the global feature model library being made accessible to the requesting device and enabling decoding of the encoded video data at the requesting device;

wherein the global feature model library is formed by;

receiving the one or more input videos, each input video being different from the subject video stream;

for each of the input videos, generating feature information and a respective feature model; and

storing in a data storage device or on cloud storage the feature models generated from the input videos, the data store or cloud storage providing pertinent feature models to the feature-based compression process and the requesting device.

2. A method as claimed in claim 1 wherein the global feature model library is further formed by:

for each input video, identifying and indexing features in the input video, the indexed features forming the respective feature model, said indexing including, for each identified feature, indicating location in the input video of the identified feature; and

wherein the decoding of the encoded video data at the requesting device uses the indexed features and corresponding feature locations in the input videos to obtain said feature models and decode the encoded video data.

3. A method as claimed in claim 2 wherein the indexing of features is a hash-based index.

4. A method as claimed in claim 1 wherein the feature models from the global feature model library form a working model subset library that is specialized per requesting device or per subject video stream.

5. A method as claimed in claim 1 wherein the feature models from the global feature model library form a working model subset library that is a differential library with respect to state of any libraries in the requesting device.

6. A method as claimed in claim 1 wherein the feature models from the global feature model library form a working model subset library that is a predictive model library storing feature models as a function of profile of an end user of the requesting device.

7. A method as claimed in claim 6 wherein the predictive model library has modifiable (settable) parameters enabling application to a variety of demand scenarios.

8. A video data system comprising:

a model library storing video data and serving as a source of streaming video; and

a codec operatively coupled to the model library, and in response to a request for a subject video, the codec being executed by a processor to (i) encode stored video data in the model library corresponding to the requested certain subject video and to (ii) stream the encoded video data from the model library, wherein the codec applies feature-based prediction and compression using feature models from a global feature model library, wherein the global feature model library is formed by;

receiving one or more input videos, each input video being different from the stored video data in the model library corresponding to the requested certain video;

for each of the input videos, generating feature information and respective feature models, such that the codec applies feature-based compression, which includes implicitly using feature models to indicate which macroblocks in the subject video to encode by forming tracks using location information from one or more of the feature models in multiple reference frames, the tracks indicating respective locations of macroblocks in a current frame of the subject video stream being encoded, and explicitly using the pixels of the feature models to improve prediction of said macroblocks relative to a first prediction of said macroblocks, where association of the feature models from the global feature model library with macroblocks in the subject video avoids the need to segment features from non-features during the feature-based compression process of the subject video; and

storing in a data storage device or on cloud storage the feature models generated from the input videos, the data store or cloud storage providing pertinent feature models to the feature-based compression process and the requesting device.

9. A video data system as claimed in claim 8 wherein the codec:

encodes the stored video data in the model library by a feature-based compression that applies feature-based prediction based on the feature models, and

transmits the encoded video data to a requesting device, the encoded video data being the streamed video data from the model library.

10. A video data system as claimed in claim 9 wherein the feature models from the global feature model library are made accessible to the requesting device and enable decoding of the encoded video data at the requesting device.

11. A video data system as claimed in claim 9 wherein the global feature model library is further formed by:

for each input video, identifying and indexing features in the input video, the indexed features forming the respective feature model, said indexing including, for each identified feature, indicating location in the input video of the identified feature; and

wherein decoding of the encoded video data at the requesting device uses the indexed features and corresponding feature locations in the input videos to obtain said feature models and decode the encoded video data.

12. A video data system as claimed in claim 11 wherein the indexing of features is a hash-based index.

13. A video data system as claimed in claim 8 wherein the feature models are stored in the model library.

14. A video data system as claimed in claim 8 wherein the feature models from the global feature model library form a working model subset library that is specialized per requesting device or per streamed encoded video data.

15. A video data system as claimed in claim 8 wherein the feature models from the global feature model library form a working model subset library that is a differential library with respect to state of any libraries in a requesting device.

16. A video data system as claimed in claim 8 wherein the feature models from the global feature model library form a working model subset library that is a predictive model library storing feature models as a function of profile of end-user.

17. A video data system as claimed in claim 16 wherein the predictive model library has modifiable parameters enabling application to a variety of demand scenarios.