Apparatus and method for processing video data

• US 7,426,285 B2
• Filed: 09/20/2005
• Issued: 09/16/2008
• Est. Priority Date: 09/21/2004
• Status: Expired due to Fees

• Current Assignee: Euclid Discoveries LLC
• Original Assignee: Euclid Discoveries LLC
• Inventors: Pace, Charles Paul
• Primary Examiner(s): Mehta; Bhavesh
• Assistant Examiner(s): Seth; Manav
• Application Number:
  US11/230,686
• Publication Number:
  US 20060067585A1
• Time in Patent Office: 1,092 Days
• Field of Search: 382/103, 382128-132, 382232-253, 348/169, 375240-24024
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/35 using statistical methods
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/533 Motion estimation using multistep search, e.g. 2D-log search or one-at-a-time search [OTS]
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

An apparatus and methods for processing video data are described. The invention provides a representation of video data that can be used to assess agreement between the data and a fitting model for a particular parameterization of the data. This allows the comparison of different parameterization techniques and the selection of the optimum one for continued video processing of the particular data. The representation can be utilized in intermediate form as part of a larger process or as a feedback mechanism for processing video data. When utilized in its intermediate form, the invention can be used in processes for storage, enhancement, refinement, feature extraction, compression, coding, and transmission of video data. The invention serves to extract salient information in a robust and efficient manner while addressing the problems typically associated with video data sources.

Claims

1. A computer apparatus for the purpose of generating an encoded form of video signal data from a plurality of video frames, comprising:

a means for identifying corresponding elements of an object between two or more video frames;

a means for modeling such correspondences to generate correspondence models, where the correspondence models include motion models and appearance models;

a means for segmenting and resampling pel data in said video frames associated with said object, said segmenting and resampling means utilizing said correspondence models, where the re-sampled pel data creates a representation of said object based on finite differences between said video frames, the finite differences implying an interpolant on a neighborhood of pels in said video frames; and

a means for restoring spatial positions of the re-sampled pel data, said restoring means utilizing the correspondence models;

wherein said object is one or more objects, and

said re-sampled pel data is an intermediate form of original pel data of the object for video processing.

2. The computer apparatus of claim 1 wherein the object is tracked by a tracking member, comprising:

a means of detecting the object in a video frame sequence; and

a means of tracking said object through two or more frames of the video frame sequence;

wherein said object detection means and said tracking means employ a Viola/Jones face detection algorithm.

3. The computer apparatus of claim 1 wherein:

the segmenting and resampling means includes a segmenter; and

the object is segmented from a video frame using a spatial segmentation method, implemented by;

the segmenter segmenting said pel data associated with said object from other pel data in said two or more video frames, the segmenter generating associated segmentation data and;

a means of composing said restored pel data together with the associated segmentation data to create a representation of an original video frame;

wherein said segmenter employes temporal integration.

4. The computer apparatus of claim 1 wherein the correspondence models are factored into global models, by:

a means of integrating correspondence measurements into a model of global motion;

said correspondence modeling means comprising a robust sampling consensus for the solution of a two dimensional affine motion model, and

said correspondence modeling means comprising a sampling population based on finite differences generated from block-based motion estimation between two or more video frames.

5. The computer apparatus of claim 1 wherein the resampled pel data being an intermediate form is normalized object pel data and further encoded by:

a means of decomposing said normalized object pel data into an encoded representation;

a means of recomposing said normalized object pel data from the encoded representation;

wherein said decomposing means comprises Principal Component Analysis, and

said recomposing means comprises Principal Component Analysis.

6. The computer apparatus of claim 5 wherein:

(i) the means for identifying employs tracking means for tracking multiple objects;

(ii) the correspondence modeling means integrates motion parameters in a manner that facilitates a compact parameterization of motion parameters in the correspondence models; and

(iii) the segmenting and resampling means includes (a) segmenting means for spatially

segmenting the pel data associated with said object from other pel data in the two or more video frames, the segmentation means generating segmentation data associated with the object, and (b) means for composing said restored pel data together with the associated segmentation data to form an original video frame, wherein said segmentation means includes temporal integration.

7. The computer apparatus of claim 5 wherein pel data of a non-object of the video frames are modeled,

said non-object of the video frames being a residual when said object is removed from the video frames.

8. The computer apparatus of claim 6 wherein the segmented and resampled pel data are combined with a video compression/decompression process, including:

a means of supplying the re-sampled pel data as video data to the video compression/decompression process;

a means of storing and transmitting correspondence models along with corresponding encoded pel data such that

said compression/decompression process has increased compression efficiency.

9. A method for generating an encoded form of video signal data from a plurality of video frames, comprising the computer implemented steps of:

identifying corresponding elements of an object between two or more video frames;

modeling such correspondences to generate correspondence models, where the correspondence models include motion models and appearance models;

segmenting and resampling pel data in said video frames associated with said object, said segmenting and resampling utilizing said correspondence models, where the re-sampled pel data creates a representation of said object using finite differences between said video frames, the finite differences implying an interpolant on a neighborhood of pels in said video frames; and

restoring spatial positions of the re-sampled pel data, said restoring utilizing the correspondence models;

wherein said object is one or more objects, and

said re-sampled pel data provides an intermediate form of original pel data of the object for video processing.

10. A method as claimed in claim 9 further comprising the steps of:

detecting the object in a video frame sequence; and

tracking said object through two or more frames of the video frame sequence.

11. A method as claimed in claim 9 wherein the step of segmenting and resampling includes the steps of:

segmenting said pel data associated with said object from other pel data in said two or more video frames, the segmenting generating associated segmentation data; and

composing said restored pel data together with the associated segmentation data resulting in a representation of an original video frame;

wherein said segmenting includes temporal integration.

12. A method as claimed in claim 9 wherein the correspondence models are factored into global models, further comprising the step of:

integrating correspondence measurements into a model of global motion;

said correspondence modeling including (i) a sampling consensus for the solution of a two dimensional affine motion model, and

(ii) a sampling population based on finite differences generated from block-based motion estimation between two or more video frames.

13. A method as claimed in claim 9 further comprising the steps of:

decomposing said resampled object pel data into an encoded representation;

recomposing said resampled object pel data from the encoded representation;

wherein said steps of decomposing and recomposing employing Principal Component Analysis.

14. A method as claimed in claim 13 further comprising the step of

(i) tracking multiple objects;

(ii) integrating motion parameters in a manner that facilitates a compact parameterization of motion parameters in the correspondence models; and

(iii) spatially segmenting the pel data associated with said object from other pel data in the two or more video frames, the segmenting generating segmentation data associated with the object; and

(iv) composing said restored pel data together with the associated segmentation data to form a representation of an original video frame.

15. A method as claimed in claim 13 further comprising the step of modeling residuals of the video frames, said residuals being results of a video frame when said one or more objects are removed.

16. A method as claimed in claim 13 further comprising the steps of:

supplying the re-sampled pel data as video data to a video compression/decompression process; and

storing and transmitting correspondence models along with the encoded representation of pel data such that the compression/decompression process has increased compression efficiency.