Computer method and apparatus for processing image data

• US 8,908,766 B2
• Filed: 01/04/2008
• Issued: 12/09/2014
• Est. Priority Date: 03/31/2005
• Status: Active Grant

• Current Assignee: Euclid Discoveries LLC
• Original Assignee: Euclid Discoveries LLC
• Inventors: Pace, Charles P.
• Primary Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US12/522,322
• Publication Number:
  US 20100008424A1
• Time in Patent Office: 2,531 Days
• Field of Search: 375/240.16
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/433 characterised by techniques for memory access
• CPC Class Codes: H04N 19/48 using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
• CPC Class Codes: H04N 7/127 Systems in which different parts of the picture signal frequency band are individually processed, e.g. suppressed, transposed

Abstract

A method and apparatus for image data compression includes detecting a portion of an image signal that uses a disproportionate amount of bandwidth compared to other portions of the image signal. The detected portion of the image signal result in determined components of interest. Relative to certain variance, the method and apparatus normalize the determined components of interest to generate an intermediate form of the components of interest. The intermediate form represents the components of interest reduced in complexity by the certain variance and enables a compressed form of the image signal where the determined components of interest maintain saliency. In one embodiment, the video signal is a sequence of video frames. The step of detecting includes any of: (i) analyzing image gradients across one or more frames where image gradient is a first derivative model and gradient flow is a second derivative, (ii) integrating finite differences of pels temporally or spatially to form a derivative model, (iii) analyzing an illumination field across one or more frames, and (iv) predictive analysis, to determine bandwidth consumption. The determined bandwidth consumption is then used to determine the components of interest.

Claims

1. A method for video data compression, comprising the steps of:

processing a video signal to determine bandwidth consumption of portions of the video signal;

using the determined bandwidth consumption, detecting a portion of the video signal that uses a disproportionate amount of bandwidth compared to one or more other portions of the video signal, the detected portion of the video signal resulting in determined components of interest, which are detected objects in the video signal;

modeling invariant portions of the determined components of interest; and

normalizing the modeled invariant portions of the determined components of interest to generate an intermediate form of the components of interest, the intermediate form representing the components of interest reduced in complexity enabling a compressed form of the video signal through linear decomposition, where the determined components of interest maintain saliency.

2. A method as in claim 1 wherein

the video signal is a sequence of frames;

and

the step of detecting a portion of the video signal that uses a disproportionate amount of bandwidth compared to one or more other portions of the video signal includes at least one of;

(i) analyzing image gradients across one or more frames where image gradient is a first derivative model and gradient flow is a second derivative,

(ii) integrating finite differences of pels temporally or spatially to form a derivative model,

(iii) analyzing an illumination field across one or more frames, and

(iv) predictive analysis, to determine bandwidth consumption, the determined bandwidth consumption being used to determine the components of interest.

3. A method as in claim 1 wherein the determined components of interest contain structural information including one or more of spatial features and correspondence of spatial features (motion).

4. A method as in claim 3 wherein the spatial features further include one or more of edges, corners, pixels, spatial patterns and derived patterns (SIFT).

5. A method as in claim 1 wherein modeling invariant portions of the determined components of interest involves forming one or more of a structural model or an appearance model of the determined components of interest.

6. A method as in claim 5 further comprising the step of applying geometric data analysis techniques to at least the appearance model, wherein the reduction in complexity of the components of interest enabling improved application of geometric data analysis.

7. A method as in claim 5 further comprising the step of applying geometric data analysis techniques to at least the structural model, wherein the reduction in complexity of the components of interest enables enabling improved application of geometric data analysis.

8. A method as claimed in claim 1 wherein the certain variance is one or more of global structure, global motion and pose, local deformation and illumination.

9. A method as in claim 8 further comprising the step of applying geometric data analysis techniques to the generated intermediate form, wherein the intermediate form provides a representation of the components of interest having a the reduction in complexity, of the components of interest enables improved application of geometric data analysis.

10. A method as claimed in claim 9 wherein the geometric data analysis techniques include the linear decomposition or nonlinear decomposition.

11. A method as claimed in claim 10 wherein the linear decomposition employs one or more of:

sequential PCA, power factorization, generalized PCA, and progressive PCA.

12. A method as claimed in claim 11 wherein progressive PCA includes wavelet transform techniques combined with PCA.

13. A computer implemented method as claimed in claim 1 wherein the step of encoding, further including encoding, with respect to certain variance, the determined components of interest to generate the intermediate form of the components of interest, where the certain variance is one or more of global structure, global motion and pose, local deformation and illumination.

14. Apparatus for video data compression, comprising:

a detector configured to use bandwidth consumption of portions of a video signal to determine components of interest, including detecting a portion of the video signal that uses a disproportionate amount of bandwidth compared to one or more other portions of the video signal, resulting in determined components of interest, which are detected ob objects;

a modeler configured to model invariant portions of the determined components of interest and;

a normalizer configured to normalize the modeled invariant portions of the determined components of interest to generate an intermediate form of the components of interest, the intermediate form representing the components of interest reduced in complexity and enabling a compressed form of the video signal through linear decomposition, where the determined components of interest maintain saliency.

15. Apparatus as claimed in claim 14 wherein

the video signal is a sequence of frames;

and

the detector further performs at least one of;

(i) analyzing image gradients across one or more frames where image gradient is a first derivative model and gradient flow is a second derivative,

(ii) integrating finite differences of pels temporally or spatially to form a derivative model,

(iii) analyzing an illumination field across one or more frames, and

(iv) predictive analysis, to determine bandwidth consumption, the determined bandwidth consumption being used to determine the components of interest.

16. Apparatus as claimed in claim 14 wherein the normalizer forms a structural model and an appearance model of the determined components of interest.

17. Apparatus as claimed in claim 16 further comprising an analyzer, the analyzer applying geometric data analysis to at least one of the appearance model and the structural model, wherein the intermediate form provides a representation of the components of interest having a reduction in complexity, the components of interest enabling improved application of geometric data analysis.

18. Apparatus as claimed in claim 14 wherein the certain variance is one or more of global structure, global motion and pose, local deformation and illumination.

19. Apparatus as claimed in claim 14 further comprising an analyzer, the analyzer applying geometric data analysis techniques to the intermediate form, wherein the intermediate form provides a representation of the components of interest having a reduction in complexity, the components of interest enabling improved application of geometric data analysis;

wherein the geometric data analysis techniques include any of linear decomposition and nonlinear decomposition, and

wherein the linear decomposition employs any of;

sequential PCA, power factorization, generalized PCA, and progressive PCA.

20. Apparatus as claimed in claim 19 wherein progressive PCA includes wavelet transform techniques combined with PCA.

21. Apparatus for video data compression, comprising;

an object detector configured to process a video signal to determine bandwidth consumption of portions of the video signal, the object detector configured to detect portions of a video signal for encoding, the detector filtering the video signal for one or more portions using a disproportionate amount of bandwidth compared to one or more other portions of the video signal, where the detected portion of the video signal is a determined component of interest, which is a detected object, for use in encoding by an encoder;

a modeler configured to model invariant portions of the determined components of interest; and

the encoder configured to process the modeled invariant portions of the determined components of interest having a disproportionate amount of bandwidth to generate an intermediate form of the invariant portions of the determined components of interest, the encoder configured to encode the intermediate form into a compressed form through linear decomposition.

22. Apparatus as in claim 21 wherein the intermediate form provides a representation of the components of interest having a reduction in complexity as compared to its original form detected in the video signal, the components of interest enabling improved application of geometric data analysis.

23. Apparatus as claimed in claim 21 wherein the encoder is configured to normalize the modeled invariant portions of the determined components of interest to generate the intermediate form of the components of interest, where the modeled invariant portions of the determined components of interest model global structure, global motion and pose, local deformation and illumination.

24. A computer implemented method of video data compression, comprising:

processing a video signal to determine bandwidth consumption of portions of the video signal;

detecting portions of the video signal by filtering the video signal for one or more portions using a disproportionate amount of bandwidth compared to one or more other portions of the video signal, where the detected portion of the video signal is a determined component of interest for use in encoding, which is a detected object;

modeling invariant portions of the determined components of interest;

processing the modeled invariant portions of the determined components of interest to generate an intermediate form of the determined components of interest; and

using the intermediate form of the invariant portions of the determined components of interest to provide an encoding through linear decomposition, the resulting linear decomposition encoding providing a compressed form of the invariant portions of the determined components of interest.

25. A computer implemented method as claimed in claim 24 wherein the intermediate form provides a representation of the components of interest having a reduction in complexity as compared to its original form detected in the video signal, the components of interest enabling improved application of geometric data analysis.

26. Apparatus for video data compression, comprising:

an object detector configured to process a video signal to determine bandwidth consumption of portions of the video signal, the object detector configured to detect portions of the video signal using a disproportionate amount of bandwidth compared to one or more other portions of the video signal, where each of the detected portions of the video signal is a component of interest being a detected object;

a modeler configured to model invariant portions of the components of interest requiring a disproportionate amount of bandwidth; and

an encoder configured to process the modeled invariant portions of the determined components of interest requiring a disproportionate amount of bandwidth into a linear subspace suitable for linear decomposition to generate an encoded form of the determined components of interest.

27. A computer implemented method of video data compression, comprising:

processing a video signal to determine bandwidth consumption of portions of the video signal;

detecting components of interest in the video signal by filtering the video signal for one or more portions using a disproportionate amount of bandwidth compared to one or more other portions of the video signal, where the detected portion of the video signal is a determined component of interest for use in decoding, which is a detected object;

modeling invariant portions of the components of interest requiring a disproportionate amount of bandwidth; and

normalizing the invariant portions of the determined components of interest requiring a disproportionate amount of bandwidth into a linear subspace suitable to generate a compressed form of the determined components of interest via linear decomposition.

28. A video data encoder comprising:

an encoder stored on a non-transitory computer readable medium, the encoder configured to process a video signal to determine bandwidth consumption of portions of the form of the video signal, the encoder configured to detect components of interest in the video signal for encoding by filtering the video signal for one or more portions using a disproportionate amount of bandwidth compared to one or more other portions of the video signal, where the detected portion of the video signal using a disproportionate amount of bandwidth is a determined component of interest for use in encoding, the determined component of interest resulting in a detected object; and

the encoder configured to model invariant portions of the determined components of interest requiring a disproportionate amount of bandwidth, the modeled invariant portions of the determined components of interested being processed into a linear subspace to enable an encoded compressed form of the determined components of interest through linear decomposition for use in encoding the video signal.

29. A video processing system comprising:

an object detector stored on a non-transitory computer readable medium, the object detector configured to detect portions of video signal for objects of interest by filtering the video signal based on bandwidth consumption to identify one or more portions using a disproportionate amount of bandwidth compared to one or more other portions of the video signal, where a respective detected portion of the video signal is a determined component of interest;

a modeler configured to model the determined components of interest requiring a disproportionate amount of bandwidth using deformation and appearance models;

a normalizer configured to normalize the determined components of interest models to enable encoding through linear decomposition; and

the encoder configured to encode the normalized determined components of interest requiring to generate an encoded form of the determined components of interest through linear decomposition, such that the resulting encoding provides a compressed form of the determined components of interest of the video signal, where the determined components of interest maintain saliency.