Motion-adaptive modelling of scene content for very low bit rate model-assisted coding of video sequences

• US 5,764,803 A
• Filed: 04/03/1996
• Issued: 06/09/1998
• Est. Priority Date: 04/03/1996
• Status: Expired due to Term

• Current Assignee: Alcatel-Lucent USA, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Rabiner, Wendi Beth, Jacquin, Arnaud Eric
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US08/549,994
• Time in Patent Office: 797 Days
• Field of Search: 345/441, 345/501, 348/384, 348/398, 348/400, 348/402, 348/422, 348/416, 348/417, 348/418, 382/199, 382/232, 382/236, 382/238, 382/241, 382/243, 382/253, 382/282, 382/283
• US Class Current: 382/236
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/126 Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A method and apparatus for video coding whereby a region of an image which includes a predetermined object such as a person'"'"'s face in the foreground portion of the image is automatically determined. Specifically, the foreground portion of the image is identified, and one or more predetermined (geometric) shapes (e.g., ellipses) are compared with the shapes of objects found in the foreground portion of the image. The foreground portion of an image may be determined by performing a global motion estimation of the overall image to detect global image movement resulting, for example, from camera pan and zoom. That portion of the image whose movement is consistent with the estimated global motion may be identified as the background portion, with the remainder of the image identified as the foreground portion. The identified region of the image which includes the predetermined object and portions of the image which do not include the predetermined object may be coded with differing levels of coding accuracy (e.g., using different quantization levels), such that if the identified region contains, for example, a person'"'"'s face, the quality of the coding of the face may be improved relative to the quality of the coding of other portions of the image.

Claims

1. A method of coding a video signal, the video signal comprising a succession of frames, a given one of the frames comprising an image including a foreground portion and a background portion, the foreground portion of the image of the given frame including a predetermined object, the method comprising the steps of:

automatically determining the foreground portion of the image of the given frame;

automatically determining a region of interest within the foreground portion of the image of the given frame, said region of interest including at least a portion of the predetermined object, by matching one or more predetermined shapes to the predetermined object in the foreground portion of the image of the given frame;

coding the determined region of interest with a first coder, the first coder having a first coding error; and

coding at least a portion of the image of the given frame, which portion excludes the determined region of interest, with a second coder, the second coder having a second coding error, wherein said first coding error is smaller than said second coding error.

2. The method of claim 1 wherein the step of determining the region of interest comprises performing edge extraction on at least a portion of the image of the given frame to determine a shape of the predetermined object in the foreground portion of the image.

3. The method of claim 1 wherein the step of determining the foreground portion of the image of the given frame comprises

comparing the image of the given frame with an image comprised in a frame previous to the given frame, and

determining a portion of the image of the given frame which has moved relative to a corresponding portion of the frame previous to the given frame.

4. The method of claim 1 wherein the step of determining the foreground portion of the image of the given frame comprises

determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised in a frame previous to the given frame,

determining a background portion of the image of the given frame, said background portion having exhibited motion in accordance with said one or more global motion estimation parameters as applied to a corresponding portion of the frame previous to the given frame, and

determining the foreground portion of the image of the given frame based on the determined background portion.

5. The method of claim 1 further comprising the steps of

determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised

in a frame subsequent to the given frame,

determining a search region of the image of the frame subsequent to the given frame based on the determined region of interest of the image of the given frame and on said one or more global motion estimation parameters,

automatically determining a region of interest of the frame subsequent to the given frame, said region of interest of the frame subsequent to the given frame including at least a portion of the predetermined object, by matching at least one of said one or more predetermined shapes to the predetermined object within the search region of the image of the subsequent frame,

coding the determined region of interest of the image of the frame subsequent to the given frame with the first coder, and

coding at least a portion of the image of the frame subsequent to the given frame, which portion excludes the determined region of interest of the image of the frame subsequent to the given frame, with the second coder.

6. The method of claim 1 further comprising the steps of

determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised in a frame subsequent to the given frame,

selecting at least one of said one or more predetermined shapes based on

the determined region of interest of the image of the given frame and on said one or more global motion estimation parameters,

automatically determining a region of interest of the frame subsequent to the given frame, said region of interest of the frame subsequent to the given frame including at least a portion of the predetermined object, by matching the selected at least one of said one or more predetermined shapes to the predetermined object in the image of the subsequent frame,

coding the determined region of interest of the image of the frame subsequent to the given frame with the first coder, and

coding at least a portion of the image of the frame subsequent to the given frame, which portion excludes the determined region of interest of the image of the frame subsequent to the given frame, with the second coder.

7. The method of claim 1 further comprising the steps of

determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised in a frame subsequent to the given frame,

determining a region of interest of the image of the frame subsequent to the given frame based on the determined region of interest of the image of the given frame and on said one or more global motion estimation parameters,

coding the determined region of interest of the image of the frame subsequent to the given frame with the first coder, and

coding at least a portion of the image of the frame subsequent to the given frame, which portion excludes the determined region of interest of the image of the frame subsequent to the given frame, with the second coder.

8. The method of claim 1 wherein the predetermined object comprises a person'"'"'s head and wherein each of the one or more predetermined shapes comprises an ellipse.

9. The method of claim 1 wherein the predetermined object comprises a person'"'"'s body and wherein each of the one or more predetermined shapes comprises an ellipse.

10. The method of claim 1 wherein the first coder comprises a waveform coder having a first quantizer and the second coder comprises a waveform coder having a second quantizer, the first quantizer providing a finer quantization than the second quantizer.

11. A video coder for coding a video signal, the video signal comprising a succession of frames, a given one of the frames comprising an image including a foreground portion and a background portion, the foreground portion of the image of the given frame including a predetermined object, the apparatus comprising:

means for automatically determining the foreground portion of the image of the given frame;

means for automatically determining a region of interest within the foreground portion of the image of the given frame, said region of interest including at least a portion of the predetermined object, by matching one or more predetermined shapes to the predetermined object in the foreground portion of the image of the given frame;

a first coder for coding the determined region of interest, the first coder having a first coding error; and

a second coder for coding at least a portion of the image of the given frame, which portion excludes the determined region of interest, the second coder having a second coding error, wherein said first coding error is smaller than said second coding error.

12. The video coder of claim 11 wherein the means for determining the region of interest comprises means for performing edge extraction on at least a portion of the image of the given frame to determine a shape of the predetermined object in the foreground portion of the image.

13. The video coder of claim 11 wherein the means for determining the foreground portion of the image of the given frame comprises

means for comparing the image of the given frame with an image comprised in a frame previous to the given frame, and

means for determining a portion of the image of the given frame which has moved relative to a corresponding portion of the frame previous to the given frame.

14. The video coder of claim 11 wherein the means for determining the foreground portion of the image of the given frame comprises

means for determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised in a frame previous to the given frame,

means for determining a background portion of the image of the given frame, said background portion having exhibited motion in accordance with said one or more global motion estimation parameters as applied to a corresponding portion of the frame previous to the given frame, and

means for determining the foreground portion of the image of the given frame based on the determined background portion.

15. The video coder of claim 11 further comprising

means for determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised in a frame subsequent to the given frame,

means for determining a search region of the image of the frame subsequent to the given frame based on the determined region of interest of the image of the given frame and on said one or more global motion estimation parameters,

means for automatically determining a region of interest of the frame subsequent to the given frame, said region of interest of the frame subsequent to the given frame including at least a portion of the predetermined object, by matching at least one of said one or more predetermined shapes to the predetermined object within the search region of the image of the subsequent frame,

means for coding the determined region of interest of the image of the frame subsequent to the given frame with the first coder, and

means for coding at least a portion of the image of the frame subsequent to the given frame, which portion excludes the determined region of interest of the image of the frame subsequent to the given frame, with the second coder.

16. The video coder of claim 11 further comprising

means for determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised in a frame subsequent to the given frame,

means for selecting at least one of said one or more predetermined shapes based on the determined region of interest of the image of the given frame and on said one or more global motion estimation parameters,

means for automatically determining a region of interest of the frame subsequent to the given frame, said region of interest of the frame subsequent to the given frame including at least a portion of the predetermined object, by matching the selected at least one of said one or more predetermined shapes to the predetermined object in the image of the subsequent frame,

means for coding the determined region of interest of the image of the frame subsequent to the given frame with the first coder, and

means for coding at least a portion of the image of the frame subsequent to the given frame, which portion excludes the determined region of interest of the image of the frame subsequent to the given frame, with the second coder.

17. The video coder of claim 11 further comprising

means for determining one or more global motion estimation parameters based on a comparison of the image of the given frame with an image comprised in a frame subsequent to the given frame,

means for determining a region of interest of the image of the frame subsequent to the given frame based on the determined region of interest of the image of the given frame and on said one or more global motion estimation parameters,

means for coding the determined region of interest of the image of the frame subsequent to the given frame with the first coder, and

means for coding at least a portion of the image of the frame subsequent to the given frame, which portion excludes the determined region of interest of the image of the frame subsequent to the given frame, with the second coder.

18. The video coder of claim 11 wherein the predetermined object comprises a person'"'"'s head and wherein each of the one or more predetermined shapes comprises an ellipse.

19. The video coder of claim 11 wherein the predetermined object comprises a person'"'"'s body and wherein each of the one or more predetermined shapes comprises an ellipse.

20. The video coder of claim 11 wherein the first coder comprises a waveform coder having a first quantizer and the second coder comprises a waveform coder having a second quantizer, the first quantizer providing a finer quantization than the second quantizer.