Mosaic generation and sprite-based coding with automatic foreground and background separation

• US 6,751,350 B2
• Filed: 03/05/2001
• Issued: 06/15/2004
• Est. Priority Date: 03/31/1997
• Status: Expired due to Term

• Current Assignee: Rakuten, Inc. (Rakuten Group, Inc.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Crinon, Regis J., Sezan, Muhammed Ibrahim
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): Alavi, Amir
• Application Number:
  US09/799,739
• Publication Number:
  US 20020191846A1
• Time in Patent Office: 1,198 Days
• Field of Search: 382/164, 382/170, 382/171, 382/173, 382/224, 382/227, 382/232, 382/233, 382/236, 382/238, 382/239, 382/284, 382/276, 382/294, 382/243, 348/586, 348/588, 348/564
• US Class Current: 382/173
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

An automatic segmentation system distinguishes foreground and background objects by first encoding and decoding a first image at a first time reference. Macroblocks are extracted from a second image at a second time reference. The macroblocks are mapped to pixel arrays in the decoded first image. Frame residuals are derived that represent the difference between the macroblocks and the corresponding pixel arrays in the previously decoded image. A global vector representing camera motion between the first and second images is applied to the macroblocks in the second image. The global vectors map the macroblocks to a second pixel array in the first decoded image. Global residuals between the macroblocks and the second mapped image arrays in the first image are derived. When the global residuals are compared with the frame residuals to determine which macroblocks are classified as background and foreground. The macroblocks classified as foreground are then blended into a mosaic.

Claims

1. A method for automatically segmenting foreground and background objects in images, comprising:

receiving a first image associated with a first time reference;

extracting macroblocks from a second image associated with a second time reference;

mapping the macroblocks with corresponding arrays in the first image according to macroblock local vectors;

deriving frame residuals between the macroblocks and the corresponding arrays in the first image;

identifying the macroblocks as different frame prediction types according to particular types of local vectors used for mapping the macroblocks to the first image;

deriving multiple local motion vectors mapping different subportions of the macroblocks to subimage arrays in the first image;

deriving residuals by comparing the subportions of the macroblocks with the mapped subimage arrays;

identifying macroblocks as a submacroblock prediction types according to the derived residuals of the subportions of the macroblocks; and

classifying the submacroblock prediction type macroblocks as foreground.

2. A method for automatically segmenting foreground and background objects in images, comprising:

receiving a first image associated with a first time reference;

extracting macroblocks from a second image associated with a second time reference;

mapping the macroblocks with corresponding arrays in the first image according to macroblock local vectors;

deriving frame residuals between the macroblocks and the corresponding arrays in the first image; and

identifying the macroblocks as different frame prediction types according to particular types of local vectors used for mapping the macroblocks to the first image;

mapping macroblocks to portions of the first image according to global motion vectors;

deriving global residuals between the macroblocks and the mapped portions in the first image; and

classifying the macroblocks as foreground or background by comparing the global residuals with the frame residuals.

3. A method according to claim 2 including transmitting an encoded bit stream that includes the identified prediction types of the macroblocks, local motion vectors and global motion vectors mapping, and the residuals for the identified prediction types.

4. An encoder, comprising:

a processor generating frame residuals by using local motion vectors to compare macroblocks in a first frame with pixel arrays in a second frame, generating global residuals by using global motion parameters to compare the macroblocks in the first frame with the pixel arrays in the second frame, and generating mosaic residuals by using the global motion parameters to compare the macroblocks in the first frame with pixel arrays in a mosaic,

the processor identifying the macroblocks as mosaic prediction type when the mosaic residuals are used for encoding the macroblocks and as frame prediction type when the frame residuals are used for encoding the macroblocks,

the processor classifying the frame prediction type macroblocks as foreground or background by comparing the global residuals with the frame residuals and classifying the mosaic prediction type as background.

5. The encoder according to claim 4 wherein the processor blends the macroblocks classified as background into the mosaic.

6. The encoder according to claim 4 wherein the processor transmits an encoded bit stream including the identified macroblock prediction type, the motion vectors associated with the identified macroblock prediction type, and the residuals associated with the identified macroblock prediction type.

7. The encoder according to claim 4 wherein the processor identifies the macroblocks as different frame prediction types according to particular types of local vectors used for mapping the macroblocks to the first frame.

8. The encoder according to claim 4 wherein the processor identifies macroblocks as foreground that are mapped to the first frame with multiple local motion vectors.

9. A method for encoding an image, comprising:

generating frame residuals by using local motion vectors to compare macroblocks in a first frame with pixel arrays in a second frame;

generating global residuals by using global motion parameters to compare the macroblocks in the first frame with the pixel arrays in the second frame;

generating mosaic residuals by using the global motion parameters to compare the macroblocks in the first frame with pixel arrays in a mosaic; and

identifying the macroblocks as mosaic prediction type when the mosaic residuals are used for encoding the macroblocks and as frame prediction type when the frame residuals are used for encoding the macroblocks.

10. A method according to claim 9 including classifying the frame prediction type macroblocks as foreground or background by comparing the global residuals with the frame residuals and classifying the mosaic prediction type as background.

11. A system for encoding an image, comprising:

a processor configured to derive frame residuals between macroblocks and the corresponding arrays in a first image, map macroblocks to portions of the first image according to global motion vectors and derive global residuals between the macroblocks and the mapped portions in the first image; and

the processor further configured to classify the macroblocks as foreground or background by comparing the global residuals with the frame residuals.

12. A system according to claim 11 wherein the first image is associated with a first time reference and the macroblocks are extracted from a second image associated with a second time reference, the processor mapping the macroblocks with corresponding arrays in the first image according to macroblock local vectors and identifying the macroblocks as different frame prediction types according to particular types of local vectors used for mapping the macroblocks to the first image.

13. A method for automatically identifying objects in images, comprising:

deriving multiple local motion vectors mapping different subportions of macroblocks to subimage arrays in a first image;

deriving residuals by comparing the subportions of the macroblocks with the mapped subimage arrays;

identifying macroblocks as a submacroblock prediction types according to the derived residuals of the subportions of the macroblocks; and

classifying the submacroblock prediction type macroblocks as foreground.

14. A method according to claim 13 including:

associating the first image with a first time reference;

extracting the macroblocks from a second image associated with a second time reference;

mapping the macroblocks with corresponding arrays in the first image according to macroblock local vectors;

deriving frame residuals between the macroblocks and the corresponding arrays in the first image; and

identifying the macroblocks as different frame prediction types according to particular types of local vectors used for mapping the macroblocks to the first image.