Video image compositing techniques

• US 5,262,856 A
• Filed: 06/04/1992
• Issued: 11/16/1993
• Est. Priority Date: 06/04/1992
• Status: Expired due to Fees

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Lippman, Andrew B., McLean, Patrick G., Bender, Walter R.
• Primary Examiner(s): Groody, James J.
• Assistant Examiner(s): Metjahic, Safet
• Application Number:
  US07/893,454
• Time in Patent Office: 530 Days
• Field of Search: 358/133, 358/136, 358/141, 358/105, 358/183, 358/22, 358/87, 358/12, 358/135, 358/140, 358/11
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/0122 the input and the output signals having different aspect ratios

Abstract

A method of processing video data created for display in one aspect ratio in a plurality of different sequences of frames thereof, each frame in a sequence containing background data which represents at least a part of the same relatively stationary background data in the sequence. The background data in each frame of a sequence is stored and processed so as to produce combined background data representing the background image in the sequence in a different aspect ratio, without the presence of moving objects therein. Data representing moving objects in each frame of a sequence are stored together with selected portions of the background data which surround the moving objects in each frame. The data representing moving objects in each frame are correctly positioned in the combined background data, which is used for each frame of the sequence, by matching the selected portion of the background data surrounding the moving objects with corresponding portions of the combined background data to determine the correct positions of the moving objects in the background image. The background data and the matching portions of the moving objects are composited in accordance with such correct position determination to form a composite image for each frame in which the moving objects are correctly positioned in the combined background image.

Claims

1. A method for processing video data in a plurality of different sequences of frames thereof, each frame in a particular sequence of frames including relatively motionless background data representing at least a part of the entire relatively motionless background data in said sequence and moving object data representing moving objects in said sequence, said method comprising the steps of

storing the background and moving object data in each frame of a particular sequence of frames;

aligning common portions of the stored background data in each frame to provide stored combined background data which represents a composite relatively motionless background image which contains substantially the entire relatively motionless background data in said sequence and is at least as wide as the background image in any one frame of said sequence thereof;

processing the aligned background data to remove the moving object data therefrom;

storing said processed background data as combined background data representing said composite background image.

2. A method in accordance with claim 1 and further including the steps of

processing the data in each frame of said sequence of frames to provide moving object data representing objects in each frame of said sequence of frames, which move relative to the relatively motionless background data over said sequence of frames, together with data representing selected portions of the background data which at least partially surround each moving object in each said frame.

3. A method in accordance with claim 2 and further including the steps of

matching the selected portions of said background data which at least partially surround the object data representing the moving objects in each frame with corresponding portions of the combined background data to determine the correct position of said moving objects in each frame in said combined background data;

and

compositing said matching portions of the background at least partially surrounding said moving objects into the combined background data in each frame in accordance with said correct position determination so as to form composited image data for each frame of said sequence of frames in which said moving objects are correctly positioned in said combined background data in each frame of said sequence of frames.

4. A method in accordance with claim 3 and further including the steps of

transmitting said combined background data representing said composite background image to a receiver for use in displaying said composite background image in each frame of said sequence of frames whereby each displayed frame includes at least a portion of substantially the same composite background image;

transmitting said moving object data and the portions of background data at least partially surrounding said moving object data to a receiver for use in displaying in each frame of said sequence thereof the moving object data for each said frame composited into the combined background data for each frame;

receiving at said receiver the transmitted combined background data and the transmitted moving object data with said portions of at least partially surrounding background data; and

performing said matching step and said compositing step at said receiver.

5. A method in accordance with claim 3 wherein said matching step is performed at a transmitter and further includes the steps of

transmitting said combined background data representing said composite background image, said moving object data with the portions of background data at least partially surrounding said moving object data, and information representing said determined correct position of said moving object in said combined background data for each frame of said sequence of frames to a receiver;

receiving at said receiver said combined background data, said moving object data with said at least partially surrounding background data, and said correct position information; and

performing said compositing step at said receiver.

6. A method in accordance with claim 1 wherein said aligning step includes

storing said background data in each frame in a buffer so that common portions thereof are aligned when background data in a frame is shifted relative to background data in a previous frame;

combining the stored background data in all of the aligned frames in a manner so as to produce combined background data representing the substantially entire background data in said sequence of frames.

7. A method in accordance with claim 6 and further including

comparing background data in one frame relative to background data in a previous frame, and when the shift of the background data in said one frame is above a selected threshold, storing the background data in said one frame in said buffer with an offset from said previous frame representing said shift.

8. A method in accordance with claim 7 wherein, in said storing step, the background data in each frame is stored in a single widened buffer so that the background data in a frame which is shifted from the background data in a previous frame is stored in said widened buffer at a selected offset position therein, portions of said buffer at each frame containing no data.

9. A method in accordance with claim 7 wherein, in said storing step, background data is stored in a narrow buffer so that the background data in a frame which is shifted from background data in a previous frame is stored in said narrow buffer, there being no portions of said narrow buffer at each frame not containing said data, and further storing a coded number representing the offset of the data in each frame.

10. A method in accordance with claim 1 wherein said processing step includes

removing pixels representing moving object data in each frame of said sequence of frames and storing said removed pixels in a moving object buffer to provide a sequence of stored frames containing only the moving objects in each frame of said sequence;

storing the relatively motionless background data remaining when said moving object data is removed in each frame in a single widened composite background buffer, said stored relatively motionless background data representing substantially the entire background data in said sequence of frames.

11. A method in accordance with claim 3 wherein the number of frames in which said moving object data is composited into the combined background data depends on how often the moving object moves over said sequence of frames.

12. A method in accordance with claim 11 wherein the moving object data for objects which move relatively often over said sequence of frames are composited into said combined background data more frequently than the moving object data for objects which move relatively less often.

13. A method in accordance with claim 4 wherein the combined background data are transmitted once for a sequence of frames, moving object data for objects which move relatively often over the sequence of frames are transmitted relatively frequently for the sequence of frames, and moving object data for objects which move less often over the sequence of frames are transmitted less frequently for the sequence of frames.

14. A method in accordance with claim 13 wherein moving object data for objects which move the most frequently over the sequence of frames are transmitted at each frame of the sequence and moving object data for objects which do not move the most frequently over the sequence of frames are transmitted more than once but less frequently than at each frame.