System for editing compressed image sequences

• US 6,104,441 A
• Filed: 04/29/1998
• Issued: 08/15/2000
• Est. Priority Date: 04/29/1998
• Status: Expired due to Term

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Company (HP Inc.)
• Inventors: Vasudev, Bhaskaran, Wee, Susie J.
• Primary Examiner(s): Peng, John K.
• Assistant Examiner(s): NATNAEL, PAULOS M
• Application Number:
  US09/069,596
• Time in Patent Office: 839 Days
• Field of Search: 348/400-403, 348/409-412, 348/415, 348/700, 345/328, 382/235, 386/4, 386/33, 386/52, 386/111, 386/112
• US Class Current: 375/240.15
• CPC Class Codes: G11B 2220/2512 Floppy disks
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/238 Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams hybrid fiber coaxial [HFC] networks for downstream channel allocation for video distribution H04L12/2801; flow control in packet networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip

Abstract

This disclosure provides for an image editing system that permits manipulation of compressed image formats without full decompression to the image domain. In particular, an editor manipulates discrete cosine transform ("DCT") coefficients for a dependent frame in a compressed image sequence to incorporate DCT representations of anchor frames upon which the dependent frame depends. Once converted, the frame can simply be reordered in the compressed domain without violating temporal dependencies, subject to bit rate matching. In one embodiment, an image sequence can be cut to leave remaining image frames which are then altered, so as to eliminate temporal dependencies upon eliminated frames. In another embodiment, an image sequence can be processed in the compressed domain for reverse-play. The editor provided by this disclosure permits real-time compressed domain editing applications, with reduced requirements for computational bandwidth.

Claims

1. A method of editing a compressed input image sequence and producing therefrom a compressed output image sequence, the input sequence including at least one group of images (that does not depend upon any other images for reconstruction from a compressed format), the group of images having at least one compressed dependent frame that contains data for reconstructing one image based upon data from a second image in the group of images, said method using a computing device and comprising:

receiving with the compressed input image sequence;

using the computing device to convert a dependent frame to a compressed data format that does not depend upon the second image without decompressing all images in the group of images to an image domain;

editing data using the computing device, by altering temporal order of an image represented by the dependent frame with respect to at least one other image in the sequence; and

using the computing device to produce the compressed output image sequence from the data which has been edited by the computing device.

2. A method according to claim 1, further comprising using the computing device to produce the compressed output image sequence in a format that is compliant with a Moving Pictures Expert Group ("MPEG") Standard.

3. A method according to claim 1, wherein using the computing device to convert the dependent frame includes:

determining discrete cosine transform ("DCT") coefficients for the dependent frame; and

modifying the DCT coefficients to incorporate image data corresponding to the second image, to yield modified DCT coefficients;

wherein the dependent frame is thereby modified so as to no longer depend upon data values from the second image for decompression.

4. A method according to claim 1, further comprising clipping first and second compressed input image sequences and splicing them together to form a third image sequence.

5. A method according to claim 4, wherein splicing includes:

clipping a tail frame of the first sequence by converting the tail frame to eliminate any temporal dependencies upon future frames in the first sequence;

clipping a lead frame of the second sequence by converting the lead frame to eliminate any temporal dependencies upon prior frames in the second sequence; and

splicing the first and second sequences together to form a third sequence, by ordering an image represented by the lead frame immediately after an image represented by the tail frame.

6. A method according to claim 1, further comprising using the computing device to edit the compressed input image sequence such that the compressed output image sequence represents a reverse-play-order of images represented by the input image sequence.

7. A method according to claim 1, wherein the compressed input image sequence is compliant with a Moving Pictures Expert Group ("MPEG") Standard, including independent ("I") frames, prediction ("P") frames and bidirectional ("B") frames, and wherein using the computing device to convert the dependent frame to a compressed data format includes:

using the computing device to perform a conversion upon the dependent frame, including one of

conversion of a P frame to an I frame by removing dependence of the P frame upon data from any other frame,

conversion of a B frame to a B frame having no dependence upon data from future frames, such that the B frame as converted only depends upon one or more prior frames in the sequence of data, and

conversion of a B frame to a B frame having no dependence upon data from prior frames, such that the B frame as converted only depends upon one or more future frames in the sequence of data.

8. A method according to claim 1, wherein using the computing device to convert the dependent frame includes converting the dependent frame without decompressing any image to an image domain.

9. An apparatus that converts an input sequence of compressed input images into an output sequence of compressed images, the input sequence including at least one group of images (that does not depend upon any other images for reconstruction from a compressed format), the group of images having at least one compressed dependent frame that contains data for reconstructing one image based upon data from a second image in the group of images, said apparatus comprising:

memory that receives the input sequence; and

an editor that receives user commands to edit the input sequence and that, responsive to user commands, alters the temporal order of at least one image from the compressed image sequence (including an image represented by the dependent frame) with respect to another image in the input sequence, the editor including one of firmware, hardware and software that

converts the dependent frame to a compressed data format that does not depend upon the second image, without decompressing all images in the group of images to an image domain,

alters the temporal order of at least one image in the sequence (including an image represented by the dependent frame), and

formats the compressed output image sequence to reflect the altered temporal order.

10. An apparatus according to claim 9, further comprising means for splicing a satellite feed of images with a second image sequence, to produce a third, composite image sequence.

11. An apparatus according to claim 9, further comprising means for cutting a first image sequence and splicing a remaining portion of the first image sequence to a second image sequence within a compressed domain.

12. An apparatus according to claim 9, further comprising means for formatting the compressed output image sequence to represent the input image sequence in reverse-play-order.

13. An apparatus, comprising:

means for receiving a compressed input image sequence;

means for manipulating the compressed input image sequence in a compressed domain, without decompressing any image in the input image sequence to a image domain, including one of means for cutting the compressed input image sequence and splicing remaining images of the compressed image sequence to a second image sequence and means for reversing play-order of images represented by the input image sequence; and

means for producing a compressed output image sequence.

14. An apparatus operative to receive compressed image data complying with a Moving Pictures Experts'"'"' Group (M.P.E.G.) Standard and including information representing a sequence of independent (I) frames, prediction (P) frames and bidirectional (B) frames, to edit temporal order of images represented by the sequence in response to a set of one or more commands, and to provide an output of image data that is compressed and compliant with a MPEG Standard, said apparatus comprising:

machine readable media; and

instructions stored on the machine readable media that instruct a machine to

receive data representing I, P and B frames,

perform, with respect to a particular frame selected in response to the set of one or more commands, a conversion including one of

(a) conversion of a P frame to an I frame by removing dependence of the P frame upon data from any other frame,

(b) conversion of a B frame to a B frame having no dependence upon data from future frames, such that the B frame as converted only depends upon one or more prior frames in the sequence of data, and

(c) conversion of a B frame to a B frame having no dependence upon data from prior frames, such that the B frame as converted only depends upon one or more future frames in the sequence of data, and

alter temporal order of at least one image represented by the sequence (including an image represented by the dependent frame) with respect to another image in response to the commands, and produce therefrom the output of compressed data,

wherein the conversion is automatically selected (in response to the set of one or more commands and in response to whether the particular frame is an I, P or B frame) to eliminate a temporal dependency between any frame representing the at least one image which is to be altered in temporal order with respect to another image.

15. An apparatus according to claim 14, wherein the particular frame is part of a group of pictures (GOP) and wherein the instructions instruct a machine to perform the conversion by:

converting the particular frame to an image domain; and

re-converting, once a particular image represented by the particular frame has been altered in temporal order with respect to another image, the particular image to one of an I frame and P frame and B frame format;

whereby the sequence of images has been edited to produce a MPEG compliant output without fully decompressing an entire GOP to an image domain.

16. An apparatus according to claim 14, wherein the instructions instruct a machine to perform the conversion in a transform-domain, without decompressing the particular frame to an image domain.

17. An apparatus according to claim 14, wherein the instructions instruct a machine to perform the conversion by directly altering motion vectors and discrete cosine transform coefficients, without performing an inverse discrete cosine transformation.

18. An apparatus according to claim 14, wherein said apparatus further provides capability to play a sequence of images backward and wherein:

the instructions instruct a machine to alter temporal order by reversing temporal dependencies of each of a plurality of frames selected from the group of I, P and B frames.

19. An apparatus according to claim 14, wherein said apparatus further provides capability to splice a first sequence of images with a second sequence of images, and wherein:

the instructions instruct a machine to clip a tail frame of the first sequence by converting the tail frame to eliminate any temporal dependencies upon future frames in the first sequence;

the instructions instruct a machine to clip a head frame of the second sequence by converting the head frame to eliminate any temporal dependencies upon prior frames in the second sequence; and

the instruction instruct a machine to splice the first and second sequences together to form a third sequence, in which an image represented by the head frame immediately follows an image represented by the tail frame when the third sequence is decompressed.

20. An apparatus according to claim 19, wherein the instructions further instruct a machine to change bit rates of the head frame.

21. An apparatus according to claim 19, wherein the instructions further instruct a machine to match bit rates of the head frame to the tail frame by further quantizing one of the head frame and the tail frame.

22. An apparatus according to claim 19, wherein the instructions further instruct a machine to match bit rates of the head frame to the tail frame by further converting one of (a) an I frame to a P frame and (b) a P frame to a B frame.