Compression and editing of movies by multi-image morphing

• US 6,285,794 B1
• Filed: 04/17/1998
• Issued: 09/04/2001
• Est. Priority Date: 04/17/1998
• Status: Expired due to Term

• Current Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wainer, Michael, Georgiev, Todor
• Primary Examiner(s): Chang, Jon
• Assistant Examiner(s): Johnson, Timothy M.
• Application Number:
  US09/062,474
• Time in Patent Office: 1,236 Days
• Field of Search: 382/107, 382/190, 382/203, 382/225, 382/232, 382/235, 382/243, 382/264, 382/308, 345/433, 345/436, 345/441, 345/434, 345/475, 348/580, 348/699, 348/700, 375/240.08
• US Class Current: 382/243
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00

Abstract

A computer-implemented method for representing a movie having a sequence of frames includes generating a pixel level frame to frame correspondence; representing the original frame sequence as a multi-image morph; and deriving from the multi-image morph a basis of feature vectors describing principal changes among frames and representing the movie in terms of the basis.

Claims

1. A method for representing a movie having an original frame sequence, comprising:

generating a pixel level frame to frame correspondence;

representing the original frame sequence as a multi-image morph, wherein representing the original frame sequence as a multi-image morph comprises representing each frame in the sequence as a point (a₁, a₂, . . . , a_n) in an n-dimensional affine morph space defined by a set of warping mappings W_i and associated color definitions f_i (1≦

i≦

n) , where each a_i is a number and each W_i and f_i is an operator on a planar image space, the coefficients a_i satisfying the constraint a₁+a₂+ . . . +a_n=1;

deriving from the multi-image morph a morph basis of feature vectors describing principal changes among frames; and

representing the movie in terms of the morph basis.

5. The method of claim 1, wherein the correspondence generating step processes the frames using an optical flow method.

6. The method of claim 1, wherein the deriving step uses a principal components method.

7. The method of claim 6, further comprising compensating for vibrations of movie frames represented in terms of the morph basis by discarding one or more high weight principal components.

8. The method of claim 1, wherein the movie has a first frame rate, further comprising resampling the movie represented in terms of the morph basis at a second frame rate.

9. The method of claim 8, wherein the movie is represented as a curve defined in an affine morph space and resampling the movie comprises calculating points on the curve at intervals defined by the second frame rate.

10. The method of claim 1, wherein the movie has a character with a gesture, further comprising editing the gesture.

11. The method of claims 3 or 1, further comprising compressing the movie by using a subset of the basis.

12. The method of claims 3 or 1, further comprising displaying the movie by performing a multi-image morphing.

13. The method of claim 1, further comprising:

selecting a compression level; and

using the compression level to limit the number of feature vectors in the morph basis.

15. The method of claim 1, further comprising representing the original frame sequence as a parametric curve fit to the points (a₁, a₂, . . . , a_n) in the morph space.

2. A computer program stored on a computer-readable medium for compressing a movie having an original frame sequence, the program comprising instructions to:

generate a pixel level frame to frame correspondence;

represent the original frame sequence as a multi-image morph, wherein instructions to represent the original frame sequence as a multi-image morph comprise instructions to represent each frame in the sequence as a point (a₁, a₂, . . . , a_n) in an n-dimensional affine morph space defined by a set of warping mappings W_i and associated color definitions f_i (1≦

i≦

n), where each a_i is a number and each W_i and f_i is an operator on a planar image space, the coefficients a_i satisfying the constraint a₁+a₂+ . . . +a_n=1;

derive from the multi-image morph a morph basis of feature vectors describing principal changes among frames; and

represent the movie in terms of the morph basis.

14. The method of claim 2, further comprising:

constructing a set of new basis frames from the basis of feature vectors, each new basis frame corresponding to one of the feature vectors in the basis of feature vectors, wherein the new basis frames are not in the original frame sequence.

16. The program of claims 4 or 2, wherein the correspondence generating instruction processes the frames using an optical flow method.

17. The program of claims 4 or 2, wherein the deriving instruction uses a principal components method.

18. The program of claim 17, further comprising instructions for compensating for vibrations of movie frames represented in terms of the morph basis by discarding one or more high weight principal components.

19. The program of claim 2, wherein the movie has a first frame rate, further comprising instructions for resampling the movie represented in terms of the morph basis at a second frame rate.

20. The program of claim 19, wherein the movie is represented as a curve defined in an affine morph space and resampling the movie comprises calculating points on the curve at intervals defined by the second frame rate.

21. The program of claims 4 or 2, wherein the movie has a character with a gesture, further comprising instructions for editing the gesture.

22. The program of claims 4 or 2, further comprising instructions to compress the movie by using a subset of the basis.

23. The program of claims 4 or 2, further comprising instructions to display the movie by performing a multi-image morphing.

24. The program of claim 2, further comprising instructions to:

select a compression level; and

use the compression level to limit the number of feature vectors in the morph basis.

25. The program of claim 2, further comprising instructions to:

construct a set of new basis frames from the basis of feature vectors, each new basis frame corresponding to one of the feature vectors in the basis of feature vectors, wherein the new basis frames are not in the original frame sequence.

26. The program of claim 2, further comprising instructions to represent the original frame sequence as a parametric curve fit to the points (a₁, a₂, . . . , a_n) in the morph space.

27. The program of claim 2 for generating a motion blur in the movie, the program further comprising instructions to:

calculate an average color to substitute for an original color for each point in a blurred frame in the movie, the average color being calculated by averaging colors taken from each basis frame along a velocity field generated by a tangent vector and the warping transformations at the blurred frame, the tangent vector being a vector tangent to a curve representing the movie in a morph space defined by the morph basis.

3. A method for representing a movie having an original frame sequence and for generating a motion blur in the movie, the method comprising:

generating a pixel level frame to frame correspondence;

representing the original frame sequence as a multi-image morph;

deriving from the multi-image morph a morph basis of feature vectors describing principal changes among frames;

representing the movie in terms of the morph basis; and

calculating an average color to substitute for an original color for each point in a blurred frame in the movie, the average color being calculated by averaging colors taken from each basis frame along a velocity field generated by a tangent vector and the warping transformations at the blurred frame, the tangent vector being a vector tangent to a curve representing the movie in a morph space defined by the morph basis.

4. A computer program stored on a computer-readable medium for compressing a movie having an original frame sequence and for generating a motion blur in the movie, the program comprising instructions to:

generate a pixel level frame to frame correspondence;

represent the original frame sequence as a multi-image morph;

derive from the multi-image morph a morph basis of feature vectors describing principal changes among frames;

represent the movie in terms of the morph basis; and

calculate an average color to substitute for an original color for each point in a blurred frame in the movie, the average color being calculated by averaging colors taken from each basis frame along a velocity field generated by a tangent vector and the warping transformations at the blurred frame, the tangent vector being a vector tangent to a curve representing the movie in a morph space defined by the morph basis.