Layered representation for image coding

• US 5,706,417 A
• Filed: 05/24/1995
• Issued: 01/06/1998
• Est. Priority Date: 05/27/1992
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Adelson, Edward H.
• Primary Examiner(s): Zimmerman, Mark K.
• Application Number:
  US08/449,091
• Time in Patent Office: 958 Days
• Field of Search: 395/128, 395/129, 395/135, 395/152, 395/164, 395/173, 395/174, 395/175, 395/960, 345/113, 345/114, 345/122, 382/236
• US Class Current: 345/640
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00

Abstract

A moving image representation format and encoding and decoding techniques. In the format, the image is represented as a series of layers, each layer comprising one or more maps containing information about the image. In decoding the information, the maps are combined in an order dictated by their ordinal depth in the image. Each layer comprises 1) an intensity map representing the intensity of the pixels in that layer, 2) an attenuation map by which the image intensities of the layers below it are multiplied while the intensity map corresponding to its layer is multiplied by its complement, 3) a delta map that describes the temporal derivative of the points in the corresponding intensity map, and 4) a velocity map which describes how all points in the layer are warped over time.

Claims

1. A apparatus for generating a digital representation of a two-dimensional image sequence recorded in a plurality of frames, comprising:

data processing means for encoding said image sequence, including

means for separating said image sequence into a plurality of layers, wherein each layer corresponds to at least a portion of at least one object in an image in said sequence, and if said at least one object includes an occulted portion, said layer also includes at least part of said occulted portion,

means for generating an intensity map corresponding to each layer, each intensity map at least partially describing said layer at a fixed point in time,

means for generating at least one attenuation map, each attenuation map corresponding to a layer and describing a manner of attenuating points in the intensity maps of layers underlying said corresponding layer, and

means for ordering said layers in terms of each layer'"'"'s depth in said image;

memory means for storing said intensity maps and said attenuation maps;

said data processing means further comprising means for generating a velocity map associated with a layer of said image, said velocity map at least partially describing how the intensity map and attenuation map, if any, of the associated layer is warped as a function of time;

said data processing means further comprising means for generating a delta map associated with a layer of said image, said delta map describing a temporal change of locations in the intensity map of the associated layer; and

said velocity map of said corresponding layer, if any, also describing how said delta map is transformed as a function of time.

2. An apparatus as set forth in claim 1 wherein said data processing means further comprises means for generating a depth map associated with a layer, said depth map describing the distance of points in said associated layer from a fixed plane parallel to said two-dimensional image sequence.

3. An apparatus as set forth in claim 2 wherein at least one of said various means for generating maps defines at least one of said maps as a continuous function of coordinates of said two dimensions, which function defines the values of a two dimensional array of points.

4. An apparatus as set forth in claim 3 wherein at least one of said various means for generating said maps defines at least one of said maps as a two-dimensional array of data points.

5. An apparatus as set forth in claim 2 wherein said data processing means further comprises:

means for generating a contrast change map associated with a layer of said image, said contrast change map describing a change in contrast of said intensity map of said associated layer as a function of time,

means for generating a blur map associated with a layer of said image, said blur map describing a space variant filter which describes blur in said intensity map of said associated layer, and

means for generating a surface orientation map associated with a layer of said image, said surface orientation map describing surface normals of said intensity map of said associated layer.

6. An apparatus as set forth in claim 5 wherein each of said various means for generating maps defines at least one of said maps as a continuous function of coordinates of said two dimensions, which function defines the values of two dimensional array of points.

7. An apparatus as set forth in claim 6 wherein each of said various means for generating maps defines at least one of said maps as a two-dimensional array of data points.

8. An apparatus as set forth in claim 7 wherein individual ones of said various maps have different dimensions.

9. An apparatus as set forth in claim 7 wherein individual ones of said various maps have different resolutions.

10. An apparatus as set forth in claim 9 wherein each of said various means for generating maps further comprise means for temporal median filtering said maps in order to generate a map of higher resolution than said frames.

11. An apparatus as set forth in claim 9 wherein each of said various means for generating maps further comprise means for performing a Laplacian pyramid combination on said maps in order to generate a map of higher resolution than said frames.

12. An apparatus as set forth in claim 7 wherein different ones of said velocity and contrast change maps are defined as a function of different frame rates.

13. An apparatus as set forth in claim 7 wherein each of said various means for generating maps further comprise data compression means for reducing the quantity of data stored in memory for each map.

14. An apparatus as set forth in claim 7 wherein at least one of said intensity maps at least partially defines an image larger than the image of any single frame of said image sequence.

15. An apparatus as set forth in claim 7 further comprising means for generating an intensity map and an attenuation map corresponding to a layer for an object which does not appear in the recorded image sequence, whereby an object may be added to said image sequence upon decoding.

16. An apparatus as set forth in claim 7 wherein said means for decoding further includes means for temporally interpolating a velocity map whereby a velocity of an object in said image sequence can be altered without altering the frame rate.

17. The apparatus of claim 1 wherein each of said various means for generating maps further comprises data compression means for reducing the quantity of data stored in said memory means for one or more maps.

18. An apparatus for encoding a first two-dimensional image sequence comprising at least one frame into a digital representation and decoding said digital representation into a second two-dimensional image sequence comprising at least one frame, said apparatus comprising:

data processing means for encoding said image sequence, including

means for separating said image sequence into a plurality of layers, wherein each layer corresponds to at least a portion of at least one object in an image in said sequence, and if said at least one object includes an occulted portion, said layer also includes at least part of said occulted portion,

means for generating an intensity data map corresponding to each layer, each data map describing at least partially said layer at a fixed point time in time,

means for generating at least one attenuation map, each attenuation map corresponding to a layer and describing a manner of attenuating points in the intensity maps of layers underlying said corresponding layer, and

means for ordering said layers in terms of each layer'"'"'s depth in said image;

memory means for storing said intensity maps and said attenuation maps;

decoding means for combining said various maps so as to generate said second two-dimensional image sequence based on said first two-dimensional image sequence;

said data processing means comprising means for generating a velocity map associated with a layer of said image, said velocity map at least partially describing how the intensity map and attenuation map, if any, of the corresponding layer is warped as a function of time;

said data processing means further comprising means for generating a delta map ciated with a layer of said image, said delta map describing a temporal change of the points in the intensity map of the corresponding layer; and

said velocity map of said corresponding layer, if any, also describing how said delta map is transformed as a function of time.

19. An apparatus as set forth in claim 18 wherein said decoding means comprises:

means for adding each delta map to the intensity map of the corresponding layer to produce a first sum,

means for warping each said first sum by the velocity map of the corresponding layer to produce a warped intensity map,

means for warping each attenuation map by the velocity map of the corresponding layer to produce a warped attenuation map,

means for multiplying each said warped intensity map by the complement of the warped attenuation map of the corresponding layer to produce a first product,

means for multiplying each warped intensity map by the warped attenuation map corresponding to every layer which overlies the layer corresponding to said warped intensity map to produce a second product for each layer for which there is an overlying layer, and

means for adding each said second product to the first product of the immediately overlying layer.

20. An apparatus as set forth in claim 19 wherein said first image sequence is of a first frame rate and wherein said apparatus further comprises means for interpolating said velocity and delta maps, whereby the frame rate of said second image sequence is different than said first frame rate.

21. The apparatus of claim 18 wherein each of said various means for generating maps further comprises data compression means for reducing the quantity of data stored in said memory means for one or more maps.

22. A method for encoding a first two-dimensional image sequence recorded in a plurality of frames into a digital representation and decoding said digital representation into a second image sequence, comprising the steps of:

separating said image into a plurality of layers, wherein each layer corresponds to at least a portion of at least one object in an image in said sequence, and if said at least one object includes an occulted portion, said layer also includes at least part of said occulted portion;

generating an intensity map corresponding to each layer, each intensity map at least partially describing said layer as a two-dimensional matrix of points at a fixed point in time;

ordering said layers in terms of each layer'"'"'s depth in said image;

generating at least one attenuation map, each attenuation map corresponding to a layer and describing a manner of attenuating points in the intensity maps of layers underlying the corresponding layer;

combining said maps to produce said second image sequence;

one or more of said steps of generating a map further comprising performing a data compression on data comprising one or more of the maps generated in said one or more steps in order to reduce the quantity of data in said one or more maps;

generating a velocity map associated with a layer of said image, said velocity map at least partially describing how the intensity map and attenuation map, if any, of the corresponding layer is transformed as a function of time; and

generating a delta map associated with a layer of said image, said delta map describing a temporal change of locations in the intensity map of the corresponding layer.

23. A method as set forth in claim 22 wherein said step of ordering said layers comprises generating a depth map.

24. A method as set forth in claim 22 wherein said step of combining said maps comprises:

generating a complement of each attenuation map,

multiplying each intensity map by the complement of the attenuation map of the corresponding layer to produce a first product,

multiplying each intensity map by the attenuation map of every layer overlying said layer of said intensity map to produce a plurality of second products, and

adding each said second product to the first product of the immediately overlying layer.

25. A method as set forth in claim 22 wherein said step of combining said maps further comprises:

adding each delta map to the intensity map of the corresponding layer to produce a first sum,

warping each said first sum by the velocity map of the corresponding layer to produce a warped intensity map,

warping each attenuation map by the velocity map of the corresponding layer to produce a warped attenuation map,

multiplying each said warped intensity map by the complement of the warped attenuation map of the corresponding layer to produce a first product,

multiplying each warped intensity map by the warped attenuation map corresponding to every layer which overlies the layer corresponding to said warped intensity map to produce a second product for each layer for which there is an overlying layer, and

adding each said second product to the first product of the immediately overlying layer.

26. A method as set forth in claim 25 further comprising the steps of:

generating a contrast change map associated with a layer of said image, said contrast change map describing a change in contrast of the intensity map of the corresponding layer as a function of time,

generating a blur map associated with a layer of said image, said blur map describing a space variant filter which describes blur in an intensity map of the corresponding layer, and

generating a surface orientation map associated with a layer of said image, said surface orientation map describing surface normals of an intensity map of the corresponding layer.

27. A method for encoding a first two-dimensional image sequence recorded in a plurality of frames into a digital representation and decoding said digital representation into a second image sequence, comprising the steps of:

separating said image into a plurality of layers, wherein each layer corresponds to at least a portion of at least one object in an image in said sequence, and if said at least one object includes an occulted portion, said layer also includes at least part of said occulted portion;

generating an intensity map corresponding to each layer, each intensity map at least partially describing said layer as a two-dimensional matrix of points at a fixed point in time;

ordering said layers in terms of each layer'"'"'s depth in said image;

generating at least one attenuation map, each attenuation map corresponding to a layer and describing a manner of attenuating points in the intensity maps of layers underlying the corresponding layer;

combining said maps to produce said second image sequence;

one or more of said steps of generating a map further comprising performing a data compression on data comprising one or more of the maps generated in said one or more steps in order to reduce the quantity of data in said one or more maps;

generating a velocity map associated with a layer of said image, said velocity map at least partially describing how the intensity map and attenuation map, if any, of the corresponding layer is transformed as a function of time; and

said first image sequence being of a first frame rate and the step of combining said maps further including interpolating said velocity maps, whereby the frame rate of said second image sequence is different than the frame rate of said first image sequence.