Upsampling and downsampling of motion maps and other auxiliary maps in a tiered signal quality hierarchy

• US 9,300,980 B2
• Filed: 11/23/2011
• Issued: 03/29/2016
• Est. Priority Date: 11/10/2011
• Status: Active Grant

• Current Assignee: Nova International Limited
• Original Assignee: Guido Meardi, Luca Rossato
• Inventors: Rossato, Luca, Meardi, Guido
• Primary Examiner(s): Werner, David N
• Application Number:
  US13/303,554
• Publication Number:
  US 20130294514A1
• Time in Patent Office: 1,588 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/192 the adaptation method, adaptation tool or adaptation type being iterative or recursive
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

Certain configurations herein include changing the resolution of an auxiliary map (e.g., a motion map, a z-map, etc.) at a first level of quality to obtain an auxiliary map at a second level of quality. For example, changing the resolution can include receiving a respective auxiliary map of one or more vectors at one or more lower levels of quality and progressively refining, via novel operations, the auxiliary map to higher or lower levels of quality in a hierarchy.

Claims

1. An image processing method of generating a signal encoded at multiple different levels of quality within a hierarchy, the image processing method comprising:

selecting an element from a rendition of an auxiliary map at a first level of quality in the hierarchy;

subdividing the selected element in the auxiliary map at the first level of quality into multiple sub-elements of an auxiliary map at a second level of quality in the hierarchy, the second level of quality higher than the first level of quality; and

based at least in part on auxiliary information assigned to at least one element in the auxiliary map at the first level of quality, generating in the auxiliary map at the second level of quality respective auxiliary information for each of the sub-elements at the second level of quality, the auxiliary information in the auxiliary map at the second level of quality for each of the sub-elements specifying motion at a higher resolution than a resolution of the sub-elements in the auxiliary map at the second level of quality.

2. The method as in claim 1 further comprising:

selecting multiple elements from a rendition of the auxiliary map at the first level of quality;

aggregating the multiple elements at the first level of quality into a parent element, the parent element being at a third level of quality in the hierarchy for the auxiliary map, the third level of quality lower than the first level of quality; and

based at least in part on auxiliary information assigned to at least one element in the auxiliary map at the first level of quality, generating respective auxiliary information for the parent element at the third level of quality of the auxiliary map.

3. The method as in claim 1, wherein the auxiliary map at the second level of quality is also an auxiliary map with respect to the signal at the second level of quality, the method further comprising:

creating a hierarchy of auxiliary maps at multiple different levels of quality, each auxiliary map including auxiliary information for the signal at a corresponding level of quality.

4. The method as in claim 3, wherein the auxiliary information supports a process of reconstructing the signal based on reference image information at a respective level of quality.

5. The method as in claim 1, wherein the auxiliary map is a motion map including motion vectors, the motion vectors indicating a relationship from one element to another.

6. The method as in claim 5, wherein motion vector coordinates are expressed using a polar coordinate system.

7. The method as in claim 1, wherein the auxiliary map is a z-map;

and

wherein the auxiliary information includes auxiliary elements indicating a depth of elements of the signal to which the auxiliary map refers at a corresponding level of quality.

8. The method as in claim 1, wherein the auxiliary map is a class map and auxiliary elements include information related to a class assigned to each element of the signal to which the auxiliary map pertains.

9. The method as in claim 1, wherein the auxiliary information associated to each plane element of the auxiliary map includes, along with auxiliary vectors, meta-information relative to the contents of auxiliary vectors;

and

the meta-information associated to each plane element of the auxiliary map includes statistical information with respect to the auxiliary vectors.

10. The method as in claim 1 further comprising:

identifying a signal element corresponding to a given element in the auxiliary map;

identifying a set of elements in the signal at the first level of quality that are contiguous with the identified signal element at the first level of quality;

analyzing auxiliary vectors associated with the identified set of signal elements; and

generating a respective auxiliary vector for a particular sub-element at the second level of quality in the auxiliary map depending on auxiliary vectors associated with the identified set of elements.

11. The method as in claim 10, wherein the respective auxiliary vector at the second level of quality is obtained by applying a linear filter to the auxiliary vectors associated with the identified set of elements at the first level of quality.

12. The method as in claim 1, wherein responsive to detecting that a majority of elements at the first level of quality bordering a particular sub-element of a given element at the first level of quality are assigned a respective auxiliary vector value within a range, assigning a calculated auxiliary vector value to a particular sub-element of the multiple sub-elements, the calculated auxiliary vector value derived from a weighted average of the values of auxiliary vectors of neighboring elements, weights of which depend on a similarity of each auxiliary vector with respect to the respective auxiliary vector.

13. The method as in claim 1 further comprising:

producing the auxiliary map based at least in part on settings information of a current image in the signal, the auxiliary map pertaining to the current image.

14. The method as in claim 1 further comprising:

producing the auxiliary map based at least in part on settings information of a reference image in the signal, the auxiliary map pertaining to a current image in the signal that is derived at least in part based on the reference image.

15. The method as in claim 1, wherein leveraging settings information includes:

identifying color information for sub-elements in a reference image at the second level of quality based on at least one auxiliary vector associated with the selected element at the first level of quality;

utilizing the color information to generate respective auxiliary vectors for the sub-elements, values of the respective auxiliary vectors following color transitions as indicated by the color information.

16. The method as in claim 1 further comprising:

generating the auxiliary map at the first level of quality by applying, to the contents of auxiliary vectors of the map at the first level of quality, a filter characterized by a kernel obtained by multiplying a linear kernel with a kernel based on image settings.

17. The method as in claim 16, wherein the linear kernel is a higher-order filter selected from a group consisting of:

a bilinear filter, a bicubic filter, a Lanczos filter, a Mitchell-Netravali filter, and a Gaussian filter.

18. The method as in claim 1 further comprising:

applying, to the coordinates of auxiliary vectors of the auxiliary map at a first level of quality, a filter characterized by a kernel obtained by multiplying a non-linear kernel with a kernel that is a function of image settings.

19. The method in claim 1 further comprising:

generating auxiliary map based at least in part on meta-information associated with auxiliary vectors.

20. The method as in claim 19, wherein meta-information includes a parameter selected from the group consisting of:

precision of each auxiliary vector,

covariance information of each vector, and

quality of the vector.

21. The method as in claim 1 further comprising:

leveraging image setting information and meta-information to produce a rendition of the auxiliary map at a second level of quality by applying, to the auxiliary vectors of the map at a first level of quality, a filter whose kernel is obtained by multiplying a linear kernel with a kernel dependent on image settings and then with a kernel that is a function of auxiliary vector meta-information.

22. The method as in claim 1 further comprising:

calculating meta-information for the auxiliary vectors of the auxiliary map at a third level of quality by filtering meta-information of the auxiliary map a lower level of quality leveraging a kernel that is a function of the kernel used to produce auxiliary vector information.

23. The method as in claim 1 further comprising:

refining the auxiliary map via upsampling and generating a hierarchy of motion maps.

24. The method as in claim 1 further comprising:

decoding an auxiliary map, the decoding including;

using steps recited in claim 1 to upsample an auxiliary map, adding reconstruction data after each upsample operation to reconstruct a rendition of the auxiliary map at the higher level of quality.

25. The method as in claim 24 wherein:

the auxiliary map is a motion map.

26. The method as in claim 1 further comprising:

decoding the auxiliary map in accordance with a multi-level tier-based decoding protocol.

27. The method as in claim 1 further comprising:

selecting a sub-element from the multiple sub-elements in the auxiliary map at the second level of quality;

subdividing the selected sub-element into multiple sub-sub elements in an auxiliary map at a third level of quality in the hierarchy, the third level of quality higher than the second level of quality; and

based at least in part on auxiliary information assigned to sub-elements in the auxiliary map at the second level of quality, generating in the auxiliary map at the third level of quality respective auxiliary information for each of the sub-sub elements at the third level of quality, the auxiliary information in the auxiliary map at the third level of quality for each of the sub-sub-elements specifying motion at a higher resolution than a resolution of the sub-sub-elements in the auxiliary map at the third level of quality.

28. The method as in claim 27, wherein the auxiliary information for each of the sub-elements in the auxiliary map of the second level of quality includes a respective motion vector.

29. The method as in claim 28, wherein the auxiliary information for each of the sub-sub-elements in the auxiliary map at the third level of quality includes a respective motion vector.

30. The method as in claim 1 further comprising:

downsampling a motion map associated with the signal to produce the auxiliary map at the first level of quality.

31. The method as in claim 30 further comprising:

producing residual data associated with the auxiliary maps at the different levels of quality, the residual data providing a way of refining settings of motion vectors in the auxiliary maps during upsampling by a decoder.

32. Computer-readable storage hardware having instructions stored thereon, the instructions, when carried out by an image processing device, causes the image processing device to perform operations of:

selecting an element from a rendition of an auxiliary map at a first level of quality in a hierarchy;

subdividing the selected element in the auxiliary map at the first level of quality into multiple sub-elements of an auxiliary map at a second level of quality in the hierarchy, the second level of quality higher than the first level of quality; and

based at least in part on auxiliary information assigned to at least one element in the auxiliary map at the first level of quality, generating in the auxiliary map at the second level of quality respective auxiliary information for each of the sub-elements at the second level of quality, the auxiliary information in the auxiliary map at the second level of quality for each of the sub-elements at a higher resolution than a resolution of the sub-elements specifying motion in the auxiliary map at the second level of quality.