UPSAMPLING AND DOWNSAMPLING OF MOTION MAPS AND OTHER AUXILIARY MAPS IN A TIERED SIGNAL QUALITY HIERARCHY

• US 20130294514A1
• Filed: 11/23/2011
• Published: 11/07/2013
• Est. Priority Date: 11/10/2011
• Status: Active Grant

• Current Assignee: Nova International Limited
• Original Assignee: Guido Meardi, Luca Rossato
• Inventors: Rossato, Luca, Meardi, Guido
• Granted Patent: US 9,300,980 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/192 the adaptation method, adaptation tool or adaptation type being iterative or recursive
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

Certain configurations herein include changing the resolution of an auxiliary map (e.g., a motion map, a z-map, etc.) at a first level of quality to obtain an auxiliary map at a second level of quality. For example, changing the resolution can include receiving a respective auxiliary map of one or more vectors at one or more lower levels of quality and progressively refining, via novel operations, the auxiliary map to higher or lower levels of quality in a hierarchy.

Claims

1. A method of generating an auxiliary map at a second level of quality in a hierarchy based on an auxiliary map at a first level of quality and a signal encoded at multiple different levels of quality in a hierarchy, the method comprising:

selecting an element from a rendition of the auxiliary map at the first level of quality;

subdividing the selected element at the first level of quality into multiple sub-elements, the multiple sub-elements being at the second level of quality in the hierarchy for the auxiliary map, the second level of quality higher than the first level of quality; and

based at least in part on auxiliary information assigned to at least one element in the auxiliary map at the first level of quality, generating respective auxiliary information for each of the sub-elements at the second level of quality of the auxiliary map.

2. The method as in claim 1, wherein the second level of quality is higher than the first level of quality, the method comprising:

selecting multiple elements from a rendition of the auxiliary map at the first level of quality;

aggregating the multiple elements at the first level of quality into a parent element, the parent element being at a third level of quality in the hierarchy for the auxiliary map, the third level of quality lower than the first level of quality; and

based at least in part on auxiliary information assigned to at least one element in the auxiliary map at the first level of quality, generating respective auxiliary information for the parent element at the third level of quality of the auxiliary map.

3. The method as in claim 1, wherein the auxiliary map at the second level of quality is also an auxiliary map with respect to the signal at the second level of quality, the method further comprising:

creating a hierarchy of auxiliary maps at multiple different levels of quality, each auxiliary map including auxiliary information for the signal at a corresponding level of quality.

4. The method as in claim 3, wherein the auxiliary information supports a process of reconstructing the signal based on reference image information at a respective level of quality.

5. The method as in claim 1, wherein the auxiliary map is a motion map including motion vectors, the motion vectors indicating a relationship from one element to another.

6. The method as in claim 1, wherein the auxiliary map is a z-map;

and

wherein the auxiliary information includes auxiliary elements indicating a depth of elements of the signal to which the auxiliary map refers at a corresponding level of quality.

7. The method as in claim 1, wherein the auxiliary map is a class map and auxiliary elements include information related to a class assigned to each element of the signal to which the auxiliary map pertains.

8. The method as in claim 1, wherein the auxiliary information associated to each plane element of the auxiliary map includes, along with auxiliary vectors, meta-information relative to the contents of auxiliary vectors;

and

the meta-information associated to each plane element of the auxiliary map includes statistical information with respect to the auxiliary vectors.

9. The method as in claim 5, wherein motion vector coordinates are expressed using a polar coordinate system.

10. The method as in claim 1 further comprising:

identifying a signal element corresponding to a given element in the auxiliary map;

identifying a set of elements in the signal at the first level of quality that are contiguous with the identified signal element at the first level of quality;

analyzing auxiliary vectors associated with the identified set of signal elements; and

generating a respective auxiliary vector for a particular sub-element at the second level of quality in the auxiliary map depending on auxiliary vectors associated with the identified set of elements.

11. The method as in claim 10, wherein the respective auxiliary vector at the second level of quality is obtained by applying a linear filter to the auxiliary vectors associated with the identified set of elements at the first level of quality.

12. The method as in claim 1, wherein responsive to detecting that a majority of elements at the first level of quality bordering a particular sub-element of a given element at the first level of quality are assigned a substantially same auxiliary vector value within a range, assigning a calculated auxiliary vector value to a particular sub-element of the multiple sub-elements, the calculated auxiliary vector value derived from a weighted average of the values of auxiliary vectors of neighboring elements, weights of which depend on a similarity of each auxiliary vector with respect to the substantially same auxiliary vector.

13. The method as in claim 1 further comprising:

producing the auxiliary map based at least in part on settings information of a current image in the signal, the auxiliary map pertaining to the current image.

14. The method as in claim 1 further comprising:

producing the auxiliary map based at least in part on settings information of a reference image in the signal, the auxiliary map pertaining to a current image in the signal that is derived at least in part based on the reference image.

15. The method as in claim 14 further comprising:

leveraging settings information from the reference image by using a higher resolution rendition of the reference image adapted by means of a lower resolution motion map.

16. The method as in claim 1, wherein leveraging settings information includes:

identifying color information for sub-elements in the reference image at the second level of quality based on at least one auxiliary vector associated with the selected element at the first level of quality;

utilizing the color information to generate respective auxiliary vectors for the sub-elements, values of the respective auxiliary vectors following color transitions as indicated by the color information.

17. The method as in claim 1 further comprising:

leveraging image settings information of either the current image or a reference image in the context of a tier-based hierarchy of auxiliary maps that refer to tier-based hierarchy of current images and reference images.

18. The method as in claim 1 further comprising:

producing an auxiliary map at a first level of quality by applying to the contents of auxiliary vectors of the map at the first level of quality a filter characterized by a kernel obtained by multiplying a linear kernel with a kernel based on image settings.

19. The method as in claim 18, wherein the linear kernel is a higher-order filter selected from a group consisting of:

a bilinear filter, a bicubic filter, a Lanczos filter, a Mitchell-Netravali filter, and a Gaussian filter).

20. The method as in claim 1 further comprising:

applying, to the coordinates of auxiliary vectors of the auxiliary map at a first level of quality, a filter characterized by a kernel obtained by multiplying a non-linear kernel with a kernel that is a function of image settings.

21. The method as in claim 1 further comprising:

producing an auxiliary map by also leveraging meta-information relative to auxiliary vectors.

22. The method as in claim 21, wherein meta-information includes parameters that indicate the precision of each auxiliary vector, or the covariance information of each vector, or any suitable value representing the relative or absolute quality of the vector.

23. The method as in claim 1 further comprising:

leveraging image setting information and meta-information to produce a rendition of the auxiliary map at a second level of quality by applying, to the auxiliary vectors of the map at a first level of quality, a filter whose kernel is obtained by multiplying a linear kernel with a kernel dependent on image settings and then with a kernel that is a function of auxiliary vector meta-information.

24. The method as in claim 1 further comprising:

calculating meta-information for the auxiliary vectors of the auxiliary map at the new level of quality by filtering meta-information of the auxiliary map at the previous level of quality leveraging a kernel that is a function of the kernel used to produce auxiliary vector information.

25. The method as in claim 1 further comprising:

refining the auxiliary map via upsampling and generating a hierarchy of motion maps.

26. The method as in claim 1 further comprising:

decoding an auxiliary map, the decoding including;

using steps recited in claim 1 to upsample an auxiliary map, adding reconstruction data after each upsample operation to reconstruct a rendition of the auxiliary map at the higher level of quality.

27. The method as in claim 26 wherein:

the auxiliary map is a motion map.

28. The method as in claim 1 further comprising:

decoding the auxiliary map in accordance with a multi-level tier-based decoding protocol.

29. Computer-readable storage hardware having instructions stored thereon, the instructions, when carried out by a processing device, causing the processing device to perform operations of:

selecting an element from a rendition of an auxiliary map at a first level of quality;

subdividing the selected element at the first level of quality into multiple sub-elements, the multiple sub-elements being at a second level of quality in the hierarchy for the auxiliary map, the second level of quality higher than the first level of quality; and

based at least in part on auxiliary information assigned to at least one element in the auxiliary map at the first level of quality, generating respective auxiliary information for each of the sub-elements at the second level of quality of the auxiliary map.