Directed interpolation and data post-processing

• US 9,729,899 B2
• Filed: 04/20/2010
• Issued: 08/08/2017
• Est. Priority Date: 04/20/2009
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Tourapis, Alexandros, Pahalawatta, Peshala V., Stec, Kevin J., Leontaris, Athanasios
• Primary Examiner(s): Werner, David N
• Application Number:
  US13/255,308
• Publication Number:
  US 20120026288A1
• Time in Patent Office: 2,667 Days
• Field of Search: 348 43, 37524001-20403, 37520408, 37520421-20428
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/172 image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/865 with detection of the former encoding block subdivision in decompressed video
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/895 : in combination with error concealment
• CPC Class Codes: H04N 21/4347 : Demultiplexing of several video streams
• CPC Class Codes: H04N 7/0102 : involving the resampling of the incoming video signal
• CPC Class Codes: H04N 7/0117 : involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407
• CPC Class Codes: H04N 7/0135 : involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59

Abstract

An encoding device evaluates a plurality of processing and/or post-processing algorithms and/or methods to be applied to a video stream, and signals a selected method, algorithm, class or category of methods/algorithms either in an encoded bitstream or as side information related to the encoded bitstream. A decoding device or post-processor utilizes the signaled algorithm or selects an algorithm/method based on the signaled method or algorithm. The selection is based, for example, on availability of the algorithm/method at the decoder/post-processor and/or cost of implementation. The video stream may comprise, for example, downsampled multiplexed stereoscopic images and the selected algorithm may include any of upconversion and/or error correction techniques that contribute to a restoration of the downsampled images.

Claims

1. A method for receiving an input video stream and encoding an output video stream with a signal indicative of a processing technique to be utilized on the output video stream, the method comprising:

receiving an input video stream;

converting the format of the input video stream to comprise a multiplexed stream of two downsampled stereoscopic views of the input video stream, wherein the multiplexed stream comprises a checkerboard arrangement;

selecting non-predefined regions within the multiplexed stream wherein the selected regions are based upon a desired characteristic, said desired characteristic further comprising detected motion within the region and texture within the region;

encoding spatial-temporal metadata associated with the non-predefined regions, the metadata characterizing the spatial-temporal characteristics of the non-predefined regions;

filtering each selected region with a plurality of filters;

selecting a desired filter for each selected region based upon a selection criterion, said selection criterion further comprise at least one of motion detection for possible temporal filtering and texture analysis for spatial filtering;

encoding filter metadata regarding the selected filter for the non-predefined regions; and

embedding a signal, the signal based on the spatial-temporal metadata and filter metadata, into the output video stream, the signal identifies at least one of at least part of a decoding process to be applied to the output video stream.

2. The method according to claim 1, wherein the selected region has a desired characteristic comprising at least one of texture, contour, edge, contrast, dynamic range, motion within the region and brightness of the region.

3. The method according to claim 1, wherein said embedding further comprises embedding information about an overlap of at least one adaptively selected region with at least another region.

4. The method according to claim 1, wherein the processing technique comprises resolution conversion comprising at least one of spatial resolution conversion and temporal resolution conversion.

5. The method according to claim 1, wherein the processing technique comprises scaling comprising at least one of de-interlacing, temporal interpolation, and spatial interpolation.

6. The method according to claim 1, wherein the processing technique comprises error concealment comprising a prediction of motion of a region in error.

7. The method according to claim 1, wherein the views comprise at least one of stereographic views, two different images, a 2D image and depth information, multiple views of a 2D scene, and multiple views of a 3D scene.

8. The method according to claim 1, wherein the views are compressed and multiplexed within the data stream in a standardized motion picture format capable of single view video streams.

9. The method according to claim 8, wherein compression of the views further comprises at least one of a filtering and decimation of the views.

10. The method according to claim 8, wherein the downsampling comprises at least one of horizontal, vertical, and quincunx sampling.

11. The method according to claim 1, further comprising the step of selecting at least one processing technique based on the indicative signal and applying the selected processing technique to the region of the image or video scene;

and

wherein the step of selecting is performed by a decoder configured to decode the data stream by applying the selected processing technique to the region.

12. The method according to claim 1, wherein the processing technique comprises a series of processing techniques comprising at least one of an interpolation process, filtering, and a de-blocking process.

13. The method according to claim 1, wherein the signal comprises a signal representing a class or type of processing techniques, and the method further comprises the steps of selecting at least one processing technique from the signaled class or type and applying the selected technique(s) to the region.

14. A post-processing device, comprising:

a video input configured to receive a decoded video stream, the video stream comprising a plurality of video scenes, wherein further the video stream comprises a multiplexed stream of two downsampled stereoscopic views of the input video stream, wherein the multiplexed stream comprises a checkerboard arrangement;

an instruction mechanism configured to one of accept a directive instruction from a side information channel of the video stream and identify a directive instruction from the decoded video stream;

a post-processor configured to post-process the decoded video stream based on the directive instruction; and

a video output configured to output the post-processed video stream,

wherein the directive instruction comprises spatial-temporal metadata associated with at least one adaptively selected region in the video of arbitrary size, comprising information about at least the shape and size, detected motion and texture of the adaptively selected region, and filter metadata associated with a selected filter applied to at least one adaptively selected regions, the selected filter being selected among a plurality of filters wherein the selection is based upon a selection criterion, said selection criterion further comprise at least one of motion detection for possible temporal filtering and texture analysis for spatial filtering; and

further wherein the decoded checkerboard multiplexed stream is processed back into a full resolution image.

15. The post-processing device according to claim 14, wherein the directive instruction comprises different parameters to be applied to as a processing method for processing each region.

16. The post-processing device according to claim 14, wherein the post-processing device is integrated into at least one of a display, a set-top box, and a media player.

17. The post-processing device according to claim 14, wherein the directive instruction identifies regions of different texture qualities and processing methods specifically for each texture quality.

18. The post-processing device according to claim 14, wherein the directive instruction comprises a filter identification selected after testing a plurality of filters on the decoded video stream when creating an encoded bitstream.

19. The post-processing device according to claim 14, wherein the directive instruction comprises a filter identification comprising an identification of an entire category of filters, and the post-processor is further configured to select a filter from the category of filters and utilize the selected filter to post-process the decoded video stream.

20. The post-processing device according to claim 19, wherein the post-processor is further configured to select a filter having a property closest to a property of the category of filters and utilize the selected filter as at least part of the post-process.

21. The post-processing device according to claim 18, wherein the post-processor is further configured to select a filter known to have a property that is at least one of close to or better than the filter identified by the filter identification.

22. The post-processing device according to claim 14, further comprising a post-process analyzer configured to analyze at least one of the decoded video stream and the video stream prior to decoding, and, based on the analysis, direct at least part of at least one of a decoding process and the post processor.