Moving picture encoding system, moving picture encoding method, moving picture encoding program, moving picture decoding system, moving picture decoding method, moving picture decoding program, moving picture reencoding system, moving picture reencoding method, moving picture reencoding program

• US 10,218,995 B2
• Filed: 04/21/2015
• Issued: 02/26/2019
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: Advanced Coding Technologies LLC
• Original Assignee: JVC Kenwood Corporation
• Inventors: Sakazume, Satoru
• Primary Examiner(s): Senfi, Behrooz M
• Assistant Examiner(s): Picon-Feliciano, Ana
• Application Number:
  US14/692,138
• Publication Number:
  US 20150229942A1
• Time in Patent Office: 1,407 Days
• Field of Search: 37524012, 37524018, 375 E7243, 375 E7226
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Abstract

A first super-resolution enlarger (103) works on moving pictures input with a standard resolution, implementing a process for a super-resolution enlargement including information on frequency components in the spatial and temporal directions that has been potentially contained in the input moving pictures but unable to express to a sufficient degree by the standard resolution, and provides super-resolution enlarged signals, which are returned to the standard resolution at a first resolution converter (104). The super-resolution enlarged signals as returned to the standard resolution are encoded at a second encoder (107). A first encoder (102) encodes moving pictures input with the standard resolution, and a multiplexer (109) multiplexes a sequence of encoded bits from a first encoder (102), a sequence of encoded bits from the second encoder (107), and the like. The second encoder (107) employs local decoded signals in the first encoder (102) or processed signals thereof, as reference signals.

Claims

1. A moving picture encoding system comprising:

a first encoder configured to work on a subsequence of a sequence of moving pictures with a standard resolution to implement a first combination of processes for an encoding and a decoding to create a first sequence of encoded bits and a set of decoded pictures with the standard resolution;

a first super-resolution enlarger configured to work on the subsequence of the sequence of moving pictures with the standard resolution to implement an interpolation of pixels with a first enlargement to create a set of super-resolution enlarged pictures with a first resolution higher than the standard resolution;

a first resolution converter configured to work on the set of super-resolution enlarged pictures to implement a process for a first resolution conversion to create a set of super-resolution enlarged and converted pictures with a standard resolution;

a second super-resolution enlarger configured to acquire the set of decoded pictures with the standard resolution from the first encoder to work on the sequence of decoded pictures to implement an interpolation of pixels with a second enlargement to create a set of super-resolution enlarged decoded pictures with a second resolution higher than the standard resolution;

a second resolution converter configured to work on the set of super-resolution enlarged decoded pictures to implement a process for a second resolution conversion to create a set of super-resolution enlarged and converted decoded pictures with a standard resolution; and

a second encoder configured to;

have the set of super-resolution enlarged and converted pictures from the first resolution converter as a set of encoding target pictures, the set of decoded pictures from the first encoder as a set of first reference pictures, and the set of super-resolution enlarged and converted decoded pictures from the second resolution converter as a set of second reference pictures,

select one of the set of first reference pictures and the set of second reference pictures to create reference picture selection information to identify the set of selected reference pictures to implement a second process for encoding to create a second sequence of encoded bits based on the set of encoding target pictures and the set of selected reference pictures, and

implement a third process for encoding for the reference picture selection information to create a sequence of encoded bits of the reference picture selection information,

wherein the set of encoding target pictures, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.

5. The moving picture encoding system according to claim 1, wherein the first super-resolution enlarger comprises:

a positioner configured to work in a processing for super-resolution enlargement, on a combination of a base picture constituting a base therein and one or more observation pictures to be based on, to make a positioning to pixel positions of a desirable high resolution, for an interval-unequal sampling to create nonhomogeneous high resolution pictures;

an interpolator configured to work on nonhomogeneous high resolution pictures created at the positioner, to implement a process for a prescribed nonhomogeneous interpolation to create interpolated pictures with a desirable high resolution;

an estimated picture creator configured to acquire interpolated pictures created at the interpolator, to implement thereon a process for a prescribed reconstruction to create estimated pictures with a desirable resolution; and

a repetition determiner configured to acquire a nonhomogeneous high resolution picture from the positioner and a nonhomogeneous estimated picture from the estimated picture creator, for employment of the acquired pictures to follow a prescribed determination method to determine whether or not a repetition of the processing for super-resolution enlargement is necessary, and work depending on a result thereof, to operate as the repetition is necessary, to provide the interpolator with information for a control to continue the processing, and operate as the repetition is unnecessary, to provide the estimated picture creator with information for a control to output a set of estimated pictures with a high resolution after super-resolution enlargement.

6. The moving picture encoding system according to claim 1, further comprising

a multiplexer configured to operate complying with a prescribed syntax structure to multiplex the first sequence of encoded bits from the first encoder, the second sequence of encoded bits from the second encoder, and information on encoding parameters of respective types used in the encoding at the first encoder and the encoding at the second encoder.

7. The moving picture encoding system according to claim 1, further comprising:

a first accumulation buffer configured to work in a course before the first encoder and the first super-resolution enlarger, to accumulate subsequences of the sequence of moving pictures with the standard resolution;

a second accumulation buffer configured to work in a course between the first encoder and the second encoder, to accumulate sets of decoded pictures from the first encoder;

an accumulation controller configured to work for detections of buffer accumulation amounts at the first accumulation buffer, the second accumulation buffer, the first encoder, and the second encoder, to control the buffer accumulation amounts; and

a code rate controller configured to work on bases of the detections of buffer accumulation amounts at the accumulation controller, to control code rates at the first encoder and the second encoder.

10. The moving picture encoding system according to claim 1, wherein the second encoder divides an encoding target picture into a plurality of regions to select one of the set of first reference pictures and the set of second reference pictures for each divided region.

2. A moving picture decoding system comprising:

a demultiplexer configured to work on a sequence of input encoded bits to implement a process for a prescribed demultiplexing to output at least a first and a second sequence of encoded bits;

a first decoder configured to acquire the first sequence of encoded bits obtained with a standard resolution at the demultiplexer to implement thereon a process for a prescribed first decoding to create a sequence of decoded pictures with a standard resolution;

a first super-resolution enlarger configured to acquire the sequence of decoded pictures created with a standard resolution at the first decoder to work on the sequence of decoded pictures to implement an interpolation of pixels with a first enlargement to create a sequence of super-resolution enlarged decoded pictures with a first resolution higher than a standard resolution;

a first resolution converter configured to acquire the sequence of super-resolution enlarged decoded pictures created at the first super-resolution enlarger to work on the sequence of super-resolution enlarged decoded pictures to implement a process for a prescribed resolution conversion to create a sequence of super-resolution decoded pictures with a standard resolution;

a second decoder configured to acquire the second sequence of encoded bits obtained with a standard resolution at the demultiplexer as a set of decoding targets, the sequence of decoded pictures created with the standard resolution at the first decoder as a set of first reference pictures, and the sequence of super-resolution decoded pictures created with the standard resolution at the first resolution converter as a set of second reference pictures, and select one of the set of first reference pictures and the set of second reference pictures based on reference picture selection information to implement a combination of processes for a prescribed prediction and a prescribed second decoding being a decoding with an extension of the standard resolution, to create a sequence of super-resolution pictures decoded with the standard resolution based on the set of decoding targets and the set of selected reference pictures; and

a second resolution converter configured to acquire the sequence of decoded pictures with the standard resolution from the first decoder to work on the sequence of decoded pictures to implement an interpolation of pixels with the second enlargement to create a sequence of enlarged decoded pictures with a high resolution as a second resolution higher than the standard resolution,

wherein the set of decoding targets, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.

11. The moving picture decoding system according to claim 2, wherein the second decoder divides a decoding target into a plurality of regions to select one of the set of first reference pictures and the set of second reference pictures for each divided region.

3. A moving picture decoding method comprising:

a step of implementing a process for a prescribed demultiplexing on a sequence of input encoded bits, outputting at least a first and a second sequence of encoded bits;

a step of acquiring the first sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing, implementing thereon a process for a prescribed first decoding, creating a sequence of decoded pictures with the standard resolution;

a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding, working on the sequence of decoded pictures to implement an interpolation of pixels with a first enlargement, creating a sequence of super-resolution enlarged decoded pictures with a first resolution higher than the standard resolution;

a step of acquiring the sequence of super-resolution enlarged decoded pictures created through the process for the prescribed super-resolution enlargement working on the sequence of super-resolution enlarged decoded pictures to implement a process for a first resolution conversion, creating a sequence of super-resolution decoded pictures with a standard resolution;

a step of acquiring the second sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing as a set of decoding targets, the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding as a set of first reference pictures, and the sequence of super-resolution decoded pictures created with the standard resolution through the process for the prescribed resolution conversion as a set of second reference pictures, and selecting one of the set of first reference pictures and the set of second reference pictures based on reference picture selection information to implement a combination of processes for a prescribed prediction and a prescribed second decoding being a decoding with an extension of the standard resolution, based on the set of decoding targets and the set of selected reference pictures to create a sequence of super-resolution pictures decoded with the standard resolution; and

a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding working on the sequence of decoded pictures to implement an interpolation of pixels with the second enlargement, creating a sequence of enlarged decoded pictures with a high resolution as a second resolution higher than the standard resolution,

wherein the set of decoding targets, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.

4. A recording medium storing a moving picture decoding program comprising a non-transitory computer-readable medium configured to have a computer execute:

a step of implementing a process for a prescribed demultiplexing on a sequence of input encoded bits, outputting at least a first and a second sequence of encoded bits;

a step of acquiring the first sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing, implementing thereon a process for a prescribed first decoding, creating a sequence of decoded pictures with the standard resolution;

a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding, working on the sequence of decoded pictures to implement an interpolation of pixels with a first enlargement, creating a sequence of super-resolution enlarged decoded pictures with a first resolution higher than the standard resolution;

a step of acquiring the sequence of super-resolution enlarged decoded pictures created through the process for the prescribed super-resolution enlargement working on the sequence of super-resolution enlarged decoded pictures to implement a process for a first resolution conversion, creating a sequence of super-resolution decoded pictures with a standard resolution;

a step of acquiring the second sequence of encoded bits obtained with a standard resolution through the process for the prescribed demultiplexing as a set of decoding targets, the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding as a set of first reference pictures, and the sequence of super-resolution decoded pictures created with the standard resolution through the process for the prescribed resolution conversion as a set of second reference pictures, and selecting one of the set of first reference pictures and the set of second reference pictures based on reference picture selection information to implement a combination of processes for a prescribed prediction and a prescribed second decoding being a decoding with an extension of the standard resolution, based on the set of decoding targets and the set of selected reference pictures to create a sequence of super-resolution pictures decoded with the standard resolution; and

a step of acquiring the sequence of decoded pictures created with the standard resolution through the process for the prescribed first decoding, working on the sequence of decoded pictures to implement an interpolation of pixels with the second enlargement creating a sequence of enlarged decoded pictures with a high resolution as a second resolution higher than the standard resolution,

wherein the first resolution is different from the second resolution,

wherein the standard resolution of the second sequence of encoded bits is the same as the standard resolution of the super-resolution decoded pictures,

and wherein the set of decoding targets, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.

8. A moving picture encoding method comprising:

a step of implementing a first combination of processes for an encoding and a decoding on a sequence of moving pictures with a standard resolution, creating a first sequence of encoded bits and a set of decoded pictures with the standard resolution;

a step of implementing an interpolation of pixels with a first enlargement on the sequence of moving pictures with the standard resolution, creating a set of super-resolution enlarged pictures with a first resolution higher than a standard resolution;

a step of implementing a process for a first resolution conversion on the set of super-resolution enlarged pictures, creating a set of super-resolution enlarged and converted pictures with a standard resolution;

a step of acquiring the set of decoded pictures with the standard resolution to work on the sequence of decoded pictures to implement an interpolation of pixels with a second enlargement to create a set of super-resolution enlarged decoded pictures with a second resolution higher than the standard resolution;

a step of working on the set of super-resolution enlarged decoded pictures to implement process for a second resolution conversion to create a set of super-resolution enlarged and converted decoded pictures with a standard resolution; and

a step of;

having the set of super-resolution enlarged and converted pictures as a set of encoding target pictures, the set of decoded pictures as a set of first reference pictures, and the set of super-resolution enlarged and converted decoded pictures as a set of second reference pictures,

selecting one of the set of first reference pictures and the set of second reference pictures to create reference picture selection information to identify the set of selected reference pictures to implement a second process for encoding to create a second sequence of encoded bits based on the set of encoding target pictures and the set of selected reference pictures, and

implementing a third process for encoding for the reference picture selection information to create a sequence of encoded bits of the reference picture selection information,

wherein the set of encoding target pictures, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.

9. A recording medium storing a moving picture encoding program comprising a non-transitory computer-readable medium configured to have a computer execute:

a step of implementing a first combination of processes for an encoding and a decoding on a sequence of moving pictures with a standard resolution, creating a first sequence of encoded bits and a set of decoded pictures with the standard resolution;

a step of implementing an interpolation of pixels with a first enlargement on the sequence of moving pictures with the standard resolution, creating a set of super-resolution enlarged pictures with a first resolution higher than the standard resolution;

a step of implementing process for a first resolution conversion on the set of super-resolution enlarged pictures, creating a set of super-resolution enlarged and converted pictures with a standard resolution;

a step of acquiring the set of decoded pictures with the standard resolution from the first encoder to work on the sequence of decoded pictures to implement an interpolation of pixels with a second enlargement to create a set of super-resolution enlarged decoded pictures with a second resolution higher than the standard resolution;

a step of working on the set of super-resolution enlarged decoded pictures to implement a process for a second resolution conversion to create a set of super-resolution enlarged and converted decoded pictures with a standard resolution; and

a step of;

having the set of super-resolution enlarged and converted pictures as a set of encoding target pictures, the set of decoded pictures as a set of first reference pictures, and the set of super-resolution enlarged and converted decoded pictures as a set of second reference pictures,

selecting one of the set of first reference pictures and the set of second reference pictures to create reference picture selection information to identify the set of selected reference pictures to implement a second process for encoding to create a second sequence of encoded bits based on the set of encoding target pictures and the set of selected reference pictures, and

implementing a third process for encoding for the reference picture selection information to create a sequence of encoded bits of the reference picture selection information,

wherein the set of encoding target pictures, the set of first reference pictures, and the set of second reference pictures have the same value in spatial resolution.