Three-dimension picture coding system

• US 5,696,551 A
• Filed: 09/27/1996
• Issued: 12/09/1997
• Est. Priority Date: 09/27/1995
• Status: Expired due to Term

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Katto, Jiro
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): DIN, LUANNE PAN
• Application Number:
  US08/720,378
• Time in Patent Office: 438 Days
• Field of Search: 348/42, 348/43, 348/421, 348/699, 348/384, 348/47, 382/162, 382/164, 382/167, 382/236, 382/246
• US Class Current: 348/43
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: H04N 11/046 DPCM
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects

Abstract

In order to improve coding efficiency, a segmentation circuit performs segmentation of an input picture by clustering referring to distance data calculated from pixel values of the input data, statistic information and a disparity compensated predictive picture, a statistic information calculator calculates an average and dispersion of pixel values of each segment according to pixel values of the input data and segmentation data from the segmentation circuit, a disparity calculator calculates disparity vectors of a horizontal dimension for minimizing error of the disparity compensated picture from pixel values of a reference picture, those of the input picture and the segmentation data, and a disparity compensating predictor generates the disparity compensated predictive picture.

Claims

1. A disparity extracting unit having at least one disparity extractor, each of said at least one disparity extractor comprising a segmentation circuit, a statistic information calculator, a disparity calculator and a disparity compensating predictor;

wherein;

said segmentation circuit performs segmentation of an input picture having disparity to a reference picture, by clustering each pixel of said input picture into one of segments whereof initial coordinates are predetermined, referring to distance data calculated from pixel values of said each pixel, statistic information delivered from said statistic information calculator, and pixel values, corresponding to said each pixel, of a disparity compensated predictive picture generated by said disparity compensating predictor;

said statistic information calculator calculates, for every of said segments, said statistic information including an average and dispersion of color components of pixels clustered into each of said segments and a center of gravity of said each of said segments, from said pixel values of said each pixel and segmentation data for said each pixel delivered from said segmentation circuit;

said disparity calculator calculates a disparity vector of a horizontal dimension, for each of said segments, which minimizes differential values between said input picture and said disparity compensated predictive picture, from pixel values of said reference picture, pixel values of said input picture, and said segmentation data delivered from said segmentation circuit; and

said disparity compensating predictor generates said disparity compensated predictive picture for said input picture from pixel values of said reference picture, and disparity vectors supplied from said disparity calculator.

2. A three dimension picture coding system having at least one disparity extractor and at least two encoders, each of said at least one disparity extractor comprising a segmentation circuit, a statistic information calculator, a disparity calculator and a disparity compensating predictor;

wherein;

said segmentation circuit performs segmentation of an input picture having disparity to a reference picture, by clustering each pixel of said input picture into one of segments whereof initial coordinates are predetermined, referring to distance data calculated from pixel values of said each pixel, statistic information delivered from said statistic information calculator, and pixel values, corresponding to said each pixel, of a disparity compensated predictive picture generated by said disparity compensating predictor;

said statistic information calculator calculates, for every of said segments, said statistic information including an average and dispersion of color components of pixels clustered into each of said segments and a center of gravity of said each of said segments, from said pixel values of said each pixel and segmentation data for said each pixel delivered from said segmentation circuit;

said disparity calculator calculates a disparity vector of a horizontal dimension, for each of said segments, which minimizes differential values between said input picture and said disparity compensated predictive picture, from pixel values of said reference picture, pixel values of said input picture, and said segmentation data delivered from said segmentation circuit;

said disparity compensating predictor generates said disparity compensated predictive picture for said input picture from pixel values of said reference picture, and disparity vectors supplied from said disparity calculator;

one of said at least two encoders encodes said reference picture; and

each of the others of said at least two encoders encodes disparity vectors finally supplied from said disparity calculator of each of said at least one disparity extractor.

3. A three dimensional picture coding system recited in claim 2, further comprising:

at least one differentiator, each of said at least one differentiator outputting differential data between said reference picture and said disparity compensated predictive picture finally generated by said disparity compensating predictor of each of said at least one disparity extractor; and

at least one encoder, each of said at least one encoder encoding said differential data output from each of said at least one differentiator.

4. A three dimension picture coding system recited in claim 2, further comprising at least two picture enhancers:

one of said at least two picture enhancer outputting an enhanced picture to be used for said reference picture; and

each of the others of said at least two picture enhancer outputting an enhanced picture used for said input picture for each of said at least one disparity extractor.

5. A three dimension picture coding system recited in claim 2, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data; and

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder.

6. A three dimension picture coding system recited in claim 3, further comprising at least two picture enhancers:

one of said at least two picture enhancer outputting an enhanced picture to be used for said reference picture; and

each of the others of said at least two picture enhancer outputting an enhanced picture used for said input picture for each of said at least one disparity extractor.

7. A three dimension picture coding system recited in claim 6, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data;

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder;

at least one third decoder, each of said at least one third decoder decoding output of each of said at least one encoder; and

at least one adder, each of said at least one adder adding output of each of said at least one third decoder with output of corresponding each of said at least one disparity compensating predictors.

8. A three dimension picture coding system recited in claim 3, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data;

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder;

at least one third decoder, each of said at least one third decoder decoding output of each of said at least one encoder; and

at least one adder, each of said at least one adder adding output of each of said at least one third decoder with output of corresponding each of said at least one disparity compensating predictors.

9. A three dimension picture coding system recited in claim 4, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data; and

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder.

10. A disparity extracting unit having at least one disparity extractor, each of said at least one disparity extractor comprising grouping means, a segmentation circuit, a statistic information calculator, a disparity calculator and a disparity compensating predictor;

wherein;

said grouping means group pixels of an input picture having disparity to a reference picture into units, each of said units having a form of rectangle including a form of a short line;

said segmentation circuit performs segmentation of said input picture, by clustering each of said units of said input picture into one of segments whereof initial coordinates are predetermined, referring to distance data calculated from pixel values of said each of said units, statistic information delivered from said statistic information calculator, and pixel values, corresponding to said each of said units, of a disparity compensated predictive picture generated by said disparity compensating predictor;

said statistic information calculator calculates, for every of said segments, said statistic information including an average and dispersion of color components of pixels of said units clustered into each of said segments and a center of gravity of said each of said segments, from said pixel values of said each of said units and segmentation data for said each of said units delivered from said segmentation circuit;

said disparity calculator calculates a disparity vector of a horizontal dimension, for each of said segments, which minimizes differential values between said input picture and said disparity compensated predictive picture, from pixel values of said reference picture, pixel values of said input picture, and said segmentation data delivered from said segmentation circuit; and

said disparity compensating predictor generates said disparity compensated predictive picture for said input picture from pixel values of said reference picture, and disparity vectors supplied from said disparity calculator.

11. A three dimension picture coding system having at least one disparity extractor and at least two encoders, each of said at least one disparity extractor comprising grouping means, a segmentation circuit, a statistic information calculator, a disparity calculator and a disparity compensating predictor;

wherein;

said grouping means group pixels of an input picture having disparity to a reference picture into units, each of said units having a form of rectangle including a form of a short line;

said segmentation circuit performs segmentation of said input picture, by clustering each of said units of said input picture into one of segments whereof initial coordinates are predetermined, referring to distance data calculated from pixel values of said each of said units, statistic information delivered from said statistic information calculator, and pixel values, corresponding to said each of said units, of a disparity compensated predictive picture generated by said disparity compensating predictor;

said statistic information calculator calculates, for every of said segments, said statistic information including an average and dispersion of color components of pixels of said units clustered into each of said segments and a center of gravity of said each of said segments, from said pixel values of said each of said units and segmentation data for said each of said units delivered from said segmentation circuit;

said disparity calculator calculates a disparity vector of a horizontal dimension, for each of said segments, which minimizes differential values between said input picture and said disparity compensated predictive picture, from pixel values of said reference picture, pixel values of said input picture, and said segmentation data delivered from said segmentation circuit;

said disparity compensating predictor generates said disparity compensated predictive picture for said input picture from pixel values of said reference picture, and disparity vectors supplied from said disparity calculator;

one of said at least two encoders encodes said reference picture; and

each of the others of said at least two encoders encodes disparity vectors finally supplied from said disparity calculator of each of said at least one disparity extractor.

12. A three dimension picture coding system recited in claim 11, further comprising:

at least one differentiator, each of said at least one differentiator outputting differential data between said reference picture and said disparity compensated predictive picture finally generated by said disparity compensating predictor of each of said at least one disparity extractor; and

at least one encoder, each of said at least one encoder encoding said differential data output from each of said at least one differentiator.

13. A three dimension picture coding system recited in claim 11, further comprising at least two picture enhancers:

one of said at least two picture enhancer outputting an enhanced picture to be used for said reference picture; and

each of the others of said at least two picture enhancer outputting an enhanced picture used for said input picture for each of said at least one disparity extractor.

14. A three dimension picture coding system recited in claim 11, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data; and

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder.

15. A three dimension picture coding system recited in claim 12, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data;

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder;

at least one third decoder, each of said at least one third decoder decoding output of each of said at least one encoder; and

at least one adder, each of said at least one adder adding output of each of said at least one third decoder with output of corresponding each of said at least one disparity compensating predictors.

16. A three dimension picture coding system recited in claim 13, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data; and

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder.

17. A picture enhancer comprising a segmentation circuit, a first statistic information calculator, at least one second statistic information calculators, a focus discriminator and an enhanced picture reproducer;

wherein;

said segmentation circuit performs segmentation of one of at least two input pictures, each of said at least two input pictures differently focused with each other, by clustering each pixel of said one of said at least two input picture into one of segments whereof initial coordinates are predetermined, referring to distance data calculated from pixel values of said each pixel, and statistic information delivered from one of said first statistic information calculator;

said first statistic information calculator calculates, for every of said segments, said statistic information of said one of said at least two input pictures including an average and dispersion of color components of pixels clustered into each of said segments and a center of gravity of said each of said segments, from said pixel values of said each pixel and segmentation data for said each pixel delivered from said segmentation circuit;

each of said at least one second statistic information calculator calculates, for every of said segments, said statistic information of each of the others of said one of said at least two input pictures including an average and dispersion of color components of pixels clustered into each of said segments, from said pixel values of said each pixel and segmentation data for said each pixel delivered from said segmentation circuit;

said focus discriminator selects, for each segment determined by said segmentation circuit, one of said at least two input pictures which has a focused segment giving a highest value of clearness for said each segment, calculating said clearness from statistic information concerning to said at least two input pictures delivered from said first statistic information calculator and said at least one second statistic information calculator; and

said enhanced picture reproducer reproduces a clearness enhanced picture with focused segments of said at least two input pictures, each of said focused segments selected by said focus discriminator for each segment determined by said segmentation circuit.

18. A three dimension picture coding system recited in claim 17, further comprising at least two picture enhancers:

one of said at least two picture enhancer outputting an enhanced picture to be used for said reference picture; and

each of the others of said at least two picture enhancer outputting an enhanced picture used for said input picture for each of said at least one disparity extractor.

19. A three dimension picture coding system recited in claim 18, further comprising:

a first decoder for decoding output of said one of said at least two encoders for generating a reference picture;

at least one second decoder, each of said at least one second decoder decoding output of said each of the other of said at least two encoders for generating disparity data;

at least one disparity compensating predictors, each of said at least one disparity compensating predictors representing a disparity compensated predictive picture from said reference picture and said disparity data generated by each of said at least one second decoder;

at least one third decoder, each of said at least one third decoder decoding output of each of said at least one encoder; and

at least one adder, each of said at least one adder adding output of each of said at least one third decoder with output of corresponding each of said at least one disparity compensating predictors.