Gain and offset correction for efficient stereoscopic coding and improved display

• US 6,111,596 A
• Filed: 10/15/1996
• Issued: 08/29/2000
• Est. Priority Date: 12/29/1995
• Status: Expired due to Term

• Current Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Kollarits, Richard Virgil, Haskell, Barin Geoffry, Puri, Atul
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Philippe, Gims S.
• Application Number:
  US08/732,822
• Time in Patent Office: 1,414 Days
• Field of Search: 348/43, 348/42, 348/47, 348/51, 348/229, 348/54, 348/48, 348/44, 348/52, 348/409, 348/46, 356/12, 382/154, 382/152, 382/160, 382/168, 382/169
• US Class Current: 348/42
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 13/133 Equalising the characteristics of different image components, e.g. their average brightness or colour balance
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/302 for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/597 : specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0077 : Colour aspects
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 : Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 : Synchronisation or controlling aspects

Abstract

A method for increasing compression efficiency of a stereoscopic image encoder employed for compressively encoding first and second signals representing respective first and second views of a stereoscopic image, comprises: providing a first histogram of the first view and a second histogram of the second view, the first and second histograms representing a frequency of occurrence of a characteristic component of the first and second views, respectively; comparing the first histogram of the first view with the second histogram of the second view and estimating the mismatch in gain, a, and mismatch in offset, b, between the first and second views; correcting the stereoscopic image by multiplying the gain a and the second signal to obtain a gain corrected second signal for the stereoscopic image and adding the offset b to the gain corrected second signal to obtain a gain and offset corrected second signal, the gain and offset corrected second signal being obtained prior to encoding to increase encoding efficiency. The quality of the resulting stereoscopic image formed from decoding the first signal and gain and offset corrected second signal is greatly improved.

Claims

1. An apparatus for correcting global mismatch in gain between first and second views forming a stereoscopic image comprising:

a) means for generating a first histogram of said first view and a second histogram of said second view, each said first and second histogram representing a frequency of occurrence of a characteristic component of said first and second views, respectively;

b) means for comparing said first histogram of said first view with said second histogram of said second view and estimating mismatch in gain, a, between said first and second views; and

,

c) means for correcting the first and second views of said stereoscopic image by multiplying said mismatch in gain a and a second signal representing the second view and generating a gain corrected second signal of said stereoscopic image such that said gain corrected second signal and a first signal representing the first view form a stereoscopic image of improved quality.

2. The apparatus according to claim 1, wherein said first view and said second view are digitized representations of left and right views of the stereoscopic image, respectively, said characteristic component of said first and second views being luminance amplitude levels of said left and right views.

3. The apparatus according to claim 1, wherein said first view and said second view are digitized representations of left and right views of tee stereoscopic image, respectively, said characteristic component of said first and second views being color component levels of said left and right views.

4. The apparatus according to claim 1, wherein said means for comparing said first histogram of said first view with said second histogram of said second view includes means for obtaining a first set of characteristic points, x_L1, x_L2, from said first histogram, and obtaining a second set of characteristic points, x_R1, x_R2, from said second view, said second set of points substantially corresponding to said first set of points, further including means for calculating the mismatch in gain, a, according to ##EQU13##

5. The apparatus according to claim 4, wherein said first set of characteristic points, x_L1, x_L2, of said first histogram is a pair selected from the group consisting of:

distinct endpoints of said first histogram, first and second maxima extrema points of said first histogram, a maxima extrema point and a local minima point of said first histogram, and first and second local minima points of said first histogram.

6. The apparatus according to claim 5, further including means for estimating and correcting global mismatch in offset, b, between said first and second views including means for calculating b=x_L2 -ax_R2, and, means for adding said offset b to said gain corrected second signal and generating a gain and offset corrected second view of said stereoscopic image.

7. The apparatus according to claim 1, wherein said means for comparing said first histogram of said first view with said second histogram of said second view includes means for calculating a statistical first left moment, I_1L, of said first histogram, a statistical first right moment, I_1R, of said second histogram, a statistical second right moment, I_2R, of said second histogram, and a statistical joint moment, I_2LR between said first and second histograms;

said statistical first left moment, I_1L, being calculated as said statistical first right moment, I_1R, being calculated as ##EQU14## said statistical second right moment, I_2R, being calculated as ##EQU15## and, said statistical joint moment, I_2LR between said first and second histograms being calculated as ##EQU16## where p(x_i) represents the frequency of occurrence of the characteristic component at x_i.

8. The apparatus according to claim 7, wherein said means for estimating mismatch in gain, a, includes means for calculating ##EQU17##

9. The apparatus according to claim 8, further including means for estimating and correcting global mismatch in offset, b, between said first and second views including means for calculating b=I_1L -aI_1R and, means for adding said mismatch in offset b to said gain corrected second view and generating a gain and offset corrected second view of said stereoscopic image.

10. The apparatus according to claim 7, wherein said means for comparing said first histogram of said first view with said second histogram of said second view includes means for computing a statistical second left moment, I_2L, of said second histogram by calculating wherein said means for estimating mismatch in gain, a, includes means for calculating ##EQU18##

11. The apparatus according to claim 10, further including means for estimating and correcting global mismatch in offset, b, between said first and second views including means for calculating b=I_1L -aI_1R and, means for adding said mismatch in offset b to said gain corrected second view and generating a gain and offset corrected second view of said stereoscopic image.

12. An apparatus for increasing compression efficiency of a stereoscopic image encoder employed for compressively encoding first and second signals representing respective first and second views of a stereoscopic image, said apparatus comprising:

a) means for generating a first histogram of said first view and a second histogram of said second view, said first and second histograms representing a frequency of occurrence of a characteristic component of said first and second views, respectively;

b) means for comparing said first histogram of said first view with said second histogram of said second view and for estimating mismatch in gain, a, and mismatch in offset, b, between said first and second views; and

,

c) means for multiplying said mismatch in gain a to said second signal to obtain a gain corrected second signal for said stereoscopic image and means for adding said mismatch in offset b to said gain corrected second signal to obtain a gain and offset corrected second signal,

whereby compression efficiency is increased when said gain and offset corrected second signal and said first signal are input to the stereoscopic image encoder.

13. The apparatus according to claim 12, wherein said first view and said second view are digitized representations of left and right views of the stereoscopic image, respectively, said characteristic component of said first and second views being luminance amplitude levels of said left and right views.

14. The apparatus according to claim 12, wherein said first view and said second view are digitized representations of left and right views of the stereoscopic image, respectively, said characteristic component of said first and second views being color component levels of said left and right views.

15. The apparatus according to claim 12, wherein said means for comparing said first histogram of said first view with said second histogram of said second view includes means for obtaining a first set of characteristic points, x_L1, x_R1, from said first histogram, and obtaining a second set of characteristic points, x_R1, x_R2, from said second histogram, said second set of points substantially corresponding to said first set of points, further including means for calculating the mismatch in gain, a, according to ##EQU19##

16. The apparatus according to claim 15, wherein said first set of characteristic points, x_L1,x_L2, of said first histogram is a pair selected from the group consisting of:

distinct endpoints of said first histogram, first and second maxima extrema points of said first histogram, a maxima extrema point and a local minima point of said first histogram, and first and second local minima points of said first histogram.

17. The apparatus according to claim 15, wherein said means for estimating said mismatch in offset, b, between said first and second views includes means for calculating b=x_L2 =ax_R2.

18. The apparatus according to claim 12, wherein said means for comparing said first histogram of said first view with said second histogram of said second view includes means for calculating a statistical first left moment, I_1L, of said first histogram, a statistical first right moment, I_1R, of said second histogram, a statistical second right moment, I_2R, of said second histogram, and a statistical joint moment, I_2LR between said first and second histograms;

said statistical first left moment, I_1L, being calculated as said statistical first right moment, I_1R, being calculated as ##EQU20## said statistical second right moment, I_2R, being calculated as ##EQU21## and, said statistical joint moment, I_2LR between said first and second histograms being calculated as ##EQU22## where p(x_i) represents the frequency of occurrence of the characteristic component at x_i.

19. The apparatus according to claim 18, wherein said means for estimating the mismatch in gain, a, includes means for calculating ##EQU23##20.

20. The apparatus according to claim 19, wherein said means for estimating the mismatch in offset, b, between said first and second views odes means for calculating b=I_1L -aI_1R.

21. The apparatus according to claim 18, wherein said means for comparing said first histogram of said first view with said second histogram of said second view includes means for computing a statistical second left moment, I_2L, of said second histogram by calculating said means for estimating mismatch in gain, a, includes means for calculating ##EQU24##

22. The apparatus according to claim 21, wherein said means for estimating mismatch in offset, b, between said first arid second views includes means for calculating b=I_1L -aI_1R.

23. The apparatus according to claim 12, wherein the encoder includes an MPEG-1 encoder for encoding said first signal and an MPEG-2 Temporal scalability encoder for encoding said gain and offset corrected second signal.

24. The apparatus according to claim 12, wherein the encoder includes a non-scalable MPEG-2 encoder for encoding said first signal and an MPEG-2 Temporal scalability encoder for encoding said gain and offset corrected second signal.

25. The apparatus according to claim 12, wherein the encoder includes an H.263 encoder for encoding said first signal and an MPEG-4 utilizing disparity/motion compensated encoder for encoding said gain and offset corrected second signal.

26. A method for estimating and correcting global mismatch in gain and offset between first and second signals representing first and second views of a stereoscopic image comprising the steps of:

a) providing a first histogram of said first view and a second histogram of said second view, said first and second histograms representing a frequency of occurrence of a characteristic component of said first and second views, respectively;

b) comparing said first histogram of said first view with said second histogram of said second view and estimating mismatch in gain, a, and mismatch in offset, b, between said first and second views;

c) correcting the first and second views of said stereoscopic image by multiplying said mismatch in gain a and said second signal to obtain a gain corrected second signal for said stereoscopic image and adding said mismatch in offset b to said gain corrected second signal to obtain a gain and offset corrected second signal such that when said gain and offset corrected second signal and said first signal are combined, a stereoscopic image of improved quality is formed.

27. The method according to claim 26, wherein said first view and said second view are digitized representations of left and right views of the stereoscopic image, respectively, said characteristic component of said first and second views being luminance amplitude levels of said left and right views.

28. The method according to claim 26, wherein said first view and said second view are digitized representations of left and right views of the stereoscopic image, respectively, said characteristic component of said first and second views being color component levels of said left and right views.

29. The method according to claim 26, wherein said step (b) of comparing said first histogram of said first view with said second histogram of said second view includes finding a first set of identifiable characteristic points, x_L1, x_L2, from said first histogram, and finding a second set of characteristic points, x_R1, x_R2, from said second histogram, said second set of points substantially corresponding to said first set of points, and wherein estimating mismatch in gain, a, includes calculating ##EQU25##30.

30. The method according to claim 29, wherein said first set x_L1, x_L2, of characteristic identifiable points from said first histogram a pair selected from the group consisting of:

distinct endpoints of said first histogram, first and second maxima extrema points of said first histogram, a maxima extrema point and a local minima point of said first histogram, and first and second local minima points of said first histogram.

31. The method according to claim 29, wherein said step of estimating mismatch in offset, b, between said first and second includes calculating b=x_L2 -ax_R2.

32. The method according to claim 26, wherein said step (b) of comparing said first histogram of said first view with said second histogram of said second view includes the step of:

applying a technique of linear minimization of mean square error estimates between first and second views by computing a statistical first left moment, I_1L, of said first histogram, a statistical first right moment, I_1R, of said second histogram, a statistical second right moment, I_2R, of said second histogram, and a statistical joint moment, I_2LR between said first and second histograms;

said statistical first left moment, I_1L, being calculated as said statistical first right moment, I_1R, being calculated as ##EQU26## said statistical second right moment, I_2R, being calculated as ##EQU27## and, said statistical joint moment, I_2LR between said first and second histograms being calculated as ##EQU28## where p(x_i) represents the frequency of occurrence of the characteristic component at x_i.

33. The method according to claim 32, wherein said mismatch in gain, a, is estimated by calculating ##EQU29##

34. The method according to claim 33, wherein said step of estimating mismatch in offset, b, includes calculating b=I_1L -aI_1R.

35. The method according to claim 32, wherein said step (b) of comparing said first histogram of said first view with said second histogram of said second view includes the step of:

comparing geometric shapes of said first and second histograms by further computing a statistical second left moment, I_2L, of said second histogram by calculating and estimating said mismatch in gain, a, by calculating ##EQU30##

36. The method according to claim 35, wherein said step of estimating mismatch in offset, b, includes calculating b=I_1L -aI_1R.

37. The method according to claim 26, further including the steps of compressively encoding said first and gain and offset corrected second signals for transmission or storage in a compressed state, and decoding said compressed fist and compressed gain and offset corrected second signals to form the stereoscopic image, wherein said method steps a), b) and c) are performed prior to encoding to improve coding efficiency.

38. The method according to claim 26, further including the steps of compressively encoding a first and second original signals for transmission or storage in a compressed state, and decoding said compressed first and second original signals to form said first and second signals, wherein said method steps a), b) and c) are performed after decoding of said first and second original signals to provide a stereoscopic image of improved quality.

39. A method for increasing compression efficiency of a stereoscopic image encoder employed for compressively encoding first and second signals representing respective first and second views of a stereoscopic image comprising the steps of:

a) providing a first histogram of said first view and a second histogram of said second view, said first and second histograms representing a frequency of occurrence of a characteristic component of said first and second views, respectively;

b) comparing said first histogram of said first view with said second histogram of said second view and estimating mismatch in gain, a, and mismatch in offset, b, between said first and second views;

c) correcting said stereoscopic image by multiplying said gain a to said second signal to obtain a gain corrected second signal for said stereoscopic image and adding said offset b to said gain corrected second signal to obtain a gain and offset corrected second signal,

wherein said gain and offset corrected second signal is obtained prior to encoding to increase encoding efficiency.

40. The method according to claim 39, wherein said first view and said second view are digitized representations of left and right views of the stereoscopic image, respectively, said characteristic component of said first and second views being luminance amplitude levels of said left and right views.

41. The method according to claim 39, wherein said first view and said second view are digitized representations of left and right views of the stereoscopic image, respectively, said characteristic component of said first and second views being color component levels of said left and right views.

42. The method according to claim 39, wherein said step (b) of comparing said first histogram of said first view with said second histogram of said second view includes finding a first set of characteristic points, x_L1, x_L2, from said first histogram, and finding a second set of characteristic points, x_R1, x_R2, from said second history said second set of points substantially corresponding to said first set of points, and wherein estimating the mismatch in gain, a, includes calculating ##EQU31##

43. The method according to claim 42, wherein said first set x_L1, x_L2, of identifiable points in said first histogram is a pair selected from the group consisting of:

distinct endpoints of said first histogram, first and second maxima extrema points of said first histogram, a maxima extrema point and a local minima point of said first histogram, and first and second local minima points of said first histogram.

44. The method according to claim 42, further including the step of determining global mismatch in offset, b, between said first and second view includes calculating b=x_L2 -ax_R2 and, adding said offset b to said gain corrected second signal to obtain a gain and offset corrected second signal such that when said gain and offset corrected second signal and said first signal are combined, a stereoscopic image of improved quality is formed.

45. The method according to claim 39, wherein said step (b) of comparing said first histogram of said first view with said second histogram of said second view includes the step of:

applying a technique of linear minimization of mean square error estimates between said first and second views by computing a statistical first left moment, I_1L, of said first histogram, a statistical first right moment, I_1R, of said second histogram, a statistical second right moment, I_2R, of said second histogram, and a statistical joint moment, I_2LR between said first and second histograms;

said statistical first left moment, I_1L, being calculated as said statistical first right moment, I_1R, being calculated as ##EQU32## said statistical second right moment, I_2R, being calculated as ##EQU33## and, said statistical joint moment, I_2LR between said first and second histograms being calculated as ##EQU34## where p(x_i) represents the frequency of occurrence of the characteristic component at x_i.

46. The method according to claim 45, wherein said mismatch in gain, a, is estimated by calculating ##EQU35##

47. The method according to claim 46, wherein said step of estimating mismatch in offset, b, between said first and second views includes calculating b=I_1L -aI_1R.

48. The method according to claim 45 wherein said step (b) of comparing said first histogram of said first view with said second histogram of said second view includes the step of:

comparing geometric shapes of said first and second histograms by further computing a statistical second left moment, I_2L, of said second histogram by calculating and estimating said mismatch in gain, a, by calculating a ##EQU36##

49. The method according to claim 48, wherein said step of estimating mismatch in offset, b, between said first and second views includes calculating b=I_1L -aI_1R.

50. An apparatus for correcting mismatch in gain and offset between multiple views of a multi-view and a reference view of the multi-view, the multiple views represented by multiple signals, one view per signal, the reference view represented by a reference signal, the apparatus comprising:

a) means for generating a reference histogram of the reference signal and for generating multiple histograms from the multiple signals, each histogram representing a characteristic component of corresponding signal;

b) means for comparing each of said multiple histograms with the reference histogram and for estimating mismatches in gain and offset between each of said multiple histograms and the reference histogram;

c) means for gain correcting each of said multiple signals by multiplying each of said multiple signals and corresponding estimated mismatch in gain to obtain multiple gain corrected signals;

d) means for offset correcting each of said multiple gain corrected signals by adding to each of said multiple gain corrected signals corresponding estimated mismatch in offset, thereby obtaining multiple offset and gain corrected signals that can be combined with the reference signal to produce a multi-view video of improved quality.