3D cameras for HDR

• US 9,420,200 B2
• Filed: 11/04/2015
• Issued: 08/16/2016
• Est. Priority Date: 12/27/2010
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Ninan, Ajit, Daly, Scott, Ward, Gregory John, Atkins, Robin
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Hilaire, Clifford
• Application Number:
  US14/932,543
• Publication Number:
  US 20160057334A1
• Time in Patent Office: 286 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/58 involving two or more exposures

Abstract

High dynamic range 3D images are generated with relatively narrow dynamic range image sensors. Input frames of different views may be set to different exposure settings. Pixels in the input frames may be normalized to a common range of luminance levels. Disparity between normalized pixels in the input frames may be computed and interpolated. The pixels in the different input frames may be shifted to, or stay in, a common reference frame. The pre-normalized luminance levels of the pixels may be used to create high dynamic range pixels that make up one, two or more output frames of different views. Further, a modulated synopter with electronic mirrors is combined with a stereoscopic camera to capture monoscopic HDR, alternating monoscopic HDR and stereoscopic LDR images, or stereoscopic HDR images.

Claims

1. A method comprising:

receiving a left input frame created with a first luminance range and a right input frame created with a second luminance range, the left input frame comprising a plurality of left input scanlines, the right input frame comprising a plurality of right input scanlines, each individual left input scanline in the plurality of left input scanlines corresponding to an individual right input scanline in the plurality of right input scanlines, and the first luminance range being different from the second luminance range;

determining a first left input scanline in the plurality of left input scanlines and a first right input scanline in the plurality of right input scanlines, the first left input scanline corresponding to the first right input scanline, the first left input scanline comprising a first set of left input pixels, and the first right input scanline comprising a first set of right input pixels;

selecting a first subset of left input pixels among the first set of left input pixels;

selecting a first subset of right input pixels among the first set of right input pixels;

computing, with the first subset of left input pixels and the first subset of right input pixels, first input scanline disparity between the first subset of left input pixels and the first subset of right input pixels;

generating, based at least in part on the first input scanline disparity, a first left output scanline for a left high dynamic range (HDR) output frame and a first right output scanline for a right HDR output frame;

wherein the first subset of left input pixels is selected, based on one or more criteria, from the set of left input pixels;

wherein the one or more criteria comprise an upper luminance level and wherein zero or more pixels, in the set of left input pixels, that are above the upper luminance level are excluded from being included in the first subset of left input pixels.

2. The method of claim 1, further comprising:

normalizing the first subset of left input pixels with the first subset of right input pixels to a common luminance range;

using the first subset of left input pixels as normalized and the second subset of right input pixels as normalized to generate the first input scanline disparity.

3. The method of claim 1, further comprising generating by interpolation, based at least in part on the first input scanline disparity, additional input scanline disparity for remaining left pixels in the first set of left input pixels that are not in the first subset of left input pixels and for remaining right pixels in the first set of right input pixels that are not in the first subset of right input pixels.

4. The method of claim 1, wherein generating, based at least in part on the first input scanline disparity, a first left output scanline for a left HDR output frame and a first right output scanline for a right HDR output frame includes merging the first set of left input pixels and the first set of right input pixels, using pre-normalized luminance levels of the first set of left input pixels and pre-normalized luminance levels of the first set of right input pixels, to form the first left output scanline and the first right output scanline based at least in part on the first input scanline disparity.

5. The method of claim 1, wherein the first left input frame comprises a portion of a left view image on an image plane, and wherein the first right input frame comprises a portion of a right view image on the same image plane.

6. The method of claim 5, wherein the image plane is that of a stereoscopic camera, wherein the left view image is captured with a left camera element of the stereoscopic camera, and wherein the right view image is captured with a right camera element of the stereoscopic camera.

7. The method of claim 5, wherein the image plane is not that of a stereoscopic camera, wherein the left view image pertains to a converted left view from an original left view image captured with a left camera element of the stereoscopic camera, and wherein the right view image pertains to a converted right view from an original right view image captured with a right camera element of the stereoscopic camera.

8. The method of claim 1, wherein at least one of the first luminance range and the second luminance range comprises one or more of an aperture setting, a shutter speed setting, or an image luminance sensitivity setting.

9. The method of claim 1, wherein the left input frame is derived from an original left image, wherein the right input frame is derived from an original right image, and wherein the original left image and the original right image are taken at the same time.

10. The method of claim 1, wherein the left HDR output frame and the right HDR output frame comprises a stereoscopic image.

11. The method of claim 10, wherein the stereoscopic image is one in a time sequence of stereoscopic images that form a video.

12. The method of claim 1, further comprising generating, based at least in part on the first input scanline disparity, an output scanline for an output frame with a view differing from those of the left HDR output frame and the right HDR output frame.

13. The method of claim 1, further comprising:

determining a second left input scanline in the plurality of left input scanlines and a second right input scanline in the plurality of right input scanlines, the second input scanline corresponding to the second right input scanline, the second left input scanline comprising a second set of left input pixels, and the second right input scanline comprising a second set of right input pixels;

selecting a second subset of left input pixels among the second set of left input pixels;

selecting a second subset of right input pixels among the second set of right input pixels;

computing, with the second subset of left input pixels and the second subset of right input pixels, second input scanline disparity between the second subset of left input pixels and the second subset of right input pixels;

generating, based at least in part on the second input scanline disparity, a second left output scanline for the left HDR output frame and a second right output scanline for the right HDR output frame.

14. The method of claim 13, further comprising:

in response to completing generating all left output scanlines for the left HDR output frame, removing one or more left visual artifacts from the left HDR output frame;

in response to completing generating all right output scanlines for the right HDR output frame, removing one or more right visual artifacts from the right HDR output frame.

15. A method comprising:

receiving a left input frame created with a first luminance range and a right input frame created with a second luminance range, the left input frame comprising a plurality of left input scanlines, the right input frame comprising a plurality of right input scanlines, each individual left input scanline in the plurality of left input scanlines corresponding to an individual right input scanline in the plurality of right input scanlines, and the first luminance range being different from the second luminance range;

determining a first left input scanline in the plurality of left input scanlines and a first right input scanline in the plurality of right input scanlines, the first left input scanline corresponding to the first right input scanline, the first left input scanline comprising a first set of left input pixels, and the first right input scanline comprising a first set of right input pixels;

selecting a first subset of left input pixels among the first set of left input pixels;

selecting a first subset of right input pixels among the first set of right input pixels;

generating, without computing disparity between pixels in the left input frame and the right input frame, a first left output scanline for a left high dynamic range (HDR) output frame and a first right output scanline for a right HDR output frame;

wherein the first subset of left input pixels is selected, based on one or more criteria, from the set of left input pixels;

wherein the one or more criteria comprise an upper luminance level and wherein zero or more pixels, in the set of left input pixels, that are above the upper luminance level are excluded from being included in the first subset of left input pixels.

16. The method of claim 15, wherein the first left input frame comprises a portion of a left view image on an image plane, and wherein the first right input frame comprises a portion of a right view image on the same image plane.

17. The method of claim 16, wherein the image plane is that of a stereoscopic camera, wherein the left view image is captured with a left camera element of the stereoscopic camera, and wherein the right view image is captured with a right camera element of the stereoscopic camera.

18. The method of claim 16, wherein the image plane is not that of a stereoscopic camera, wherein the left view image pertains to a converted left view from an original left view image captured with a left camera element of the stereoscopic camera, and wherein the right view image pertains to a converted right view from an original right view image captured with a right camera element of the stereoscopic camera.

19. The method of claim 15, wherein at least one of the first luminance range and the second luminance range comprises one or more of an aperture setting, a shutter speed setting, or an image luminance sensitivity setting.

20. The method of claim 15, wherein the left input frame is derived from an original left image, wherein the right input frame is derived from an original right image, and wherein the original left image and the original right image are taken at the same time.

21. The method of claim 15, wherein the left HDR output frame and the right HDR output frame comprises a stereoscopic image.

22. The method of claim 21, wherein the stereoscopic image is one in a time sequence of stereoscopic images that form a video.

23. The method of claim 15, further comprising:

determining a second left input scanline in the plurality of left input scanlines and a second right input scanline in the plurality of right input scanlines, the second input scanline corresponding to the second right input scanline, the second left input scanline comprising a second set of left input pixels, and the second right input scanline comprising a second set of right input pixels;

selecting a second subset of left input pixels among the second set of left input pixels;

selecting a second subset of right input pixels among the second set of right input pixels;

generating, without computing disparity, a second left output scanline for the left HDR output frame and a second right output scanline for the right HDR output frame.

24. The method of claim 23, further comprising:

in response to completing generating all left output scanlines for the left HDR output frame, removing one or more left visual artifacts from the left HDR output frame;

in response to completing generating all right output scanlines for the right HDR output frame, removing one or more right visual artifacts from the right HDR output frame.

25. A method comprising:

receiving a left input frame created with a first luminance range and a right input frame created with a second luminance range, the left input frame comprising a plurality of left input scanlines, the right input frame comprising a plurality of right input scanlines, each individual left input scanline in the plurality of left input scanlines corresponding to an individual right input scanline in the plurality of right input scanlines, and the first luminance range being different from the second luminance range;

determining a first left input scanline in the plurality of left input scanlines and a first right input scanline in the plurality of right input scanlines, the first left input scanline corresponding to the first right input scanline, the first left input scanline comprising a first set of left input pixels, and the first right input scanline comprising a first set of right input pixels;

selecting a first subset of left input pixels among the first set of left input pixels;

selecting a first subset of right input pixels among the first set of right input pixels;

computing, with the first subset of left input pixels and the first subset of right input pixels, first input scanline disparity between the first subset of left input pixels and the first subset of right input pixels;

generating, based at least in part on the first input scanline disparity, a first left output scanline for a left high dynamic range (HDR) output frame and a first right output scanline for a right HDR output frame;

wherein the first subset of left input pixels is selected, based on one or more criteria, from the set of left input pixels;

wherein the one or more criteria comprise a lower luminance level and wherein zero or more pixels, in the set of left input pixels, that are under the lower luminance level are excluded from the first subset of left input pixels.

26. The method of claim 25, further comprising:

normalizing the first subset of left input pixels with the first subset of right input pixels to a common luminance range;

using the first subset of left input pixels as normalized and the second subset of right input pixels as normalized to generate the first input scanline disparity.

27. The method of claim 25, further comprising generating by interpolation, based at least in part on the first input scanline disparity, additional input scanline disparity for remaining left pixels in the first set of left input pixels that are not in the first subset of left input pixels and for remaining right pixels in the first set of right input pixels that are not in the first subset of right input pixels.

28. The method of claim 25, wherein generating, based at least in part on the first input scanline disparity, a first left output scanline for a left HDR output frame and a first right output scanline for a right HDR output frame includes merging the first set of left input pixels and the first set of right input pixels, using pre-normalized luminance levels of the first set of left input pixels and pre-normalized luminance levels of the first set of right input pixels, to form the first left output scanline and the first right output scanline based at least in part on the first input scanline disparity.

29. The method of claim 25, wherein the first left input frame comprises a portion of a left view image on an image plane, and wherein the first right input frame comprises a portion of a right view image on the same image plane.

30. The method of claim 29, wherein the image plane is that of a stereoscopic camera, wherein the left view image is captured with a left camera element of the stereoscopic camera, and wherein the right view image is captured with a right camera element of the stereoscopic camera.

31. The method of claim 29, wherein the image plane is not that of a stereoscopic camera, wherein the left view image pertains to a converted left view from an original left view image captured with a left camera element of the stereoscopic camera, and wherein the right view image pertains to a converted right view from an original right view image captured with a right camera element of the stereoscopic camera.

32. The method of claim 25, wherein at least one of the first luminance range and the second luminance range comprises one or more of an aperture setting, a shutter speed setting, or an image luminance sensitivity setting.

33. The method of claim 25, wherein the left input frame is derived from an original left image, wherein the right input frame is derived from an original right image, and wherein the original left image and the original right image are taken at the same time.

34. The method of claim 25, wherein the left HDR output frame and the right HDR output frame comprises a stereoscopic image.

35. The method of claim 25, wherein the stereoscopic image is one in a time sequence of stereoscopic images that form a video.

36. The method of claim 25, further comprising generating, based at least in part on the first input scanline disparity, an output scanline for an output frame with a view differing from those of the left HDR output frame and the right HDR output frame.

37. The method of claim 25, further comprising:

determining a second left input scanline in the plurality of left input scanlines and a second right input scanline in the plurality of right input scanlines, the second input scanline corresponding to the second right input scanline, the second left input scanline comprising a second set of left input pixels, and the second right input scanline comprising a second set of right input pixels;

selecting a second subset of left input pixels among the second set of left input pixels;

selecting a second subset of right input pixels among the second set of right input pixels;

computing, with the second subset of left input pixels and the second subset of right input pixels, second input scanline disparity between the second subset of left input pixels and the second subset of right input pixels;

generating, based at least in part on the second input scanline disparity, a second left output scanline for the left HDR output frame and a second right output scanline for the right HDR output frame.

38. The method of claim 37, further comprising:

in response to completing generating all left output scanlines for the left HDR output frame, removing one or more left visual artifacts from the left HDR output frame;

in response to completing generating all right output scanlines for the right HDR output frame, removing one or more right visual artifacts from the right HDR output frame.

39. A method comprising:

receiving a left input frame created with a first luminance range and a right input frame created with a second luminance range, the left input frame comprising a plurality of left input scanlines, the right input frame comprising a plurality of right input scanlines, each individual left input scanline in the plurality of left input scanlines corresponding to an individual right input scanline in the plurality of right input scanlines, and the first luminance range being different from the second luminance range;

determining a first left input scanline in the plurality of left input scanlines and a first right input scanline in the plurality of right input scanlines, the first left input scanline corresponding to the first right input scanline, the first left input scanline comprising a first set of left input pixels, and the first right input scanline comprising a first set of right input pixels;

selecting a first subset of left input pixels among the first set of left input pixels;

selecting a first subset of right input pixels among the first set of right input pixels;

generating, without computing disparity between pixels in the left input frame and the right input frame, a first left output scanline for a left high dynamic range (HDR) output frame and a first right output scanline for a right HDR output frame;

wherein the first subset of left input pixels is selected, based on one or more criteria, from the set of left input pixels;

wherein the one or more criteria comprise an upper luminance level and wherein zero or more pixels, in the set of left input pixels, that are above the upper luminance level are excluded from being included in the first subset of left input pixels.

40. The method of claim 39, wherein the first left input frame comprises a portion of a left view image on an image plane, and wherein the first right input frame comprises a portion of a right view image on the same image plane.

41. The method of claim 40, wherein the image plane is that of a stereoscopic camera, wherein the left view image is captured with a left camera element of the stereoscopic camera, and wherein the right view image is captured with a right camera element of the stereoscopic camera.

42. The method of claim 40, wherein the image plane is not that of a stereoscopic camera, wherein the left view image pertains to a converted left view from an original left view image captured with a left camera element of the stereoscopic camera, and wherein the right view image pertains to a converted right view from an original right view image captured with a right camera element of the stereoscopic camera.

43. The method of claim 39, wherein at least one of the first luminance range and the second luminance range comprises one or more of an aperture setting, a shutter speed setting, or an image luminance sensitivity setting.

44. The method of claim 39, wherein the left input frame is derived from an original left image, wherein the right input frame is derived from an original right image, and wherein the original left image and the original right image are taken at the same time.

45. The method of claim 39, wherein the left HDR output frame and the right HDR output frame comprises a stereoscopic image.

46. The method of claim 45, wherein the stereoscopic image is one in a time sequence of stereoscopic images that form a video.

47. The method of claim 39, further comprising:

determining a second left input scanline in the plurality of left input scanlines and a second right input scanline in the plurality of right input scanlines, the second input scanline corresponding to the second right input scanline, the second left input scanline comprising a second set of left input pixels, and the second right input scanline comprising a second set of right input pixels;

selecting a second subset of left input pixels among the second set of left input pixels;

selecting a second subset of right input pixels among the second set of right input pixels;

generating, without computing disparity, a second left output scanline for the left HDR output frame and a second right output scanline for the right HDR output frame.

48. The method of claim 47, further comprising:

in response to completing generating all left output scanlines for the left HDR output frame, removing one or more left visual artifacts from the left HDR output frame;

in response to completing generating all right output scanlines for the right HDR output frame, removing one or more right visual artifacts from the right HDR output frame.