Method and Apparatus for Enhanced Digital Imaging

• US 20160292842A1
• Filed: 11/18/2013
• Published: 10/06/2016
• Est. Priority Date: 11/18/2013
• Status: Abandoned Application

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: SEN, Sumeet, KORHONEN, Janne, PYLKKANEN, Tom
• Application Number:
  US15/034,894
• Publication Number:
  US 20160292842A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position

Abstract

Method, apparatus and computer programs are disclosed for forming synthesized panning of images. Based on a pair of digital images taken by a pair of digital cameras and their calibration information, a disparity map is formed for image objects in the pair of digital images. A combined image is formed using the pair of digital images. The combined image is segmented, using the disparity map, to include a foreground region and a background region. A sequence of synthesized panning images is formed so that for each combined image; a perspective shift is applied between the foreground region and background region; and a shifting portion of the perspective shifted image is cropped.

Claims

1. An apparatus comprising:

at least one memory configured to store calibration information;

two digital image capture units configured to take a respective pair of digital images at given offset from one another, with overlapping fields of view so that some image objects may appear in each of the pair of digital images;

the at least one memory being further configured to store the pair of digital images;

a processor configured to;

produce, based the calibration information and the pair of digital images, a disparity map for the pair of digital images;

form a combined image using the pair of digital images;

segment the combined image, using the disparity map, to comprise a foreground region and a background region;

form a sequence of synthesized panning images so that for each combined image;

a perspective shift is applied between at least one of the the foreground region and background region; and

a shifting portion of the perspective shifted image is cropped.

6. The apparatus of claim 1, wherein the two digital image capture units are formed of a common digital camera and of an optical image splitter with two offset and substantially parallel image input ports.

7. The apparatus of claim 1, wherein the pair of digital images have substantially overlapping fields of view.

10. The apparatus of claim 1, wherein the processor is configured to enable a user of the apparatus to identify the foreground region to facilitate the segmentation.

11. The apparatus of claim 10, wherein the processor is further configured to form an animation file of the sequence of the synthesized panning images.

12. The apparatus of claim 1, further comprising an optical image stabilization unit configured to optically stabilize at least one of the digital images of the pair of digital images.

13. The apparatus of claim 12, wherein the processor is further configured to control the optical image stabilization unit and to control the image capture units so as to take multiple images shifting the view affected by optical image stabilization unit from one image to another in the direction of the synthesized panning.

14. The apparatus of claim 12, wherein processor is configured to control the optical image stabilization unit to perform both image stabilization and the shifting of the view.

15. The apparatus of claim 14, wherein the processor is configured to cause the optical image stabilization to be performed to the extent possible after the shifting of the view.

16. The apparatus of claim 12, wherein the processor is further configured to cause the digital image capture units to take a plurality of the pairs of the digital images and to cause the optical image stabilization unit to perform the shifting of the view differently for different pairs of digital images.

18. The apparatus of claim 16, wherein the processor is further configured to perform the forming of the combined image using the plurality of pairs of digital images.

19. The apparatus of claim 12, wherein the processor is further configured to use the changing mutual geometry of the image capture units to facilitate the producing of the disparity map or to refine the disparity map.

20. The apparatus of claim 1, further comprising a display.

21. The apparatus of claim 20, wherein the processor is configured to cause presenting a preview on the display to illustrate synthesized panning that can be produced with current view of the image capture units.

22. The apparatus of claim 20, further comprising a user input.

23. The apparatus of claim 22, wherein the processor is further configured to enable user determination of at least one parameter and to use the at least one parameter in any one or more of:

the producing of the disparity map;

the forming of the combined image;

the segmenting of the combined image; and

the forming of the sequence of synthesized panning images.

24. The apparatus of claim 22, wherein the user input comprises a touch screen.

25. The apparatus of claim 24, wherein the processor is configured to at least partly form the at least one parameter by recognizing a gesture such as swiping on the touch screen.

2-5. -5. (canceled)

8-9. -9. (canceled)

17. (canceled)

26. An apparatus comprising:

at least one memory configured to store calibration information;

two digital image capture units at given offset from one another and with overlapping fields of view so that some image objects may appear in images taken with each of the two image capture units;

a processor configured to cause the apparatus, for forming video image, to sequentially;

cause the two digital image capture units to capture a pair of digital images;

store in the at least one memory the captured pair of digital images;

produce, based the calibration information and the pair of digital images, a disparity map for the pair of digital images;

form a combined image from the pair of digital images;

segment the combined image, using the disparity map, to comprise a foreground region and a background region;

form, from the sequentially formed combined images, synthesized panning images so that for each combined image;

a perspective shift is applied between the foreground region and background region; and

a shifting portion of the image is cropped.

27-30. -30. (canceled)

31. A method comprising:

storing calibration information;

taking by two digital image capture units a respective pair of digital images at given offset from one another, with overlapping fields of view so that some image objects may appear in each of the pair of digital images;

storing the pair of digital images;

producing, based the calibration information and the pair of digital images, a disparity map for the pair of digital images;

forming a combined image using the pair of digital images;

segmenting the combined image, using the disparity map, to comprise a foreground region and a background region;

forming a sequence of synthesized panning images so that for each combined image;

a perspective shift is applied between the foreground region and background region; and

a shifting portion of the perspective shifted image is cropped.

35. The method of claim 31, wherein the pair of digital images have substantially overlapping fields of view.

38. The method of claim 31, further comprising enabling a user to identify the foreground region to facilitate the segmentation.

39. The method of claim 31, further comprising forming an animation file of the sequence of the synthesized panning images.

40. The method of claim 31, further comprising optically stabilizing at least one of the digital images of the pair of digital images.

41. The method of claim 31, further comprising controlling the optical image stabilizing unit and the image capture units to take multiple images shifting the view affected by optical image stabilizing from one image to another in the direction of the synthesized panning.

42. The method of claim 40, comprising performing both image stabilizing and the shifting of the view.

43. The method of claim 42, comprising performing the optical image stabilizing to the extent possible after the shifting of the view.

44. The method of claim 40, further comprising taking a plurality of the pairs of the digital images and performing the optical image stabilizing to perform the shifting of the view differently for different pairs of digital images.

46. The method of claim 44, further comprising performing the forming of the combined image using the plurality of pairs of digital images.

47. The method of claim 40, further comprising using the changing mutual geometry of the image capture units to facilitate the producing of the disparity map or to refine the disparity map.

48. The method of claim 31, further comprising displaying a preview to illustrate synthesized panning that can be produced with current view of the image capture units.

49. The method of claim 31, further comprising receiving a user determination of at least one parameter and using the at least one parameter in any one or more of:

the producing of the disparity map;

the forming of the combined image;

the segmenting of the combined image; and

the forming of the sequence of synthesized panning images.

50. The method of claim 49, wherein the at least one parameter determined at least partly by recognizing a gesture of a user on a touch screen.

32-34. -34. (canceled)

36-37. -37. (canceled)

45. (canceled)

51-57. -57. (canceled)

58. A computer program, comprising:

code for storing calibration information;

code for taking by two digital image capture units a respective pair of digital images at given offset from one another, with overlapping fields of view so that some image objects may appear in each of the pair of digital images;

code for storing the pair of digital images;

code for producing, based the calibration information and the pair of digital images, a disparity map for the pair of digital images;

code for forming a combined image using the pair of digital images;

code for segmenting the combined image, using the disparity map, to comprise a foreground region and a background region; and

code for forming a sequence of synthesized panning images so that for each combined image;

a perspective shift is applied between the foreground region and background region; and

a shifting portion of the perspective shifted image is cropped;

when the computer program is run on a processor.

59. (canceled)