Methods and apparatus relating to image stabilization

• US 9,686,471 B2
• Filed: 10/31/2014
• Issued: 06/20/2017
• Est. Priority Date: 11/01/2013
• Status: Active Grant

• Current Assignee: Blue River Technology Inc. (Deere & Company)
• Original Assignee: Light Labs Inc.
• Inventors: Shroff, Sapna A, Laroia, Rajiv
• Primary Examiner(s): OSINSKI, MICHAEL S
• Application Number:
  US14/530,682
• Publication Number:
  US 20150163408A1
• Time in Patent Office: 963 Days
• CPC Class Codes: G02B 27/58 Optics for apodization or superresolution; Optical synthetic aperture systems
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 23/15 Image signal generation with circuitry for avoiding or correcting image misregistration
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Methods and apparatus for processing images captured by a camera device including multiple optical chains, e.g., camera modules, are described. Three, 4, 5 or more optical chains maybe used. Different optical chains capture different images due to different perspectives. Multiple images, e.g., corresponding to different perspectives, are captured during a time period and are combined to generate a composite image. In some embodiments one of the captured images or a synthesized image is used as a reference image during composite image generation. The image used as the reference image is selected to keep the perspective of sequentially generated composite images consistent despite unintentional came movement and/or in accordance with an expected path of travel. Thus, which camera module provides the reference image may vary over time taking into consideration unintended camera movement. Composite image generation may be performed external to the camera device or in the camera device.

Claims

1. A method of processing a sequence of image data captured by a moving camera, the method comprising:

detecting motion of the moving camera, said moving camera including multiple optical chains or being a light field camera, said moving camera configured for performing image synthesis from any of a plurality of points of view; and

determining a point of view to be used for generating an image based on the detected motion of the moving camera; and

synthesizing an image, said synthesized image being based on said determined point of view; and

outputting said synthesized image.

2. The method of claim 1,

wherein detecting motion of the moving camera includes tracking motion of the moving camera along a path;

wherein determining a point of view includes determining points of view to be used for synthesizing a sequence of images based on said path of motion; and

wherein synthesizing an image is performed as part of synthesizing a sequence of images, said synthesized sequence of images being based on said determined sequence of points of view; and

wherein outputting said synthesized images is performed as part of outputting sequence of images.

3. The method of claim 2, wherein said plurality of points of view are within a synthetic aperture region corresponding to a synthetic aperture supported by a combination of apertures of different optical chains of said camera, said camera including at least three optical chains.

4. The method of claim 1, wherein said method is performed by said moving camera or on a processing system which receives images from said moving camera, said processing system including at least a processor synthesizing images corresponding to different points of view, a synthesized image corresponding to a point in time being generated by said processor from images captured by different camera modules of said camera at said point in time.

5. The method of claim 1, further comprising performing as part of a stabilization operation, the steps of:

cropping one or more images captured by the camera based on inadvertent motion of the camera.

6. The method of claim 2,

wherein said synthesized sequence of images is a video sequence;

and

wherein determining the sequence of points of view to be used for synthesizing the sequence of images of said video based on said path of motion includes;

performing a smoothing operation on the tracked path of motion to produce a smoothed path of motion; and

using said smoothed path to determine said sequence of points of view.

7. The method of claim 2, wherein determining the sequence of points of view to be used for synthesizing the sequence of images includes:

performing a smoothing operation on the tracked path of motion to produce a smoothed path of motion; and

using said smoothed path to determine said sequence of points of view.

8. The method of claim 7, wherein said smoothed path is a straight line.

9. The method of claim 7, wherein said smoothed path is a smooth curve.

10. The method of claim 1, wherein detecting the motion of the moving camera includes monitoring an output of at least one of an accelerometer and gyroscope included in said camera or comparing an image captured during a first time period to an image captured during a second time period.

11. The method of claim 4,

wherein performing a track stabilization operation includes performing adjustments to cropping of said images and adjustments to the sequence of points of view based on measurements generated by at least one of a gyroscope or an accelerometer included in the camera;

and

where said camera is a handheld camera that is manually moved along a path, wherein said path is a straight line.

12. The method of claim 1, wherein a synthetic aperture of said camera includes expected points of view corresponding to a range of inadvertent track deviations expected to be encountered by a handheld camera.

13. A camera system comprising:

an image capture device including a plurality of camera modules or a light field camera configured to capture a sequence of image data as the image capture device is moved;

a tracking module configured to detect image capture device motion;

a module configured to determine a point of view to be used for synthesizing an image based on the detected motion of the image capture device; and

a synthesization module configured to synthesize said image from multiple images captured by said image capture device in parallel based on the determined point of view; and

an output module configured to output said synthesized image.

14. The camera system of claim 13,

wherein said tracking module is configured to track a path of motion of the image capture device over time;

wherein the module is configured to determine a sequence of points of view to be used for synthesizing a sequence of images based on said path of motion of the image capture device, said determined point of view being one of the points of view in said sequence; and

wherein the synthesization module is configured to synthesize a sequence of images, said synthesized sequence of images being based on said determined sequence of points of view, said synthesized image being an image in said sequence of images; and

wherein the output module is configured to output said synthesized sequence of images.

15. The camera system of claim 14, wherein one point of view in said sequence of point of views is used for each synthesized image in said synthesized sequence of images.

16. The camera system of claim 14, wherein said image capture system is a moving camera that supports image synthesis from any of a plurality of points of view.

17. The camera system of claim 16, wherein different points of view in said plurality of points of view correspond to different optical chains of said image capture device, each optical chain having a single point of view at a given image capture time.

18. The camera system of claim 14,

wherein said points of view are within a synthetic aperture region of said camera;

wherein said synthetic aperture region is a region which corresponds to apertures of multiple camera modules included in said camera; and

wherein said synthetic aperture region is a circular region having a diameter approximately the diameter of a circular area including the outermost lenses of said multiple camera modules.

19. A non-transitory machine readable medium including processor executable instructions which when executed by a processor control an apparatus including said processor to:

detect motion of the moving camera, said moving camera including multiple optical chains or being a light field camera, said moving camera configured for performing image synthesis from any of a plurality of points of view; and

determine a point of view to be used for generating an image based on the detected motion of the moving camera; and

synthesize an image, said synthesized image being based on said determined point of view; and

output said synthesized image.