Apparatus and methods for camera alignment model calibration

• US 9,639,935 B1
• Filed: 09/16/2016
• Issued: 05/02/2017
• Est. Priority Date: 05/25/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Douady-Pleven, Bruno Cesar, Kraak, Michael Serge Andre, Derbanne, Thomas Armel
• Primary Examiner(s): HSIA, SHERRIE Y
• Application Number:
  US15/268,038
• Time in Patent Office: 228 Days
• Field of Search: 348175, 348188, 348 95, 348 36, 348 39, 348 47, 348 48, 348135, 348148, 348584, 382275, 382284, 382103, 382151
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 19/10 using adaptive coding
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/90 : using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/61 : the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

Generating a calibrated camera alignment model for image capture devices having overlapping fields-of-view may include identifying a camera alignment model describing a first candidate alignment path for a defined location in a first input frame and a second candidate alignment path for the defined location in a second input frame, identifying a third candidate alignment path spatially adjacent to the first candidate alignment path, identifying a fourth candidate alignment path spatially adjacent to the second candidate alignment path, identifying a first point along the first candidate alignment path or the second candidate alignment path corresponding to a second point along the third candidate alignment path or the fourth candidate alignment path, and updating the camera alignment model based on the first point, the second point, or both.

Claims

1. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:

receiving a first input frame captured by a first image capture device of an image capture apparatus, the first image capture device having a first field-of-view;

receiving a second input frame captured by a second image capture device of the image capture apparatus, the second image capture device having a second field-of-view such that a first region of the first field-of-view, corresponding to a first region of the first input frame, overlaps a second region of the second field-of-view, corresponding to a second region of the second input frame; and

generating a calibrated camera alignment model for the first image capture device and the second image capture device, wherein generating the calibrated camera alignment model includes;

identifying a camera alignment model for the first image capture device and the second image capture device, wherein the camera alignment model includes information describing a first alignment path for a defined location in the first region of the first input frame and a second alignment path for the defined location in the second region of the second input frame;

identifying the first alignment path as a first candidate alignment path in the first input frame;

identifying a second candidate alignment path in the first input frame spatially adjacent to the first candidate alignment path in a first lateral direction;

identifying the second alignment path as a third candidate alignment path in the second input frame;

identifying a fourth candidate alignment path in the second input frame spatially adjacent to the third candidate alignment path in a second lateral direction;

identifying a first point along the first candidate alignment path or the second candidate alignment path corresponding to a second point along the third candidate alignment path or the fourth candidate alignment path;

on a condition that the first point is a point along the second candidate alignment path;

generating an updated first alignment path by updating the first alignment path based on the second candidate alignment path;

omitting the first alignment path from the calibrated camera alignment model; and

including the updated first alignment path in the calibrated camera alignment model;

on a condition that the second point is a point along the fourth candidate alignment path;

generating an updated second alignment path by updating the second alignment path based on the fourth candidate alignment path;

omitting the second alignment path from the calibrated camera alignment model; and

including the updated second alignment path in the calibrated camera alignment model; and

outputting or storing the calibrated camera alignment model.

2. The non-transitory computer-readable storage medium of claim 1, wherein the first point is a point along the second candidate alignment path on a condition that a relative physical orientation of the first image capture device and the second image capture device concurrent with receiving the first input frame differs from a relative physical orientation of the first image capture device and the second image capture device concurrent with the camera alignment model.

3. The non-transitory computer-readable storage medium of claim 1, wherein the first input frame is one of a sequence of input frames captured by the first image capture device for a defined temporal period, further comprising:

omitting generating a calibrated camera alignment model based on each frame from the sequence of input frames other than the first input frame.

4. The non-transitory computer-readable storage medium of claim 1, wherein the camera alignment model is a previously generated calibrated camera alignment model calibrated based on input frames received prior to receiving the first input frame and reviving the second input frame.

5. The non-transitory computer-readable storage medium of claim 1, wherein the defined location is a defined relative longitude.

6. The non-transitory computer-readable storage medium of claim 5, wherein the first alignment path indicates:

a first end of the first alignment path at a first portion of the first region of the first input frame proximate to a location in the first input frame corresponding to the defined relative longitude and a defined relative latitude; and

a second end of the first alignment path at a second portion of the first region of the first input frame proximate to a location in the first input frame corresponding to the defined relative longitude and a distal relative latitude.

7. The non-transitory computer-readable storage medium of claim 6, wherein the defined relative latitude is the relative equator, and the distal relative latitude is a latitude a defined distance from the equator distal to an optical center of the first image capture apparatus.

8. The non-transitory computer-readable storage medium of claim 7, wherein the distal relative latitude is at an edge of the first input frame.

9. The non-transitory computer-readable storage medium of claim 1, wherein the camera alignment model includes information describing a plurality of alignment paths for a plurality of defined locations in the first region of the first input frame, wherein each alignment path from the plurality of alignment paths corresponds with a respective defined location from the plurality of defined locations, and wherein the first alignment path is one of the plurality of alignment paths and the defined location is one of the plurality of defined locations.

10. The non-transitory computer-readable storage medium of claim 9, wherein each defined location from the plurality of defined locations corresponds with a respective distinct longitude.

11. The non-transitory computer-readable storage medium of claim 9, wherein a distance between the first candidate alignment path and the second candidate alignment path is within half of a distance between the first alignment path and an adjacent alignment path from the plurality of alignment paths in the first lateral direction.

12. The non-transitory computer-readable storage medium of claim 1, wherein identifying the first point includes:

generating a plurality of match quality metrics including a respective match quality metric for each point from a plurality of points along the first candidate alignment path, wherein the plurality of points along the first candidate alignment path includes the first point, and wherein each point from the plurality of points along the first candidate alignment path corresponds with a distinct relative latitude, and wherein determining the respective match quality metric for the first point includes determining a difference between a first block of pixels from the first input frame centered on the first point and a second block of pixels from the second input frame centered on the second point; and

identifying the first point from the plurality of points along the first candidate alignment path the match quality metric for the first point is a minimal match quality metric from the plurality of match quality metrics such that a smoothness criterion is satisfied.

13. A method comprising:

identifying, by an image signal processor of an image capture apparatus, a first input frame captured by a first image capture device of the image capture apparatus;

identifying, by the image signal processor, a second input frame captured by a second image capture device of the image capture apparatus, wherein the first frame includes a first overlap region including content and the second input frame includes a second overlap region including the content;

calibrating a camera alignment model, wherein calibrating the camera alignment model includes;

identifying a camera alignment model for the first image capture device and the second image capture device, wherein the camera alignment model includes information describing a first alignment path for a defined location in the first region of the first input frame and a second alignment path for the defined location in the second region of the second input frame;

identifying a first candidate alignment path for aligning a first portion of the first overlap region with a second portion of the second overlap region, wherein a previously generated camera alignment model indicates the first candidate alignment path;

identifying a second candidate alignment path spatially adjacent to the first candidate alignment path in a first lateral direction for aligning the first portion of the first overlap region with the second portion of the second overlap region; and

in response to a determination to align the first portion of the first overlap region with the second portion of the second overlap region based on the second candidate alignment path;

generating a calibrated camera alignment model based on the previously generated camera alignment model and the second candidate alignment path;

aligning the first portion of the first overlap region with the second portion of the second overlap region based on the second candidate alignment path; and

outputting or storing the calibrated camera alignment model;

generating a combined image based on the calibrated camera alignment model; and

outputting or storing the combined image.

14. The method of claim 13, wherein calibrating the camera alignment model includes:

determining to align the first portion of the first overlap region with the second portion of the second overlap region based on the second candidate alignment path on a condition that a relative physical orientation of the first image capture device and the second image capture device concurrent with a temporal location associated with the first input frame differs from a relative physical orientation of the first image capture device and the second image capture device concurrent with the previously generated camera alignment model.

15. The method of claim 13, wherein the first input frame is one of a sequence of input frames captured by the first image capture device for a defined temporal period, further comprising:

omitting calibrating the camera alignment model based on each input frame from the sequence of input frames other than the first input frame.

16. The method of claim 13, wherein the previously generated camera alignment model is a previously calibrated camera alignment model calibrated based on input frames associated with a temporal location preceding a temporal location associated with the first input frame.

17. An image capture apparatus comprising:

a first image capture device having a first field-of-view;

a second image capture device having a second field-of-view, wherein the first field-of-view overlaps the second field-of-view;

an image signal processor configured to;

receive a first series of input frames captured by the first image capture device;

receive a second series of input frames captured by the second image capture device, wherein each input frame from the first series of input frames corresponds temporally with a respective corresponding input frame from the second series of input frames; and

generate a series of combined frames based on the first series of input frames, the second series of input frames, and a camera alignment model, wherein generating the series of combined frames includes periodically calibrating the camera alignment model, wherein calibrating the camera alignment model includes identifying a change in relative orientation between the first image capture device and the second image capture device based on a first input frame from the first series of input frames and a first corresponding input frame from the second series of input frames; and

a memory for storing the series of combined frames.

18. The image capture apparatus of claim 17, wherein calibrating the camera alignment model includes:

identifying a first alignment path from the camera alignment model, the first alignment path extending from a first pixel of the first frame, the first pixel corresponding to a relative longitude and a midpoint between the first field-of-view and the second field-of-view, the path extending to a second pixel of the first frame at an edge of the first frame, such that for a latitude along the path, a longitude of the path differs from the relative longitude by an amount based on an expected relative orientation of the first image capture device and the second image capture device indicated by the camera alignment model;

identifying a second alignment path corresponding to and longitudinally offset from the first alignment path; and

identifying the change in relative orientation in response to generating a first combined frame from the series of combined frames based on the first input frame, the first corresponding input frame, and a portion of the second alignment path.

19. The image capture apparatus of claim 18, wherein the first series of input frames has a frame rate, the second series of input frames has the frame rate, and periodically calibrating the camera alignment model includes calibrating the camera alignment model includes at a camera alignment calibration rate that is less than half the frame rate.

20. The image capture apparatus of claim 19, wherein the frame rate is at least 24 frames per second and the camera alignment calibration rate is one frame per second.