Image sensor alignment in a multi-camera system accelerator architecture

• US 9,628,718 B2
• Filed: 09/30/2016
• Issued: 04/18/2017
• Est. Priority Date: 10/01/2013
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Campbell, Scott Patrick, Mobbs, Paul, Adsumilli, Balineedu Chowdary, Chawla, Sumit
• Primary Examiner(s): LAM, HUNG H
• Application Number:
  US15/282,306
• Publication Number:
  US 20170019606A1
• Time in Patent Office: 200 Days
• Field of Search: 3482181, 3482221, 348294, 34833302
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 1/6019 using look-up tables H04N1/6025 takes precedence
• CPC Class Codes: H04N 1/6027 Correction or control of colour gradation or colour contrast H04N1/6058 takes precedence
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/1883 the unit relating to sub-band structure, e.g. hierarchical level, directional tree, e.g. low-high [LH], high-low [HL], high-high [HH]
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6811 : based on the image signal
• CPC Class Codes: H04N 23/683 : performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/73 : by influencing the exposure time
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 : Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 : for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/75 : Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/76 : Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

An image capture accelerator performs accelerated processing of image data. In one embodiment, the image capture accelerator includes accelerator circuitry including a pre-processing engine and a compression engine. The pre-processing engine is configured to perform accelerated processing on received image data, and the compression engine is configured to compress processed image data received from the pre-processing engine. In one embodiment, the image capture accelerator further includes a demultiplexer configured to receive image data captured by an image sensor array implemented within, for example, an image sensor chip. The demultiplexer may output the received image data to an image signal processor when the image data is captured by the image sensor array in a standard capture mode, and may output the received image data to the accelerator circuitry when the image data is captured by the image sensor array in an accelerated capture mode.

Claims

1. A camera system, comprising:

a first image sensor chip configured to produce first image data representative of light incident upon the first image sensor chip; and

an image signal processor chip (“

ISP”

) comprising;

one or more inputs configured to receive the first image data and to receive a second sub-band component representative of second image data from a second camera system, the camera system and the second camera system comprising at least partially overlapping fields of view;

a compression engine configured to decimate the first image data into a plurality of image sub-band components including a first sub-band component representative of the first image data; and

an alignment engine configured to adjust one or both of the fields of view of the camera system and the second camera system based on a comparison of the first sub-band component and the second sub-band component.

2. The camera system of claim 1, wherein the first image sub-band component comprises a first low/low sub-band component, and wherein the second image sub-band component comprises a second low/low sub-band component, the first low/low sub-band component comprising a resolution lower than the first image data, and the second low/low sub-band component comprising a resolution lower than the second image data.

3. The camera system of claim 1, wherein comparing the first sub-band component and the second sub-band component comprises aligning the first sub-band component and the second sub-band component.

4. The camera system of claim 3, wherein aligning the first sub-band component and the second sub-band component comprises determining a pixel difference between the first sub-band component and the second sub-band component.

5. The camera system of claim 4, wherein adjusting one or both of the fields of view of the camera system and the second camera system comprises adjusting one of the fields of view by the determined pixel difference between the first sub-band component and the second sub-band component.

6. The camera system of claim 1, wherein the second camera system comprises a compression engine configured to decimate the second image data to produce the second image sub-band component, and wherein the second camera system is configured to output the second image sub-band component to the ISP in response to the second image data being captured by the second camera system.

7. The camera system of claim 1, wherein the camera system is configured as a master camera system and wherein the second camera system is configured as a slave camera system.

8. A method for aligning cameras in a multi-camera system, comprising:

capturing, by a first image sensor chip of a first camera, light incident upon the first image sensor chip to produce first image data representative of light incident upon the first image sensor chip;

decimating, by a compression engine of the first camera, the first image data to produce a first sub-band component representative of the first image data;

receiving, by an input of the first camera, a second sub-band component from a second camera representative of second image data captured by the second camera, the first camera and the second camera having at least partially overlapping fields of view;

comparing, by the first camera, the first sub-band component and the second sub-band component; and

adjusting, by an alignment engine of the first camera, one or both of the fields of view of the first camera and the second camera based on the comparison of the first sub-band component and the second sub-band component.

9. The method of claim 8, wherein the first image sub-band component comprises a first low/low sub-band component, and wherein the second image sub-band component comprises a second low/low sub-band component, the first low/low sub-band component comprising a resolution lower than the first image data, and the second low/low sub-band component comprising a resolution lower than the second image data.

10. The method of claim 8, wherein comparing the first sub-band component and the second sub-band component comprises aligning the first sub-band component and the second sub-band component.

11. The method of claim 10, wherein aligning the first sub-band component and the second sub-band component comprises determining a pixel difference between the first sub-band component and the second sub-band component.

12. The method of claim 11, wherein adjusting one or both of the fields of view of the camera and the second camera comprises adjusting one of the fields of view by the determined pixel difference between the first sub-band component and the second sub-band component.

13. The method of claim 8, wherein the second camera comprises a compression engine configured to decimate the second image data to produce the second image sub-band component, and wherein the second camera is configured to output the second image sub-band component to the ISP in response to the second image data being captured by the second camera.

14. The method of claim 8, wherein the first camera is configured as a master camera and wherein the second camera is configured as a slave camera.

15. An integrated circuit chip for aligning cameras in a multi-camera system, comprising:

a first input configured to receive a first sub-band component representative of first

image data captured by a first image sensor chip of a first camera;

a second input configured to receive a second sub-band component representative of second image data captured by a second image sensor chip of a second camera, the first camera and the second camera having at least partially overlapping fields of view;

a comparison engine configured to compare the first sub-band component and the second sub-band component;

an alignment engine configured to adjust one or both of the fields of view of the first camera and the second camera based on the comparison of the first sub-band component and the second sub-band component; and

wherein the first image sub-band component comprises a first low/low sub-band component, and wherein the second image sub-band component comprises a second low/low sub-band component, the first low/low sub-band component comprising a resolution lower than the first image data, and the second low/low sub-band component comprising a resolution lower than the second image data.

16. The integrated circuit chip of claim 15, wherein comparing the first sub-band component and the second sub-band component comprises aligning the first sub-band component and the second sub-band component.

17. The integrated circuit chip of claim 16, wherein aligning the first sub-band component and the second sub-band component comprises determining a pixel difference between the first sub-band component and the second sub-band component.

18. The integrated circuit chip of claim 17, wherein adjusting one or both of the fields of view of the camera system and the second camera system comprises adjusting one of the fields of view by the determined pixel difference between the first sub-band component and the second sub-band component.

19. The integrated circuit chip of claim 15, wherein the first input and the second input comprise the same input.