IMAGE SENSOR ALIGNMENT IN A MULTI-CAMERA SYSTEM ACCELERATOR ARCHITECTURE

• US 20160173785A1
• Filed: 02/20/2016
• Published: 06/16/2016
• Est. Priority Date: 10/01/2013
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Campbell, Scott Patrick, Mobbs, Paul, Adsumilli, Balineedu Chowdary, Chawla, Sumit
• Granted Patent: US 9,485,417 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 1/6019 using look-up tables H04N1/6025 takes precedence
• CPC Class Codes: H04N 1/6027 Correction or control of colour gradation or colour contrast H04N1/6058 takes precedence
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/1883 the unit relating to sub-band structure, e.g. hierarchical level, directional tree, e.g. low-high [LH], high-low [HL], high-high [HH]
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6811 : based on the image signal
• CPC Class Codes: H04N 23/683 : performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/73 : by influencing the exposure time
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 : Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 : for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/75 : Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/76 : Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

An image capture accelerator performs accelerated processing of image data. In one embodiment, the image capture accelerator includes accelerator circuitry including a pre-processing engine and a compression engine. The pre-processing engine is configured to perform accelerated processing on received image data, and the compression engine is configured to compress processed image data received from the pre-processing engine. In one embodiment, the image capture accelerator further includes a demultiplexer configured to receive image data captured by an image sensor array implemented within, for example, an image sensor chip. The demultiplexer may output the received image data to an image signal processor when the image data is captured by the image sensor array in a standard capture mode, and may output the received image data to the accelerator circuitry when the image data is captured by the image sensor array in an accelerated capture mode.

Claims

1. A camera system, comprising:

a first image sensor chip configured to produce first image data representative of light incident upon the first image sensor chip;

an image capture accelerator chip (“

ICA”

) comprising;

a compression engine configured to decimate the first image data into a plurality of first image sub-band components; and

one or more outputs configured to output the first image data and a first image sub-band component; and

an image signal processor chip (“

ISP”

), the ICA coupled between the first image sensor chip and the ISP, the ISP comprising;

one or more inputs configured to received the first image data, the first image sub-band component, and a second image sub-band component representative of second image data from a second camera system, the second image data representative of light incident upon a second image sensor chip of the second camera system; and

an alignment engine configured to adjust a first field of view of the first image sensor chip or a second field of view of the second image sensor chip based on the first image sub-band component and the second image sub-band component.

2. The camera system of claim 1, wherein the first image sub-band component comprises a first low/low sub-band component, and wherein the second image sub-band component comprises a second low/low sub-band component, the first low/low sub-band component comprising a resolution lower than the first image data, and the second low/low sub-band component comprising a resolution lower than the second image data.

3. The camera system of claim 1, wherein the first image sub-band component comprises a first high/high sub-band component, and wherein the second image sub-band component comprises a second high/high sub-band component.

4. The camera system of claim 1, wherein the ICA is configured to receive the second image data, wherein the compression engine is further configured to decimate the second image data to produce the second image sub-band component, and wherein the second output is further configured to output the second image sub-band component.

5. The camera system of claim 1, wherein the second camera system comprises a second ICA chip configured to decimate the second image data to produce the second image sub-band component, and to provide the second image sub-band component to the ISP.

6. The camera system of claim 1, wherein the first field of view and the second field of view at least partially overlap.

7. The camera system of claim 1, wherein the alignment engine is further configured to align the first sub-band component and the second sub-band component, and wherein the adjustment of the first field of view or the second field of view is based on the alignment.

8. A method for aligning image sensors in a multi-camera system, comprising:

capturing, by a first image sensor chip, light incident upon the first image sensor chip to produce first image data representative of the captured light;

receiving, by an image capture accelerator chip (“

ICA”

), the first image data from the first image sensor chip;

outputting, by the ICA, the first image data to an image signal processor chip (“

ISP”

);

decimating, by the ICA, the first image data into a plurality of image sub-band components;

outputting, by the ICA, a first image sub-band components of the plurality of image sub-band components to the ISP;

accessing, by the ISP, a second image sub-band component representative of second image data from a second camera system, the second image data representative of light incident upon a second image sensor chip of the second camera system; and

adjusting, by the ISP, a first field of view of the first image sensor chip and a second field of view of the second image sensor chip based on the first image sub-band component and the second image sub-band component.

9. The method of claim 8, wherein the first image sub-band component comprises a first low/low sub-band component, and wherein the second image sub-band component comprises a second low/low sub-band component, the first low/low sub-band component comprising a resolution lower than the first image data, and the second low/low sub-band component comprising a resolution lower than the second image data.

10. The method of claim 8, wherein the first image sub-band component comprises a first high/high sub-band component, and wherein the second image sub-band component comprises a second high/high sub-band component.

11. The method of claim 8, wherein the ICA is further configured to receive the second image data, to decimate the second image data to produce the second image sub-band component, and to output the second image sub-band component.

12. The method of claim 8, wherein the second camera system comprises a second ICA chip configured to decimate the second image data to produce the second image sub-band component, and to provide the second image sub-band component to the ISP.

13. The method of claim 8, wherein the first field of view and the second field of view at least partially overlap.

14. The method of claim 8, wherein the ISP is further configured to align the first sub-band component and the second sub-band component, and wherein the adjustment of the first field of view and the second field of view is based on the alignment.

15. A multi-camera system comprising, comprising:

a first camera comprising;

a first image sensor chip corresponding to a first field of view, the first camera configured to produce first image data representative of light incident upon the first image sensor chip;

a first image capture accelerator chip (“

ICA”

) configured to output the first image data for processing and to decimate the first image data to produce a first image sub-band component; and

a first image signal processing chip (“

ISP”

) configured to process the first image data;

a second camera comprising;

a second image sensor chip corresponding to a second field of view, the second camera configured to produce second image data representative of light incident upon the second image sensor chip;

a second image capture accelerator chip (“

ICA”

) configured to output the second image data for processing and to decimate the second image data to produce a second image sub-band component; and

a second image signal processing chip (“

ISP”

) configured to process the second image data; and

an alignment engine configured to adjust the first field of view of the first image sensor chip or the second field of view of the second image sensor chip based on the first image sub-band component and the second image sub-band component.

16. The multi-camera system of claim 15, wherein the first image sub-band component comprises a first low/low sub-band component, and wherein the second image sub-band component comprises a second low/low sub-band component, the first low/low sub-band component comprising a resolution lower than the first image data, and the second low/low sub-band component comprising a resolution lower than the second image data.

17. The multi-camera system of claim 15, wherein the first image sub-band component comprises a first high/high sub-band component, and wherein the second image sub-band component comprises a second high/high sub-band component.

18. The multi-camera system of claim 15, wherein the first field of view and the second field of view at least partially overlap.

19. The multi-camera system of claim 15, wherein the alignment engine is further configured to align the first sub-band component and the second sub-band component, and wherein the adjustment of the first field of view or the second field of view is based on the alignment.

20. The multi-camera system of claim 15, wherein the alignment engine is implemented within the first ISP or the second ISP.