Low power framework for controlling image sensor mode in a mobile image capture device

• US 10,225,511 B1
• Filed: 12/30/2015
• Issued: 03/05/2019
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lim, Suk Hwan, Aguera-Arcas, Blaise
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US14/984,869
• Time in Patent Office: 1,161 Days
• Field of Search: 3482201, 3482311-2313, 348364, 348372
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/426 using memory downsizing methods
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/65 : Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/8042 : involving data reduction
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for controlling image sensor mode in a mobile image capture device. On example low power frame work includes a scene analyzer that analyzes a scene depicted by a first image and, based at least in part on such analysis, causes an image sensor control signal to be provided to an image sensor to adjust at least one of the frame rate and the resolution of the image sensor.

Claims

1. A resource-efficient image capture device, the image capture device comprising:

an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;

an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;

a temporary image buffer that temporarily stores the one or more first images; and

a scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;

wherein the scene analyzer is operable to determine a desirability of the scene depicted by the at least one of the one or more first images;

wherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to increase at least one of the frame rate or the resolution at which the image sensor provides the raw image data; and

wherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is not desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to decrease at least one of the frame rate or the resolution at which the image sensor provides the raw image data.

2. The image capture device of claim 1, wherein the image sensor is further operable to adjust, based at least in part on the image sensor control signal, at least one of an exposure time, a gain, a region of interest, a crop, and a binning mode of the image sensor, and wherein the scene analyzer causes, based at least in part on the analysis of the scene, the image sensor control signal to be provided to the image sensor to adjust at least one of the exposure time, the gain, the region of interest, the crop, and the binning mode of the image sensor.

3. The image capture device of claim 1, wherein:

the image capture device is operable in at least a low resolution capture mode, a high resolution capture mode, and a burst capture mode;

during a regular mode of operation, the image capture device periodically transitions between the low resolution capture mode and the high resolution capture mode; and

when the scene analyzer determines that a particular scene depicted by a particular image is desirable, the scene analyzer causes the image capture device to transition to the burst capture mode.

4. A method for operating an image capture device, the method comprising:

outputting, by an image sensor of the image capture device, raw image data, the image sensor capable of adjusting at least one of a frame rate and a resolution in response to an image sensor control signal;

streaming, by an imaging pipeline of the image capture device, the raw image data from the image sensor;

processing, by the imaging pipeline, the raw image data to generate a first image;

analyzing, by the image capture device, a scene depicted by the first image; and

providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to cause the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;

wherein analyzing, by the image capture device, the scene depicted by the first image comprises determining, by the image capture device, that the scene depicted by the first image has one or more desirable characteristics; and

wherein providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal comprises providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to cause the image sensor to increase at least one of the frame rate or the resolution.

5. The method of claim 4, further comprising:

providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to adjust at least one of an exposure time, a gain, a region of interest, a crop, and a binning mode of the image sensor.

6. The method of claim 4, wherein:

analyzing, by the image capture device, the scene depicted by the first image comprises determining, by the image capture device, that the scene depicted by the first image has one or more desirable characteristics; and

providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to cause the image sensor to adjust at least one of the frame rate and the resolution comp rises selecting, by the image capture device, one of a plurality of different capture modes in which to operate, the plurality of different capture modes respectively having a plurality of different combinations of frame rates and resolutions.

7. A resource-efficient image capture device, the image capture device comprising:

an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, a frame rate of the image sensor;

an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;

a temporary image buffer that temporarily stores the one or more first images; and

a scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust the frame rate of the image sensor;

wherein the scene analyzer is operable to determine a desirability of the scene depicted by the at least one of the one or more first images;

wherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to increase the frame rate at which the image sensor provides the raw image data; and

wherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is not desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to decrease the frame rate at which the image sensor provides the raw image data.

8. The image capture device of claim 7, wherein the image sensor is further operable to adjust, based at least in part on the image sensor control signal, at least one of an exposure time, a gain, a region of interest, a crop, and a binning mode of the image sensor, and wherein the scene analyzer causes, based at least in part on the analysis of the scene, the image sensor control signal to be provided to the image sensor to adjust at least one of the exposure time, the gain, the region of interest, the crop, and the binning mode of the image sensor.

9. A resource-efficient image capture device, the image capture device comprising:

an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;

an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;

a temporary image buffer that temporarily stores the one or more first images;

a scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor, wherein, in at least one of a plurality of capture modes, the scene analyzer determines, based at least in part on the analysis of the scene depicted by the at least one of the one or more first images, whether to store the one or more first images in a non-volatile memory or to discard the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory;

an image compression component that, in the at least one of the plurality of capture modes, compresses the one or more first images only when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory, such that the image compression component expends energy on compressing only the first images selected for storage at the non-volatile memory; and

the non-volatile memory to which the one or more first images are written after compression.

10. The image capture device of claim 9, wherein, in at least a high resolution capture mode:

the image sensor provides high resolution raw image data;

the image capture device further comprises a temporary raw data buffer that temporarily stores the high resolution raw image data;

the image capture device further comprises a down sampler that streams the high resolution raw image data from the image sensor and down samples the high resolution raw image data to output low resolution raw image data;

the imaging pipeline streams the low resolution raw image data from the down sampler and processes the low resolution raw image data to generate low resolution versions of the one or more first images;

the temporary image buffer temporarily stores the low resolution versions of the one or more first images; and

the scene analyzer analyzes at least one of the low resolution versions of the one or more first images to determine whether to store the one or more first images in the non-volatile memory or to discard the one or more first images.

11. The image capture device of claim 10, wherein, in at least the high resolution capture mode, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory, the imaging pipeline receives the high resolution raw image data from the temporary raw data buffer and processes the high resolution raw image data to generate high resolution versions of the one or more first images, and the image compression component compresses the high resolution versions of the one or more first images for storage at the non-volatile memory.

12. The image capture device of claim 10, wherein, in at least a low resolution capture mode:

the image sensor performs binning to provide low resolution raw image data;

the imaging pipeline streams the low resolution raw image data from the image sensor and processes the low resolution raw image data to generate low resolution versions of the one or more first images;

the temporary image buffer temporarily stores the low resolution versions of the one or more first images; and

the scene analyzer analyzes at least one of the low resolution versions of the one or more first images to determine whether to non-temporarily store the low resolution versions of the one or more first images or to discard the low resolution versions of the one or more first image.

13. The image capture device of claim 9, wherein the temporary image buffer comprises dynamic random-access memory.

14. A resource-efficient image capture device, the image capture device comprising:

an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;

an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;

a temporary image buffer that temporarily stores the one or more first images;

a scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;

an image compression component that compresses images; and

a non-volatile memory to which images are written after compression;

wherein, in at least a burst capture mode of operation;

the imaging pipeline streams the raw image data from the image sensor and processes the raw image data to generate a second image; and

the second image is provided to the image compression component for compression and then stored at the non-volatile memory without being analyzed by the scene analyzer.

15. The image capture device of claim 14, wherein, in at least the burst capture mode, the image capture device captures high resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured high resolution images.

16. A resource-efficient image capture device, the image capture device comprising:

an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;

an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;

a temporary image buffer that temporarily stores the one or more first images; and

a scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;

wherein the scene analyzer comprises;

at least one neural network that receives the first image as an input and outputs at least one descriptor of a desirability of the scene depicted by the first image; and

a mode controller that selects, based at least in part on the at least one descriptor, one of a plurality of capture states, wherein the image capture device thereafter operates in the capture state selected by the mode controller.