Low power framework for processing, compressing, and transmitting images at a mobile image capture device

• US 9,838,641 B1
• Filed: 12/30/2015
• Issued: 12/05/2017
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lim, Suk Hwan, Aguera-Arcas, Blaise
• Primary Examiner(s): Vu, Ngoc-Yen
• Application Number:
  US14/984,825
• Time in Patent Office: 706 Days
• Field of Search: 3482221, 34820799
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/65 Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for processing, compressing, and transmitting images at a mobile image capture device. One example low power framework includes a scene analyzer that analyzes a scene depicted by a first image and determines whether to store the first image in a non-volatile memory or to discard the first image from a temporary image buffer without storing the first image in the non-volatile memory.

Claims

1. An image capture device that has a low-power framework for processing imagery, the image capture device comprising:

an image sensor that provides high resolution raw image data;

a temporary raw data buffer that temporarily stores the high resolution raw image data;

a down sampler that down samples the high resolution raw image data to output low resolution raw image data;

an imaging pipeline that streams the low resolution raw image data from the down sampler and processes the low resolution raw image data to generate low resolution versions of one or more first images;

a temporary image buffer that temporarily stores the low resolution versions of the one or more first images;

a scene analyzer that analyzes at least a portion of a scene depicted by at least one of the low resolution versions one or more first images and determines whether to store the one or more first images in a non-volatile memory or to discard the low resolution versions of the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory;

an image compression component that compresses the one or more first images when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory; and

the non-volatile memory to which the one or more first images are written after compression.

2. The image capture device of claim 1, wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory:

the image compression component compresses the low resolution versions of the one or more first images; and

the image capture device writes the low resolution versions of the one or more first images to the non-volatile memory after compression.

3. The image capture device of claim 1, wherein, in a base capture mode of operation, the image sensor periodically transitions between providing high resolution raw image data and low resolution raw image data.

4. The image capture device of claim 1, wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory:

the imaging pipeline receives the high resolution raw image data from the temporary raw data buffer and processes the high resolution raw image data to generate high resolution versions of the one or more first images;

the image compression component compresses the high resolution versions of the one or more first images; and

the image capture device writes the high resolution versions of the one or more first images to the non-volatile memory after compression.

5. The image capture device of claim 4, wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory, the low resolution versions of the one or more first images are also compressed and written to the non-volatile memory.

6. The image capture device of claim 1, wherein, in at least a second mode of operation:

the imaging pipeline streams the raw image data from the image sensor and processes the raw image data to generate a second image;

the second image is provided to the image compression component for compression and then storage at the non-volatile memory without being analyzed by the scene analyzer.

7. The image capture device of claim 6, wherein the second mode of operation comprises a burst mode of operation in which the image capture device captures high resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured high resolution images.

8. The image capture device of claim 6, wherein the second mode of operation comprises a hyper-lapse mode of operation in which the image capture device captures mid resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured mid resolution images.

9. The image capture device of claim 1, wherein the scene analyzer comprises:

at least one neural network that receives the at least one of the one or more first images as an input and outputs at least one descriptor of a desirability of the scene depicted by at least one of the one or more first images; and

a save controller that determines, based at least in part on the at least one descriptor, whether to store the one or more first images in the non-volatile memory or to discard the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory.

10. The image capture device of claim 1, wherein the image capture device comprises a mobile image capture device that, at least in operation, continuously captures and processes imagery.

11. A method for processing imagery, the method comprising:

outputting, by an image sensor of an image capture device, high resolution raw image data;

temporarily storing, by a temporary raw data buffer of the image capture device, the high resolution raw image data; and

down sampling the high resolution raw image data to output low resolution raw image data;

processing, by the imaging pipeline, the low resolution raw image data to generate a low resolution version of a first image;

temporarily storing, by a temporary image buffer of the image capture device, the low resolution version of the first image;

analyzing a scene depicted by the low resolution version of the first image;

determining based at least in part on the analysis of the scene depicted by the low resolution version of the first image whether to store the first image in the non-volatile memory or to discard the low resolution version of the first image from the temporary image buffer without storing the first image in the non-volatile memory; and

when it is determined that the first image should be stored in the non-volatile memory;

compressing the first image; and

writing the first image to the non-volatile memory after compression.

12. The method of claim 11, wherein, when it is determined that the first image should be stored in the non-volatile memory:

compressing the first image comprises compressing the low resolution version of the first image; and

writing the first image to the non-volatile memory comprises writing the low resolution version of the first image to the non-volatile memory after compression.

13. The method of claim 11, further comprising, when it is determined that the first image should be stored in the non-volatile memory:

receiving, by the imaging pipeline;

the high resolution raw image data from the temporary raw data buffer;

processing, by the imaging pipeline, the high resolution raw image data to generate a high resolution version of the first image;

wherein compressing the first image comprises compressing the high resolution version of the first image; and

wherein writing the first image to the non-volatile memory comprises writing the high resolution version of the first image to the non-volatile memory after compression.

14. The method of claim 11, wherein;

in at least one mode of operation, the method comprises;

outputting, by the image sensor of an image capture device, low resolution raw image data rather than high resolution raw image data.

15. The method of claim 11, further comprising:

streaming, by the imaging pipeline, a second set of raw image data from the image sensor;

processing, by the imaging pipeline, the second set of raw image data to generate a second image; and

without providing the second image to the scene analyzer;

compressing the second image; and

writing the second image to the non-volatile memory after compression.

16. A mobile image capture device that has a low power image capture framework that is operable in a plurality of different capture modes, the image capture framework comprising:

an image sensor operable to output raw image data of different resolutions depending on the capture mode;

one or more image processing components operable to process the raw image data into one or more images;

one or more temporary buffers operable to temporarily store the one or more images;

a scene analyzer operable to analyze a scene depicted by at least one of the one or more images and determine, based at least in part on such analysis, whether to store such image at a non-volatile memory or to discard such image without storing such image at the non-volatile memory;

an image compression component operable to compress one or more images; and

the non-volatile memory operable to store non-temporary copies of one or more images,

wherein in a first capture mode of the plurality of different capture modes;

the scene analyzer analyzes each image to determine whether to store such image at the non-volatile memory or to discard such image without storing such image at the non-volatile memory; and

only images that the scene analyzer has determined should be stored in the non-volatile memory are provided to the image compression component for compression and then written to the non-volatile memory; and

wherein in a second capture mode of the plurality of different capture modes, all of the images are provided to the image compression component for compression and then written to the non-volatile memory without being analyzed by the scene analyzer.

17. The mobile image capture device of claim 16, wherein, in at least one of the plurality of capture modes:

the image sensor provides high resolution raw image data for each of the images;

the one or more temporary buffers comprise a temporary raw data buffer that temporarily stores the high resolution raw image data for each of the images;

the image capture framework further comprises a down sampler that streams the high resolution raw image data from the image sensor and down samples the high resolution raw image data to output low resolution raw image data;

the one or more image processing components stream the low resolution raw image data from the down sampler and process the low resolution raw image data to generate a low resolution version of each of the images;

the one or more temporary buffers temporarily store the low resolution version of each of the images;

the scene analyzer analyzes the low resolution version of each of the images to determine whether to store such image at the non-volatile memory or to discard such image; and

when the scene analyzer determines that a particular image should be stored in the non-volatile memory;

the one or more image processing components receive the high resolution raw image data the temporary raw data buffer and process the high resolution raw image data to generate a high resolution version of the particular image;

the image compression component compresses the high ion version of the particular image; and

the mobile image capture device writes the high resolution version of the particular image to the non-volatile memory after compression.

18. The mobile image capture device of claim 16, wherein the second capture mode comprises a burst capture mode.

19. The mobile image capture device of claim 16, wherein the first capture mode comprises a base capture mode in which the image sensor periodically transitions between providing high resolution raw image data and low resolution raw image data.

20. The mobile image capture device of claim 16, wherein the second mode of operation comprises a hyper-lapse mode of operation in which the image capture device captures mid resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured mid resolution images.

21. An image capture device that has a low-power framework for processing imagery, the image capture device comprising:

an image sensor that performs binning to provide low resolution raw image data;

an imaging pipeline that streams the low resolution raw image data and processes the low resolution raw image data to generate low resolution versions of one or more first images;

a temporary image buffer that temporarily stores the low resolution versions the one or more first images;

a scene analyzer that analyzes at least a portion of a scene depicted by at least one of the low resolution versions of the one or more first images and determines whether to store the lower resolution versions of the one or more first images in a non-volatile memory or to discard the low resolution versions of the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory;

an image compression component that compresses the low resolution versions of the one or more first images when the scene analyzer determines that the low resolution versions of the one or more first images should be stored in the non-volatile memory; and

the non-volatile memory to which the low resolution versions of the one or more first images are written after compression.