Systems and methods that leverage deep learning to selectively store images at a mobile image capture device

• US 9,836,484 B1
• Filed: 12/30/2015
• Issued: 12/05/2017
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bialynicka-Birula, Iwona, Aguera-Arcas, Blaise, Ramage, Daniel, McMahan, Hugh Brendan, Lange, Oliver Fritz, Fortuna, Emily Anne, Tyamagundlu, Divya, Holbrook, Jess, Kohlhepp, Kristine, Payne, Juston, Duleba, Krzysztof, Vanik, Benjamin, Lentz, Alison, Clapper, Jon Gabriel, Lovejoy, Joshua Denali, Donsbach, Aaron Michael
• Primary Examiner(s): Hannett, James
• Application Number:
  US14/984,683
• Time in Patent Office: 706 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 1/2137 with temporary storage before final recording, e.g. in a frame buffer
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/65 Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. The mobile image capture device is operable to input an image into at least one neural network and to receive at least one descriptor of the desirability of a scene depicted by the image as an output of the at least one neural network. The mobile image capture device is operable to determine, based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of such image in a non-volatile memory of the mobile image capture device or to discard a first copy of such image from a temporary image buffer without storing the second copy of such image in the non-volatile memory.

Claims

1. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising:

a network interface;

a power source;

an image sensor;

at least one processor; and

a memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;

wherein the plurality of deep neural networks comprise one or more of;

a face detection deep neural network that detects a presence of one or more faces in the scene of each input image;

a face recognition deep neural network that matches one or more faces in the scene of each input image to one or more other faces;

a face attributes deep neural network that detects various facial characteristics of one or more faces in the scene of each input image;

an image content deep neural network that outputs one or more semantic labels that describe the scene of each input image; and

a photo quality deep neural network that outputs a photo score that describes various photographic quality characteristics of each input image; and

wherein the mobile image capture device is operable to;

capture a first image that depicts a scene;

maintain a first copy of the first image in the temporary image buffer;

input the first image into at least one of the plurality of deep neural networks;

receive at least one descriptor of the desirability of the scene depicted by the first image as an output from the at least one of the plurality of deep neural networks into which the first image is input; and

determine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image.

2. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to:

operate the image sensor in a plurality of different capture modes that respectively correspond to a plurality of different resolutions and frame rates,

select, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of the plurality of different capture modes; and

switch operation of the image sensor to the selected capture mode.

3. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to select the at least one of the plurality of deep neural networks into which the first image is input.

4. The mobile image capture device of claim 1, wherein the plurality of deep neural networks comprise a plurality of feed-forward deep neural networks.

5. The mobile image capture device of claim 1, wherein the plurality of deep neural networks comprise at least one convolutional neural network.

6. The mobile image capture device of claim 1, wherein each of the plurality of deep neural networks outputs at least one annotation for each input image, the at least one annotation for each input image indicative of the desirability of the scene depicted by such image.

7. The mobile image capture device of claim 1, wherein the plurality of deep neural networks comprise a multi-headed deep neural network that receives a single set of inputs and provides a plurality of outputs, wherein the plurality of outputs respectively include a plurality of descriptors of the desirability of the scene of each input image.

8. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to, prior to inputting the first image into the at least one of the plurality of deep neural networks:

input the first image into a recurrent deep neural network that analyzes only a portion of the first image;

receive as output from the recurrent deep neural network an indication of whether the first image should be input into the at least one of the plurality of deep neural networks, for further analysis, such that die recurrent deep neural network operates as a prefilter to the plurality of deep neural networks; and

receive as output from the recurrent deep neural network a description of which portion of a next image the recurrent deep neural network should analyze.

9. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to receive data that describes a set of entities having an elevated importance to a user of the mobile image capture device, wherein the at least descriptor output by the at least one of the plurality of deep neural networks indicates whether a member of the set of entities is depicted in the scene of the first image.

10. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to:

receive, from another mobile image capture device that is proximately located, an excitement signal that indicates that the other mobile image capture device has recently captured an image that depicts a desirable scene; and

determine, based at least in part on the excitement signal, Whether to store a second Copy of a recently captured image in the non-volatile memory of the mobile image capture device or to discard a first copy of such image from the temporary image buffer without storing a second copy of such image in the non-volatile memory.

11. A resource-efficient mobile image capture device that, at least in operation, continuously captures imagery, the mobile image capture device comprising:

a network interface;

a power source;

an image sensor;

at least one processor;

a memory; and

a scene analyzer that includes;

at least one neural network that receives a first image captured by the image sensor and outputs at least one descriptor of a desirability of a scene depicted by the first image;

a save controller that determines, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the memory of the mobile image capture device or to discard the first image without storing a second copy of the first image; and

a mode controller that;

selects, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of a plurality of different capture modes of the mobile image capture device, wherein the plurality of different capture modes respectively correspond to a plurality of different resolutions and frame rates; and

switches operation of the image sensor to the selected capture mode.

12. The mobile image capture device of claim 11, wherein the scene analyzer further comprises an attention model that:

analyzes only a portion of the first image;

outputs an indication, based at least in part on the analysis of the portion of the first image, whether the first image should be further analyzed in its entirety; and

outputs a description of which portion of a second image the attention model should analyze, the second image temporally subsequent to the first image.

13. The mobile image capture device of claim 11, wherein:

the scene analyzer comprises a plurality of multi-layer non-linear models; and

the scene analyzer further comprises a model selector that selects at least one of the multi-layer non-linear models into which the first image is input.

14. The mobile image capture device of claim 11, wherein the scene analyzer comprises a multi-headed deep neural network that receives a single set of inputs and provides a plurality of sets of outputs, wherein the plurality of sets of outputs respectively include a plurality of descriptors of the desirability of the scene of each input image.

15. A method to selectively retain images, the method comprising:

capturing, by a mobile image capture device, an image that depicts a scene;

maintaining, by the mobile image capture device, a first copy of the image in a temporary image buffer of the mobile image capture device;

inputting, by the mobile image capture device, the image into at least one neural network to determine a desirability of the scene depicted by the image, the at least one neural network stored in a memory of the mobile image capture device, wherein inputting, by the mobile image capture device, the image into at least one neural network comprises;

inputting, by the mobile image capture device, the image into a recurrent neural network that analyzes only a portion of the image;

receiving, by the mobile image capture device, as output from the recurrent neural network a probability that the image warrants further analysis; and

when the probability exceeds a threshold, inputting the image into at least one feed-forward deep neural network;

receiving, by the mobile image capture device, at least one descriptor of the desirability of the scene depicted by the image as an output of the at least one neural network, wherein the descriptor of the desirability of the scene is received from the at least one feed-forward deep neural network; and

determining, by the mobile image capture device based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of the image in a nonvolatile memory of the mobile image capture device or to discard the first copy of the image from the temporary image buffer without storing a second copy of the image in the non-volatile memory.

16. The method of claim 15, wherein:

inputting, by the mobile image capture device, the image into at least one neural network comprises;

selecting, by the mobile image capture device, one or more of a plurality of deep neural networks; and

inputting, by the mobile image capture device, the image into the selected one or more deep neural networks;

receiving, by the mobile image capture device, at least one descriptor of the desirability of the scene comprises receiving, by the mobile image capture device, at least one annotation from each of the selected one or more deep neural networks, the at least one annotation for each of the selected one or more deep neural networks indicative of the desirability of the scene depicted by such image; and

determining, by the mobile image capture device based at least in part on the at least one descriptor of the desirability of the scene of each image, whether to store a second copy of such image in the non-volatile memory of the mobile image capture device comprises determining, by the mobile image capture device based at least in part on the at least one annotation received from each of the selected one or more deep neural networks, whether to store a second copy of such image in the non-volatile memory of the mobile image capture device or to discard the first copy of such image from the temporary image buffer without storing a second copy of such image in the non-volatile memory.

17. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising:

a network interface;

a power source;

an image sensor;

at least one processor; and

a memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;

wherein the mobile image capture device is operable to;

capture a first image that depicts a scene;

maintain a first copy of the first image in the temporary image buffer;

input the first image into at least one of the plurality of deep neural networks;

receive at least one descriptor of the desirability of the scene depicted by the first image as an output from the at least one of the plurality of deep neural networks into which the first image is input; and

determine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image; and

wherein the mobile image capture device is further operable to;

operate the image sensor in a plurality of different capture modes that respectively correspond to a plurality of different resolutions and frame rates,

select, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of the plurality of different capture modes; and

switch operation of the image sensor to the selected capture mode.

18. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising:

a network interface;

a power source;

an image sensor;

at least one processor; and

a memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;

wherein the plurality of deep neural networks comprise a multi-headed deep neural network that receives a single set of inputs and provides a plurality of outputs, wherein the plurality of outputs respectively include a plurality of descriptors of the desirability of the scene of each input image; and

wherein the mobile image capture device is operable to;

capture a first image that depicts a scene;

maintain a first copy of the first image in the temporary image buffer;

input the first image into the multi-headed deep neural network;

receive the plurality of descriptors of the desirability of the scene depicted by the first image as an output from the multi-headed deep neural network into which the first image is input; and

determine, based at least in part on the plurality of descriptors of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image.

19. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising:

a network interface;

a power source;

an image sensor;

at least one processor; and

a memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;

wherein the mobile image capture device is operable to;

capture a first image that depicts a scene;

maintain a first copy of the first image in the temporary image buffer, wherein to maintain the first copy the mobile image capture device maintains temporary data sufficient to generate high resolution copy of the first image;

input the first image into at least one of the plurality of deep neural networks, wherein to input the first image the mobile image capture device inputs a low resolution copy of the first image into the at least one neural network;

receive at least one descriptor of the desirability of the scene depicted by the first image as an output from the at least one of the plurality of deep neural networks into which the first image is input; and

determine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image, wherein to determine whether to store the second copy, the mobile image capture device determines, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store the high resolution copy of the first image in the non-volatile memory of the mobile image capture device or to discard the temporary data without storing the high resolution copy of the first image.

20. A resource-efficient mobile image capture device that, at least in operation, continuously captures imagery, the mobile image capture device comprising:

a network interface;

a power source;

an image sensor;

at least one processor;

a memory; and

a scene analyzer that includes;

at least one neural network that receives a first image captured by the image sensor and outputs at least one descriptor of a desirability of a scene depicted by the first image; and

a save controller that determines, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the memory of the mobile image capture device or to discard the first image without storing a second copy of the first image;

wherein the scene analyzer comprises a plurality of multi-layer non-linear models; and

wherein the scene analyzer further comprises a model selector that selects at least one of the multi-layer non-linear models into which the first image is input.