Systems and methods for selective retention and editing of images captured by mobile image capture device

• US 10,732,809 B2
• Filed: 03/06/2018
• Issued: 08/04/2020
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bialynicka-Birula, Iwona, Aguera-Arcas, Blaise, Ramage, Daniel, McMahan, Hugh Brendan, Lange, Oliver Fritz, Fortuna, Emily Anne, Tyamagundlu, Divya, Holbrook, Jess, Kohlhepp, Kristine, Payne, Juston, Duleba, Krzysztof, Vanik, Benjamin, Lentz, Alison, Clapper, Jon Gabriel, Lovejoy, Joshua Denali, Donsbach, Aaron Michael
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US15/913,793
• Publication Number:
  US 20180196587A1
• Time in Patent Office: 882 Days
• Field of Search: None
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 1/215 Recording a sequence of still pictures, e.g. burst mode

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. The mobile image capture device is operable to input an image into at least one neural network and to receive at least one descriptor of the desirability of a scene depicted by the image as an output of the at least one neural network. The mobile image capture device is operable to determine, based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of such image and/or one or more contemporaneously captured images in a non-volatile memory of the mobile image capture device or to discard a first copy of such image from a temporary image buffer without storing the second copy of such image in the non-volatile memory.

Claims

1. A system for selective retention of images, the system comprising:

a mobile image capture device comprising a network interface, a power source, an image sensor, at least one processor, and a memory that stores a machine-learned face recognition model usable to recognize faces depicted by images, the memory comprising at least a temporary image buffer and a non-volatile memory, the memory storing data descriptive of a set of faces that have an elevated importance to a user of the mobile image capture device, the mobile image capture device configured to;

capture a first image that depicts one or more faces;

maintain a first copy of the first image in the temporary image buffer;

input the first image into the machine-learned face recognition model;

receive an output of the machine-learned face recognition model that describes the one or more faces depicted by the first image and recognized by the machine-learned face recognition model;

determine, based at least in part on the output of the machine-leafed face recognition model, whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user; and

determine, based at least in part on whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image from the temporary image buffer without storing a second copy of the first image in the non-volatile memory.

2. The system of claim 1, wherein:

the output of the machine-learned face recognition model comprises one or more embeddings respectively associated with the one or more faces depicted by the first image; and

to determine whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, the mobile image capture device is configured to respectively compare the embedding associated with each of the one or more faces to each of a set of embeddings respectively associated with the set of faces that have elevated importance to the user.

3. The system of claim 1, wherein the machine-learned face recognition model comprises a deep neural network.

4. The system of claim 1, wherein:

the memory of the mobile image capture device further stores a machine-learned face tracking model usable to perform face tracking across a plurality of input images and provide a bounding box descriptive of a location of a tracked face within a given input image of the plurality of input images; and

to input the first image into the machine-learned face recognition model, the mobile image capture device is configured to;

extract one or more face crops from the first image based on one or more first bounding boxes output by the machine-learned face tracking model for the first image; and

input the one or more face crops into the machine-learned face recognition model.

5. The system of claim 1, wherein the mobile image capture device is further configured to automatically populate the set of faces that have elevated importance to the user with one or more additional faces that are frequently recognized by the machine-learned face recognition model within images captured by the mobile image capture device.

6. The system of claim 1, wherein at least some of the faces included in the set of faces that have elevated importance to the user have been selected by the user of the mobile image capture device from existing images.

7. The system of claim 1, wherein the mobile image capture device is further configured to:

receive a plurality of existing images associated with the user;

identify one or more additional faces that are frequently recognized by the machine-learned face recognition model within the plurality of existing images; and

automatically add the one or more additional faces to the set of faces that have elevated importance to the user.

8. The system of claim 1, wherein, when at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, the mobile image capture device is configured to increase a probability that the second copy of the first image is stored in the non-volatile memory.

9. The system of claim 1, wherein, when at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user, the mobile image capture device is configured to:

automatically determine a recommended composition for the first image based at least in part on a location associated with the at least one face that is included in the set of faces, wherein the recommended composition for the first image comprises a suggested crop of the first image that includes the at least one face that is included in the set of faces.

10. The system of claim 1, wherein, when at least one of the one or more faces depicted by the first image is not included in the set of faces that have elevated importance to the user, the mobile image capture device is further configured to:

automatically determine a recommended composition for the first image, wherein the recommended composition comprises at least one of a blurring of or a cropping to exclude the at least one face that is not included in the set of faces.

11. The system of claim 1, further comprising a user computing device communicatively coupled to the mobile image capture device, wherein the user computing device is configured to provide an interactive user interface that enables the user to edit the set of faces that have an elevated importance to the user.

12. The system of claim 1, further comprising a user computing device, the user computing device configured to:

selectively communicatively connect to the mobile image capture device;

select one or more of a plurality of images stored in the non-volatile memory of the user computing device for transfer, wherein images that depict faces included in the set of faces that have elevated importance to the user are preferentially selected for transfer relative to images that do not depict faces included in the set of faces;

receive from the mobile image capture device the selected one or more images; and

provide a user interface that enables a user to edit the one or more images received from the mobile image capture device.

13. The system of claim 1, wherein the mobile image capture device is configured to:

maintain data sufficient to generate a high resolution copy of the first image in a temporary raw data buffer;

input a low resolution copy of the first image into the machine-learned face recognition model; and

determine, based at least in part on the output of the machine-learned face recognition model, whether to store a high resolution copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image from the temporary image buffer without storing a high resolution copy of the first image in the non-volatile memory.

14. An image capture device, comprising:

one or more processors;

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the image capture device to perform operations, the operations comprising;

capturing a first image that depicts one or more faces;

storing the first image in a volatile memory of the image capture device;

inputting the first image into a machine-learned face recognition model;

receiving an output of the machine-learned face recognition model that describes the one or more faces depicted by the first image and recognized by the machine-learned face recognition model;

determining, based at least in part on the output of the machine-learned face recognition model, whether at least one of the one or more faces depicted by the first image is included in a set of faces that have elevated importance to a user, the set of faces including at least a first face that is associated with an individual that is separate and distinct from the user, wherein the first face was selected for inclusion in the set of faces by the user; and

determining whether to store the first image in a non-volatile memory of the image capture device based at least in part on whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user;

in response to determining to store the first image in the non-volatile memory, storing the first image in the non-volatile memory; and

in response to determining to not store the first image in the non-volatile memory, discarding the first image from the volatile memory without storing the first image in the non-volatile memory.

15. The image capture device of claim 14, wherein:

the output of the machine-learned face recognition model comprises one or more embeddings respectively associated with the one or more faces depicted by the first image; and

determining whether at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user comprises respectively comparing the embedding associated with each of the one or more faces to each of a set of embeddings respectively associated with the set of faces that have elevated importance to the user.

16. The image capture device of claim 14, wherein the machine-learned face recognition model comprises a deep neural network.

17. The image capture device of claim 14, wherein the operations further comprise:

inputting the first image into a machine-learned face tracking model that performs face tracking across a plurality of input images; and

receiving a bounding box from the machine-learned face tracking model that is descriptive of a location of a tracked face within the first image;

wherein inputting the first image into the machine-learned face recognition model comprises;

extracting a face crop from the first image based on the bounding box output by the machine-learned face tracking model; and

inputting the face crop into the machine-learned face recognition model.

18. The image capture device of claim 14, wherein the operations further comprise automatically populating the set of faces that have elevated importance to the user with one or more additional faces that are frequently recognized by the machine-learned face recognition model within images captured by the image capture device.

19. The image capture device of claim 14, wherein the operations further comprise, when at least one of the one or more faces depicted by the first image is included in the set of faces that have elevated importance to the user:

automatically determining a recommended composition for the first image based at least in part on a location associated with the at least one face that is included in the set of faces, wherein the recommended composition for the first image comprises a suggested crop of the first image that includes the at least one face that is included in the set of faces.

20. An image capture device that, at least in operation, captures imagery, the image capture device comprising:

a network interface;

a power source;

an image sensor;

at least one processor;

a memory that stores data descriptive of a set of faces that have elevated importance, the set of faces including at least a first face that is associated with an individual that is different from a user of the image capture device, wherein the user of the image capture device selected the first face for inclusion in the set of faces; and

a scene analyzer that includes;

a machine-learned face recognition model that recognizes one or more faces in a first image captured by the image sensor; and

a save controller that determines, based at least in part on whether the one or more faces recognized by the machine-learned face recognition model match at least one face in the set of faces that have elevated importance, whether to store the first image in a non-volatile memory of the image capture device or to discard the first image from a volatile memory of the image capture device without storing the first image in the non-volatile memory of the image capture device.