Systems and methods for selective retention and editing of images captured by mobile image capture device

• US 9,836,819 B1
• Filed: 12/30/2015
• Issued: 12/05/2017
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Donsbach, Aaron Michael, Vanik, Benjamin, Clapper, Jon Gabriel, Lentz, Alison, Lovejoy, Joshua Denali, Fritz, III, Robert Douglas, Duleba, Krzysztof, Zhang, Li, Payne, Juston, Fortuna, Emily Anne, Bialynicka-Birula, Iwona, Aguera-Arcas, Blaise, Ramage, Daniel, McMahan, Hugh Brendan, Lange, Oliver Fritz, Holbrook, Jess
• Primary Examiner(s): Liew, Alex
• Application Number:
  US14/984,628
• Time in Patent Office: 706 Days
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. The mobile image capture device is operable to input an image into at least one neural network and to receive at least one descriptor of the desirability of a scene depicted by the image as an output of the at least one neural network. The mobile image capture device is operable to determine, based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of such image and/or one or more contemporaneously captured images in a non-volatile memory of the mobile image capture device or to discard a first copy of such image from a temporary image buffer without storing the second copy of such image in the non-volatile memory.

Claims

1. A system for selective retention and editing of images, the system comprising:

a continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising a network interface, a power source, an image sensor, at least one processor, and a memory that stores at least one neural network usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory, the mobile image capture device operable to;

capture a first image that depicts a scene;

maintain a first copy of the first image in the temporary image buffer;

input the first image into the at least one neural network;

receive at least one descriptor of the desirability of the scene depicted by the first image as an output of the at least one neural network;

determine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image from the temporary image buffer without storing a second copy of the first image in the non-volatile memory; and

transmit at least one of a plurality of images stored in the non-volatile memory to a user computing device.

2. The system of claim 1, further comprising the user computing device, the user computing device operable to:

selectively communicatively connect to the mobile image capture device;

select one or more of the plurality of images stored in the non-volatile memory of the user computing device for transfer;

receive from the mobile image capture device the selected one or more images; and

provide a user interface that enables a user to edit the one or more images received from the mobile image capture device.

3. The system of claim 1, wherein the at least one neural network comprises at least one convolutional neural network.

4. The system of claim 1, wherein:

the at least one neural network comprises a plurality of deep neural networks that respectively analyze a plurality of different characteristics of each input image;

the mobile image capture device is operable to input the first image into at least one of the plurality of deep neural networks; and

each of the plurality of deep neural networks outputs at least one annotation for each image input into such deep neural network, the at least one annotation for each input image indicative of the desirability of the scene depicted by such image.

5. The system of claim 4, wherein the mobile image capture device periodically receives one or more updated versions of one or more of the plurality of deep neural networks from one or more of the user computing device and a server computing device, the one or more updated versions of the one or more of the plurality of deep neural networks trained on additional data.

6. The system of claim 1, wherein the mobile image capture device is operable to:

maintain data sufficient to generate a high resolution copy of the first image in a temporary raw data buffer;

input a low resolution copy of the first image into the at least one neural network; and

determine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a high resolution copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image from the temporary image buffer without storing a high resolution copy of the first image in the non-volatile memory.

7. The system of claim 1, further comprising the user computing device, wherein the mobile image capture device further comprises a wide angle lens;

and wherein the user computing device is operable to dewarp the one or more images received from the mobile image capture device to obtain one or more narrow angle images.

8. The system of claim 1, wherein:

the mobile image capture device is operable to transmit an excitement signal that is wirelessly receivable by one or more other mobile image capture devices that are located proximate to the mobile image capture device; and

the excitement signal indicates that the scene depicted by the first image has a large desirability.

9. The system of claim 8, wherein, in response to receipt of the excitement signal, the one or more other mobile image capture devices perform at least one of the following:

increase a probability that the other mobile image capture device will store a copy of a recently captured image in the non-volatile memory;

attempt to identify a match for one or more embeddings included in the excitement signal within the recently captured image; and

increase one or more of a frame rate or a resolution of an image sensor of the other mobile image capture device.

10. The system of claim 1, wherein to communicatively connect:

the user computing device is operable to visually display a pattern; and

the mobile image capture device is operable to;

capture imagery of the pattern; and

in response to capturing the imagery of the pattern, communicatively connect to the user computing device.

11. The system of claim 1, further comprising the user computing device, the user computing device operable to:

analyze at least one descriptor of the desirability of the scene depicted by each of the plurality of images stored in the non-volatile memory of the mobile image capture device; and

determine, based at least in part on the at least one descriptor of the desirability of the scene depicted by each of the plurality of images stored in the non-volatile memory of the mobile image capture device, whether to request transfer of such image.

12. The system of claim 1, wherein:

the mobile image capture device is operable to periodically provide a capture visualization to the user computing device for display by the user computing device;

display of the capture visualization by the user computing device is indicative of present operation of the mobile image capture device; and

the capture visualization is based at least in part on one or more images recently captured by the mobile image capture device.

13. The system of claim 1, further comprising the user computing device, wherein:

the mobile image capture device is operable to perform face recognition with respect to captured images to recognize one or more faces depicted in the captured images; and

the user computing device is operable to;

enable the user to select one or more of the recognized faces as having a heightened importance to the user; and

communicate information to the mobile image capture device that enables the mobile image capture device to increase a probability of storing subsequently captured images which depict the one or more selected faces.

14. The system of claim 1, further comprising:

the user computing device; and

a server computing device for image synchronization, storage, and sharing, the user computing device selectively communicatively coupleable to the server computing device;

wherein the user computing device uploads to the server computing device only images that are requested for upload to the server computing device during or as a result of a user-controlled editing session.

15. The system of claim 1, further comprising the user computing device, and wherein:

the mobile image capture device is operable to perform face recognition with respect to captured images to recognize one or more faces depicted in the captured images; and

the user computing device is operable to;

enable the user to share the one or more images received from the mobile image capture device with one or more contacts;

determine one or more relationships between the one or more contacts with whom the one or more images are shared and the one or more faces depicted in the shared images; and

suggest, with respect to subsequent images received from the mobile image capture device in which the one or more faces are recognized, that the user share such subsequent images with particular ones of the one or more contacts based at least in part on the one or more relationships.

16. A system for selective retention and editing of images, the system comprising:

a continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising a network interface, a power source, an image sensor, at least one processor, a memory that includes at least a temporary image buffer and a non-volatile memory, and a scene analyzer, the mobile image capture device operable to;

capture a plurality of images that respectively depict a plurality of scenes;

maintain a first copy of each of the plurality of images in the temporary image buffer;

determine, by the scene analyzer, at least one descriptor of the desirability of the scene depicted by each image;

determine, based at least in part on the at least one descriptor of the desirability of the scene of each image, whether to store a second copy of such image in the non-volatile memory of the mobile image capture device or to discard the first copy of such image from the temporary image buffer without storing a second copy of such image in the non-volatile memory; and

a user computing device, the user computing device comprising at least one processor, a memory, and a transfer controller, the user computing device operable to;

selectively communicatively connect to the mobile image capture device;

determine, by the transfer controller based at least in part on the at least one descriptor of the desirability of the scene depicted by each image stored in the non-volatile memory of the mobile image capture device, whether to request transfer of such image from the mobile image capture device; and

receive one or more images requested for transfer from the mobile image capture device.

17. The system of claim 16, wherein:

the scene analyzer comprises a plurality of deep neural networks that respectively analyze a plurality of different characteristics of an input image;

the mobile image capture device is operable to input each image into at least one of the plurality of deep neural networks;

each of the plurality of deep neural networks outputs at least one annotation for each input image, the at least one annotation for each input image indicative of the desirability of the scene depicted by such image; and

the transfer controller determines whether to request transfer of each image stored in the non-volatile memory of the mobile image capture device based at least in part on the one or more annotations provided for such image by the at least one of the plurality of deep neural networks into which such image was input.

18. The system of claim 17, wherein the user computing device determines a recommended composition for at least one of the transferred images based at least in part on the at least one annotation associated with such image, the recommended composition including only a portion of the at least one of the transferred images.

19. The system of claim 16, wherein the transfer controller performs an optimization algorithm to select which of the images stored in the non-volatile memory of the mobile image capture device to request for transfer, the optimization algorithm having a plurality of associated objectives, the plurality of associated objectives including at least a diversity of settings and a diversity of depicted persons.

20. A method to selectively retain and transfer images, the method comprising:

capturing, by a mobile image capture device, a first image that depicts a scene;

maintaining, by the mobile image capture device, a first copy of the first image in a temporary image buffer of the mobile image capture device;

inputting, by the mobile image capture device, the first image into at least one neural network to determine a desirability of the scene depicted by the first image, the at least one neural network stored in a memory of the mobile image capture device;

receiving, by the mobile image capture device, at least one descriptor of the desirability of the scene depicted by the first image as an output of the neural network;

determining, by the mobile image capture device based at least in part on the at least one descriptor of the desirability of the scene, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image from the temporary image buffer without storing a second copy of the first image in the non-volatile memory; and

transferring, by the mobile image capture device, one or more of a plurality of images stored in the non-volatile memory of the mobile image capture device to a user computing device.