Low power framework for controlling image sensor mode in a mobile image capture device

• US 10,728,489 B2
• Filed: 08/22/2018
• Issued: 07/28/2020
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Donsbach, Aaron Michael, Vanik, Benjamin, Clapper, Jon Gabriel, Lentz, Alison, Lovejoy, Joshua Denali, Fritz, III, Robert Douglas, Duleba, Krzysztof, Zhang, Li, Payne, Juston, Fortuna, Emily Anne, Bialynicka-Birula, Iwona, Aguera-Arcas, Blaise, Ramage, Daniel, McMahan, Benjamin James, Lange, Oliver Fritz, Holbrook, Jess
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US16/109,708
• Publication Number:
  US 20180367752A1
• Time in Patent Office: 706 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/426 using memory downsizing methods
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/65 : Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/8042 : involving data reduction
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal

Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for controlling image sensor mode in a mobile image capture device. On example low power frame work includes a scene analyzer that analyzes a scene depicted by a first image and, based at least in part on such analysis, causes an image sensor control signal to be provided to an image sensor to adjust at least one of the frame rate and the resolution of the image sensor.

Claims

1. A computer-implemented method, comprising:

obtaining, by one or more server computing devices, data descriptive of a pre-trained artificial neural network, the pre-trained artificial neural network having been previously trained based at least in part on a set of first training images;

receiving, by the one or more server computing devices, a set of user training images provided by a user, the set of user training images including at least one image not included in the set of first training images, wherein at least one of the set of user training images has been edited by the user resulting in photographic re-composition;

re-training, by the one or more server computing devices, the pre-trained artificial neural network based at least in part on the set of user training images to form a re-trained artificial neural network; and

transmitting, by the one or more server computing devices, the re-trained artificial neural network to a user computing device associated with the user for implementation at the user computing device.

2. The computer-implemented method of claim 1, wherein at least one of the set of user training images has been hand-labeled by the user.

3. The computer-implemented method of claim 1, wherein the user computing device comprises an image capture device associated with the user.

4. The computer-implemented method of claim 3, wherein the set of user training images comprise images previously captured by the image capture device.

5. The computer-implemented method of claim 1, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise convolutional artificial neural networks.

6. The computer-implemented method of claim 1, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image classification artificial neural networks.

7. The computer-implemented method of claim 1, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise face detection artificial neural networks.

8. The computer-implemented method of claim 1, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image content artificial neural networks.

9. A training computing system for training personalized artificial neural networks based on user-submitted training data, the training computing system comprising one or more server computing devices configured to perform operations, the operations comprising:

obtaining data descriptive of a pre-trained artificial neural network, the pre-trained artificial neural network having been previously trained based at least in part on a set of first training images;

receiving a set of user training images selected by a user, the set of user training images including at least one image not included in the set of first training images, wherein at least one of the set of user training images has been edited by the user resulting in photographic re-composition;

re-training the pre-trained artificial neural network based at least in part on the set of user training images to form a re-trained artificial neural network; and

transmitting the re-trained artificial neural network to a user computing device associated with the user for implementation at the user computing device.

10. The training computing system of claim 9, wherein at least one of the set of user training images has been hand-labeled by the user.

11. The training computing system of claim 9, wherein the user computing device comprises an image capture device associated with the user.

12. The training computing system of claim 11, wherein the set of user training images comprise images previously captured by the image capture device.

13. The training computing system of claim 9, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise convolutional artificial neural networks.

14. The training computing system of claim 9, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image classification artificial neural networks.

15. The training computing system of claim 9, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise face detection artificial neural networks.

16. The training computing system of claim 9, wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image content artificial neural networks.

17. A user computing system comprising:

an image capture system comprising an artificial neural network, the image capture system configured to capture images based at least in part on an output of the artificial neural network; and

one or more computing devices configured to;

obtain a set of images captured by the image capture device;

receive user input that edits one or more of the set of images captured by the image capture device to form a set of edited images having photographic re-composition based on the user input; and

after receiving the user input, provide the set of edited images to a training computing system; and

wherein the image capture system is configured to;

receive and store a re-trained version of the artificial neural network that has been re-trained by the training computing system based on the set of edited images provided to the training computing system by the one or more computing devices; and

capture images based at least in part on an output of the re-trained version of the artificial neural network.

18. The user computing system of claim 17, wherein the image capture system comprises a mobile image capture device.