Systems and methods for content tagging

• US 10,157,333 B1
• Filed: 08/25/2016
• Issued: 12/18/2018
• Est. Priority Date: 09/15/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Wang, Xiaoyu, Xu, Ning, Zhang, Ning, Carvalho, Vitor R., Li, Jia
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): RICE, ELISA M
• Application Number:
  US15/247,697
• Time in Patent Office: 845 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/09 Recognition of logos
• CPC Class Codes: H04N 23/63 by using electronic viewfinders

Abstract

Systems, methods, devices, media, and computer readable instructions are described for local image tagging in a resource constrained environment. One embodiment involves processing image data using a deep convolutional neural network (DCNN) comprising at least a first subgraph and a second subgraph, the first subgraph comprising at least a first layer and a second layer, processing, the image data using at least the first layer of the first subgraph to generate first intermediate output data; processing, by the mobile device, the first intermediate output data using at least the second layer of the first subgraph to generate first subgraph output data, and in response to a determination that each layer reliant on the first intermediate data have completed processing, deleting the first intermediate data from the mobile device. Additional embodiments involve convolving entire pixel resolutions of the image data against kernels in different layers if the DCNN.

Claims

1. A method of image tagging on a mobile device comprising:

accessing, by one or more processors of the mobile device, image data for a first image;

initiating, by the mobile device, processing of the image data using a deep convolutional neural network (DCNN) executed by the one or more processors, the DCNN comprising at least a first subgraph and a second subgraph, the first subgraph comprising at least a first layer and a second layer;

processing, by the mobile device, the image data using at least the first layer of the first subgraph to generate first intermediate output data;

processing, by the mobile device, the first intermediate output data using at least the second layer of the first subgraph to generate first subgraph output data;

in response to a determination that each layer reliant on the first intermediate output data have completed processing, deleting the first intermediate output data from the mobile device;

processing, by the mobile device, the first subgraph output data using a first layer of the second subgraph to generate second intermediate data;

processing, by the mobile device, the second intermediate data using a second layer of the second subgraph to generate second subgraph output data; and

in response to a second determination that each layer reliant on the second intermediate data has completed processing, deleting the second intermediate data from the mobile device.

2. The method of claim 1 further comprising:

capturing, by an image sensor of the mobile device, the first image;

processing, by the mobile device, the first image as captured by the image sensor to generate a file comprising the image data at a first pixel resolution associated with a pixel height and a pixel width; and

storing the file in a second memory of the mobile device.

3. The method of claim 2 wherein the first layer comprises a convolutional layer;

and

wherein processing the image data using at least the first layer of the first subgraph comprises convolving at least a first kernel with the image data, wherein the first kernel comprises a kernel pixel height less than the pixel height and a kernel pixel width less than the pixel width.

4. The method of claim 3 wherein the first intermediate output data comprises a plurality of matrixes, each matrix of the plurality of output matrixes generate by convolving an associated kernel of a plurality of kernels of the first layer with the image data;

wherein the plurality of kernels comprises the first kernel.

5. The method of claim 2 further comprising:

generating a plurality of output values from the DCNN, each output value associated with a corresponding tag;

comparing each output value of the plurality of output values with a corresponding threshold; and

assigning one or more tags to the first image based on the comparison of each output value associated with the corresponding tag to the corresponding threshold.

6. The method of claim 5 further comprising:

capturing, by the mobile device, a plurality of metadata associated with the first image;

processing, by the mobile device, the assigned one or more tags and the plurality of metadata using a natural language processor to generate a set of extended visual search tags.

7. The method of claim 6 further comprising storing the first image and the set of visual search tags with a plurality of images, each having associated extended visual search tags, in a memory of the mobile device.

8. The method of claim 7 further comprising:

receiving, at an input device of the mobile device, a first search term; and

generating a set of search results by comparing, using the one or more processors of the mobile device, the first search term with the associated extended visual search tags in the memory of the mobile device.

9. The method of claim 1 further comprising:

processing the second subgraph output data using a fully connected layer converted to a convolutional layer to generate a dense prediction score map.

10. The method of claim 9 wherein the fully connected layer generates the dense prediction score map using an associated output from each convolution layer.

11. The method of claim 9 further comprising subsampling the dense prediction score map using a max-pooling operating to generate a plurality of output recognition scores, each output recognition score associated with one or more corresponding tags.

12. The method of claim 1 wherein the DCNN comprises a plurality of 16 bit weights.

13. The method of claim 1 wherein the DCNN comprises a plurality of 32 bit floating point weights compressed to 8 bit weight indices;

and

wherein processing, by the mobile device, the image data using at least the first layer of the first subgraph to generate first intermediate output data comprises;

decompressing, by the mobile device, a first set of 8 bit weight indices in response to initiation of the processing of the image data; and

processing, by the mobile device, the image data using a corresponding first set of 32 bit weights as decompressed from the 8 bit weight indices.

14. The device of claim 1 wherein the one or more processors are further configured to comprising:

generate a plurality of output values from the DCNN, each output value associated with a corresponding tag;

compare each output value of the plurality of output values with a corresponding threshold; and

assign one or more tags to the first image based on the comparison of each output value associated with the corresponding tag to the corresponding threshold;

capture a plurality of metadata associated with the first image;

process the assigned one or more tags and the plurality of metadata using a natural language processor to generate a set of extended visual search tags;

store the first image and the set of visual search tags with a plurality of images, each having associated extended visual search tags, in a memory of the mobile device;

receive, from an input device of the mobile device, a first search term; and

generate a set of search results by comparing the first search term with the associated extended visual search tags in the memory of the mobile device.

15. A mobile device for image tagging comprising:

a memory;

an image sensor coupled to the memory; and

one or more processors coupled to the memory and configured to;

access image data for a first image;

initiate processing of the image data using a deep convolutional neural network (DCNN) executed by the one or more processors, the DCNN comprising at least a first subgraph and a second subgraph, the first subgraph comprising at least a first layer and a second layer;

process the image data using at least the first layer of the first subgraph to generate first intermediate output data;

process the first intermediate output data using at least the second layer of the first subgraph to generate first subgraph output data;

in response to a determination that each layer reliant on the first intermediate output data have completed processing, delete the first intermediate output data from the mobile device;

processing the first subgraph output data using a first layer of the second subgraph to generate second intermediate data;

processing the second intermediate data using a second layer of the second subgraph to generate second subgraph output data; and

in response to a second determination that each layer reliant on the second intermediate data has completed processing, deleting the second intermediate data from the mobile device.

16. The device of claim 15 wherein the one or more processors are further configured to:

further comprising;

process the first image as captured by the image sensor to generate a file comprising the image data at a first pixel resolution associated with a pixel height and a pixel width; and

store the file in a second memory of the mobile device;

wherein the first layer comprises a convolutional layer; and

wherein processing the image data using at least the first layer of the first subgraph comprises convolving at least a first kernel with the image data, wherein the first kernel comprises a kernel pixel height less than the pixel height and a kernel pixel width less than the pixel width; and

wherein the first intermediate output data comprises a plurality of matrixes, each matrix of the plurality of output matrixes generate by convolving an associated kernel of a plurality of kernels of the first layer with the image data.

17. A non-transitory storage medium comprising instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform operations for local image tagging, the operations comprising:

capturing, by an image sensor of the mobile device, the first image;

processing the first image as captured by the image sensor to generate a file comprising the image data at a first pixel resolution associated with a pixel height and a pixel width; and

storing the file in a second memory of the mobile device;

accessing image data for a first image;

initiating processing of the image data using a deep convolutional neural network (DCNN) executed by the one or more processors, the DCNN comprising at least a first subgraph and a second subgraph, the first subgraph comprising at least a first layer and a second layer;

processing the image data using at least the first layer of the first subgraph to generate first intermediate output data;

processing the first intermediate output data using at least the second layer of the first subgraph to generate first subgraph output data;

in response to a determination that each layer reliant on the first intermediate output data have completed processing, deleting the first intermediate output data from the mobile device;

processing the first subgraph output data using a first layer of the second subgraph to generate second intermediate data;

processing the second intermediate data using a second layer of the second subgraph to generate second subgraph output data; and

in response to a second determination that each layer reliant on the second intermediate data has completed processing, deleting the second intermediate data from the mobile device.

18. The non-transitory storage medium of claim 17 wherein the instructions further cause the device to perform operations comprising:

processing, by the mobile device, the first subgraph output data using a first layer of the second subgraph to generate second intermediate data;

processing, by the mobile device, the second intermediate data using a second layer of the second subgraph to generate second subgraph output data; and

in response to a second determination that each layer reliant on the second intermediate data has completed processing, deleting the second intermediate data from the mobile device; and

processing the second subgraph output data using a fully connected layer converted to a convolutional layer to generate a dense prediction score map.

19. The non-transitory storage medium of claim 18 wherein the instructions further cause the device to perform operations comprising:

subsampling the dense prediction score map using a max-pooling operating to generate a plurality of output recognition scores, each output recognition score associated with one or more corresponding tags;

wherein the DCNN comprises a plurality of 32 bit floating point weights compressed to 8 bit weight indices; and

wherein processing, by the mobile device, the image data using at least the first layer of the first subgraph to generate first intermediate output data comprises;

decompressing, by the mobile device, a first set of 8 bit weight indices in response to initiation of the processing of the image data; and

processing, by the mobile device, the image data using a corresponding first set of 32 bit weights as decompressed from the 8 bit weight indices.

20. The non-transitory storage medium of claim 17 wherein the DCNN comprises a plurality of 16 bit weights.