Methods and systems for low-energy image classification

• US 10,055,672 B2
• Filed: 05/18/2015
• Issued: 08/21/2018
• Est. Priority Date: 03/11/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shoaib, Mohammed, Venkataramani, Swagath
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/715,554
• Publication Number:
  US 20160267358A1
• Time in Patent Office: 1,191 Days
• Field of Search: 382190, 382195, 382199, 382201, 382203, 382224, 382159, 382155
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding

Abstract

Examples of the disclosure enable efficient processing of images. In some examples, one or more interest points are identified in an image. One or more features are extracted from the identified interest points using a filter module, a gradient module, a pool module, and/or a normalizer module. The extracted features are aggregated to generate one or more vectors. Based on the generated vectors, it is determined whether the extracted features satisfy a predetermined threshold. Based on the determination, the image is classified such that the image is configured to be processed based on the classification. Aspects of the disclosure facilitate conserving memory at a local device, reducing processor load or an amount of energy consumed at the local device, and/or reducing network bandwidth usage between the local device and the remote device.

Claims

1. A computer-implemented method for classifying a plurality of images observed by a user, the method comprising executing on one or more computing devices the operations of:

identifying one or more interest points in each image of the plurality of images observed by the user;

extracting one or more features from the identified interest points using one or more of a filter module, a gradient module, a pool module, and a normalizer module;

aggregating the extracted features to generate one or more vectors;

based on the generated vectors, determining whether the extracted features satisfy a predetermined threshold;

based on the determination, classifying each image of the plurality of images observed by the user as a first image or a second image;

transmitting a set of first images, and not any of the second images, to a different computing system for processing, including one or more of recognizing the extracted features, understanding the set of first images, and generating one or more actionable items;

receiving, in response to the transmission of the set of first images, data related, in part, to the processed set of first images; and

presenting, via a user interface, the received data to the user.

2. The computer-implemented method of claim 1, wherein identifying one or more interest points comprises detecting one or more corners in the image, wherein a first corner corresponds to a first interest point.

3. The computer-implemented method of claim 1, wherein extracting one or more features comprises smoothing, by the filter module, one or more pixels associated with the interest points.

4. The computer-implemented method of claim 1, wherein extracting one or more features comprises:

computing, by the gradient module, one or more gradients along a first axis and a second axis perpendicular to the first axis; and

based on the computed gradients, generating, by the gradient module, an output array including one or more feature maps.

5. The computer-implemented method of claim 1, wherein extracting one or more features comprises pooling, by the pool module, one or more feature maps along a grid, wherein the feature maps correspond to the extracted features.

19. The computer-implemented method of claim 1, wherein the received data relating to the classified image pertains to one or more of the recognized extracted features, understanding of the image, and the generated one or more actionable items.

20. The computer-implemented method of claim 1, further comprising utilizing the received data in one or more of advertising and a navigation application.

6. A mobile device comprising:

a sensor module configured to capture data corresponding to a plurality of images observed by a user;

a memory area storing computer-executable instructions for classifying the plurality of images observed by the user; and

a processor configured to execute the computer-executable instructions to;

extract one or more features from the plurality of images observed by the user, a quantity of extracted features associated with a desired power consumption of the mobile device;

determine whether the extracted features satisfy a predetermined threshold;

based on the determination, classify each of the plurality of images observed by the user as a first image or a second image;

transmit a set of first images, and not any of the second images, to a different computing system for processing, including one or more of recognizing the extracted features, understanding the first set of images, and generating one or more actionable items;

receive, in response to the transmission of the set of first images, data related, in part, to the processed set of first images; and

present, via a user interface, the received data to the user.

7. The mobile device of claim 6, wherein the processor is further configured to execute the computer-executable instructions to detect one or more corners in the plurality of images, wherein the features are extracted from the detected corners.

8. The mobile device of claim 6, wherein the processor is further configured to execute the computer-executable instructions to smooth one or more pixels associated with the plurality of images, wherein the pixels correspond to the extracted features.

9. The mobile device of claim 6, wherein the processor is further configured to execute the computer-executable instructions to:

compute one or more gradients along a first axis and a second axis perpendicular to the first axis; and

based on the computed gradients, generate an output array including one or more feature maps, such that the output array includes a predetermined number of feature maps having a predetermined size.

10. The mobile device of claim 6, wherein the processor is further configured to execute the computer-executable instructions to pool one or more feature maps along a grid, wherein the feature maps correspond to the extracted features.

11. A computing device comprising:

a sensor module configured to capture data corresponding to one or more images observed by a user;

a feature computation module configured to;

identify one or more interest points in the one or more images observed by the user;

extract one or more features from the identified interest points; and

aggregate the extracted features to generate one or more vectors; and

a feature classification module configured to;

based on the generated vectors, determine whether the extracted features satisfy a predetermined threshold;

based on the determination, classify the one or more images observed by the user into a first set of images and a second set of images;

transmit the first set of images to a server, and not the second set of images, the server configured to process the first set of images including one or more of recognizing the extracted features, understanding the first set of images, and generating one or more actionable items;

receive, in response to the transmission of the first set of images, data related, in part, to the processed first set of images; and

present, via a user interface, the received data to the user.

12. The computing device of claim 11, wherein the feature computation module is configured to detect one or more corners in the one or more images, wherein a first corner corresponds to a first interest point.

13. The computing device of claim 11, wherein the feature computation module is configured to:

compute one or more gradients along a first axis and a second axis perpendicular to the first axis; and

based on the computed gradients, generate an output array including a predetermined number of feature maps.

14. The computing device of claim 11, wherein the feature computation module is configured to pool one or more feature maps along a grid, wherein the feature maps correspond to the extracted features.

15. The computing device of claim 11, wherein one or more of the feature computation module and the feature classification module are configured to process one image in parallel with processing another image.

16. The computing device of claim 11, wherein the feature computation module is configured to process an interest point in parallel with processing another interest point.

17. The computing device of claim 11, wherein the feature computation module is configured to process a pixel in parallel with processing another pixel.

18. The computing device of claim 11, wherein one or both of the feature computation module and the feature classification module includes a plurality of submodules, wherein a first submodule of the plurality of submodules is configured to process a first set of data to generate a first output and transmit the first output to a second submodule of the plurality of submodules such that the first submodule is configured to process a second set of data in parallel with the second submodule processing the first output.