Context-awareness through biased on-device image classifiers

• US 10,268,886 B2
• Filed: 05/18/2015
• Issued: 04/23/2019
• Est. Priority Date: 03/11/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shoaib, Mohammed, Liu, Jie, Li, Jin
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/715,555
• Publication Number:
  US 20160267324A1
• Time in Patent Office: 1,436 Days
• Field of Search: 382190, 382195, 382199, 382224
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: H04N 1/00204 with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324

Abstract

Examples of the disclosure enable efficient processing of images. One or more features are extracted from a plurality of images. Based on the extracted features, the plurality of images are classified into a first set including a plurality of first images and a second set including a plurality of second images. One or more images of the plurality of first images are false positives. The plurality of first images and none of the plurality of second images are transmitted to a remote device. The remote device is configured to process one or more images including recognizing the extracted features, understanding the images, and/or generating one or more actionable items. Aspects of the disclosure facilitate conserving memory at a local device, reducing processor load or an amount of energy consumed at the local device, and/or reducing network bandwidth usage between the local device and the remote device.

Claims

1. A computer-implemented method for classifying one or more images, the method comprising executing on one or more computing devices the operations of:

configuring an image classifier executable on a computing device based in part on at least one of a power requirement of a specific application, a performance requirement of the application, power available to the computing device, and computational resources available at the computing device;

receiving a plurality of different images generated by the application for processing at the image classifier;

extracting one or more features from each of the plurality of different images;

based on the extracted features, classifying the plurality of different images into a first set including a plurality of first images and a second set including a plurality of second images, one or more images of the plurality of first images being false positives, the number of false positive images being based in part on the configuration of the image classifier; and

transmitting the plurality of first images and none of the plurality of second images from the computing device to a remote device, wherein the remote device is configured to process the one or more images of the plurality of first images including;

recognizing the one or more extracted features including arbitrary objects, understanding the one or more images by applying an image-understanding algorithm to the one or more images based on the recognition of the arbitrary objects, and generating one or more actionable items based on the understanding, wherein the one or more actionable items comprise a navigational aid to enable a user to detect and avoid obstacles.

2. The computer-implemented method of claim 1, wherein extracting one or more features comprises identifying one or more interest points, the features extracted from the identified interest points.

3. The computer-implemented method of claim 1, wherein extracting one or more features from the plurality of images comprises extracting features from one image in parallel with extracting features from another image, and the plurality of images comprises at least one of traffic signals, pedestrians, or navigational hazards.

4. The computer-implemented method of claim 1, wherein classifying the plurality of images comprises classifying the plurality of second images into the second set such that no image classified into the second set is a true positive.

5. The computer-implemented method of claim 1, wherein classifying the plurality of images comprises classifying one image in parallel with classifying another image.

6. The computer-implemented method of claim 1, further comprising filtering the plurality of images such that the first set is partitioned from the second set.

7. The computer-implemented method of claim 1, further comprising receiving, from the remote device, the actionable items.

8. A mobile device comprising:

a sensor module configured to capture data corresponding to a plurality of different images, one or more images of the plurality of different images associated with an object;

a memory area storing computer-executable instructions for classifying the plurality of different images; and

a processor configured to execute the computer-executable instructions to;

configure an image classifier based in part on at least one of a power requirement of a specific application, a performance requirement of the application, power available to the mobile device, and computational resources available at the mobile device;

determine, at the image classifier, whether a first image of the plurality of different images associated with the application is associated with the object, the image classifier being biased towards determining that the first image is associated with the object, wherein the extent of the biasing is based on the configuration of the image classifier; and

transmit the first image from the computing device to a remote device configured to process the first image including recognizing extracted features including arbitrary objects, understanding the images by applying an image-understanding algorithm to the first image based on the recognition of the arbitrary objects, and generating one or more actionable items, the one or more actionable items comprise a navigational aid to enable a user to detect and avoid obstacles.

9. The mobile device of claim 8, wherein the sensor module is configured to stream the plurality of images.

10. The mobile device of claim 8, wherein the processor is configured to execute the computer-executable instructions to:

extract one or more features from the plurality of images; and

based on the extracted features, classify the plurality of images into a first set of images and a second set of images.

11. The mobile device of claim 10, wherein the processor is configured to execute the computer-executable instructions to identify one or more interest points, the features extracted from the identified interest points.

12. The mobile device of claim 8, wherein the processor is configured to execute the computer-executable instructions to partition the first image from a second image of the plurality of images.

13. The mobile device of claim 8, wherein the processor is configured to execute the computer-executable instructions to determine whether the first image is associated with the object in parallel with determining whether a second image of the plurality of images is associated with the object.

14. The mobile device of claim 8, wherein the processor is configured to execute the computer-executable instructions to receive, from the remote device, the actionable items.

15. A computing device comprising:

an image sensor configured to capture data corresponding to a plurality of different images, one or more images of the plurality of different images associated with an object;

an image classifier configured based in part on at least one of a power requirement of a specific application, a performance requirement of the application, power available to the computing device, and computational resources available at the computing device to;

classify the plurality of different images associated with the application into a first set including a plurality of first images and a second set including a plurality of second images, one or more images of the plurality of first images being false positives, the number of false positives being based in part on the configuration of the image classifier, and

transmit the plurality of first images and none of the plurality of second images from the computing device to a remote device, the remote device configured to process one or more images including recognizing one or more features including arbitrary objects, understanding the images by applying an image-understanding algorithm to the one or more images based on the recognition of the arbitrary objects, and generating one or more actionable items, the one or more actionable items comprise a navigational aid to enable a user to detect and avoid obstacles.

16. The computing device of claim 15, wherein the image classifier is configured to partition the first set from the second set.

17. The computing device of claim 15, wherein the image classifier is biased such that no image classified into the second set is a true positive.

18. The computing device of claim 15, the image classifier is configured to extract one or more features from the plurality of images, wherein the plurality of images are classified based on the extracted features.

19. The computing device of claim 18, wherein the image classifier is configured to identify one or more interest points in the plurality of images, wherein the features are extracted based on the identified interest points.

20. The computing device of claim 15, wherein the image classifier is configured to classify one image in parallel with classifying another image.