ADAPTABLE IMAGE SEARCH WITH COMPUTER VISION ASSISTANCE

• US 20150213058A1
• Filed: 01/24/2014
• Published: 07/30/2015
• Est. Priority Date: 01/24/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ambardekar, Amol Ashok, Huybregts, Christopher Leonard, Wall, Larry, Houshangi, Damoun, Pathak, Hrishikesh
• Granted Patent: US 9,619,488 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 10/87 using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

A computing device having adaptable image search and methods for operating an image recognition program on the computing device are disclosed herein. An image recognition program may receive a query from a user and a target image within which a search based on the query is to be performed using one or more of a plurality of locally stored image recognition models, which are determined to be able to perform the search with sufficiently high confidence. The query may comprise text that is typed or converted from speech. The image recognition program performs the search within the target image for a target region of the target image using at least one selected image recognition model stored locally, and returns a search result to the user.

Claims

1. A computing device having adaptable image search, the computing device comprising:

non-volatile memory configured to store a plurality of image recognition models;

an image recognition program executed by a processor of the computing device and configured to;

receive a query from a user, the query comprising text that is typed or converted from speech;

receive a target image within which a search based on the query is to be performed;

rank the image recognition models by confidence level for performing the search based on the query within the target image;

determine whether any of the image recognition models is above a confidence threshold for performing the search locally on the processor of the computing device; and

upon determining that at least one of the image recognition models is above the confidence threshold,

select at least one highly ranked image recognition model;

perform the search within the target image for a target region of the target image using at least one selected image recognition model; and

return a search result to the user.

2. The computing device of claim 1, wherein the target image is a single image or one or a plurality of image frames that constitute a portion of video.

3. The computing device of claim 1, wherein each image recognition model includes at least one of the following:

an image recognition algorithm, an optical character recognition (OCR) algorithm, and a keyword matching algorithm.

4. The computing device of claim 1, wherein the image recognition program uses location information about the user.

5. The computing device of claim 1, wherein the target image is an image or video of the user'"'"'s current surroundings and the query indicates a desired product or destination.

6. The computing device of claim 1, wherein the computing device comprises a camera and wherein the target image is captured by the camera.

7. The computing device of claim 1, wherein the computing device is a smartphone or tablet.

8. The computing device of claim 1, wherein the computing device is a watch or other wearable device.

9. The computing device of claim 1, wherein the image recognition program displays the search result on a head-up display.

10. The computing device of claim 1, wherein the query includes a directive to search for target regions in the target image or in other images that are similar to the target region in the target image.

11. A method for operating an image recognition program on a computing device having adaptable image search, the method comprising:

receiving a query from a user, the query comprising text that is typed or converted from speech;

receiving a target image within which a search based on the query is to be performed;

ranking a plurality of image recognition models by confidence level for performing the search based on the query within the target image, wherein the image recognition models are stored in non-volatile memory of the computing device;

determining whether any of the image recognition models is above a confidence threshold for performing the search locally on a processor of the computing device; and

upon determining that at least one of the image recognition models is above the confidence threshold,

selecting at least one highly ranked image recognition model;

performing the search within the target image for a target region of the target image using at least one selected image recognition model; and

returning a search result to the user.

12. The method of claim 11, wherein each image recognition model includes at least one of the following:

an image recognition algorithm, an optical character recognition (OCR) algorithm, and a keyword matching algorithm.

13. The method of claim 11, wherein the image recognition program uses location information about the user.

14. The method of claim 11, wherein the target image is an image or video of the user'"'"'s current surroundings and the query indicates a desired product or destination.

15. The method of claim 11, wherein the computing device comprises a camera and wherein the target image is captured by the camera.

16. The method of claim 11, wherein the computing device is a smartphone or tablet.

17. The method of claim 11, wherein the image recognition program displays the search result on a head-up display.

18. The method of claim 11, further comprising, upon determining that none of the image recognition models is above the confidence threshold:

connecting to a web service including a plurality of image recognition models;

receiving a ranking of image recognition models of the web service by confidence level for performing a search based on the query within the target image;

determining whether any of the image recognition models of the web service is above a confidence threshold for performing the search; and

upon determining that at least one of the image recognition models of the web service is above the confidence threshold,

selecting at least one highly ranked image recognition model of the web service;

downloading at least one selected image recognition model to the computing device;

performing the search within the target image for a target region of the target image using at least one downloaded image recognition model; and

returning a search result to the user.

19. The method of claim 11, further comprising, upon determining that none of the image recognition models is above the confidence threshold:

performing a web image search in an image search engine, wherein the web image search is based on the query;

returning a predetermined number of example images;

selecting at least one example image from the predetermined number of example images;

creating an image recognition model based on at least one selected example image;

storing the created image recognition model in the non-volatile memory of the computing device;

performing the search within the target image for a target region of the target image using the created image recognition model; and

returning a search result to the user.

20. A computing device having adaptable image search, the computing device comprising:

non-volatile memory configured to store a plurality of image recognition models;

a camera, wherein a target image is captured by the camera and wherein the target image is a single image or one or a plurality of image frames that constitute a portion of video;

a head-up display; and

an image recognition program executed by a processor of the computing device and configured to;

receive a query from a user, the query comprising text that is typed or converted from speech;

receive the target image within which a search based on the query is to be performed;

rank the image recognition models by confidence level for performing the search based on the query within the target image;

determine whether any of the image recognition models is above a confidence threshold for performing the search locally on the processor of the computing device; and

upon determining that at least one of the image recognition models is above the confidence threshold,

select at least one highly ranked image recognition model;

perform the search within the target image for a target region of the target image using at least one selected image recognition model;

return a search result to the user; and

display the search result on the head-up display.