Feature-based candidate selection

• US 9,767,156 B2
• Filed: 08/30/2012
• Issued: 09/19/2017
• Est. Priority Date: 08/30/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wang, Xin-Jing, Scott, Matthew Robert, Chen, Xi
• Primary Examiner(s): Uddin, Md. I
• Application Number:
  US13/701,008
• Publication Number:
  US 20150161126A1
• Time in Patent Office: 1,846 Days
• Field of Search: 707728, 707748, 707759, 707767, 707769
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs

Abstract

A candidate for selection or insertion may be based on extracted candidate features and reference features. The candidate features may be scored and ranked while candidates may be removed from consideration based on criteria. The remaining candidates may be presented for selection and/or insertion.

Claims

1. A system comprising:

one or more processors; and

memory storing modules that, when executed by the one or more processors, cause the one or more processors to;

receive search criteria including at least a current user input in a user interface, the user interface associated with an input method editor (IME) interface associated with a host application, the host application selected from a plurality of host applications based at least in part on an indication from a user;

receive one or more candidates returned by a search engine based at least in part on the search criteria, the search criteria including a content of the current user input and a context of the current user input, the context of the current user input including at least a relevance of the current user input to a preceding user input in a same session of the host application, the preceding user input received at the host application prior to the current user input, the relevance of the current user input to the preceding user input based at least in part on a spatial relationship between the current user input and the preceding user input in the host application;

select one or more references from the one or more candidates returned by the search engine;

assign corresponding scores to the one or more candidates based at least in part on a comparison between one or more candidate features extracted from the one or more candidates and one or more reference features extracted from the one or more references, the corresponding scores further based in part on the context of the current user input;

rank the one or more candidates based at least in part on the corresponding scores;

remove a first candidate from the one or more candidates when a score of the corresponding scores of the first candidate is below a threshold score;

select one or more selected candidates from the one or more candidates based at least in part on the corresponding scores;

provide, in the user interface, the one or more selected candidates for user selection;

receive the user selection of a user selected candidate from the one or more selected candidates; and

provide to the host application the user selected candidate.

2. The system of claim 1, wherein the current user input comprises one or more of a textual input, a gesture, a motion, or a voice input.

3. The system of claim 1, wherein the one or more candidates returned by the search engine are returned in a ranked order.

4. The system of claim 3, wherein the one or more candidates returned by the search engine are further based in part on results information, the results information based at least in part on one or more of a set of top ranked results, a set of tail ranked results, or a set duplicate results.

5. The system of claim 1, wherein the one or more candidates comprise one or more of a candidate image, a candidate video, a candidate audio file, or a textualized graphic.

6. The system of claim 1, wherein the one or more reference features comprise average features of the one or more references.

7. The system of claim 1, wherein at least one of the corresponding scores is assigned based at least in part on a classifier, and wherein the classifier assigns the at least one of the corresponding scores based at least in part on a learned model that maps features to a score value.

8. The system of claim 1, wherein at least one of the corresponding scores is assigned based at least in part on a classifier, wherein the classifier assigns the at least one of the corresponding scores based at least in part on a regression model.

9. The system of claim 1, wherein an individual candidate of the one or more candidates with a corresponding score greater than the threshold score value is available for selection.

10. The system of claim 1, wherein the select the one or more selected candidates from the one or more candidates based at least in part on the corresponding scores comprises a comparison between a candidate and another candidate, wherein a candidate duplicative or within a similarity threshold of another candidate is made unavailable for selection.

11. The system of claim 10, wherein the comparison between the candidate and the another candidate is based at least in part on one or more of local features, global features, or pixel-level comparisons.

12. The system of claim 1, the memory storing modules that further cause the one or more processors to:

determine that a first score of the corresponding scores associated with a first candidate of the one or more selected candidates is within a similarity threshold of a second score of the corresponding scores associated with a second candidate of the one or more selected candidates; and

remove the second candidate from the one or more selected candidates.

13. A system comprising:

one or more processors;

an input method editor (IME) interface, the IME interface including a user interface, the user interface to receive a textual input and to provide an output to a host application, the host application selected from a plurality of host applications based at least in part on an indication from a user;

an analyzer executable by the one or more processors to analyze a content of the textual input and a context of the textual input, the context of the textual input including at least a relevancy of the textual input to a preceding textual input in a user conversation, the relevancy of the textual input to the preceding textual input based at least in part on a spatial relationship between the textual input and the preceding textual input in the user conversation;

a candidate manager executable by the one or more processors to gather one or more candidate images based at least in part on the content of the textual input and the context of the textual input, and to select one or more reference images from the one or more candidate images based at least in part on search results returned from a search engine;

an extractor executable by the one or more processors to extract one or more reference features from the one or more reference images and to extract one or more candidate features from the one or more candidate images;

a classifier executable by the one or more processors to;

assign one or more scores to the one or more candidate images based at least in part on a comparison between the one or more candidate features and the one or more reference features; and

rank the one or more candidate images based at least in part on the one or more scores;

a reducer executable by the one or more processors to remove a first candidate image from the one or more candidate images based at least in part on a score of a candidate compared to a threshold score value; and

an insertion manager executable by the one or more processors to provide the one or more candidate images for selection in the user interface.

14. The system of claim 13, further comprising a duplicate remover executable by the one or more processors to remove a second candidate image from the one or more candidates based at least in part on one or more of a duplicity of a candidate to another candidate or a similarity of a candidate to another candidate.

15. The system of claim 13, wherein a reference image of the one or more reference images is selected based in part on a number of duplicate images or similar images associated with the reference image.

16. A method comprising:

receiving a current textual input from a user through an input method editor (IME) interface, the IME interface including a user interface and providing output for a selected host application of a plurality of host applications, the selected host application selected based at least in part on an indication from the user;

analyzing, by a computing device, a content of the current textual input and a context of the current textual input, the context of the current textual input based at least in part on a type of the selected host application;

gathering one or more candidate images from a search engine based at least in part on the context of the current textual input and the content of the current textual input, the context of the current textual input including at least a relevancy of the current textual input to a preceding textual input in a user conversation, the preceding textual input received prior to the current textual input, the relevancy of the current textual input to the preceding textual input based at least in part on a spatial relationship between the current textual input and the preceding textual input in the user conversation;

selecting one or more reference images from the one or more candidate images based at least in part on results from the search engine;

extracting one or more reference features from the one or more reference images;

extracting one or more candidate features from the one or more candidate images;

assigning a score to the one or more candidate images based at least in part on the one or more candidate features;

ranking the one or more candidate images based at least in part on the score;

removing a first candidate image from the one or more candidate images when the score of the first candidate image is below a threshold score;

removing a second candidate image from the one or more candidate images based at least in part on a comparison of features between the second candidate and another candidate;

providing a ranked subset of remaining candidate images of the one or more candidate images, to the user, for selection through the IME interface; and

in response to receiving a selection of an image of the ranked subset of remaining candidate images, providing the image to the selected host application.

17. The method of claim 16, wherein the results from the search engine comprise one or more of a set of top ranked results, a set of tail ranked results, or a set duplicate results.

18. The method of claim 16, wherein the comparison of features between the second candidate and the another candidate is based at least in part on one or more of local features, global features, or a pixel-level comparison.

19. The method of claim 16, further comprising inserting the image into the user conversation associated with the host application, the image replacing at least a portion of text included in the user conversation.