SEARCHING TEXT VIA FUNCTION LEARNING

• US 20140129494A1
• Filed: 11/06/2013
• Published: 05/08/2014
• Est. Priority Date: 11/08/2012
• Status: Active Grant

• Current Assignee: Georges Harik
• Original Assignee: Georges Harik
• Inventors: Harik, Georges
• Granted Patent: US 10,467,530 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06N 3/126 Evolutionary algorithms, e.g. genetic algorithms or genetic programming

Abstract

A method which does not rely on explicit inverted indices is provided to search for documents in a corpus of documents responsive to a textual search query. The method includes (a) selecting a program that is customized by setting values for a plurality of parameters, the program structured to receive the textual search query as input and to provide as output values indicating the relevance of the documents in the corpus to the search query; (b) training the program using a machine learning technique; and (c) applying the trained program to the textual search query. The program may be based on a structure that is developed based on a genetic programming technique.

Claims

1. A method for searching documents in a corpus of documents based on a textual search query, the method comprising:

selecting a program that is characterized by a set of program parameters, the program structured to receive the textual search query as input and to provide as output values indicating the relevance of the documents in the corpus to the textual search query;

training the program using a machine learning technique based on training data from a subset of the corpus of documents; and

executing the trained program using the textual search query as input.

2. The method of claim 1, wherein the program is trained according to a training method which comprises repeating the steps of:

selecting a document from the subset of the corpus as a test document;

extracting one or more text sequences from the selected test document;

running the program using the extracted text sequences as a textual search query;

evaluating an error function that relates the output values of the program with predetermined values assigned to the test document for each textual search query ran; and

resetting the values of the program parameters based on the evaluated error function.

3. The method of claim 2, wherein each predetermined value represents a relevance score for the test document in response to one of the extracted text sequence as a search query.

4. The method of claim 2, wherein the extracted text sequence comprises one or more of:

(i) a title of the test document, (ii) one or more sequences of words appearing in the test sequence, and (iii) anchortext of links in other documents in the subset of documents pointing to the test document.

5. The method of claim 1, wherein the output values of the program comprises a probability distribution indicating the relevance of each document in the corpus to the textual search query.

6. The method of claim 1, wherein the machine learning technique comprises application of the stochastic gradient descent technique.

7. The method of claim 1, wherein the program has a structure that is determined using a genetic programming technique.

8. The method of claim 1, wherein executing the trained program comprises:

dividing the corpus of documents into a plurality of partitions;

executing the trained program on each of the partitions to obtain for each partition corresponding output values indicating the relevance of the documents in the partition to the textual search query; and

combining the corresponding output values for the partitions to provide combined output values indicating the relevance of the documents in the corpus to the textual search query.

9. The method of claim 8, wherein combining the corresponding output values further comprises the step of, for each partition, scoring the documents associated with the corresponding output values using an inverted index, and combining the documents from the partitions according to the scores.