METHOD AND SYSTEM FOR ADAPTIVE DISCOVERY OF CONTENT ON A NETWORK

• US 20090210407A1
• Filed: 02/15/2008
• Published: 08/20/2009
• Est. Priority Date: 02/15/2008
• Status: Active Grant

• Current Assignee: University of Utah Research Foundation (State of Utah)
• Original Assignee: University of Utah Research Foundation (State of Utah)
• Inventors: Barbosa, Luciano, Freire, Juliana
• Granted Patent: US 8,965,865 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method is provided for identifying documents that include a searchable form relevant to a topic. A document is received. If the received document comprises a form is determined. A form includes a field presented to a user requesting information from the user. If the received document is determined to comprise a form, a determination is made concerning whether or not the form is a searchable form. A searchable form returns non-trivial information to a requester in response to a submission of the form. If the form is determined to be a searchable form, a determination is made concerning whether or not the form is relevant to an identified topic. If the form is determined to be relevant to the identified topic, the document is identified as a searchable form relevant to the identified topic.

Claims

1. A device for identifying documents that include a searchable form relevant to a topic, the device comprising:

a processor;

a computer-readable medium operably coupled to the processor, the computer-readable medium comprising instructions that, upon execution by the processor, perform operations comprising

(a) receiving a document;

(b) determining if the received document comprises a form, wherein the form includes a field presented to a user requesting information from the user;

(c) if the received document is determined to comprise a form, determining if the form is a searchable form, wherein a searchable form returns non-trivial information to a requestor in response to a submission of the form;

(d) if the form is determined to be a searchable form, determining if the form is relevant to an identified topic; and

(e) if the form is determined to be relevant to the identified topic, identifying the document as a searchable form relevant to the identified topic.

2. A computer-readable medium comprising computer-readable instructions therein that, upon execution by a processor, cause the processor to identify documents that include a searchable form relevant to a topic, the instructions configured to cause a computing device to:

(a) receive a document;

(b) determine if the received document comprises a form, wherein the form includes a field presented to a user requesting information from the user;

(c) if the received document is determined to comprise a form, determine if the form is a searchable form, wherein a searchable form returns non-trivial information to a requestor in response to a submission of the form;

(d) if the form is determined to be a searchable form, determine if the form is relevant to an identified topic; and

(e) if the form is determined to be relevant to the identified topic, identify the document as a searchable form relevant to the identified topic.

3. A method of identifying documents that include a searchable form relevant to a topic, the method comprising:

(a) receiving a document;

(b) determining if the received document comprises a form, wherein the form includes a field presented to a user requesting information from the user;

(c) if the received document is determined to comprise a form, determining if the form is a searchable form, wherein a searchable form returns non-trivial information to a requester in response to a submission of the form;

(d) if the form is determined to be a searchable form, determining if the form is relevant to an identified topic; and

(e) if the form is determined to be relevant to the identified topic, identifying the document as a searchable form relevant to the identified topic.

4. The method of claim 3, further comprising outputting the identified document wherein outputting comprises at least one of printing the identified document, displaying the identified document, storing a hyperlink to the identified document in a memory for use as an input to an application, and storing the identified document in a memory for use as an input to an application.

5. The method of claim 3, further comprising assigning the received document to a cluster of a plurality of clusters based on a similarity value.

6. The method of claim 3, wherein determining if the received document comprises a form comprises identifying a form indicator in text between markup language tags used to partition the received document.

7. The method of claim 3, wherein determining if the form is a searchable form comprises identifying a search indicator in text between markup language tags used to partition the received document.

8. The method of claim 3, further comprising, before (c), identifying structural features of the form wherein determining if the form is a searchable form is based on the identified structural features.

9. The method of claim 8, wherein the structural features may be selected from the group consisting of a number of hidden tags, a number of radio tags, a number of file inputs, a number of submit tags, a number of image inputs, a number of buttons, a number of resets, a number of password tags, a number of textboxes, and a submission method for the form.

10. The method of claim 3, wherein determining if the form is relevant to an identified topic comprises analyzing text between markup language tags used to partition the form in the received document.

11. The method of claim 3, further comprising:

(f) if the form is determined to be relevant to the identified topic, identifying a hyperlink to the received document;

(g) updating a link classifier based on the identified hyperlink, the link classifier estimating a distance between a received hyperlink and a potentially searchable form;

(h) ranking a plurality of hyperlinks based on the updated link classifier;

(i) selecting a next hyperlink from the ranked plurality of hyperlinks;

(j) receiving a second document pointed to by the selected next hyperlink; and

(k) repeating (b)-(j) with the received second document as the received document until a stop condition is satisfied.

12. The method of claim 11, further comprising, before (h), allocating the plurality of hyperlinks to a plurality of queues based on an estimated number of link levels to reach a potentially searchable form wherein ranking the plurality of hyperlinks is within the plurality of queues.

13. The method of claim 12, wherein selecting the next hyperlink comprises selecting a highest ranked hyperlink from a queue of the plurality of queues associated with a minimum estimated number of link levels.

14. The method of claim 11, further comprising, after (i), removing the selected next hyperlink from the plurality of hyperlinks.

15. The method of claim 14, wherein the stop condition comprises a comparison of a number of the plurality of hyperlinks with a stop number.

16. The method of claim 11, further comprising, before (a), receiving the plurality of hyperlinks.

17. The method of claim 11, further comprising, after (d), identifying a document hyperlink from the received document and adding the identified document hyperlink to the plurality of hyperlinks.

18. The method of claim 11, wherein (g)-(h) are repeated only after satisfaction of a learning threshold test.

19. The method of claim 18, wherein the learning threshold test comprises a number of identified relevant forms.

20. A method of identifying documents relevant to a topic, the method comprising:

(a) receiving a document;

(b) determining if the received document is relevant to an identified topic;

(c) if the received document is determined to be relevant to the identified topic, identifying the received document as a relevant document;

(d) identifying a hyperlink to the relevant document;

(e) updating a link classifier based on the identified hyperlink, the link classifier estimating a distance between a received hyperlink and a potentially relevant document;

(f) ranking a plurality of hyperlinks based on the updated link classifier;

(g) selecting a next hyperlink from the ranked plurality of hyperlinks;

(h) linking to the selected next hyperlink to receive a second document; and

(i) repeating (b)-(h) with the received second document as the received document until a stop condition is satisfied.

21. The method of claim 20, further comprising outputting the relevant document wherein outputting comprises at least one of printing the relevant document, displaying the relevant document, storing a hyperlink to the relevant document in a memory for use as an input to an application, and storing the relevant document in a memory for use as an input to an application.

22. The method of claim 20, further comprising assigning the relevant document to a cluster of a plurality of clusters based on a similarity value.

23. The method of claim 20, further comprising, before (f), allocating the plurality of hyperlinks to a plurality of queues based on an estimated number of link levels to reach a potentially relevant document wherein ranking the plurality of hyperlinks is within the plurality of queues.

24. The method of claim 23, wherein selecting the next hyperlink comprises selecting a highest ranked hyperlink from a queue of the plurality of queues associated with a minimum estimated number of link levels.

25. The method of claim 20, further comprising, after (g), removing the selected next hyperlink from the plurality of hyperlinks.

26. The method of claim 25, wherein the stop condition comprises a comparison of a number of the plurality of hyperlinks with a stop number.

27. The method of claim 20, further comprising, identifying a document hyperlink from the relevant document and adding the identified document hyperlink to the plurality of hyperlinks.

28. The method of claim 20, wherein (e)-(f) are repeated only after satisfaction of a learning threshold test.

29. The method of claim 28, wherein the learning threshold test comprises a number of identified relevant documents.