System and method for efficient control and capture of dynamic database content

• US 20070192442A1
• Filed: 12/04/2006
• Published: 08/16/2007
• Est. Priority Date: 07/24/2001
• Status: Active Grant

• Current Assignee: Brightplanet Corp II, Inc.
• Original Assignee: Brightplanet Corp. (Brightplanet Corp II, Inc.)
• Inventors: Tiahrt, Thomas, Bushee, William
• Granted Patent: US 7,676,555 B2
• Time in Patent Office: Days
• US Class Current: 709/219
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system and method for efficient control and capture of dynamic database content for rapidly providing a user with a highly relevant collection of documents related to a query. The system and method for efficient control and capture of dynamic database content includes a computer system with a storage means for facilitating the retention and recall of dynamic database content and a communications means for facilitating bi-directional communication of the computer system with local or distributed networks; an executory module operationally coupled to the computer system for controlling the storage means and the communications means as well as directing the system for the efficient control and capture of dynamic database content to a plurality of pre-selected Internet sites; and a capture module which is in communication with the executory module and facilitates selection of the plurality of Internet sites associated with a query submitted by a user of the system.

Claims

1-45. -45. (canceled)

46. A method for efficiently controlling and capturing dynamic database content comprising:

obtaining a query from a user, said query to be used to search for relevant information in a specified area;

pre-processing said query;

routing said query to a plurality of said databases, each one of said databases using said query to locate relevant information for the user;

obtaining results from each one of said plurality of databases;

processing said results;

storing said results for use and analysis by the user.

47. The method of claim 46, wherein said step of preprocessing further comprises:

obtaining a database search listing, said database search listing containing a plurality of databases determined to be relevant for specified domain areas;

comparing said query to said database search listing to determine the best matches from all candidate databases;

selecting the most relevant databases to submit said query from the user.

48. The method of claim 46, wherein said step of preprocessing further comprises:

invoking a thread manager to facilitate the submission of said query to said plurality of databases, said thread manager coordinating communication between an executory module and a capture module, said executory module controlling bi-directional communications with said databases;

obtaining a thread count parameter for limiting the maximum number of coexisting threads allowed by said thread manager;

establishing a plurality of threads between said capture module and said executory module such than a plurality of coexisting sequential control flows are created between said capture module and said executory module;

establishing a plurality of sockets for said executory module to generate bi-directional connectivity between said executory module and said plurality of appropriate databases;

assigning each one of said plurality of databases to a specified socket;

obtaining a query from a queue;

routing said query from said capture module through an associated one of said plurality of threads to said executory module and from said executory module through an associated one of said plurality of sockets to an associated one of said plurality of appropriate databases;

waiting for a response from said one of said plurality of appropriate databases;

capturing results returned by each one of said plurality of

databases for processing and storage.

49. The method of claim 46, wherein said step of preprocessing further comprises:

building a query queue to store each one of a plurality of queries until each one of said queries is routed to said plurality of databases;

queuing said plurality of queries to be submitted to each one of said plurality of databases;

routing each query in turn to each one of said plurality of databases

50. The method of claim 46, wherein said step of obtaining results further comprises:

capturing links to relevant documents returned from one of said databases via a results page, said links providing a URL path to obtain each one of said documents;

examining each one of said URLs to determine if a redirection is used to reach an ultimate location for anyone of said documents;

following each one of said URLs to determine if subsequent redirects are used;

capturing one of said documents from said database;

capturing an ultimate URL portion associated with said document;

capturing a source URL portion identifying which one of said databases was a referring source of said document;

deleting any intervening URLs between said source url portion and said ultimate URL portion.

51. The method of claim 46, wherein said step of obtaining results further comprises:

obtaining a aging parameter from the user, said aging parameter defining the maximum age of a document to be considered current;

checking each URL for a document returned as a result by one of said databases with documents stored in a document storage and retrieval portion, said document storage and retrieval portion storing all documents captured during a search;

determining if said document is located in said document storage and retrieval portion;

downloading said document if said document is not located in said document storage and retrieval portion;

examining record related information for said document if said document is located within said document storage and retrieval portion;

verifying an age of said document from said record related information;

comparing said age of said document with said aging parameter;

querying a server associated with said document for the existence of modifications to said document if said document IS older than the maximum age determined by said aging parameter;

downloading said document if said document has been modified since the last time the document was downloaded.

52. The method of claim 46, wherein said step of processing further comprises:

obtaining an exclusion list detailing terms, phrases, and sources to be excluded from responses to said query;

comparing each responses associated with said query to said exclusion list;

deleting any responses which match terms, phrases, and sources on said exclusion list.

53. The method of claim 46, wherein said step of processing further comprises:

obtaining an inclusion list detailing terms, phrases, and sources which must be contained in each response to said query;

comparing each response associated with said query to said inclusion list;

deleting any response which does not include terms, phrases, and sources on said inclusion list.

54. The method of claim 46, wherein said step of processing further comprises:

scoring each document returned from each of said databases for relevance compared to said query;

recording a plurality of scores, each score associated with one of said responses from each one of said plurality of databases;

obtaining a minimum acceptable score for narrowing the documents to be considered;

comparing each one of said plurality of scores with said minimum acceptable score;

deleting documents which have an associated score numerically less than said minimum acceptable score.