Source editing, internationalization, advanced configuration wizard, and summary page selection for information automation systems

• US 7,908,260 B1
• Filed: 12/31/2007
• Issued: 03/15/2011
• Est. Priority Date: 12/29/2006
• Status: Active Grant

• Current Assignee: Brightplanet Corp II, Inc.
• Original Assignee: Brightplanet Corp II, Inc.
• Inventors: Bushee, William J.
• Primary Examiner(s): Fleurantin; Jean B
• Application Number:
  US12/006,241
• Time in Patent Office: 1,170 Days
• Field of Search: 707 1- 10, 707100-1041, 707200-206, 707600-831, 709/217, 709/219, 704/7, 704/9
• US Class Current: 707/705
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A source manager includes an editor program that can be used to edit an existing source record via a graphical user interface (GUI). Test Action and Test Source functions allow a user to test enter a query and to test a source expeditiously. A conversion tool converts existing sources to the design and format to reconcile data scattered among the source engine data and source partition record. For handling internationalization issues, aspects of the invention include persistently storing the source'"'"'s encoding type during the configuration process, and then using that encoding type later during the deep harvest phase. According to another aspect of the invention a solution for selecting a summary passage for a particular source is provided. Other aspects of the invention include solutions for character encoding, “Next Links” recognition and “Next Results” handling.

Claims

1. An improved information automation system for deep web harvesting of data the information automation system including computer hardware and being configured to connect with individual ones of a plurality of separately-searchable deep web sources and, for each deep web source, to (a) automatically analyze a search query entry form and search result output format;

(b) based on that analysis, automatically develop a source-specific query format and search result collection process corresponding to that deep web source; and

(c) automatically store each source-specific query format as a source record associated with a corresponding deep web source;

wherein for performing a new deep web harvest, the system is configured to accept input of new search criteria and, based on a plurality of source records, formulate a plurality of distinct queries based on a corresponding plurality of source records and on the new search criteria, the improvement comprising;

a user interface device configured to display a visual output and to accept user input including the input of the new search criteria, the user interface device being programmed to;

display, via the visual output, contents of a first automatically-configured source record in a user-readable form;

facilitate user editing of certain portions of the first source record via the interaction with user-readable form; and

facilitate functionality testing, via user interaction with the user-readable form, of a sample automated interaction between the information automation system and the deep web source that corresponds to the first source record, with the automatic interaction being conducted based on the first source record being edited;

wherein the sample automated interaction is user-initiatable via the user-readable form; and

wherein in response to an initiation of the sample automated interaction, at least a portion of a harvesting operation is automatically performed, and a sample result set of the sample automated interaction is displayed via the user interface device for review by the user.

2. The improved information automation system of claim 1, wherein the user-readable form includes a sample query entry portion that accepts sample search criteria entered by a user, and from which a first type of sample automated interaction is initiatable by the user, wherein the first type of automated transaction executes a first sample query at the first source based on the sample search criteria, and displays an output of initial results of the first sample query that are produced by the first source for review by the user.

3. The improved information automation system of claim 2, wherein the first type of sample automated interaction omits harvesting operations beyond obtaining initially-displayed results of the first sample query.

4. The improved information automation system of claim 2, wherein the output of initial results is displayed for review by the user in a separate window from the user-readable form, and wherein following execution of the first type of sample interaction, the user-readable form remains displayed on the user interface device for further editing by the user.

5. The improved information automation system of claim 2, wherein the user-readable form includes source test initiation portion from which a second type of sample automated interaction is initiatable by the user, wherein the second type of automated transaction executes a second sample query at the first source based user-specified search criteria, and displays an output of harvested results of the second sample query that are produced by the first source for review by the user, wherein the harvested results include initially-displayed search results provided by the first source and subsequently-displayed search results provided by the first source in response to additional automated interaction with the first source.

6. The improved information automation system of claim 1, wherein the functionality testing includes separate selectable testing of (i) specific query action functionality of the first source record that returns only initially-displayed search results in response to a sample query automatically executed at the first source;

and (ii) source harvesting functionality of the first source record that returns initially-displayed search results provided by the first source in response to a sample query, and subsequently-displayed search results provided by the first source in response to the sample query and further in response to additional automated interaction with the first source.

7. The improved information automation system of claim 1, wherein the user interface is programmed to facilitate user editing of at least one field of the user-readable form that corresponds to at least one parameter of the first source record selected from the group consisting of:

title, encoding, language, search results removal flag, action string, action method type, Boolean operator configuration, next link identification, filter link identification, source authentication, or any combination thereof.

8. The improved information automation system of claim 1, wherein the user interface is programmed to:

accept a deep web harvest query entry from a user, the deep web harvest query being applicable to a plurality of different sources and having an original set of Boolean operators; and

facilitate user editing to set source-specific Boolean operators for the first source record; and

wherein the information automation system is programmed to automatically-re-form the web harvest query directed at the first source such that the source-specific Boolean operators are used in that query in place of the original set of Boolean operators.

9. The improved information automation system of claim 1, wherein the user interface is programmed to accept a deep web harvest query entry from a user, the deep web harvest query being applicable to a plurality of different sources and having an original syntax that includes a set of original Boolean operators;

and

wherein the information automation stem is programmed to determine whether the first source utilizes a query syntax that differs from the original syntax and, in response to a determination that the first source utilizes a query syntax that omits Boolean operators, to automatically-re-form the web harvest query directed at the first source into a new syntax that differs from the original syntax such that the new syntax reflects logic of the original syntax based on the set of original Boolean operators but without the presence of any Boolean operators.

10. The improved information automation system of claim 9, wherein the information automation stem is programmed to determine whether the first source utilizes a query syntax that differs from the original syntax based on a language associated with the first source.

11. The improved information automation system of claim 9, wherein the information automation stem is programmed to re-form the web harvest query for the first source by removing the Boolean operators from the web harvest query, and by inserting or removing at least one space between search tokens based on the set of original Boolean operators.

12. The improved information automation system of claim 1, wherein the user interface is programmed to accept a deep web harvest query entry from a user, the deep web harvest query being applicable to a plurality of different sources and having an original syntax that includes a set of non-English language search tokens and a set of English language Boolean operators.

13. The improved information automation system of claim 1, further comprising:

a storage arrangement in which source records are stored, the storage arrangement being a data storage space that is a part of a search engine program stored on a computer-readable medium.

14. The improved information automation system of claim 1, wherein the user interface is programmed to facilitate automatic configuration of a source record based on automatic analysis of a user'"'"'s interaction with a source to which that source record corresponds.

15. The improved information automation system of claim 14, wherein the user interface is programmed to obtain a copy of an HTML page of the source, and to monitor a user'"'"'s interaction with that copy of the HTML page.

16. The improved information automation system of claim 15, wherein the user interface is programmed to dynamically adjust a document object model (DOM) tree of the HTML page without modifying any original forms of the HTML page.

17. The improved information automation system of claim 15, wherein in response to the user'"'"'s interaction with the copy of the HTML page, the information automation system determines which data entry fields of the HTML page should be set in accordance with execution of a query, and which action button should be activated to initiate that query.

18. The improved information automation system of claim 1, wherein the user interface is programmed to accept a deep web harvest query entry from a user, the deep web harvest query being applicable to a plurality of different sources and having an original syntax that includes a set of original Boolean operators;

and

wherein the information automation stem is programmed to determine a type of character encoding used by the first source, and to re-form the deep web harvest query for the first source to correspond with the type of character encoding determined to be used by the first source.

19. In a computer-implemented information automation system for deep web harvesting of data in which the information automation system includes computer hardware and is configured to connect with individual ones of a plurality of separately-searchable deep web sources and, for each deep web source, to (a) automatically analyze a search query entry form and search result output format;

(b) based on that analysis, automatically develop a source-specific query format and search result collection process corresponding to that deep web source; and

(c) automatically store each source-specific query format as a source record associated with a corresponding deep web source;

wherein for performing a new deep web harvest, the system is configured to accept input of new search criteria and, based on a plurality of source records, formulate a plurality of distinct queries based on a corresponding plurality of source records and on the new search criteria, a method for interactively editing source records, the method comprising;

displaying, via a user interface device, contents of a first automatically-configured source record, the contents being displayed in a user-readable form;

providing, via the user-readable form, editable fields for modifying content of the first source record;

providing, via the user-readable form, a functionality testing initiation control, such that, when activated, the functionality testing initiation control initiates a sample automated interaction between the information automation system and the deep web source that corresponds to the first source record, with the automatic interaction being conducted based on the first source record being edited;

in response to an initiation of the sample automated interaction, automatically performing at least a portion of a harvesting operation, and displaying, via the user interface device, a sample result set of the sample automated interaction.

20. The method of claim 19, further comprising:

obtaining a copy of an HTML page of a source to which a source record corresponds, and displaying the copy of the HTML page via the user interface device;

automatically monitoring a user'"'"'s interaction with the copy of the HTML page;

automatically analyzing the monitored interaction with the copy of the HTML page to determine which fields in the HTML page are relevant to querying the source and browsing query results provided by the source; and

automatically configuring the source record based on the fields in the HTML page that are determined to be relevant to querying the source and browsing the query results.