Indexing system and method

• US 20030191737A1
• Filed: 01/14/2003
• Published: 10/09/2003
• Est. Priority Date: 12/20/1999
• Status: Active Grant

• Current Assignee: Youramigo Limited
• Original Assignee: Youramigo Limited
• Inventors: Steele, Robert James, Powers, David Martin
• Granted Patent: US 7,987,165 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method for generating an index of data available from a server, including processing data on the server to access data items for a central index, the data items including network addresses and terms, compiling an index file including the data items, and transmitting the index file to the central index. The processing may include locating database query statements in the data, and the data items then include input tuples for the statements. The index is accessible from servers, and includes page entries including a program address for a program for generating a dynamic page and input tuples for submission to the program to generate the page, and search entries identifying the dynamic pages and identifying the tuples corresponding to search terms. A search engine operable on the index, is able to access the search entries to identify dynamic pages corresponding to search terms of a search query, and access the page entries to generate addresses for the dynamic pages identified, the addresses being generated on the basis of the program address and the tuples.

Claims

1. A method for generating an index of data available from a server, including:

processing data on said server to access data items for a central index, said data items including network addresses and terms;

compiling an index file including said data items; and

transmitting said index file to said central index.

2. A method as claimed in claim 1, wherein said processing includes determining changes in said data items, and said index file is an index delta file comprising said changes in said data items.

3. A method as claimed in claim 1, wherein said processing includes locating database query statements in said data and said data items include input tuples for said statements.

4. A method as claimed in claim 3, wherein said data items include additional data for accessing a database corresponding to said statements.

5. A method as claimed in claim 4, wherein said additional data includes the network address of a form handling program.

6. A method as claimed in claim 5, wherein the additional data includes the network address of a form page, and details on the input fields for the tuples and columns of the database.

7. A method as claimed in claim 6, wherein the additional data includes terms from the form page and terms from dynamic pages generated by the form handling program in response to tuples submitted on the form page.

8. A method as claimed in claim 6, wherein said additional data includes additional details for accessing the database, such as database type and name, and any authentication data for access, such as a username and password.

9. A method as claimed in claim 3, wherein said processing includes processing data requests received at said server to extract said tuples.

10. A method as claimed in claim 3, wherein said data items include output tuples.

11. A method as claimed in claim 10, wherein said processing includes processing data requests received and responses generated at said server to extract said tuples.

12. A method as claimed in claim 3, wherein said processing includes generating dynamic pages using said statements and said input tuples, and said data items include terms of said dynamic pages.

13. A method as claimed in claim 1, wherein said processing includes processing dynamic pages transmitted from said server in response to data requests and said data items include terms of said dynamic pages.

14. A method as claimed in claim 3, wherein said processing includes identifying dynamic pages on the basis of tags associated with said query statements.

15. A method as claimed in claim 14, wherein said tags are form tags.

16. A method as claimed in claim 14, wherein said tags are applet tags, and said compiling includes generating a submit applet to submit said input tuples to an applet identified by an applet tag, and said data items include a network address for the submit applet.

17. A method as claimed in claim 2, wherein said determining includes installing database triggers to detect changes in columns of a database accessible by query statements in said data, said changes including said changes in said columns.

18. A method as claimed in claim 2, wherein said processing includes generating link pairs from said data, said link pairs including a source network address and target network address and said data items include said pairs.

19. A method as claimed in claim 18, including detecting a change in at least one of said pairs, and sending a change notification to a location corresponding to said source address of said pair.

20. A method as claimed in claim 19, including receiving said change notification and adjusting said data associated with said source address on the basis of said change.

21. A method as claimed in claim 20, wherein said adjusting includes replacing tags with said target address with tags to a new target address.

22. A method as claimed in claim 18, including detecting a change in at least one of said pairs, and sending a change notification to a location corresponding to said target address.

23. A method as claimed in claim 22, including receiving said change notification and establishing a link referral page on the basis of said change at a server associated with said target address.

24. A method as claimed in claim 1, wherein said processing includes accessing statistical data on said server, such as relating to data requests received at said server and data responses sent from said server, and said data items include statistical data.

25. A method as claimed in any one of the preceding claims, wherein said method is executed by an agent for said central index.

26. A method as claimed in claim 25, wherein said agent is on said server.

27. An agent having components for executing the steps of the method as claimed in any one of claim 1 to 24.

28. A method as claimed in any one of claims 1 to 24, including:

receiving said index file at said central index, which has an index database; and

maintaining said index database on the basis of entries in said index file, said index database being adapted for use by a search engine.

29. A method as claimed in claim 28, wherein said index database includes index entries of said data items, said entries of said index file and said index database correspond to pages of data having said data items.

30. A method as claimed in claim 29, wherein said index database includes a lexicon of terms of said data items, and said index database is accessible by said search engine to construct network addresses to generate dynamic pages on the basis of said data items, in response to a search engine query received by said search engine.

31. An indexing system having components for executing the steps of the method as claimed in any one of claims 1 to 24 and 28 to 30.

34. An indexing system including:

an agent as claimed in claim 27;

an index as claimed in claim 32;

and

a search engine as claimed in claim 33.

68. A method as claimed in claim 18, including detecting a change in at least one of said pairs and adjusting at least one network address stored in a user'"'"'s system on the basis of said change.

69. A method as claimed in claim 18, including detecting a change in at least one of said pairs and adjusting at least one network address stored as a bookmark for a browser.

71. A browser agent as claimed in claim 70, wherein said indexing system is as claimed in anyone of claims 34 to 50.

32. An index of data accessible from servers, including:

page entries including a program address for a program for generating a dynamic page and input tuples for submission to the program to generate the page; and

search entries identifying the dynamic pages and identifying the tuples corresponding to search terms.

33. A search engine operable on the index claimed in claim 32, including:

means for accessing the search entries to identify dynamic pages corresponding to search terms of a search query; and

means for accessing the page entries to generate addresses for the dynamic pages identified, said addresses being generated on the basis of said program address and said tuples.

72. A browser agent as claimed in claim 70, wherein said index is as claimed in claim 32

35. An indexing system, including:

a server for providing access to at least one site;

a server agent for creating an index file of data relating to the site; and

a central index for storing index information from the index file, wherein the server agent initiates communication with the central index for transfer of the index file.

36. An indexing system as claimed in claim 35, wherein the server agent is adapted to review the at least one site and compile an index delta file, representing changes to the at least one site, the delta file being transmitted to the central index for updating the index information.

37. An indexing system as claimed in claim 36, including a plurality of servers and associated server agents arranged to transmit a respective index file and/or index delta file to the central index.

38. An indexing system as claimed in claim 37, wherein each delta file includes information on any change affecting the validity of links in the sites of the respective servers.

39. An indexing system as claimed in claim 38, wherein the server agents, upon identifying said change, are arranged to transmit notification of the change to a corresponding server associated with another site that includes an invalidated link resulting from the change.

40. An indexing system as claimed in claim 39, wherein the central index is adapted to identify a site affected by the change and to transmit notification to a corresponding server associated with that site.

41. An indexing system as claimed in claim 40, wherein the notification is transmitted to the respective server agent of the corresponding server which, upon receipt of the notification, is adapted to effect action to update information on the corresponding server and/or relevant site.

42. An indexing system as claimed in claim 40, wherein the notification is transmitted to an administrator of the corresponding server and, upon receipt of the notification, the administrator is able to effect action to update information on the corresponding server and/or relevant site.

43. An indexing system as claimed in claim 37, wherein at least one of the sites serves a dynamic page which is created by access to a database and the index file for dynamic pages includes possible inputs that will generate a valid page by a corresponding form handling program.

44. An indexing system as claimed in claim 43, wherein the index file further includes possible outputs and text of the dynamic page.

45. An indexing system as claimed in claim 43, wherein the server agent associated with the dynamic page is adapted to access the databases and include one or more columns of the database in the index file.

46. An indexing system as claimed in claim 43, wherein the server agent associated with the dynamic page introduces a trigger in the database to identify any changes in the database, the changes being included in the index delta file associated with the dynamic page.

47. An indexing system as claimed in claim 43, wherein the index file associated with the dynamic page includes a record of inputs of pages previously retrieved therefrom by users.

48. An indexing system as claimed in claim 43, wherein the index file associated with the dynamic pages includes an index of the full text pages of the dynamic pages, previously retrieved by users.

49. An indexing system as claimed in claim 43, wherein the index file associated with each dynamic page for which input/output tuples or row/column indexing is stored further includes an identifier for identifying the form handling program and text associated with the page.

50. An indexing system as claimed in claim 43, including a server for accessing the central index in response to search queries.

51. A method of indexing, including:

providing a server agent for indexing sites provided by a server;

compiling an index file representing site data of the sites; and

transmitting the file to a central index, wherein the server agent initiates communication with the central index for transfer of the index file.

52. A method of indexing as claimed in claim 51, wherein the server agent processes the sites and compiles an index delta file, representing changes to the one or more sites, the delta file being transmitted to the central index for updating index information held by the central index.

53. A method of indexing as claimed in claim 52, wherein the server agent records link information on the sites and any change affecting the validity of the link information is recorded in the delta file.

54. A method of indexing as claimed in claim 53, including identifying said change at one of a plurality of servers with associated server agents and transmitting, from a respective server agent, notification of the change to a corresponding server associated with another site of that link.

55. A method of indexing as claimed in claim 53, wherein the central index has a list of site addresses and, upon receipt of a delta function indicating said change, an associated site affected by the change is identified and notification is sent to the corresponding server of the affected site.

56. A method of indexing as claimed in claim 54 or 55, wherein the notification is transmitted to a relevant server agent of the corresponding server which, upon receipt of said notification, effects action to update information on the corresponding server and/or relevant site.

57. A method of indexing as claimed in claim 54 or 55, wherein the notification is transmitted to an administrator of the corresponding server and, upon receipt of the notification, the administrator is able to effect action to update information on the corresponding server and/or relevant site.

58. A method of indexing as claimed in claim 51, wherein the index file includes information relating to dynamic pages of the sites.

59. A method of indexing as claimed in claim 58, wherein the index file includes possible inputs to generate the dynamic pages.

60. A method of indexing as claimed in claim 59, wherein an index file includes possible outputs used to generate the dynamic pages.

61. A method of indexing as claimed in claim 58, wherein the server agent associated with a dynamic page identifies at least one database used to create the dynamic page, and extracts the text and input tuples for inclusion in the index file.

62. A method of indexing as claimed in claim 61, wherein the server agent extracts output tuples associated with said input tuples.

64. A method of indexing as claimed in claim 61, wherein the server agent introduces a trigger in the database to report to the server agent when changes occur in the database, the changes being included in an index delta file.

65. A method of indexing as claimed in claim 58, wherein the index file includes a record of inputs and/or outputs used to generate previously retrieved dynamic pages.

66. A method of indexing as claimed in claim 58, wherein the index file includes an index of the text of dynamic pages previously retrieved by users.

63. A method of indexing as claimed in 62, wherein said server agent extracts input and/or output columns of the database for inclusion in the index file.

67. A method for indexing dynamic pages including:

identifying at least one database accessed in producing a dynamic page;

determining the parameters and environment variables of the database;

determining a relationship between input fields of the page and the database;

identifying columns of the database that correspond to inputs; and

storing data of the columns in an index file.

70. A browser agent, executable on a user computer system, having means for communicating with an index of an indexing system to determine a change in a network address stored on said user computer system.

73. A browser agent as claimed in claim 70, wherein said network address is a URL stored as a bookmark.

74. A browser agent as claimed in claim 70, having means for adjusting said network address on the basis of said change.