System and method for enumerating arbitrary hyperlinked structures in which links may be dynamically calculable

• US 20040205587A1
• Filed: 07/19/2002
• Published: 10/14/2004
• Est. Priority Date: 08/07/2001
• Status: Abandoned Application

• Current Assignee: Iora Limited (Infonic Plc)
• Original Assignee: Iora Limited (Infonic Plc)
• Inventors: Draper, Stephen P.W.
• Application Number:
  US10/200,025
• Publication Number:
  US 20040205587A1
• Time in Patent Office: Days
• US Class Current: 715/513
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9558 Details of hyperlinks; Management of linked annotations
• CPC Class Codes: G06F 40/137 Hierarchical processing, e.g. outlines
• CPC Class Codes: G06F 40/221 Parsing markup language streams streaming G06F40/149

Abstract

A system and method for dynamically enumerating the links in arbitrary hyperlinked data structures using parsing rules are provided. A hyperlinked data structure may be enumerated by reading one or more data objects through an object access interface, parsing the one or more data objects in the data structure, identifying two or more data elements in the one or more data objects, combining one or more data elements to obtain one or more hyperlink addresses, and reading the one or more hyperlink addresses to enumerate the one or more data objects. In one embodiment, hyperlinked structures such as Websites, may be enumerated using configuration templates. The templates may include heuristic parsing rules. The definition of the templates may be independent of the evolution of the data objects over time.

Claims

1. A method for enumerating a hyperlinked data structure, comprising:

reading one or more data objects of a hyperlinked data structure through an object access interface;

parsing one or more data objects of the data structure;

identifying one or more data elements in one or more data objects;

combining two or more data elements to obtain one or more hyperlink addresses; and

reading one or more hyperlink addresses to enumerate one or more data objects.

2. The method of claim 1, wherein reading one or more data objects through an object access interface comprises reading one or more data objects over a network.

3. The method of claim 1, wherein reading one or more data objects through an object access interface comprises reading one or more data objects from a local memory medium.

4. The method of claim 1, wherein identifying one or more data elements comprises identifying one or more data elements according to at least one rule.

5. The method of claim 1, wherein identifying one or more data elements comprises identifying one or more data elements according to at least one rule, wherein at least one rules is associated with at least one configuration template.

6. The method of claim 1, wherein identifying one or more data elements comprises identifying one or more data elements according to at least one rule, wherein at least one rules is associated with at least one configuration template, and wherein the method further comprises selecting at least one configuration template based on a data object type of at least one data object.

7. The method of claim 1, wherein identifying the one or more data elements uses rules based on a data object type of the one or more data objects.

8. The method of claim 1, wherein identifying the one or more data elements uses rules based on grammar of a language of the one or more data objects.

9. The method of claim 1, wherein identifying the one or more data elements uses rules based on the structure of one or more data objects.

10. The method of claim 1, wherein identifying one or more data elements uses rules based on grammar of a language other than a language of the one or more data objects.

11. The method of claim 1, wherein identifying one or more data elements comprises pattern matching based on lexical rules.

12. The method of claim 1, wherein identifying one or more data elements comprises pattern matching based on lexical rules, wherein the lexical rules are specified using regular expressions.

13. The method of claim 1, wherein combining two or more data elements comprises combining two or more identified data elements.

14. The method of claim 1, wherein combining two or more data elements comprises combining one or more identified data elements and one or more fixed data elements.

15. The method of claim 1, wherein combining two or more data elements comprises combining one or more identified data elements and one or more fixed data elements, and wherein one or more fixed data elements are defined by a configuration template.

16. The method of claim 1, wherein combining two or more data elements comprises combining two or more data elements according to at least one rule.

17. The method of claim 1, further comprising applying at least one functional transformation to one or more data elements before combining two or more data elements.

18. The method of claim 1, wherein combining two or more data elements to obtain one or more hyperlink addresses comprises applying at least one functional transformation.

19. The method of claim 1, wherein identifying one or more data elements in one or more data objects and combining two or more data elements to obtain one or more hyperlink addresses comprises applying at least one functional transformation wherein at least one functional transformation identifies one or more data elements and combines one or more data elements to obtain one or more hyperlink addresses.

20. The method of claim 1, wherein combining two or more data elements comprises combining two or more data elements according to at least one rule, wherein at least one rule is defined by a configuration template.

21. The method of claim 1, wherein combining two or more data elements comprises combining two or more data elements according to at least one rule, wherein at least one rule is defined by a configuration template, and wherein the method further comprises selecting at least one configuration template based on a data object type of one or more data objects.

22. The method of claim 1, wherein combining two or more data elements comprises combining two or more data elements according to at least one rule, wherein the method further comprises selecting at least one rule based on a data object type of one or more data objects.

23. The method of claim 1, further comprising filtering one or more hyperlink addresses according to scoping rules.

24. The method of claim 1, wherein one or more hyperlink addresses comprise URLs.

25. The method of claim 1, wherein one or more hyperlink addresses comprise filenames.

26. The method of claim 1, wherein the data structure comprises a Website.

27. The method of claim 1, wherein the data structure comprises a file system.

28. A system configured to enumerate a hyperlinked data structure, comprising:

a network;

a CPU coupled to the network; and

a system memory coupled to the CPU, wherein the system memory stores one or more computer programs executable by the CPU;

wherein one or more computer programs are executable to;

read one or more data objects of a hyperlinked data structure through an object access interface;

parse the one or more data objects of the data structure;

identify one or more data elements in one or more data objects;

combine two or more data elements to obtain one or more hyperlink addresses; and

read one or more hyperlink addresses to enumerate one or more data objects.

29. A carrier medium comprising program instructions, wherein the program instructions are executable to implement a method for enumerating a hyperlinked data structure on a computer system, the method comprising:

reading one or more data objects of a hyperlinked data structure through an object access interface;

parsing one or more data objects of the data structure;

identifying one or more data elements in one or more data objects;

combining two or more data elements to obtain one or more hyperlink addresses; and

reading one or more hyperlink addresses to enumerate one or more data objects.