Architecture for an indexer

• US 20050165838A1
• Filed: 01/26/2004
• Published: 07/28/2005
• Est. Priority Date: 01/26/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Neumann, Andreas, Rajagopalan, Sridhar, Shekita, Eugene J., Zien, Jason Yeong, Fontoura, Marcus F.
• Granted Patent: US 7,424,467 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99937 Sorting
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

Disclosed is a technique for indexing data. A token is received. It is determined whether a data field associated with the token is a fixed width. When the data field is a fixed width, the token is designated as one for which fixed width sort is to be performed. When the data field is a variable length, the token is designated as one for which a variable width sort is to be performed.

Claims

1. A method for indexing data, comprising:

receiving a token;

determining whether a data field associated with the token is a fixed width;

when the data field is a fixed width, designating the token as one for which fixed width sort is to be performed; and

when the data field is a variable length, designating the token as one for which a variable width sort is to be performed.

2. The method of claim 1, wherein the token is variable width and further comprising:

transforming the variable width token into a fixed width token.

3. The method of claim 1, further comprising:

performing a fixed width sort on one of dual code paths and a variable width sort on the other of dual code paths.

4. The method of claim 1, further comprising:

generating a sort key that includes a token type, a token, a document identifier, a document section, and an offset in a document.

5. The method of claim 1, further comprising:

receiving different sections of a document at different times.

6. The method of claim 5, wherein the different sections include a context section and an anchor text section.

7. The method of claim 1, further comprising:

generating sort keys for each token of multiple tokens; and

using the sort keys to create posting lists that simultaneously are ordered by token and by document identifier for each token.

8. The method of claim 7, further comprising:

using the sort keys to bring together multiple sections of a document.

9. The method of claim 1, further comprising:

sorting on certain bits of a sort key containing multiple bits.

10. The method of claim 9, further comprising:

sorting on uppermost bits of the sort key.

11. A computer system including logic for indexing data, comprising:

receiving a token;

determining whether a data field associated with the token is a fixed width;

when the data field is a fixed width, designating the token as one for which fixed width sort is to be performed; and

when the data field is a variable length, designating the token as one for which a variable width sort is to be performed.

12. The computer system of claim 11, wherein the token is variable width and wherein the logic further comprises:

transforming the variable width token into a fixed width token.

13. The computer system of claim 11, wherein the logic further comprises:

performing a fixed width sort on one of dual code paths and a variable width sort on the other of dual code paths.

14. The computer system of claim 11, wherein the logic further comprises:

generating a sort key that includes a token type, a token, a document identifier, a document section, and an offset in a document.

15. The computer system of claim 11, wherein the logic further comprises:

receiving different sections of a document at different times.

16. The computer system of claim 15, wherein the different sections include a context section and an anchor text section.

17. The computer system of claim 11, wherein the logic further comprises:

generating sort keys for each token of multiple tokens; and

using the sort keys to create posting lists that simultaneously are ordered by token and by document identifier for each token.

18. The computer system of claim 17, wherein the logic further comprises:

using the sort keys to bring together multiple sections of a document.

19. The computer system of claim 11, wherein the logic further comprises:

sorting on certain bits of a sort key containing multiple bits.

20. The computer system of claim 19, wherein the logic further comprises:

sorting on uppermost bits of the sort key.

21. An article of manufacture including a program for indexing data, wherein the program causes operations to be performed, the operations comprising:

receiving a token;

determining whether a data field associated with the token is a fixed width;

when the data field is a fixed width, designating the token as one for which fixed width sort is to be performed; and

when the data field is a variable length, designating the token as one for which a variable width sort is to be performed.

22. The article of manufacture of claim 21, wherein the token is variable width and wherein the operations further comprise:

transforming the variable width token into a fixed width token.

23. The article of manufacture of claim 21, wherein the operations further comprise:

performing a fixed width sort on one of dual code paths and a variable width sort on the other of dual code paths.

24. The article of manufacture of claim 21, wherein the operations further comprise:

generating a sort key that includes a token type, a token, a document identifier, a document section, and an offset in a document.

25. The article of manufacture of claim 21, wherein the operations further comprise:

receiving different sections of a document at different times.

26. The article of manufacture of claim 25, wherein the different sections include a context section and an anchor text section.

27. The article of manufacture of claim 21, wherein the operations further comprise:

generating sort keys for each token of multiple tokens; and

using the sort keys to create posting lists that simultaneously are ordered by token and by document identifier for each token.

28. The article of manufacture of claim 27, wherein the operations further comprise:

using the sort keys to bring together multiple sections of a document.

29. The article of manufacture of claim 21, wherein the operations further comprise:

sorting on certain bits of a sort key containing multiple bits.

30. The article of manufacture of claim 29, wherein the operations further comprise:

sorting on uppermost bits of the sort key.