Dynamic lexicon

• US 20050080797A1
• Filed: 09/09/2004
• Published: 04/14/2005
• Est. Priority Date: 08/26/2002
• Status: Abandoned Application

• Current Assignee: Siftology Incorporated
• Original Assignee: Siftology Incorporated
• Inventors: Short, Gordon
• Application Number:
  US10/938,336
• Publication Number:
  US 20050080797A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 40/242 Dictionaries

Abstract

In a system for content management, a dynamic lexicon allows dictionary and lexical data at NLP (natural-language processing) engines at remote sites to stay current with table data at a central location without suffering the time loss involved in computing new tables at the remote sites, or computing new tables at the central site and distributing them. As new terms are added to the dictionary, each term is assigned a unique token identifier. A first step involves downloading extensions to the table data in real time whenever a new word or expression is encountered. A second step involves periodically updating the table data in real time with recomputed data transmitted in compact data files from the central location. Content items in the local archive are re-indexed based on the updated table data. Maintaining tokens across generations of tables allows documents in different languages to be associated without requiring translation.

Claims

1. A method of transmitting dictionary updates in a system for real-time analysis of content comprising steps of:

providing a local copy of a dictionary and associated lexical tables;

downloading extensions to said dictionary and said tables as needed to account for new terms from a central location, wherein said extensions assign implied lexical values to said new terms;

periodically downloading from said central location newly-computed lexical values for each term in said dictionary; and

re-indexing documents in a local archive in real time based on said newly-computed lexical values.

2. The method of claim 1, further comprising a step of assigning a unique identifier to each term in said dictionary.

3. The method of claim 2, wherein said unique identifier comprises a tag.

4. The method of claim 2, wherein lexical data for a term is organized in said lexical tables and associated with said unique identifier for said term.

5. The method of claim 2, further comprising a step of:

supplying an implied lexical value for a first term by borrowing a lexical value for a term closely resembling said first term.

6. The method of claim 1, wherein said step of periodically downloading comprises any of the steps of:

when the number of new terms exceeds a predetermined threshold, re-computing said lexical tables and distributing to user sites; and

when a predetermined period of time has passed, re-computing said lexical tables and distributing to user sites.

7. The method of claim 1, wherein said step of periodically downloading comprises downloading said newly-computed values in compact, tabular form.

8. The method of claim 2, wherein a lexical table comprises a vector of values to be associated with a unique identifier.

9. The method of claim 2, wherein a table is downloaded as a vector where an offset in the data comprises said unique identifier, wherein the amount of data distributed to a user'"'"'s system is kept minimized.

10. The method of claim 1, wherein the step of re-indexing documents comprises re-indexing the documents as the system is running.

11. The method of claim 1, wherein the step of re-indexing documents comprises computing a new signature for each document.

12. The method of claim 11, wherein said system runs having a mixture of documents having old lexical tables mixed with documents having new lexical tables.

13. The method of claim 11, wherein all document signatures are brought up to date.

14. A process for updating a lexicon in real time by extension, comprising steps of:

assigning a unique token to each term in a dictionary;

creating extensions to the dictionary wherein each term added to dictionary is assigned an implied lexical value;

transmitting an extension to lexical tables incorporating said implied values to a client when an analysis at said client machine first encounters a new term.

15. A process for updating a lexicon in real time by replacement comprising steps of:

assigning a unique token to each term in a dictionary;

periodically re-computing lexical values for said dictionary;

periodically downloading recomputed lexical values as vectors to a client, wherein each vector is associated with a token; and

re-indexing items in said clients archive in real time using said re-computed lexical values.

16. A method for maintaining currency of an index of content items comprising steps of:

establishing an update schedule that minimizes unbalance between old and new lexical tables;

downloading re-computed lexical values to a client;

initiating re-indexing of an archive at said client wherein items are resigned in real time; and

continuing re-indexing until the entire archive has been resigned.

17. A content management system comprising:

a server;

at least one client; and

means for dynamically transmitting dictionary updates from said server to said at least one client for real-time analysis of content.

18. The system of claim 34, wherein said means for dynamically transmitting dictionary updates from said server to said at least one client for real-time analysis of content comprises

means for downloading down loading extensions to a dictionary and said lexical tables at said client from said server as whenever a new term is encountered, wherein said extensions assign implied lexical values to said new terms;

means for periodically downloading from said server newly-computed lexical values for each term in said dictionary; and

means for re-indexing documents in a client archive in real time based on said newly-computed lexical values.

19. The system of claim 18, further comprising means for assigning a unique identifier to each term in said dictionary.

20. The system of claim 19, wherein said unique identifier comprises a token.

21. The system of claim 19, wherein lexical data for a term is organized in said lexical tables and associated with said unique identifier for said term.

22. The system of claim 19, further comprising means for:

supplying an implied lexical value for a first term by borrowing a lexical value for a term closely resembling said first term.

23. The system of claim 18, wherein said means for periodically downloading comprises means for any of:

when the number of new terms exceeds a predetermined threshold, re-computing said lexical tables and distributing to user sites; and

when a predetermined period of time has passed, re-computing said lexical tables and distributing to user sites.

24. The system of claim 18, wherein said mean for periodically downloading comprises downloading said newly-computed values in compact, tabular form.

25. The system of claim 24, wherein a table comprises a vector of values to be associated with a unique identifier.

26. The system of claim 25, wherein a table is downloaded as a vector where an offset in the data comprises said unique identifier, wherein the amount of data distributed to a user'"'"'s system is kept minimized.

27. The system of claim 18, wherein the step of re-indexing documents comprises re-indexing the documents as the system is running.

28. The system of claim 18, wherein the step of re-indexing documents comprises computing a new signature for each document.

29. The system of claim 28, wherein said system runs having a mixture of documents having old lexical tables mixed with documents having new lexical tables.

30. The system of claim 28, wherein all document signatures are brought up to date.

31. A method of associating documents across languages without translation in a content management system that includes a lexicon comprising steps of:

assigning a unique token to each term;

maintaining the same tokens from generation to generation of lexical tables;

assigning the same tokens to equivalent words, expressions or word combinations in another language so that tables for the two languages correspond; and

associating documents across languages without translating based on said corresponding tables.