System for compiling word usage frequencies

• US 8,543,373 B2
• Filed: 11/13/2008
• Issued: 09/24/2013
• Est. Priority Date: 01/13/2005
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Chen, Yen-Fu, Dunsmoir, John W.
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US12/270,082
• Publication Number:
  US 20090063478A1
• Time in Patent Office: 1,776 Days
• Field of Search: None
• US Class Current: 704/1
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models

Abstract

A system for assisting a user who is learning a language to prioritize words to be learned in order of usage frequency is disclosed. A frequency determination program running on a computer determines the frequency of usage of each word at a list of locations provided by the user. Different algorithms to identify what constitutes a word are employed depending upon the language of the source data. The total number of words at each location and their usage frequency found during the user session, along with a total number of words and their usage frequency for all user sessions performed regardless of location, are calculated and made available to the user. The user can view usage frequencies for words from a single location, a group of locations, or all user sessions performed.

Claims

1. A computer system for managing word usage frequencies, the computer system comprising:

one or more processors, one or more computer-readable memories and one or more computer-readable tangible storage devices;

program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to receive;

an identifier of a location;

an identifier of a number of document levels at the location, the document levels comprising at least one document; and

an identifier of a minimum number of words;

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, to determine whether the minimum number of words are present in the at least one document;

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, responsive to determining that the minimum number of words are present in the at least one document, to analyze all of the at least one document to determine whether the at least one document comprises Latin based words or Sino-Tibetan based words;

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, responsive to determining that the at least one document comprises the Latin based words, to populate a Latin based word list with unique Latin based words in the at least one document and to determine a frequency of each of the unique Latin based words in the at least one document;

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, responsive to determining that the at least one document comprises the Sino-Tibetan based words, to populate a Sino-Tibetan based word list with unique Sino-Tibetan based words in the at least one document and to determine a frequency of the unique Sino-Tibetan based words in the at least one document; and

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, to generate results comprising the unique Latin based words, the frequency of the unique Latin based words, the unique Sino-Tibetan based words, and the frequency of the unique Sino-Tibetan based words.

2. The computer system of claim 1, wherein the program instructions to determine whether the location comprises the Latin based words or the Sino-Tibetan based words identify a Unicode encoding for words in the location.

3. The computer system of claim 1, wherein the location is a website having hyperlinks.

4. The computer system of claim 1, further comprising program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, responsive to determining that the at least one document comprises the Latin based words, to calculate a ratio of the frequency of the Latin based words to a total number of the Latin based words.

5. The computer system of claim 1, further comprising program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, responsive to determining that the at least one document comprises the Sino-Tibetan based words, to calculate a ratio of the frequency of the Sino-Tibetan based words to a total number of the Sino-Tibetan words.

6. The computer system of claim 1, wherein the program instructions to determine the frequency of the unique Sino-Tibetan based words:

search a Sino-Tibetan word dictionary for an entry that comprises a sequence of characters in at least one of the unique Sino-Tibetan based words; and

in response to determining that the dictionary includes the entry, add the at least one of the unique Sino-Tibetan based words to the Sino-Tibetan based word list.

7. The computer system of claim 6 then comprising:

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, to convert an encoding of the at least one document from a first encoding to a second encoding, wherein the dictionary is stored in the second encoding.

8. The computer system of claim 7, wherein the dictionary is also stored in another encoding.

9. The computer system of claim 1, further comprising:

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, to identify a link in the at least one document to at least one other document;

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, to determine whether the at least one other document is present within the number of document levels at the location; and

program instructions, stored on the at least one of the one or more storage devices for execution by the at least one of the one or more processors via the at least one of the one or more memories, to cease processing of the at least one other document if the at least one other document is not present within the number of document levels at the location.

10. The computer system of claim 1, wherein the program instructions to determine the frequency of the each of the unique Latin based words in the at least one document:

determine that a unique Latin based word in the at least one document contains at least one leading punctuation mark, and in response, remove the leading punctuation mark from the Latin based word prior to populating the Latin based word list with the unique Latin based words.

11. A computer program product for managing word usage frequencies, the computer program product comprising:

one or more computer-readable tangible storage devices;

program instructions, stored on at least one of the one or more storage devices, to receive;

an identifier of a location comprising at least one document,

an identifier of a number of document levels at the location, the document levels comprising the at least one document, and

an identifier of a minimum number of words;

program instructions, stored on at least one of the one or more storage devices, to determine whether the minimum number of words are present in the at least one document;

program instructions, stored on at least one of the one or more storage devices, responsive to determining that the minimum number of words are present in the at least one document, to analyze all of the at least one document to determine whether the at least one document comprises Latin based words or Sino-Tibetan based words;

program instructions, stored on at least one of the one or more storage devices, responsive to determining that the at least one document comprises the Latin based words, to populate a Latin based word list with unique Latin based words in the at least one document and to determine a frequency of each of the unique Latin based words in the at least one document;

program instructions, stored on at least one of the one or more storage devices, responsive to determining that the at least one document comprises the Sino-Tibetan based words, to populate a Sino-Tibetan based word list with unique Sino-Tibetan based words in the at least one document and to determine a frequency of the unique Sino-Tibetan based words in the at least one document; and

program instructions, stored on at least one of the one or more storage devices, to generate results comprising the unique Latin based words, the frequency of the unique Latin based words, the unique Sino-Tibetan based words, and the frequency of the unique Sino-Tibetan based words.

12. The computer program product of claim 11, wherein the program instructions to determine whether the location comprises Latin based words or Sino-Tibetan based words identify a Unicode encoding for words in the location.

13. The computer program product of claim 11, wherein the location is a website having hyperlinks, and further comprising program instructions, stored on at least one of the one or more storage devices, responsive to determining that the at least one document comprises the Latin based words, to calculate a ratio of the frequency of the Latin based words to a total number of the Latin based words.

14. The computer program product of claim 11, wherein the program instructions to determine the frequency of the Sino-Tibetan based words:

search a Sino-Tibetan word dictionary for an entry that comprises a sequence of characters in at least one of the unique Sino-Tibetan based words; and

in response to determining that the dictionary includes the entry, add the at least one of the unique Sino-Tibetan based words to the Sino-Tibetan based word list.

15. The computer program product of claim 14 further comprising:

program instructions, stored on at least one of the one or more storage devices, to convert an encoding of the at least one document from a first encoding to a second encoding, wherein the dictionary is stored in the second encoding.

16. The computer program product of claim 15, wherein the dictionary is also stored in another encoding.

17. The computer program product of claim 11, further comprising:

program instructions, stored on the at least one of the one or more storage devices to identify a link in the at least one document to at least one other document;

program instructions, stored on the at least one of the one or more storage devices to determine whether the at least one other document is present within the number of document levels at the location; and

program instructions, stored on the at least one of the one or more storage devices, to cease processing of the at least one other document if the at least one other document is not present within the number of document levels at the location.

18. The computer program product of claim 11, wherein the program instructions to determine the frequency of the each of the unique Latin based words in the at least one document:

determine that a unique Latin based word in the at least one document contains at least one leading punctuation mark, and in response, remove the leading punctuation mark from the Latin based word prior to populating the Latin based word list with the unique Latin based words.

19. A method for managing word usage frequencies, the method comprising:

a computer receiving;

an identifier of a location;

an identifier of a number of document levels at the location, the document levels comprising at least one document; and

an identifier of a minimum number of words;

the computer determining whether the minimum number of words are present in the at least one document;

responsive to the computer determining that the minimum number of words are present in the at least one document, the computer analyzing all of the at least one document to determine whether the at least one document comprises Latin based words or Sino-Tibetan based words;

responsive to the computer determining that the at least one document comprises the Latin based words, the computer populating a Latin based word list with the Latin based words and the computer determining a frequency of the Latin based words; and

responsive to the computer determining that the at least one document comprises the Sino-Tibetan based words, the computer populating a Sino-Tibetan based word list with the Sino-Tibetan based words and to determine a frequency of the Sino-Tibetan based words; and

the computer generating results comprising the Latin based words, the frequency of the Latin based words, the Sino-Tibetan based words, and the frequency of the Sino-Tibetan based words.

20. The method of claim 19, wherein the computer determining whether the at least one document comprises Latin based words or Sino-Tibetan based words comprises the computer identifying a Unicode encoding for words in the at least one document.

21. The method of claim 19, wherein the location is a website having hyperlinks, and further comprising:

responsive to the computer determining that the at least one document comprises the Sino-Tibetan based words, the computer calculating a ratio of the frequency of the Sino-Tibetan based words to a total number of the Sino-Tibetan words.

22. The method of claim 19, wherein the computer determining the frequency of the Sino-Tibetan based words comprises:

the computer searching a Sino-Tibetan word dictionary for an entry that comprises a sequence of characters in at least one of the Sino-Tibetan based words; and

in response to the computer determining that the dictionary includes the entry, the computer adding the at least one of the Sino-Tibetan based words to the Sino-Tibetan based word list.

23. The method of claim 22 further comprising:

the computer converting an encoding of the at least one document from a first encoding to a second encoding, wherein the dictionary is stored in the second encoding.

24. The method of claim 23, wherein the dictionary is also stored in another encoding.

25. The method of claim 19, further comprising:

the computer identifying a link in the at least one document to at least one other document; and

the computer determining that the at least one other document is not present within the number of document levels at the location, and in response, the computer ceasing processing of the at least one other document.

26. The method of claim 19, wherein the computer determining the frequency of the each of the unique Latin based words in the at least one document comprises:

the computer determining that a unique Latin based word in the at least one document contains at least one leading punctuation mark, and in response, the computer removing the leading punctuation mark from the Latin based word prior to populating the Latin based word list with the unique Latin based words.