Methods and systems for determining a language of a document
First Claim
Patent Images
1. A system for automatically determining a language of a document from a set of candidate languages, the system comprising:
- a database containing probability data for a plurality of text strings each having a predetermined length equal to each other, each text string of the plurality of text strings having an associated probability value indicating a probability that the text string occurs within a language based on occurrences of the text string in all of the candidate languages;
logic for setting a negative assumption value for each of the candidate languages indicating the document is not one of the candidate languages;
an extractor for extracting a character string from the document, the character string having a length equal to the predetermined length of the plurality of text strings contained in the database; and
a language analyzer for determining a probability value for each of the candidate languages that the character string does not belong to the candidate languages by retrieving the probability value associated to the character string from the database for each or the candidate languages, and includes logic for adjusting the negative assumption value based on the probability value, the language analyzer determining that the document is one language of the candidate languages when the negative assumption value passes a threshold value.
2 Assignments
0 Petitions
Accused Products
Abstract
A system and method for determining the language of an unknown document is provided. For a set of candidate languages, a negative assumption is set for each candidate language that the document is not that language and the system attempts to prove the negative assumption is wrong. If the negative assumption fails for one language, then the document is identified as being in that language. The present system and method provides a higher degree of accuracy when determining the language of a document.
48 Citations
19 Claims
-
1. A system for automatically determining a language of a document from a set of candidate languages, the system comprising:
-
a database containing probability data for a plurality of text strings each having a predetermined length equal to each other, each text string of the plurality of text strings having an associated probability value indicating a probability that the text string occurs within a language based on occurrences of the text string in all of the candidate languages; logic for setting a negative assumption value for each of the candidate languages indicating the document is not one of the candidate languages; an extractor for extracting a character string from the document, the character string having a length equal to the predetermined length of the plurality of text strings contained in the database; and a language analyzer for determining a probability value for each of the candidate languages that the character string does not belong to the candidate languages by retrieving the probability value associated to the character string from the database for each or the candidate languages, and includes logic for adjusting the negative assumption value based on the probability value, the language analyzer determining that the document is one language of the candidate languages when the negative assumption value passes a threshold value. - View Dependent Claims (2, 3, 4)
-
-
5. A method of determining a language of a document from a set of candidate languages, the method comprising the steps of:
-
setting a null hypothesis to a true value for each candidate language indicating the document is not in the candidate language and setting a false value; extracting a text string from the document, the text string having a predetermined length; determining a contrary probability for each candidate language that the text string does not belong to the candidate language based on probabilities that the text string belongs to each of the candidate languages where the probabilities are retrieved from a database that stores probability values for a plurality of text strings each having the predetermined length, each text string of the plurality of text strings having an associated probability value for each candidate language indicating a probability that the text string occurs within a language from the candidate languages based on occurrences of the text string in all of the candidate languages; adjusting the null hypothesis for each candidate language with the contrary probability corresponding to the candidate language; and determining the document is one language from the candidate languages when the null hypothesis for the one language is disproved by approaching the false value. - View Dependent Claims (6, 7, 8, 9, 10, 11, 12)
-
-
13. A process of determining that a document is in a selected language, the process comprising the steps of:
-
setting a probability assumption indicating that the document is not in the selected language; extracting a character string from the document; and disproving the probability assumption based on a contrary probability that the character string does not belong to the selected language such that if the contrary probability fails to support the probability assumption, then the document is determined as being in the selected language. - View Dependent Claims (14, 15, 16, 17, 18, 19)
-
Specification