Recognizer of text-based work

• US 7,356,188 B2
• Filed: 04/24/2001
• Issued: 04/08/2008
• Est. Priority Date: 04/24/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Venkatesan, Ramarathnam, Malkin, Michael T.
• Primary Examiner(s): Bali; Vikkram
• Application Number:
  US09/843,255
• Publication Number:
  US 20020172425A1
• Time in Patent Office: 2,541 Days
• Field of Search: 382/176, 382/187, 382/209, 382/217, 382/218, 382/224, 382/229, 711/216, 711/164, 707/5, 707/6, 710/73, 713/176, 380/28
• US Class Current: 382/229
• CPC Class Codes: G06F 40/194 Calculation of difference between files
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

Described herein is a technology for recognizing the content of text documents. The technology determines one or more hash values for the content of a text document. Alternatively, the technology may generate a “sifted text” version of a document. In one implementation described herein, document recognition is used to determine whether the content of one document is copied (i.e., plagiarized) from another document. This is done by comparing hash values of documents (or alternatively their sifted text). In another implementation described herein, document recognition is used to categorize the content of a document so that it may be grouped with other documents in the same category. This abstract itself is not intended to limit the scope of this patent. The scope of the present invention is pointed out in the appending claims.

Claims

1. A computer-implemented method for hashing a body of text, the method comprising:

obtaining a body of text containing textual content in a computer-readable format, wherein the textual content of the obtained computer-readable formatted body of text is mutable via software tools for manipulation of textual content of bodies of text;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of the textual content of bodies of text;

deriving a hash value representative of the textual content of the filtered subtext, perceptually distinct filtered subtexts having hash values that are substantially independent of each other, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting,

wherein the filtering further comprises removing superfluous elements from the textual content, thereby leaving a remaining textual content and re-arranging the remaining textual content into a canonical format.

2. A method as recited in claim 1, wherein perceptually distinct image-based formatted, filtered subtexts have hash values that are independent of each other.

3. A method as recited in claim 1 further comprising comparing hash values of two image-based formatted, filtered subtexts to determine if such values match.

4. A computer-readable medium embedded with computer-executable instructions that, when executed by a computer, performs the method as recited in claim 3.

5. A method as recited in claim 1 further comprising comparing hash values of two image-based formatted, filtered subtexts to determine if such values substantially match.

6. A method as recited in claim 5 further comprising indicating whether such values substantially match.

7. A method as recited in claim 5 further comprising indicating suspicion of plagiarism between the two filtered subtexts when the compared hash values of the two image-based formatted, filtered subtexts substantially match.

8. A computer comprising one or more computer-readable media embedded with computer-executable instructions that, when executed by the computer, perform the method as recited in claim 1.

9. A method as recited in claim 1, wherein, before formatting, the textual content of the body of text comprises multiple words and sentences.

10. A method as recited in claim 1, wherein, before formatting, the textual content of the body of text comprises multiple words and sentences and the derived hash value is representative of the textual content of the body of text.

11. A computer-implemented method for hashing a body of text, the method comprising:

obtaining a body of text containing textual content in a computer-readable format;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of textual content of bodies of text;

deriving a hash value representative of the filtered subtext, perceptually similar filtered subtexts having proximally similar hash values, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting,

wherein the filtering further comprises removing superfluous elements from the textual content, thereby leaving a remaining textual content and re-arranging the remaining textual content into a canonical format.

12. A method as recited in claim 11 further comprising comparing hash value of an image-based formatted, filtered subtext to determine if such value is proximally near hash values of a group of image-based formatted, filtered subtexts having proximally clustered hash values.

13. A method as recited in claim 12 further comprising grouping the image-based formatted, filtered subtext with the group of image-based formatted, filtered subtexts if the hash value of such subtext is proximally near the values of the group.

14. A computer-readable medium embedded with computer-executable instructions that, when executed by a computer, performs the method as recited in claim 12.

15. A computer comprising one or more computer-readable media embedded with computer-executable instructions that, when executed by the computer, perform the method as recited in claim 12.

16. A method as recited in claim 11, wherein, before formatting, the textual content of the body of text comprises multiple words and sentences.

17. A method as recited in claim 11, wherein, before formatting, the textual content of the body of text comprises multiple words and sentences and the derived hash value is representative of the textual content of the body of text.

18. A computer-readable medium embedded with computer-executable instructions that, when executed by a computer, performs the method comprising:

obtaining a body of text containing textual content in a computer-readable format, wherein the textual content of the obtained computer-readable formatted body of text is mutable via software tools for manipulation of textual content of bodies of text;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of the textual content of bodies of text;

deriving a hash value representative of the textual content of the filtered subtext, perceptually distinct filtered subtexts having hash values that are substantially independent of each other, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting,

wherein the filtering further comprises removing superfluous elements from the textual content, thereby leaving a remaining textual content and re-arranging the remaining textual content into a canonical format.

19. A computer-readable medium embedded with computer-executable instructions that, when executed by a computer, performs the method comprising:

obtaining a body of text containing textual content in a computer-readable format, wherein the textual content of the obtained computer-readable formatted body of text is mutable via software tools for manipulation of textual content of bodies of text;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of textual content of bodies of text;

deriving a hash value representative of the filtered subtext, perceptually similar filtered subtexts having proximally similar hash values, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting,

wherein the filtering further comprises removing superfluous elements from the textual content, thereby leaving a remaining textual content and re-arranging the remaining textual content into a canonical format.

20. A computer-implemented method for hashing a body of text, the method comprising:

obtaining a body of text containing textual content in a computer-readable format, wherein the textual content of the obtained computer-readable formatted body of text is mutable via software tools for manipulation of textual content of bodies of text;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of the textual content of bodies of text;

deriving a hash value representative of the textual content of the filtered subtext, perceptually distinct filtered subtexts having hash values that are substantially independent of each other, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting,

wherein the filtering further comprises pseudo-randomly extracting elements of textual content for inclusion in the filtered subset, wherein the pseudo-random extraction is based, at least in part, upon a cryptographic key.

21. A computer-implemented method for hashing a body of text, the method comprising:

obtaining a body of text containing textual content in a computer-readable format;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of textual content of bodies of text;

deriving a hash value representative of the filtered subtext, perceptually similar filtered subtexts having proximally similar hash values, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting,

wherein the filtering further comprises pseudo-randomly extracting elements of textual content for inclusion in the filtered subset, wherein the pseudo-random extraction is based, at least in part, upon a cryptographic key.

22. A computer-readable medium embedded with computer-executable instructions that, when executed by a computer, performs the method comprising:

obtaining a body of text containing textual content in a computer-readable format, wherein the textual content of the obtained computer-readable formatted body of text is mutable via software tools for manipulation of textual content of bodies of text;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of the textual content of bodies of text;

deriving a hash value representative of the textual content of the filtered subtext, perceptually distinct filtered subtexts having hash values that are substantially independent of each other, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting,

wherein the filtering further comprises pseudo-randomly extracting elements of textual content for inclusion in the filtered subset, wherein the pseudo-random extraction is based, at least in part, upon a cryptographic key.

23. A computer-readable medium embedded with computer-executable instructions that, when executed by a computer, performs the method comprising:

obtaining a body of text containing textual content in a computer-readable format, wherein the textual content of the obtained computer-readable formatted body of text is mutable via software tools for manipulation of textual content of bodies of text;

filtering the textual content of the body of text to remove elements of the textual content, wherein the filtering act produces filtered subtext, which is a subset of the textual content of the body of text;

formatting the filtered subtext into a defined image-based format, wherein the textual content of the defined image-based formatted filtered subtext is immutable via software tools for manipulation of textual content of bodies of text;

deriving a hash value representative of the filtered subtext, perceptually similar filtered subtexts having proximally similar hash values, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting, bodies of text, deriving a hash value representative of the filtered subtext, perceptually similar filtered subtexts having proximally similar hash values, wherein the deriving comprises hashing the image-based formatted, filtered subtext resulting from the formatting, wherein the filtering further comprises pseudo-randomly extracting elements of textual content for inclusion in the filtered subset, wherein the pseudo-random extraction is based, at least in part, upon a cryptographic key, in combination with the other limitations as claimed.