Combined content indexing and data reduction

• US 9,772,981 B2
• Filed: 03/29/2006
• Issued: 09/26/2017
• Est. Priority Date: 03/29/2006
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Osmond, Roger F., Goren, Gil
• Primary Examiner(s): Liao, Jason
• Assistant Examiner(s): Mitiku, Berhanu
• Application Number:
  US11/277,790
• Publication Number:
  US 20070233707A1
• Time in Patent Office: 4,199 Days
• Field of Search: 704 1, 704 4, 704 9- 10, 707696, 707791, 707793, 707796-797, 707956-957, 707959
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

Data storage is improved by combining content indexing and data reduction in text-containing files by using common word elimination. Raw data is processed by finding words in selected files, creating an index of found words, and replacing the words in the raw data with pointers to the corresponding words in the index. Each word appears only once in the index. Consequently, the index is relatively small and the procedure is completely reversible. In particular, the index is small relative to other methods because the data is transformed in place, and the transformed data and index are used together to capture the total information about the data.

Claims

1. A computer program product stored on a non-transitory computer-readable media comprising:

logic which concurrently indexes content and reduces data in data storage, including;

logic which finds individual semantic units in at least one file in data storage, each semantic unit including at least one word;

logic which determines whether a found semantic unit is in an index, and if the found semantic unit is not in the index then adds that semantic unit to the index; and

logic which replaces found semantic units with index pointers to corresponding semantic units in the index.

2. The program product of claim 1 wherein the semantic units are words of a human language.

3. The program product of claim 2 wherein each instance of a found word occurs in the index only once.

4. The program product of claim 2 further comprising logic that selects files for processing by filtering files based on likelihood of retrieval.

5. The program product of claim 2 further comprising logic that selects files for processing by filtering files based on file extension.

6. The program product of claim 2 further comprising logic that selects files for processing by filtering files based on content.

7. The program product of claim 2 further comprising logic that selects, as a word, characters bracketed by one or more of whitespace and punctuation.

8. The program product of claim 2 further comprising logic that ranks words in the index in terms of data reduction and assign pointers to words based on pointer length and word ranking.

9. The program product of claim 2 further comprising logic that post-processes the data to provide indexing by terms other than words.

10. The program product of claim 2 further comprising logic that further compresses the processed data by syllable.

11. A method for indexing content and reducing data, comprising the steps of:

concurrently indexing content and reducing data in data storage by;

finding individual semantic units in at least one file in data storage, each semantic unit including at least one word;

in response to finding a semantic unit, determining whether the semantic unit is in an index, and if the found semantic unit is not in the index then adding that semantic unit to the index; and

replacing found semantic units with index pointers to corresponding semantic units in the index.

12. The method of claim 11 wherein the semantic unit is a word of a human language.

13. The method of claim 12 wherein each instance of a found word occurs in the index only once.

14. The method of claim 12 further comprising the step of selecting files for processing by filtering files based on likelihood of retrieval.

15. The method of claim 12 further comprising the step of selecting files for processing by filtering files based on file extension.

16. The method of claim 12 further comprising the step of selecting files for processing by filtering files based on content.

17. The method of claim 12 further comprising the step of selecting, as a word, characters bracketed by one or more of whitespace and punctuation.

18. The method of claim 12 further comprising the steps of ranking words in the index in terms of data reduction and assigning relatively shorter pointers to words having better potential data reduction ranking.

19. The method of claim 12 further comprising the steps of post-processing the data to provide indexing by terms other than words.

20. The method of claim 12 further comprising the step of compressing the processed data by syllable.

21. Apparatus for processing and storing data, including indexing content and reducing the data, comprising:

storage media that concurrently stores data and an index; and

a processor that;

finds individual semantic units in at least one file in the storage media, each semantic unit including at least one word;

determines whether a found semantic unit is in the index, and if the found semantic unit is not in the index then adds that semantic unit to the index; and

replaces found semantic units with index pointers to the corresponding semantic units in the index.

22. The apparatus of claim 21 wherein the semantic unit is a word in a human language.

23. The apparatus of claim 22 wherein each instance of a found word occurs in the index only once.

24. The apparatus of claim 22 wherein the processor selects files for processing by filtering files based on likelihood of retrieval.

25. The apparatus of claim 22 wherein the processor selects files for processing by filtering files based on file extension.

26. The apparatus of claim 22 wherein the processor selects files for processing by filtering files based on content.

27. The apparatus of claim 22 wherein the processor selects, as a word, characters bracketed by one or more of whitespace and punctuation.

28. The apparatus of claim 22 wherein the processor ranks words in the index in terms of data reduction and assigns relatively shorter pointers to words having better potential data reduction ranking.

29. The apparatus of claim 22 wherein the processor post-processes the data to provide indexing by terms other than words.

30. The apparatus of claim 22 wherein the processor further compresses the processed data by syllable.