Systems and methods for storing data and eliminating redundancy

• US 8,984,028 B2
• Filed: 05/29/2012
• Issued: 03/17/2015
• Est. Priority Date: 05/29/2012
• Status: Active Grant

• Current Assignee: Open Text Holdings, Inc. (Open Text Corporation)
• Original Assignee: Recommind Incorporated (Open Text Corporation)
• Inventors: Verschoor, Richard
• Primary Examiner(s): Bullock, Joshua
• Application Number:
  US13/482,907
• Publication Number:
  US 20130325908A1
• Time in Patent Office: 1,022 Days
• Field of Search: 707/812
• US Class Current: 707/812
• CPC Class Codes: G06F 16/1752 based on file chunks

Abstract

A method for effective data storing allowing reducing storage requirements and speeding up various storage operations by reducing storage of redundant data is disclosed. The method includes receiving new informational objects (such as computer files) for storage and breaking down the new informational objects into multiple data objects. Each data object is analyzed to determine whether an identical data object already exists in a database. If an identical data object already exists in the database, the new data object is not stored. If, on the other hand, an identical data object does not exist in the database already, the new data object is stored. Furthermore, for each informational object to be stored, relational links are generated which associate the data objects and their storage locations with the informational object such that the informational object can be easily recreated when required.

Claims

1. A method for storing data using a data storing system that comprises a processor and a memory for storing instruction that when executed by the processor cause the data storing system to perform the method, comprising:

receiving a computer file;

separating the received computer file into a plurality of data objects;

analyzing a database and each of the plurality of data objects of the computer file to determine both;

which data objects of the computer file were not previously stored in the database, and

which data objects of the computer file are already stored in the database;

generating relational links between the data objects already stored in the database and the data objects of the computer file that were previously stored in the database; and

storing in the database both;

the data objects that were not previously stored in the database, and

the generated relational links instead of the data objects that are already stored in the database.

2. The method of claim 1, further comprising identifying a type of the computer file, wherein retrieving of the plurality of data objects from the computer file is based upon the type of the computer file.

3. The method of claim 2, further comprising storing the computer file as a binary object in the database if the type of computer file is not identified.

4. The method of claim 1, further comprising:

generating relational links between the computer file and the data objects that were not previously stored in the database; and

storing in the database the relational links generated for the data objects that were not previously stored in the database.

5. The method of claim 1, wherein the relational links define storage locations associated with the already stored data objects in the database and/or an order of the already stored data objects as presented in the computer file.

6. The method of claim 1, further comprising compressing or encoding the generated relational links.

7. The method of claim 1, wherein the plurality of data objects and the relational links are stored in different databases.

8. The method of claim 1, further comprising:

tracking a frequency of usage of the plurality of data objects; and

caching one or more stored data objects based on the frequency of usage.

9. The method of claim 1, further comprising caching one or more monitored data objects based on a frequency of access.

10. The method of claim 1, wherein the plurality of data objects are stored at least as a part of a binary large object (BLOB).

11. The method of claim 1, wherein the plurality of data objects are stored at least as a part of a character large object (CLOB).

12. The method of claim 1, further comprising:

generating a computer file identifier uniquely identifying the computer file; and

generating a plurality of data object identifiers uniquely identifying the plurality of data objects.

13. The method of claim 12, wherein the relational links are associated with the computer file identifier and the plurality of data object identifiers.

14. The method of claim 1, wherein the computer file comprises one or more of the following:

a computer file, a binary object, and a program code.

15. The method of claim 1, wherein the data objects comprise one or more of the following:

a text, an image, a video, an audio, a multimedia object, a program code, a numerical value, and a data structure.

16. A system for storing data, the system comprising:

a processor; and

a memory for storing executable instructions that are executed by the processor, the

executable instructions comprising;

a communication module configured to receive a computer file;

an exploding module configured to separate the received computer file into a plurality of data objects;

a processing module configured to;

analyze a database and each of the plurality of data objects to determine both;

those data objects of the computer file that were not previously stored in the database, and

those data objects of the computer file that are already stored in the database, and

generate relational links between the data objects already stored in the database and the data objects of the computer file that were previously stored in the database; and

a storing module configured to store in the database both;

the data objects that were not previously stored in the database, and

the generated relational links instead of the data objects that are already stored in the database.

17. The system of claim 16, further comprising an identifying module configured to identify a type of the computer file, wherein retrieving of one or more data objects from the computer file is based upon the type of the computer file.

18. The system of claim 16, wherein the processing module is further configured to determine data objects of the computer file that were previously stored in the database.

19. The system of claim 16, further comprising an encoding module configured to compress or encode the one or more data objects.

20. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to:

receive a first computer file;

separate the first computer file into a plurality of data objects;

analyze a database and each of the data objects of the first computer file to determine both;

data objects of the first computer file that were already previously stored in the database as data objects of a second computer file, and

data objects of the first computer file that were not previously stored in the database as data objects of the second computer file;

generate relational links between the data objects already previously stored in the database as data objects of the second computer file and the data objects of the first computer file that were previously stored in the database as data objects of the second computer file; and

store in the database both;

the data objects of the first computer file that were not previously stored in the database as data objects of the second computer file, and

the generated relational links instead of the data objects that were already previously stored in the database as data objects of the second computer file.

21. A method for storing data using a data storing system that comprises a processor and a memory for storing instruction that when executed by the processor cause the data storing system to perform the method, comprising:

receiving a computer file;

storing the computer file as a single binary object if the computer file cannot be separated into a plurality of data objects;

separating the computer file into a plurality of data objects if the computer file is capable of being separated into the data objects;

generating relational links between the data objects and the computer file, the relational links defining an order of the data objects within the computer file that is utilized to reconstruct the computer file;

analyzing a database and each of the plurality of data objects of the computer file to determine both;

which data objects of the computer file were not previously stored in the database, and

which data objects of the computer file are already stored in the database; and

storing in the database data both;

the data objects that were not previously stored in the database, and

the generated relational links instead of the data objects that are already stored in the database.