Supporting big data in enterprise content management systems

• US 9,348,855 B2
• Filed: 02/13/2013
• Issued: 05/24/2016
• Est. Priority Date: 02/13/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Shazly, Hassan A.
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US13/766,661
• Publication Number:
  US 20140229514A1
• Time in Patent Office: 1,196 Days
• Field of Search: 707/812, 707/600, 707/607, 707/688, 707/706
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Provided are a method, computer program product, and system for storing a data stream. A distributed library server is configured with a plurality of data servers. A data type is mapped to at least one of the plurality of data servers. A portion of a data stream containing data of the data type is selected. An object part including the portion of the data stream is generated, and the object part is stored at one of the data servers mapped to the data.

Claims

1. A computer implemented method for storing a data stream, comprising:

providing a big data enterprise content management system with a distributed library server, a plurality of data servers, and a big data stream processor, wherein the distributed library server includes

     1) a library server main node that includes a data type table that identifies data types and a node table and

     2) a plurality of library server nodes, wherein each of the library server nodes contains a logical metadata table, and wherein the logical metadata table is comprised of physical metadata tables that describe storage locations for object parts, by at least one computing processor of a computing system;

mapping a data type to at least one of the plurality of data servers, by the at least one computing processor of the computing system by;

identifying the data type among the data types in the data type table;

identifying a logical metadata table of one of the library server nodes using the data type table;

identifying a physical metadata table using the logical metadata table, wherein the physical metadata table identifies a node; and

using the node table to map the node identified in the physical metadata table to the at least one of the plurality of data servers;

selecting a portion of the data stream containing data of the data type, by the at least one computing processor of the computing system;

generating an object part including the portion of the data stream, by the at least one computing processor of the computing system; and

storing the object part at the at least one of the plurality of data servers mapped to the data type of the data, by the at least one computing processor of the computing system.

2. The method of claim 1, further comprising:

storing the data stream as a collection of object parts.

3. The method of claim 1, wherein selecting further comprises:

receiving a control signal; and

upon receiving the control signal, identifying the portion.

4. The method of claim 1, wherein selecting further comprises:

determining a data format of the portion of the data stream; and

upon a change of the data format, identifying the portion.

5. The method of claim 1, further comprising:

indexing the object part with text information contained in the data stream.

6. The method of claim 1, further comprising:

processing the data stream with an executable process.

7. The method of claim 6 further comprising:

executing the executable process at a node determined by the distributed library server.

8. A system for storing a data stream, comprising:

a processor; and

a storage device connected to the processor, wherein the storage device has stored thereon a program, wherein the processor is configured to execute instructions of the program to perform operations, and wherein the operations comprise;

providing a big data enterprise content management system with a distributed library server, a plurality of data servers, and a big data stream processor, wherein the distributed library server includes

     1) a library server main node that includes a data type table that identifies data types and a node table and

     2) a plurality of library server nodes, wherein each of the library server nodes contains a logical metadata table, and wherein the logical metadata table is comprised of physical metadata tables that describe storage locations for object parts;

mapping a data type to at least one of the plurality of data servers by;

identifying the data type among the data types in the data type table;

identifying a logical metadata table of one of the library server nodes using the data type table;

identifying a physical metadata table using the logical metadata table, wherein the physical metadata table identifies a node; and

using the node table to map the node identified in the physical metadata table to the at least one of the plurality of data servers;

selecting a portion of the data stream containing data of the data type;

generating an object part including the portion of the data stream; and

storing the object part at the at least one of the plurality of data servers mapped to the data type of the data.

9. The system of claim 8, wherein the operations further comprise:

storing the data stream as a collection of object parts.

10. The system of claim 8, wherein the operation of selecting includes:

receiving a control signal; and

upon receiving the control signal, identifying the portion.

11. The system of claim 8, wherein the operation of selecting includes:

determining a data format of the portion of the data stream; and

upon a change of the data format, identifying the portion.

12. The system of claim 8, wherein the operations further comprise:

indexing the object part with text information contained in the data stream.

13. The system of claim 8, wherein the operations further comprise:

processing the data stream with an executable process.

14. The system of claim 13, wherein the operations further comprise:

executing the executable process at a node determined by the distributed library server.

15. A computer program product for storing a data stream, the computer program product comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code, executable by at least one processor of a computer is configured to perform;

providing a big data enterprise content management system with a distributed library server, a plurality of data servers, and a big data stream processor, wherein the distributed library server includes

     1) a library server main node that includes a data type table that identifies data types and a node table and

     2) a plurality of library server nodes, wherein each of the library server nodes contains a logical metadata table, and wherein the logical metadata table is comprised of physical metadata tables that describe storage locations for object parts;

mapping a data type to at least one of the plurality of data servers by;

identifying the data type among the data types in the data type table;

identifying a logical metadata table of one of the library server nodes using the data type table;

identifying a physical metadata table using the logical metadata table, wherein the physical metadata table identifies a node; and

using the node table to map the node identified in the physical metadata table to the at least one of the plurality of data servers;

selecting a portion of the data stream containing data of the data type;

generating an object part including the portion of the data stream; and

storing the object part at the at least one of the plurality of data servers mapped to the data type of the data.

16. The computer program product of claim 15, wherein the computer readable program code, executable by the at least one processor of the computer, is configured to perform:

storing the data stream as a collection of object parts.

17. The computer program product of claim 15, wherein the computer readable program code performing the selecting is configured to include:

receiving a control signal; and

upon receiving the control signal, identifying the portion.

18. The computer program product of claim 15, wherein the computer readable program code performing the selecting is configured to include:

determining a data format of the portion of the data stream; and

upon a change of the data format, identifying the portion.

19. The computer program product of claim 15, wherein the computer readable program code, executable by the at least one processor of the computer, is configured to perform:

indexing the object part with text information contained in the data stream.

20. The computer program product of claim 15, wherein the computer readable program code, executable by the at least one processor of the computer, is configured to perform:

processing the data stream with an executable process.

21. The computer program product of claim 20, wherein the computer readable program code, executable by the at least one processor of the computer, is configured to perform:

executing the executable process at a node determined by the distributed library server.