SYSTEM, APPARATUS, PROGRAM AND METHOD FOR DATA AGGREGATIONE

• US 20150142829A1
• Filed: 10/27/2014
• Published: 05/21/2015
• Est. Priority Date: 11/18/2013
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: LEE, Vivian, Hu, Bo
• Granted Patent: US 9,858,280 B2
• Time in Patent Office: Days
• US Class Current: 707/749
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

Embodiments include a method, apparatus, program, and system for distributing data items among a plurality of data storage units, the data items being an aggregation of data from a plurality of data sources. The method comprises generating a semantic description of each of the plurality of data sources; calculating, for each pair of data sources from among the plurality of data sources, a degree of similarity between the semantic descriptions of the pair of data sources; and allocating data items to data storage units in dependence upon the calculated degree of similarity between the data source of a data item being allocated and the or each data source of data items already allocated to the data storage units.

Claims

1. A method for distributing data items among a plurality of data storage units, the data items being an aggregation of data from a plurality of data sources, the method comprising:

generating a semantic description of each of the plurality of data sources;

calculating, for each pair of data sources from among the plurality of data sources, a degree of similarity between semantic descriptions of the pair of data sources; and

allocating data items to data storage units in dependence upon the degree of similarity between the data source of a data item being allocated and one of the and each data source of data items already allocated to the data storage units.

2. A method according to claim 1, wherein generating a semantic description of each of the data sources includes extracting most significant terms as a list of weighted terms from a data source which list of weighted terms is a generated semantic description.

3. A method according to claim 2, wherein the most significant terms among terms in the data source are identified by and a weight attributed to each of extracted most significant terms is calculated by a term-frequency method.

4. A method according to claim 3, wherein the degree of similarity is a value obtained by calculating a cosine similarity between generated semantic descriptions of the pair of data sources.

5. A method according to claim 1, wherein allocating data items to data storage units in dependence upon the calculated degree of similarity includes allocating a data item to be allocated to the data storage unit storing a highest proportion of the data items from the data source having a highest degree of similarity to the data source of the data item to be allocated.

6. A method according to claim 5, wherein allocating data items to data storage units in dependence upon the calculated degree of similarity includes allocating a group of data items from a same data source to the data storage unit storing a highest proportion of the data items from the data source having the highest degree of similarity to the data source of the data item to be allocated which has sufficient storage space for the group of data items.

7. A method according to claim 5, further comprising:

maintaining a record of, for each data source, an identity of one of the and each data storage unit storing data items from the data source and an indication of a proportion of the data items from the data source stored on one of the andr each identified data storage unit.

8. A method according to claim 1, wherein the data items are stored in a unified data format, wherein the unified data format is an Resource Description Framework (RDF) triple format.

9. A method according to claim 8, further comprising:

reading data from a data source and performing processing to prepare read data for storage as data items having the unified data format.

10. A method according to claim 1, further comprising, if a data storage unit selected to receive data items being allocated already stores data items from two data sources, and the degree of similarity between the semantic descriptions of the two data sources is less than a higher of the degrees of similarity between the semantic descriptions of each of the data sources and the data source of the data items to be allocated, then the data items from the one of the two data sources having a lowest degree of similarity to the data source of the data items to be allocated are removed from the data storage unit and allocated elsewhere.

11. A method according to claim 1, wherein the data items each include an identifier identifying a data source of the respective data item from among the plurality of data sources.

12. An apparatus for distributing data items among a plurality of data storage units, the data items being an aggregation of data from a plurality of data sources, the apparatus comprising:

a description generating module configured to generate a semantic description of each of the plurality of data sources;

a similarity calculation module configured to calculate, for each pair of data sources from among the plurality of data sources, a degree of similarity between semantic descriptions of the pair of data sources; and

an allocator module configured to allocate data items to data storage units in dependence upon a calculated degree of similarity between the data source of a data item being allocated and one of the and each data source of data items already allocated to the data storage units.

13. A data storage system comprising:

an apparatus according to claim 12; and

the plurality of data storage units.

14. A non-transitory computer readable storage storing a computer program which, when executed by a computing apparatus, causes the computing apparatus to perform a process for distributing data items among a plurality of data storage units, the data items being an aggregation of data from a plurality of data sources, the process comprising:

generating a semantic description of each of the plurality of data sources;

calculating, for each pair of data sources from among the plurality of data sources, a degree of similarity between semantic descriptions of the pair of data sources; and

allocating data items to data storage units in dependence upon a calculated degree of similarity between the data source of a data item being allocated and one of the and each data source of data items already allocated to the data storage units.