Data ingest optimization
First Claim
1. A computer readable storage medium comprising a non-transitory computer readable program code, wherein the computer readable program code when executed on a computer causes the computer to:
- obtain a slot map including slots for the storage of data segments in a persistent storage device, the slot map including quality tag values associated with each of the data segments applied to each slot in the slot map;
prioritize the data segments associated with the slots by weighting the quality tag values, each of which is associated with a different slot and data segment and indicates a respective degree of importance of the associated data segment, with costs of retrieving the data segments from respective data sources and probabilities of successfully retrieving valid data segments from each of the respective data sources at one or more particular future times, and output a priority queue of the data segments;
populate the slot map with retrieved data segments and output the slot map; and
direct a retrieval of the data segments from the respective data sources in an order in accordance with a determined priority of the data segments to optimize a quality of data obtainable within a critical time constraint.
0 Assignments
0 Petitions
Accused Products
Abstract
Methods and systems for optimizing the retrieval of data from multiple sources are described. A slot map including slots for the storage of data elements can be obtained. The data elements associated with the slots can be prioritized by weighting values with costs of retrieving the data elements from respective data sources. Each value can be associated with a different data element and can indicate a respective degree of importance of the associated data element. Further, the systems and methods can direct the retrieval of data elements from the respective data sources in an order in accordance with the priority of the data elements to optimize the quality of data obtainable within a critical time constraint. In addition, the retrieved data elements can be stored in corresponding slots on a storage medium.
30 Citations
12 Claims
-
1. A computer readable storage medium comprising a non-transitory computer readable program code, wherein the computer readable program code when executed on a computer causes the computer to:
-
obtain a slot map including slots for the storage of data segments in a persistent storage device, the slot map including quality tag values associated with each of the data segments applied to each slot in the slot map; prioritize the data segments associated with the slots by weighting the quality tag values, each of which is associated with a different slot and data segment and indicates a respective degree of importance of the associated data segment, with costs of retrieving the data segments from respective data sources and probabilities of successfully retrieving valid data segments from each of the respective data sources at one or more particular future times, and output a priority queue of the data segments; populate the slot map with retrieved data segments and output the slot map; and direct a retrieval of the data segments from the respective data sources in an order in accordance with a determined priority of the data segments to optimize a quality of data obtainable within a critical time constraint. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A system for optimizing the retrieval of data from multiple sources comprising:
-
a slot map generator configured to generate a slot map including slots for the storage of data segments in a persistent storage device, the slot map further including quality tag values associated with each of the data segments applied to each slot in the slot map; a priority module configured to prioritize data segments associated with the slots by weighting the quality tag values, each of which is associated with a different slot and data segment and indicates a respective degree of importance of the associated data segment, with costs of retrieving the data segments from respective data sources and probabilities of successfully retrieving valid data segments from each of the respective data sources at one or more particular future times; and a processor configured to direct a retrieval of the data segments from the respective data sources in an order in accordance with a determined priority of the data segments to optimize a quality of data obtainable within a critical resource constraint, the processor being further configured to output a priority queue of the data segments, populate the slot map with retrieved data segments, and output the slot map. - View Dependent Claims (8, 9, 10, 11, 12)
-
Specification