Advanced retry mechanism for transmitting large datasets
First Claim
1. A method for transmitting data from a source data store, comprising:
- receiving a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size;
performing at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store;
executing a first attempt to transfer the modified data chunks to the destination data store;
determining that at least one of the modified data chunks has failed to transfer to the destination data store;
storing the at least one failed data chunk in a failed chunk retry data store;
performing the at least one modification operation on the at least one failed data chunk; and
executing a second attempt to transfer the at least one failed data chunk to the destination data store.
1 Assignment
0 Petitions
Accused Products
Abstract
In non-limiting examples of the present disclosure, systems, methods and devices for transmitting data stored in a source data store to a destination data store are presented. A plurality of data chunks may be received in a first format. At least one upsert operation may be performed on each of the data chunks for converting the first format to a format compatible with a destination data store schema. A transfer of the upserted data chunks to the destination data store may be executed. A determination may be made that at least one of the upserted data chunks has failed to transfer to the destination data store. Failed data chunks may be stored in a failed chunk retry data store. The at least one upsert operation may be re-performed on failed data chunks and a re-execution of the transfer of failed data chunks to the destination data store may be made.
-
Citations
20 Claims
-
1. A method for transmitting data from a source data store, comprising:
-
receiving a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size; performing at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store; executing a first attempt to transfer the modified data chunks to the destination data store; determining that at least one of the modified data chunks has failed to transfer to the destination data store; storing the at least one failed data chunk in a failed chunk retry data store; performing the at least one modification operation on the at least one failed data chunk; and executing a second attempt to transfer the at least one failed data chunk to the destination data store. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 20)
-
-
14. A computer-readable storage device comprising executable instructions that, when executed by a processor, assist with transmitting data from a source data store, the computer-readable storage device including instructions executable by the processor for:
-
receiving a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size; performing at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store; executing a first attempt to transfer the modified data chunks to the destination data store; determining that at least one of the modified data chunks has failed to transfer to the destination data store; storing the at least one failed data chunk in a failed chunk retry data store; performing the at least one modification operation on the at least one failed data chunk; and executing a second attempt to transfer the at least one failed data chunk to the destination data store. - View Dependent Claims (15, 16)
-
-
17. A system for transmitting data from a source data store, comprising:
-
a memory for storing executable program code; and a processor, functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative to; receive a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size; perform at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store; execute a first attempt to transfer the modified data chunks to the destination data store; determine that at least one of the modified data chunks has failed to transfer to the destination data store; store the at least one failed data chunk in a failed chunk retry data store; perform the at least one modification operation on the at least one failed data chunk; and execute a second attempt to transfer the at least one failed data chunk to the destination data store. - View Dependent Claims (18, 19)
-
Specification