Advanced retry mechanism for transmitting large datasets

• US 10,454,619 B2
• Filed: 11/08/2016
• Issued: 10/22/2019
• Est. Priority Date: 11/08/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Padmanabhan, Mohan, Ray, Mayukh, Kekatpure, Aditya
• Primary Examiner(s): Dorman, Christian M
• Application Number:
  US15/346,352
• Publication Number:
  US 20180131470A1
• Time in Patent Office: 1,078 Days
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 1/08 by repeating transmission, e.g. Verdan system H04L1/1858 and H04L1/189 take precedence

Abstract

In non-limiting examples of the present disclosure, systems, methods and devices for transmitting data stored in a source data store to a destination data store are presented. A plurality of data chunks may be received in a first format. At least one upsert operation may be performed on each of the data chunks for converting the first format to a format compatible with a destination data store schema. A transfer of the upserted data chunks to the destination data store may be executed. A determination may be made that at least one of the upserted data chunks has failed to transfer to the destination data store. Failed data chunks may be stored in a failed chunk retry data store. The at least one upsert operation may be re-performed on failed data chunks and a re-execution of the transfer of failed data chunks to the destination data store may be made.

Claims

1. A method for transmitting data from a source data store, comprising:

receiving a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size;

performing at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store;

executing a first attempt to transfer the modified data chunks to the destination data store;

determining that at least one of the modified data chunks has failed to transfer to the destination data store;

storing the at least one failed data chunk in a failed chunk retry data store;

performing the at least one modification operation on the at least one failed data chunk; and

executing a second attempt to transfer the at least one failed data chunk to the destination data store.

2. The method of claim 1, further comprising determining a schema of the second format for the destination data store.

3. The method of claim 2, wherein the at least one modification operation comprises modifying the format of each of the plurality of data chunks to conform to the second format for the destination data store.

4. The method of claim 1, wherein determining that at least one of the modified data chunks has failed to transfer to the destination data store comprises receiving an indication that a single record contained in the modified data chunks has failed to transfer.

5. The method of claim 1, wherein determining that at least one of the modified data chunks has failed to transfer to the destination data store comprises receiving an indication that at least one of the modified data chunks was impacted by at least one of:

network weather, the destination data store being in a fail state, and the destination data store being in an update state.

6. The method of claim 1, further comprising:

encrypting, upon determining that at least one of the modified data chunks has failed to transfer to the destination data store, the at least one failed data chunk for storage in the failed chunk retry data store.

7. The method of claim 1, further comprising:

encrypting, upon performing the at least one modification operation on the at least one failed data chunk, the at least one failed data chunk for transfer to the destination data store.

8. The method of claim 1, further comprising:

determining whether the at least one failed data chunk has exceeded a back-off retry attempt threshold based on a number of transfer attempts to the destination data store; and

transferring, upon determining that the at least one failed data chunk has exceeded the back-off retry attempt threshold, the at least one failed data chunk to a permanently failed container.

9. The method of claim 8, wherein each subsequent attempt to transfer the at least one failed data chunk is linearly scaled based on a linear temporal scale.

10. The method of claim 8, wherein each subsequent attempt to transfer the at least one failed data chunk is exponentially scaled temporally.

11. The method of claim 8, wherein a plurality of mapping engines are assigned for transferring the plurality of data chunks to the destination data store, each of the plurality of mapping engines assigned for the transfer of a separate data chunk of the plurality of data chunks.

12. The method of claim 11, further comprising:

associating, upon determining that at least one of the modified data chunks has failed to transfer to the destination data store, metadata with the at least one failed data chunk indicating which of the plurality of mapping engines was assigned to the at least one failed data chunk.

13. The method of claim 1, wherein the failed chunk retry data store is the source data store.

20. The method of claim 1, wherein the first format corresponds to a structured format and the second format corresponds to at least one of a non-structured format or a semi-structured format.

14. A computer-readable storage device comprising executable instructions that, when executed by a processor, assist with transmitting data from a source data store, the computer-readable storage device including instructions executable by the processor for:

receiving a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size;

performing at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store;

executing a first attempt to transfer the modified data chunks to the destination data store;

determining that at least one of the modified data chunks has failed to transfer to the destination data store;

storing the at least one failed data chunk in a failed chunk retry data store;

performing the at least one modification operation on the at least one failed data chunk; and

executing a second attempt to transfer the at least one failed data chunk to the destination data store.

15. The computer-readable storage device of claim 14, wherein the instructions are further executable by the processor for determining a schema for the second format for the destination data store.

16. The computer-readable storage device of claim 15, wherein the at least one modification operation comprises modifying the format of each of the plurality of data chunks to conform to the second format for the destination data store.

17. A system for transmitting data from a source data store, comprising:

a memory for storing executable program code; and

a processor, functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative to;

receive a plurality of data chunks, wherein the plurality of data chunks are received in a first format and are of a fixed size;

perform at least one modification operation on each of the plurality of data chunks, the at least one modification operation comprising converting the first format to a second format compatible with a destination data store;

execute a first attempt to transfer the modified data chunks to the destination data store;

determine that at least one of the modified data chunks has failed to transfer to the destination data store;

store the at least one failed data chunk in a failed chunk retry data store;

perform the at least one modification operation on the at least one failed data chunk; and

execute a second attempt to transfer the at least one failed data chunk to the destination data store.

18. The system of claim 17, wherein the processor is further responsive to the computer-executable instructions and operative to encrypt, upon determining that at least one of the modified data chunks has failed to transfer to the destination data store, the at least one failed data chunk for storage in the failed chunk retry data store.

19. The system of claim 17, wherein the processor is further responsive to the computer executable instructions and operative to:

determine whether the at least one failed data chunk has exceeded a back-off retry attempt threshold based on a number of transfer attempts to the destination data store; and

transfer, upon determining that the at least one failed data chunk has exceeded the back-off retry attempt threshold, the at least one failed data chunk to a permanently failed container.