ADVANCED RETRY MECHANISM FOR TRANSMITTING LARGE DATASETS

• US 20180131470A1
• Filed: 11/08/2016
• Published: 05/10/2018
• Est. Priority Date: 11/08/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Padmanabhan, Mohan, Ray, Mayukh, Kekatpure, Aditya
• Granted Patent: US 10,454,619 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 1/08 by repeating transmission, e.g. Verdan system H04L1/1858 and H04L1/189 take precedence

Abstract

In non-limiting examples of the present disclosure, systems, methods and devices for transmitting data stored in a source data store to a destination data store are presented. A plurality of data chunks may be received in a first format. At least one upsert operation may be performed on each of the data chunks for converting the first format to a format compatible with a destination data store schema. A transfer of the upserted data chunks to the destination data store may be executed. A determination may be made that at least one of the upserted data chunks has failed to transfer to the destination data store. Failed data chunks may be stored in a failed chunk retry data store. The at least one upsert operation may be re-performed on failed data chunks and a re-execution of the transfer of failed data chunks to the destination data store may be made.

Claims

1. A method for transmitting data from a source data store, comprising:

receiving a plurality of data chunks, the plurality of data chunks received in a first format;

performing at least one upsert operation on each of the plurality of data chunks, the at least one upsert operation comprising converting the first format to a schema compatible with a destination data store;

executing a transfer of the upserted data chunks to the destination data store;

determining that at least one of the upserted data chunks has failed to transfer to the destination data store;

storing the at least one failed data chunk in a failed chunk retry data store;

re-performing the at least one upsert operation on the at least one failed data chunk; and

re-executing a transfer of the at least one failed data chunk to the destination data store.

2. The method of claim 1, further comprising determining a schema format for the destination data store.

3. The method of claim 2, wherein the at least one upsert operation comprises modifying the format of each of the plurality of data chunks to conform to the schema format for the destination data store.

4. The method of claim 1, wherein determining that at least one of the upserted data chunks has failed to transfer to the destination data store comprises receiving an indication that that a single record contained in the upserted data chunks has failed to transfer.

5. The method of claim 1, wherein determining that at least one of the upserted data chunks has failed to transfer to the destination data store comprises receiving an indication that at least one of the upserted data chunks was impacted by at least one of:

network weather, the destination data store being in a fail state, and the destination data store being in an update state.

6. The method of claim 1, further comprising:

encrypting, upon determining that at least one of the upserted data chunks has failed to transfer to the destination data store, the at least one failed data chunk for storage in the failed chunk retry data store.

7. The method of claim 1, further comprising:

encrypting, upon re-performing the at least one upsert operation on the at least one failed data chunk, the at least one failed data chunk for transfer to the destination data store.

8. The method of claim 1, further comprising:

determining whether the at least one failed data chunk has exceeded a back-off retry attempt threshold based on a number of transfer attempts to the destination data store; and

transferring, upon determining that the at least one failed data chunk has exceeded the back-off retry attempt threshold, the at least one failed data chunk to a permanently failed container.

9. The method of claim 8, wherein each subsequent attempt of the back-off retry attempt threshold is linearly scaled based on a linear temporal scale.

10. The method of claim 8, wherein each subsequent attempt of the back-off retry attempt threshold is linearly scaled temporally.

11. The method of claim 8, wherein a plurality of mapping engines are assigned for transferring the plurality of data chunks to the destination data store, each of the plurality of mapping engines assigned for the transfer of a separate data chunk of the plurality of data chunks.

12. The method of claim 11, further comprising:

associating, upon determining that at least one of the upserted data chunks has failed to transfer to the destination data store, metadata with the at least one failed data chunk indicating which of the plurality of mapping engines was assigned to the at least one failed data chunk.

13. The method of claim 1, wherein the failed chunk retry data store is the source data store.

14. A computer-readable storage device comprising executable instructions that, when executed by a processor, assist with transmitting data from a source data store, the computer-readable medium including instructions executable by the processor for:

receiving a plurality of data chunks, the plurality of data chunks received in a first format;

performing at least one upsert operation on each of the plurality of data chunks, the at least one upsert operation comprising converting the first format to a schema compatible with a destination data store;

executing a transfer of the upserted data chunks to the destination data store;

determining that at least one of the upserted data chunks has failed to transfer to the destination data store;

storing the at least one failed data chunk in a failed chunk retry data store;

re-performing the at least one upsert operation on the at least one failed data chunk; and

re-executing a transfer of the at least one failed data chunk to the destination data store.

15. The computer-readable storage device of claim 14, wherein the instructions are further executable by the processor for determining a schema format for the destination data store.

16. The computer-readable storage device of claim 15, wherein the at least one upsert operation comprises modifying the format of each of the plurality of data chunks to conform to the schema format for the destination data store.

17. The computer-readable storage device of claim 14, wherein the instructions are further executable by the processor for encrypting, upon determining that at least one of the upserted data chunks has failed to transfer to the destination data store, the at least one failed data chunk for storage in the failed chunk retry data store.

18. A system for transmitting data from a source data store, comprising:

a memory for storing executable program code; and

a processor, functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative to;

receive a plurality of data chunks, the plurality of data chunks received in a first format;

perform at least one upsert operation on each of the plurality of data chunks, the at least one upsert operation comprising converting the first format to a schema compatible with a destination data store;

execute a transfer of the upserted data chunks to the destination data store;

determine that at least one of the upserted data chunks has failed to transfer to the destination data store;

store the at least one failed data chunk in a failed chunk retry data store;

re-perform the at least one upsert operation on the at least one failed data chunk; and

re-execute a transfer of the at least one failed data chunk to the destination data store.

19. The system of claim 18, wherein the processor is further responsive to the computer-executable instructions and operative to encrypt, upon determining that at least one of the upserted data chunks has failed to transfer to the destination data store, the at least one failed data chunk for storage in the failed chunk retry data store.

20. The system of claim 18, wherein the processor is further responsive to the computer executable instructions and operative to:

determine whether the at least one failed data chunk has exceeded a back-off retry attempt threshold based on a number of transfer attempts to the destination data store; and

transfer, upon determining that the at least one failed data chunk has exceeded the back-off retry attempt threshold, the at least one failed data chunk to a permanently failed container.