System and method for representing and maintaining redundant data sets utilizing DNA transmission and transcription techniques

• US 7,194,504 B2
• Filed: 02/12/2001
• Issued: 03/20/2007
• Est. Priority Date: 02/18/2000
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Avamar Technologies, Inc. (Dell Technologies Inc.)
• Inventors: Moulton, Gregory Hagan
• Primary Examiner(s): JUNG, DAVID YIUK
• Application Number:
  US09/781,920
• Publication Number:
  US 20020010797A1
• Time in Patent Office: 2,227 Days
• Field of Search: 709200-203, 382/244, 382/232, 380/217
• US Class Current: 709/203
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC
• CPC Class Codes: H04L 69/06 Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
• CPC Class Codes: H04Q 2213/13034 A/D conversion, code compression/expansion
• CPC Class Codes: H04Q 2213/13103 Memory
• CPC Class Codes: H04Q 2213/13166 Fault prevention
• CPC Class Codes: H04Q 2213/13204 Protocols
• CPC Class Codes: H04Q 2213/13296 Packet switching, X.25, frame relay
• CPC Class Codes: H04Q 2213/13339 Ciphering, encryption, security
• CPC Class Codes: H04Q 2213/13389 LAN, internet

Abstract

A system and method for representing and maintaining redundant data sets utilizing system DNA transmission and transcription techniques comprising a system of symbolic exchange used for data transmission whose lexicon is based on the provision of unique and consistent identifiers (or symbols) for transmitted data utilizing “sticky byte” factoring. A protocol is provided for determining which entries (comprising unique identifier to corresponding data pairs) in a local lexicon (or “Primordial Data Pool”) are contained in a remote lexicon and can, therefore, be used in a transmission in lieu of the data itself. A mechanism is further provided through which the local lexicon can add newly presented unique identifier to corresponding data pairs to one or more remote lexicons.

Claims

1. A system for data transmission without loss of data integrity comprising:

a transmitting system comprising a first lexicon including a plurality of digital sequences with unique corresponding identifiers, wherein each of the digital sequences is greater in size than the corresponding unique identifiers and wherein said unique corresponding identifiers are based upon hashes of said corresponding digital sequences;

a receiving system;

a transmission medium coupling said transmitting and receiving systems, said transmitting system capable of transmitting at least one of said plurality of unique corresponding identifiers to said receiving system in lieu of said at least one corresponding digital sequence.

2. The system of claim 1 comprising:

another system capable of receiving at least one of said plurality of unique corresponding identifiers from one of said transmitting or receiving systems.

3. The system of claim 1 wherein said receiving system comprises a second lexicon including a plurality of digital sequences with unique corresponding identifiers.

4. The system of claim 2 wherein said another system comprises another lexicon including a plurality of digital sequences with unique corresponding identifiers.

5. The system of claim 4 wherein said first and second lexicons comprise information regarding content of other lexicons in said system.

6. The system of claim 1 wherein said unique corresponding identifiers are based upon sequential identifiers for said corresponding digital sequences.

7. The system of claim 1 wherein said unique corresponding identifiers are based upon a hash based file system.

8. The system of claim 1 wherein a digital sequence to be transmitted from said transmitting system to said receiving system is factored into a number of digital sequence chunks for which corresponding unique identifiers are ascribed.

9. The system of claim 8 wherein said digital sequence to be transmitted is factored by means of sticky byte factoring.

10. The system of claim 1 wherein said first lexicon further comprises information regarding contents of other lexicons in said system.

11. The system of claim 1 wherein data transmissions within said system are effectuated on a point-to-point basis.

12. The system of claim 1 wherein data transmissions within said system are effectuated on a point-to-multipoint basis.

13. The system of claim 1 wherein data transmissions are routed in said system based on metrics derived from said first lexicon.

14. A method for data transmission without loss of data integrity comprising:

providing a transmitting system comprising a first lexicon including a plurality of digital sequences with unique corresponding identifiers;

providing a receiving system;

coupling said transmitting and receiving systems through a transmission medium, said transmitting system capable of transmitting at least one of said plurality of unique identifiers to said receiving system in lieu of said at least one corresponding digital sequence; and

factoring a digital sequence to be transmitted from said transmitting system to said receiving system into a number of digital sequence chunks and ascribing corresponding unique identifiers therefor.

15. The method of claim 14 comprising:

providing another system capable of receiving at least one of said plurality of unique identifiers from one of said transmitting or receiving systems.

16. The method of claim 15 wherein said another system comprises another lexicon including a plurality of digital sequences with unique corresponding identifiers.

17. The method of claim 14 wherein said receiving system comprises a second lexicon including a plurality of digital sequences with unique corresponding identifiers.

18. The method of claim 17 wherein said first and second lexicons comprise information regarding content of other lexicons in said system.

19. The method of claim 14 wherein said unique identifiers are based upon hashes of said corresponding digital sequences.

20. The method of claim 14 wherein said unique identifiers are based upon sequential identifiers for said corresponding digital sequences.

21. The method of claim 14 wherein said unique identifiers are based upon a hash based file system.

22. The method of claim 14 wherein said step of factoring is carried out by means of sticky byte factoring.

23. The method of claim 14 wherein said first lexicon further comprises information regarding contents of other lexicons in said system.

24. The method of claim 14 wherein data transmissions within said system are effectuated on a point-to-point basis.

25. The method of claim 14 wherein data transmissions within said system are effectuated on a point-to-multipoint basis.

26. The system of claim 14 further comprising:

routing data transmissions in said system based on metrics derived from said first lexicon.

27. A system for symbolic exchange of digital sequences without loss of data integrity comprising:

first and second computer systems comprising respective first and second local lexicons including a plurality of digital sequences with unique corresponding identifiers; and

a transmission medium coupling said first and second computer systems;

said first computer system operative to transmit a first unique identifier to said second computer system in lieu of said corresponding digital sequence;

wherein said second computer system is operative to compare said first unique identifier to contents of said second local lexicon and request said first computer system to transmit said corresponding digital sequence to said first unique identifier if said first unique identifier is not present in said second lexicon; and

wherein said second computer system is operative to add said corresponding digital sequence to said first unique identifier to said second local lexicon if not previously contained therein.

28. The computer system of claim 27 wherein said second computer system is operative to add said corresponding digital sequence to said first unique identifier to said second local lexicon if not previously contained therein.

29. The computer system of claim 27 further comprising:

a third computer system comprising a third local lexicon including a plurality of digital sequences with unique corresponding identifiers, said third computer system coupled to one of said first or second computer systems and operative to receive a first unique identifier therefrom in lieu of said corresponding digital sequence.

30. The computer system of claim 27 wherein said transmission medium comprises a network.

31. The computer system of claim 30 wherein said network comprises the Internet.

32. The computer system of claim 27 wherein said unique identifiers are based upon a hash of said corresponding digital sequence.

33. The computer system of claim 27 wherein said digital sequence comprises a file.

34. The computer system of claim 27 wherein said digital sequence comprises a video stream.

35. A method for symbolic exchange of digital sequences without loss of data integrity between first and second computer systems including respective first and second local tables in memory containing a plurality of digital sequences with unique corresponding identifiers, said method comprising:

transmitting a first unique identifier from said first computer system to said second computer system in lieu of said corresponding digital sequence;

comparing said first unique identifier to said contents of said second local table; and

requesting said first computer system to transmit said corresponding digital sequence to said first unique identifier if said first unique identifier is not present in said second local table;

wherein said unique identifiers are based upon a hash of said corresponding digital sequence.

36. The method of claim 35 further comprising:

transmitting said corresponding digital sequence to said first unique identifier if said first computer system is aware that said second local table does not contain said unique identifier.

37. The method of claim 35 further comprising the step of:

adding said corresponding digital sequence to said first unique identifier to said second local table in not previously present therein.

38. The method of claim 35 further comprising the step of:

providing a third computer system comprising a third local table including a plurality of digital sequences with unique corresponding identifiers;

coupling said third computer system to one of said first or second computer systems; and

receiving a first unique identifier from said one of said first or second computer systems in lieu of said corresponding digital sequence.

39. The method of claim 35 wherein said digital sequence comprises a file.

40. The method of claim 35 wherein said digital sequence comprises a video stream.

41. The method of claim 35 further comprising with said first unique identifier generating a new digital sequence at the second computer system that comprises said corresponding digital sequence retrieved from said second local table.

42. The method of claim 41 wherein the generating of the new digital sequence comprises retrieving the corresponding digital sequence from the second local table by indexing with the first unique identifier and wherein the new digital sequence comprises additional ones of the plurality of digital sequences retrieved from second local table based on a set of the unique corresponding identifiers received from the first computer system.

43. A system for symbolic exchange of digital sequences without loss of data integrity comprising:

first and second computer systems comprising respective first and second local lexicons including a plurality of digital sequences with unique corresponding identifiers; and

a transmission medium coupling said first and second computer systems, wherein said transmission medium comprises a network;

said first computer system operative to transmit a first unique identifier to said second computer system in lieu of said corresponding digital sequence;

wherein said second computer system is operative to compare said first unique identifier to contents of said second local lexicon and request said first computer system to transmit said corresponding digital sequence to said first unique identifier if said first unique identifier is not present in said second lexicon.

44. The system of claim 43, wherein said unique identifiers are based upon a hash of said corresponding digital sequence.

45. The computer system of claim 44, wherein said digital sequence comprises a file.

46. The computer system of claim 44, wherein said digital sequence comprises a video stream.

47. A system for symbolic exchange of digital sequences without loss of data integrity comprising:

first and second computer systems comprising respective first and second local lexicons including a plurality of digital sequences with unique corresponding identifiers, wherein said digital sequences each comprise a file; and

a transmission medium coupling said first and second computer systems;

said first computer system operative to transmit a first unique identifier to said second computer system in lieu of said corresponding digital sequence;

wherein said second computer system is operative to compare said first unique identifier to contents of said second local lexicon and request said first computer system to transmit said corresponding digital sequence to said first unique identifier if said first unique identifier is not present in said second lexicon.

48. A system for symbolic exchange of digital sequences without loss of data integrity comprising:

first and second computer systems comprising respective first and second local lexicons including a plurality of digital sequences with unique corresponding identifiers, wherein said digital sequences each comprise a video stream; and

a transmission medium coupling said first and second computer systems;

said first computer system operative to transmit a first unique identifier to said second computer system in lieu of said corresponding digital sequence;

wherein said second computer system is operative to compare said first unique identifier to contents of said second local lexicon and request said first computer system to transmit said corresponding digital sequence to said first unique identifier if said first unique identifier is not present in said second lexicon.

49. A method for symbolic exchange of digital sequences without loss of data integrity between first and second computer systems including respective first and second local tables in memory containing a plurality of digital sequences with unique corresponding identifiers, said method comprising:

transmitting a first unique identifier from said first computer system to said second computer system in lieu of said corresponding digital sequence;

comparing said first unique identifier to said contents of said second local table;

requesting said first computer system to transmit said corresponding digital sequence to said first unique identifier if said first unique identifier is not present in said second local table; and

transmitting said corresponding digital sequence to said first unique identifier if said first computer system is aware that said second local table does not contain said unique identifier.