Systems and methods for securely transmitting large data files

• US 10,546,139 B2
• Filed: 07/26/2018
• Issued: 01/28/2020
• Est. Priority Date: 04/21/2017
• Status: Active Grant

• Current Assignee: Ntropy, LLC
• Original Assignee: Ntropy, LLC
• Inventors: Ellingson, John, Hansen, Robert M.
• Primary Examiner(s): Ullah, Sharif E
• Application Number:
  US16/046,564
• Publication Number:
  US 20180357431A1
• Time in Patent Office: 551 Days
• Field of Search: 726 7, 726 2, 726 21, 726 36, 713150, 713163, 713181, 380255, 380264, 380279
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 21/606 by securing the transmission between two devices or processes
• CPC Class Codes: H04L 63/0272 Virtual private networks
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/102 Entity profiles
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 12/033 of the user plane, e.g. user's traffic

Abstract

In methods, systems, and computing devices configured to implement methods of conveying a data file from a first computing device to a second computing device. A referential complex dataset (RCD) is stored in memory of a first computing device and a second computing device. The first computing device may compare bit strings within a data file to bit strings within the RCD to identify a matching bit string in the RCD, generate a set of rules for locating the matched bit string in the RCD, and transmit the rule set to the second computing device. The second computing device may receive the rule set, sequentially use each rule in the rule set to identify corresponding bit strings in the RCD in memory, and copy the identified bit strings into a memory to replicate the data file.

Claims

1. A method of conveying a data file from a first computing device to a second computing device, comprising:

analyzing one or more data files representative of the data file to be transmitted to identify long bit strings that appear at least a threshold number of times;

storing in a referential complex dataset (RCD) each long bit string that appears at least a threshold number of times;

generating an index for locating each long bit string stored in the RCD;

generating metadata corresponding to each long bit string stored in the RCD that provides information useful for comparing data files to the RCD to identify matching long bit strings;

storing the generated metadata in association with the corresponding long bit string;

transmitting the RCD to the second computing device for storage;

storing the RCD in memory of the first computing device and in memory of the second computing device;

performing by a processor of the first computing device the following operations sequentially on the data file until the entire data file has been processed;

comparing bit strings within the data file to bit strings within the RCD to identify a matching bit string in the RCD, by using metadata associated with a matched bit string in the RCD to determine whether a longer bit string including the same matched bits exists in the RCD;

generating a rule for locating the matched bit string in the RCD comprising

storing the index for the matched bit string in the rule set; and

storing the generated rule sequentially in a rule set; and

transmitting the rule set to the second computing device.

2. The method of claim 1, wherein storing the RCD in memory of the first computing device and in memory of the second computing device comprises:

generating, by the processor of the first computing device, a rule for changing the RCD stored in the memories of both the first computing device and the second computing device;

applying, by the processor of the first computing device, the generated rule to the RCD stored in memory of the first computing device to generate a changed RCD; and

transmitting the rule to the second computing device in a format that will enable a processor of the second computing device to apply the rule to the RCD stored in the memory of the second computing device to generate the changed RCD in the second computing device.

3. The method of claim 1, further comprising:

storing a subscriber RCD in memory of the first computing device and in memory of the second computing device;

performing by the processor of the first computing device the following operations sequentially on the rule set until the entire rule set has been processed;

comparing a bit string of each rule in the rule set to bit strings within the subscriber RCD to identify a matching bit string in the subscriber RCD;

generating a secondary rule for locating the matched bit string in the subscriber RCD; and

storing the generated secondary rule in a secondary rule set,

wherein transmitting the rule set to the second computing device comprises transmitting the secondary rule set to the second computing device.

4. The method of claim 3, further comprising:

receiving the secondary rule set in the second computing device;

sequentially using each secondary rule in the secondary rule set to identify corresponding bit strings in the subscriber RCD in memory;

using the identified corresponding bit strings in the subscriber RCD as rule to identify corresponding bit strings in the RCD in memory; and

copying the identified bit strings into a memory to replicate the data file.

5. The method of claim 1, wherein the data file to be transmitted is a multimedia file and analyzing one or more data files representative of the data file comprises analyzing one or more multimedia files having a similarity to the multimedia file to be transmitted.

6. The method of claim 1, further comprising:

receiving the rule set in the second computing device;

sequentially using each rule in the rule set to identify corresponding bit strings in the RCD in memory; and

copying the identified bit strings into a memory to replicate the data file.

7. The method of claim 6, further comprising:

processing the replicated data file using an application compatible with the data file processed by the first computing device.

8. A computing device, comprising:

a memory; and

a processor coupled to the memory and configured with processor-executable instructions to perform operations comprising;

analyzing one or more data files representative of the data file to be transmitted to identify long bit strings that appear at least a threshold number of times;

storing in a referential complex dataset (RCD) each long bit string that appear at least a threshold number of times;

generating an index for locating each long bit string stored in the RCD;

generating metadata corresponding to each long bit string stored in the RCD that provides information useful for comparing data files to the RCD to identify matching long bit strings;

storing the generated metadata in association with the corresponding long bit string;

transmitting the RCD to the second computing device for storage;

storing in the memory the RCD,

wherein the RCD is the same as an RCD stored in a memory of a second computing device;

performing the following operations sequentially on the data file until the entire data file has been processed;

comparing bit strings within the data file to bit strings within the RCD to identify a matching bit string in the RCD, by using metadata associated with a matched bit string in the RCD to determine whether a longer bit string including the same matched bits exists in the RCD;

generating a rule for locating the matched bit string in the RCD comprising storing the index for the matched bit string in the rule set; and

storing the generated rule sequentially in a rule set; and

transmitting the rule set to the second computing device.

9. The computing device of claim 8, wherein the processor is configured with processor-executable instructions to perform operations such that storing in the memory the RCD that is the same as the RCD stored in the memory of the second computing device comprises:

generating a rule for changing the RCD stored in memories of both the first computing device and the second computing device;

applying the generated rule to the RCD stored in the memory to generate a changed RCD; and

transmitting the rule to the second computing device in a format that will enable a processor of the second computing device to apply the rule to the RCD to generate the changed RCD in the second computing device.

10. The computing device of claim 8, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

storing a subscriber RCD in the memory of the first computing device;

providing the subscriber RCD to the second computing device for storage in the memory of the second computing device;

performing by the processor of the first computing device the following operations sequentially on the rule set until the entire rule set has been processed;

comparing a bit string of each rule in the rule set to bit strings within the subscriber RCD to identify a matching bit string in the subscriber RCD;

generating a secondary rule for locating the matched bit string in the subscriber RCD; and

storing the generated secondary rule in a secondary rule set,

wherein transmitting the rule set to the second computing device comprises transmitting the secondary rule set to the second computing device.

11. The computing device of claim 8, wherein the data file to be transmitted is a multimedia file and analyzing one or more data files representative of the data file comprises analyzing one or more multimedia files having a similarity to the multimedia file to be transmitted.

12. A non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor of a computing device to perform operations comprising:

analyzing one or more data files representative of the data file to be transmitted to identify long bit strings that appear at least a threshold number of times;

storing in a referential complex dataset (RCD) each long bit string that appears at least a threshold number of times;

generating an index for locating each long bit string stored in the RCD;

generating metadata corresponding to each long bit string stored in the RCD that provides information useful for comparing data files to the RCD to identify matching long bit strings;

storing the generated metadata in association with the corresponding long bit string;

transmitting the RCD to the second computing device for storage;

storing in a memory the RCD, wherein the RCD is the same as an RCD stored in a memory of a second computing device;

performing the following operations sequentially on the data file until the entire data file has been processed;

comparing bit strings within the data file to bit strings within the RCD to identify a matching bit string in the RCD, by using metadata associated with a matched bit string in the RCD to determine whether a longer bit string including the same matched bits exists in the RCD;

generating a rule for locating the matched bit string in the RCD; and

storing the generated rule sequentially in a rule set; and

transmitting the rule set to the second computing device.

13. The non-transitory processor-readable storage medium of claim 12, wherein the stored processor-executable instructions are configured to cause the processor of the computing device to perform operations such that storing in the memory the RCD that is the same as the RCD stored in the memory of the second computing device comprises:

generating a rule for changing the RCD stored in memories of both the first computing device and the second computing device;

applying the generated rule to the RCD stored in the memory to generate a changed RCD; and

transmitting the rule to the second computing device in a format that will enable a processor of the second computing device to apply the rule to the RCD to generate the changed RCD in the second computing device.

14. The non-transitory processor-readable storage medium of claim 12, wherein the stored processor-executable instructions are configured to cause the processor of the computing device to perform operations further comprising:

storing a subscriber RCD in the memory of the first computing device;

providing the subscriber RCD to the second computing device for storage in the memory of the second computing device;

performing by the processor of the first computing device the following operations sequentially on the rule set until the entire rule set has been processed;

comparing a bit string of each rule in the rule set to bit strings within the subscriber RCD to identify a matching bit string in the subscriber RCD;

generating a secondary rule for locating the matched bit string in the subscriber RCD; and

storing the generated secondary rule in a secondary rule set,

wherein transmitting the rule set to the second computing device comprises transmitting the secondary rule set to the second computing device.

15. The non-transitory processor-readable storage medium of claim 12, wherein the data file to be transmitted is a multimedia file and analyzing one or more data files representative of the data file comprises analyzing one or more multimedia files having a similarity to the multimedia file to be transmitted.