Method and system for providing transfer of analytic application data over a network

• US 7,162,643 B1
• Filed: 06/15/2001
• Issued: 01/09/2007
• Est. Priority Date: 06/15/2001
• Status: Expired due to Term

• Current Assignee: Informatica LLC (Informatica, Inc. (California))
• Original Assignee: Informatica Corporation
• Inventors: Tang, Zhenyu, Butsky, Volodymyr, Koritala, Sridhar C., Sankaran, Mohan
• Primary Examiner(s): Reagan; James A
• Application Number:
  US09/882,733
• Time in Patent Office: 2,034 Days
• Field of Search: 705/50, 705/51, 713/189, 713/190, 375/752, 375/343, 365/189, 365/185, 369/275, 370/342, 370/503, 707/104
• US Class Current: 713/189
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A method and system providing a high speed and secure data link for moving large amounts of data across a network, such as the data used in an analytic application. Featured are simultaneous compression and encryption of the data, as well as means for recovery in the event the network connection is lost.

Claims

1. In a multithreaded analytic application executed by a source computer system and capable of concurrent execution of multiple session threads, a method for transferring data, the method comprising:

receiving an incoming request for analytic data resident in a mass storage unit on the source computer system;

authenticating the incoming request;

spawning a session thread that reads and parses a command received via the incoming request, the command for sending the data to a second computer system; and

concurrently executing a plurality of data transformation threads within the session thread, comprising

a reader thread that reads data and writes at least a part of the data to a first data block buffer;

a compressor thread that compresses the part of the data in the first data block buffer into a compressed data block and writes the compressed data block to a second data block buffer;

an encryptor thread that encrypts the compressed data block in the second data block buffer into an encrypted and compressed data block and writes the encrypted and compressed data block to a third data block buffer; and

a writer thread that reads the encrypted and compressed data block in the third data block buffer and sends the encrypted and compressed data block to the second computer;

restoring a connection with the second computer system when an ongoing connection is lost; and

resuming transfer of data to the second computer system at the point in the data where the ongoing connection was lost.

2. The method of claim 1 further comprising:

verifying that data transfer to the second computer system is complete.

3. The method of claim 1 further comprising:

verifying that data transfer to the second computer system is without error.

4. The method of claim 1 wherein the source computer system and the second computer system are networked via the Internet.

5. The method of claim 1 wherein the data comprises data processed by an analytic application.

6. The method of claim 1 wherein the incoming request uses Extensible Markup Language (XML).

7. The method of claim 1 wherein spawning a session thread further comprises:

translating the command into a plurality of tasks;

storing the tasks in a task table in a given order; and

executing the tasks in order until a task ending the session thread is found.

8. The method of claim 1 wherein the first data block buffer and the second data block buffer are substantially equal in size and wherein enough compressed data blocks are accumulated to fill the second data block buffer before the compressor thread writes to a second data block buffer.

9. The method of claim 1 wherein the second data block buffer and the third data block buffer are substantially equal in size and wherein enough encrypted and compressed data blocks are accumulated to fill the third data block buffer before the encryptor thread writes to the third data block buffer.

10. In a first multithreaded analytic application executed by a target computer system and capable of concurrent execution of multiple session threads, a method for receiving data transferred from a source computer, the method comprising:

issuing a request for data to the source computer system on which the data resides, the source computer system executing a second multithreaded analytic application;

spawning a session thread in response to a message from the source computer system;

receiving from the source computer system at least one encrypted and compressed data block of the data; and

concurrently executing a plurality of data transformation threads within the session thread, comprising

a reader thread for writing the encrypted and compressed data block to a first data block buffer;

a decryptor thread for decrypting the encrypted and compressed data block into a compressed data block and writing the compressed data block to a second data block buffer; and

a decompressor thread for decompressing the compressed data block in the second data block buffer and writing a resultant data block to a third data block buffer;

restoring a connection with the source computer system when an ongoing connection is lost; and

resuming transfer of data from the source computer system at the point in the data where the ongoing connection was lost.

11. The method of claim 10 further comprising:

verifying that data transfer from the source computer system was complete.

12. The method of claim 10 further comprising:

verifying that data transfer from the source computer system was without error.

13. The method of claim 10 wherein the target computer system and the source computer system are networked via the Internet.

14. The method of claim 10 wherein the data comprises data processed by an analytic application.

15. The method of claim 10 wherein a plurality of encrypted and compressed data blocks accumulate before the decryptor thread executes.

16. The method of claim 10 wherein a plurality of compressed data blocks accumulate before the decompressor thread executes.

17. A source computer system comprising:

a bus;

a memory unit coupled to the bus;

a multithreaded analytic application stored in the memory unit, and comprising;

a listener object for receiving an incoming request for data resident in a mass storage unit on the source computer system;

protocol for authenticating the incoming request;

a session manager object for spawning a session thread that reads and parses a command received via the incoming request, the command for sending the data to a second computer system;

a reader channel object for reading data and writing at least a part of the data to a first data block buffer;

a compressor channel object for compressing the part of the data in the first data block buffer into a compressed data block and writing the compressed data block to a second data block buffer;

an encryptor channel object for encrypting the compressed data block in the second data block buffer into an encrypted and compressed data block and writing the encrypted and compressed data block to a third data block buffer; and

a writer channel object for reading the encrypted and compressed data block in the third data block buffer and sending the encrypted and compressed data block to the second computer, wherein the application executes the reader channel object, the compressor channel object, the encryptor channel object, and the writer channel object concurrently; and

a processor coupled to the bus, the processor configured for executing the multithreaded analytic application, wherein the processor is further configured for;

restoring a connection with the second computer system when an ongoing connection is lost; and

resuming transfer of data to the second computer system at the point in the data where the ongoing connection was lost.

18. The source computer system of claim 17 wherein the processor is further configured for verifying that data transfer to the second computer system is complete.

19. The source computer system of claim 17 wherein the processor is further configured for verifying that data transfer to the second computer system is without error.

20. The source computer system of claim 17 wherein the source computer system and the second computer system are networked via the Internet.

21. The source computer system of claim 17 wherein the data comprises data processed by an analytic application.

22. The source computer system of claim 17 wherein the incoming request uses Extensible Markup Language (XML).

23. The source computer system of claim 17 wherein the session manager object is further configured for:

translating the command into a plurality of tasks;

storing the tasks in a task table in a given order; and

executing the tasks in order until a task ending the session thread is found.

24. The source computer system of claim 17 wherein the first data block buffer and the second data block buffer are substantially equal in size and wherein the compressor channel object is further configured for:

accumulating compressed data blocks before data are written to the second data block buffer, wherein enough compressed data blocks are accumulated to fill the second data block buffer.

25. The source computer system of claim 17 wherein the second data block buffer and the third data block buffer are substantially equal in size and wherein the encryptor channel object is further configured for:

accumulating encrypted and compressed data blocks before data are written to the third data block buffer, wherein enough encrypted and compressed data blocks are accumulated to fill the third data block buffer.

26. A target computer system comprising:

a bus;

a memory unit coupled to the bus;

a multithreaded analytic application stored in the memory unit, and comprising;

a first session thread for issuing a request for data to a source computer system on which the data resides;

a session manager object for spawning a session thread in response to a message from the source computer system;

a listener object for receiving from the source computer system at least one encrypted and compressed data;

a reader channel object for reading data and writing at least part of the encrypted and compressed data to a first data block buffer;

a decryptor channel object for decrypting the encrypted and compressed data block into a compressed data block and writing the compressed data block to a second data block buffer; and

a decompressor channel object for decompressing the compressed data block in said the second data block buffer and writing a resultant data block to a third data block buffer, wherein the application executes the reader channel object, the decryptor channel object, and the decompressor channel object concurrently; and

a processor coupled to the bus, the processor configured for executing the multithreaded analytic application, wherein the processor is further configured for;

restoring a connection with the source computer system when an ongoing connection is lost; and

resuming transfer of data from the source computer system at the point in the data where the ongoing connection was lost.

27. The target computer system of claim 26 wherein the processor is further configured for verifying that data transfer from the source computer system was complete.

28. The target computer system of claim 26 wherein the processor is further configured for verifying that data transfer from the source computer system was without error.

29. The target computer system of claim 26 wherein the target computer system and the source computer system are networked via the Internet.

30. The target computer system of claim 26 wherein the data comprises data processed by an analytic application.

31. The target computer system of claim 26 wherein the decryptor channel object is further configured for accumulating encrypted and compressed data blocks before decrypting the encrypted and compressed data blocks.

32. The target computer system of claim 26 wherein the decompressor channel object is further configured for accumulating compressed data blocks before decompressing the compressed data blocks.