Method and apparatus for high speed parallel execution of multiple points of logic across heterogeneous data sources

• US 6,697,835 B1
• Filed: 10/28/1999
• Issued: 02/24/2004
• Est. Priority Date: 10/28/1999
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Unisys Corporation
• Inventors: Redding, Roger Anthony, Johnson, Thomas Winston, Poon, Koon-yui, Hanson, Charles Albin, OʼHara, Carol Jean
• Primary Examiner(s): Barot, Bharat
• Application Number:
  US09/429,354
• Time in Patent Office: 1,580 Days
• Field of Search: 709/200-203, 709/217-219, 709/227, 709/246, 709/310, 709/328, 707/2-4, 707/8-10, 707/102-103, 707/104.1, 707/200-202
• US Class Current: 709/201
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

Heterogeneous data at a plurality of remote nodes is accessed automatically in parallel at high speed from a user site using a simple script request containing a data source object name wherein the heterogeneous data is treated as a single data source object, the script further containing code representing a user-defined program to be executed on the data source object. The user-defined program may be represented by an embedded script, a user-defined script or an executable program designation. A user site agent breaks the user-generated script into new scripts appropriate for execution at the remote nodes. A messenger process transmits the new scripts to the appropriate remote nodes where respective agent processes respond to automatically access the appropriate data and to automatically execute the specified program. If the program is a user-defined script or executable, the respective agent processes access a metadata repository to obtain the specified program.

Claims

1. A method of accessing and operating upon heterogeneous data at a plurality of nodes comprising the steps of:

(1) propounding a request containing a data source object name wherein the heterogeneous data is treated as a single data source object, said request further containing code identifying at least one user-defined program to be performed on the data source object;

(2) determining whether the data source object is distributed across a plurality of nodes;

(3) if the data source object is determined to be distributed, breaking said request into a plurality of new requests, each of said new requests having a format appropriate to one of the respective nodes; and

(4) transmitting each of said new requests in parallel to its respective corresponding node, each new request containing a request to execute said user-defined program.

2. The method of claim 1 further comprising the step of executing the user-defined program concurrently on the data source object at each of the corresponding nodes.

3. The method of claim 2 wherein the said user-defined program comprises one of the following types:

a user-defined script, an embedded script, and an executable program.

4. The method of claim 3 wherein each of said new requests includes an argument containing indicia representing the type of user-defined program and wherein a remote agent process at each corresponding node the type of user-defined program and directly executes the program if it is an embedded if the program type is a user-defined script or an executable program.

5. The method of claim 4 wherein an embedded script is enclosed within inverted commas in said argument to distinguish it from a user-defined script or executable program.

6. The method of claim 5 wherein said argument is preceded by a function designator reflecting that the argument contains a user-defined program and wherein said remote agent is programmed to recognize the absence of a data source object in said request as indicating that results of execution are not to be returned for processing by said remote agent process.

7. The method of claim 6 wherein, in performing the step of determining, said first agent process consults a data source descriptor file containing a subset of data contained in a first repository of metadata.

8. The method of claim 7 wherein the data source descriptor file is created from said repository at run-time.

9. The method of claim 7 wherein the metadata comprises a collection of data source objects which reflect treatment of data stored in each respective database as a single object and wherein each of said data source objects is broken down into successive class levels.

10. The method of claim 9 wherein said class levels include a class comprising a System Node, System Server, Data Source Object, Field Desc and System Script.

11. The method of claim 2 wherein said request is in the form of a script and each said new request is in the form of a script having said format.

12. The method of claim 11 wherein said script is in the form of a Java script.

13. The method of claim 11 wherein each of said nodes has associated therewith a respective database, and a respective agent process, each respective agent process comprising code selected to execute the respective new script with respect to the data source object as it is contained in the respective database.

14. The method of claim 13 wherein each of said databases is different from the remaining respective databases.

15. The method of claim 14 wherein the respective databases comprise at least two databases, each selected from the following group:

Oracle database, NT database and SQL Server.

16. The method of claim 1 further comprising the step of determining whether said data source object is stored locally or is nondistributed but stored remotely.

17. The method of claim 1 wherein a first agent process performs the step of determining whether said data source object is distributed.

18. The method of claim 17 wherein a first messenger process cooperates with said first agent process to transmit each said new request to its respective node.

19. In apparatus for performing a plurality of user-defined programs, said apparatus comprising:

a plurality of databases, each located at a different one of a plurality of nodes, and data processing apparatus at each node, each data processing apparatus including an agent code module, each agent code module being constructed to determine whether a data source object presented in a script is distributed across a plurality of said nodes, and if said object is distributed, to break the script into a plurality of new scripts suitable for execution at a respective one said nodes, and if the script contains a code representing a user-defined program, to execute said program.

20. The apparatus of claim 19 wherein if the script contains a plurality of successive methods, said agent is further constructed to determine whether a second of said methods should be performed at the respective node on the results of execution of a first of said methods at said respective node.

21. The apparatus of claim 20 further including a messenger code module at each node for facilitating transmission of each of said new scripts to its respective node.

22. The method of claim 19 wherein the user-defined program comprises one of the following types:

a user-defined script, an embedded script, and an executable program.

23. The method of claim 22 wherein each new request includes an argument containing indicia representing the type of user-defined program and wherein said agent code module is further constructed to determine the type of user-defined program and to directly execute the program if it is an embedded script and to access a repository of metadata to retrieve a program to be executed if the program is a user-defined script or an executable program.

24. The method of claim 23 wherein an embedded script is enclosed within inverted commas in said argument to distinguish it from a user-defined script or executable program.

25. The method of claim 23 wherein said argument is preceded by a function designator reflecting that the argument contains a user-defined program and wherein said agent code module is constructed to recognize the absence of a data source object in a request as indicating that results of execution are not to be returned to a transmitting agent process.

26. The apparatus of claim 19 further including a repository of metadata at each said node, said metadata describing the contents of the data source objects available at that node.

27. The apparatus of claim 26 further including a repository application at each node for creating a data source descriptor file for use by the agent in extracting data from the database at that node.

28. An article of manufacture comprising:

a computer usable medium having computer readable program code means embodied in said medium for accessing and executing a plurality of methods on data at each of a plurality of nodes, said data being treated as a single data source object, the computer readable code means comprising;

means for receiving a request containing a data source object name wherein heterogeneous data is treated as a single data source object, said request further containing a code representing a user-defined program to be executed on the data source object; and

means for determining whether the data source object is distributed across a plurality of nodes; and

if the data source object is determined to be distributed, breaking said request into a plurality of new requests, each of said new requests including code representing said user-defined program.

29. The article of claim 28 wherein said computer readable code means further contains means for automatically executing a first method, automatically storing the results of the execution, and then automatically executing a second method upon the stored results and returning the results of execution of said second method to a node which transmitted said request.

30. The article of claim 29 wherein said computer readable code means further includes means for automatically executing a third method included in a said request on the returned results.

31. Computer executable process steps operative to control a computer and stored on a computer readable medium, comprising:

a step to receive a request containing a data source object name wherein heterogeneous data stored at a plurality of nodes is treated as a single data source object, said request further containing code representing one of a plurality of user-defined programs to be performed on the data source object;

a step to determine whether the data source object is distributed across a plurality of remote nodes; and

a step wherein, if the data source object is determined to be distributed, a request to execute one of said plurality of user-defined programs is sent to each of the remote nodes across which the data source agent is distributed.

32. The process steps of claim 31 further including:

a step wherein, if it is determined that the data source object is distributed and that first and second methods should be performed at the remote nodes, a request to perform said first and second methods is broken into a plurality of new requests, each containing code representing said first and second methods.

33. The process steps of claim 32 further including a step to return results of execution of said second method back to a location where said request to perform originated.

34. The process steps of claim 33 including a step to merge results received at said location and a step to execute a third method on those results.

35. The process steps of claim 34 wherein said first, second and third methods respectively are search, sort and e-mail.