Component model for batch computing in a distributed object environment

• US 8,024,733 B2
• Filed: 05/13/2004
• Issued: 09/20/2011
• Est. Priority Date: 05/13/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Little, Rodney Alan, High, Robert Howard Jr., Hambrick, Geoffrey Martin
• Primary Examiner(s): An; Meng A
• Assistant Examiner(s): WAI, ERIC CHARLES
• Application Number:
  US10/845,544
• Publication Number:
  US 20050256826A1
• Time in Patent Office: 2,686 Days
• Field of Search: 718/101, 714/15
• US Class Current: 718/101
• CPC Class Codes: G06F 2209/5013 Request control
• CPC Class Codes: G06F 2209/5014 Reservation
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 2209/503 Resource availability
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups

Abstract

A batch component model is provided within a distributed object environment. The batch component is designed to capture the iterative logic of a batch program as it reads from one or more input streams, invokes operations on other business component functions, and generates output to one or more output streams. Deployment descriptors express declarative policies for the component that will influence how the component is managed including the streams it uses, business components it depends on, how processing costs are accounted, and the resource demands the job will put on the system. Input streams and output streams are encapsulated in objects that hide the actual source of input and output data so that the component can be redeployed in different execution environments to different physical data sources without requiring the program to be changed. A batch container enforces the deployment policies declared for the batch component.

Claims

1. A computer-implemented method of batch processing in a batch component model within a distributed object environment, the computer-implemented method comprising:

instantiating a batch component by a processor of a data processing system, for use with a batch job within the distributed object environment;

initializing the batch component with a set of deployment descriptors and an instance of a batch container to form a contractual relationship between the batch component and the batch container, wherein the set of deployment descriptors is a set of declarative policies for the batch component;

wrapping the contractual relationship between the batch component and the batch container to form an adapter, wherein the adapter isolates the batch component from different implementations of the batch container;

dynamically computing by the batch container, for each use of a checkpoint interval, a size of the checkpoint interval for the batch job based on the set of deployment descriptors and other processing workloads;

managing operation of the batch component in the batch component model by the batch container in accordance with the set of deployment descriptors and the other processing workloads; and

committing, by the batch container on the processor, at an end of the checkpoint interval, checkpoint cursors and data of the batch job that are updated during the batch processing to a storage of the data processing system, wherein context information, including the size of the checkpoint interval and resource dependencies, is persisted and passed to downstream batch containers.

2. The computer-implemented method of claim 1, wherein the set of deployment descriptors identify at least one of one or more input stream components and one or more output stream components.

3. The computer-implemented method of claim 2, wherein the one or more input stream components are abstractions of actual underlying data technology from which the one or more input stream components receive data.

4. The computer-implemented method of claim 1, further comprising computing the size of the checkpoint interval by the batch container based on a number of records in the checkpoint interval.

5. The computer-implemented method of claim 2, wherein the one or more output stream components are abstractions of actual underlying data technology to which the one or more output stream components send data.

6. The computer-implemented method of claim 1, further comprising:

determining, by the batch container, whether a resource capacity is available to process the batch job; and

responsive to the resource capacity being unavailable, suspending the batch job by the batch container until the resource capacity becomes available.

7. The computer-implemented method of claim 1, wherein the set of deployment descriptors includes descriptors for identifying business function components, symbolic references resolved at deployment time, for conditioning a checkpoint interval and enumerating resource dependencies identify resources that are needed for the batch job.

8. The computer-implemented method of claim 7, wherein managing operation of the batch component includes allocating, by the batch container, at least a portion of the resources for at least a portion of the batch job.

9. The computer-implemented method of claim 8, wherein managing operation of the batch component further includes locking, by the batch container, at least a portion of the resources required by the batch job.

10. The computer-implemented method of claim 1, wherein managing operation of the batch component includes checkpointing periodically, by the batch container, a progress of the batch job;

and persisting, by storing in a checkpoint cursor storage, any cursors that mark the progress of the batch job, wherein an administrator is capable of querying the checkpoint cursor storage.

11. The computer-implemented method of claim 1, wherein the batch component shares business logic and data components within the distributed object environment with other batch jobs or online transaction processing components.

12. A computer-implemented method of batch processing with a batch container of a batch component model in a distributed object environment, the computer-implemented method comprising:

forming a contractual relationship between a batch component and the batch container;

wrapping the contractual relationship between the batch component and the batch container to form an adapter, wherein the adapter isolates the batch component from different implementations of the batch container;

computing, by the batch container on a processor of a data processing system, for each use of a checkpoint interval, a size of the checkpoint interval for a batch component of a batch job based on a set of deployment descriptors and other processing workloads, wherein the set of deployment descriptors is a set of declarative policies for the batch component;

running the batch component within the checkpoint interval, by the processor; and

upon completion of the checkpoint interval, committing by the batch container, data updated during the checkpoint interval and at least one checkpoint cursor to a storage of the data processing system, wherein context information, including the size of the checkpoint interval and resource dependencies, is persisted and passed to downstream batch containers.

13. The computer-implemented method of claim 12, further comprising:

increasing, by the batch container, the size of the checkpoint interval when contention for resources to run the batch job is low.

14. The computer-implemented method of claim 13, further comprising:

stopping the batch component by the batch container during a given checkpoint interval; and

responsive to the batch component stopping during a given checkpoint interval, determining by the batch container a starting point based on the at least one checkpoint cursor and restarting the batch component at the starting point.

15. The computer-implemented method of claim 12, further comprising:

responsive to a failure of the batch container to commit data, rolling back the batch job to a prior checkpoint by the batch container.

16. The computer-implemented method of claim 12, further comprising:

before running the batch component within the checkpoint interval, allocating resources to be used by the batch component during the checkpoint interval and locking the resources by the batch container.

17. The computer-implemented method of claim 16, further comprising:

upon completion of the checkpoint interval duration, unlocking the resources by the batch container.

18. The computer-implemented method of claim 12, further comprising:

restarting the batch job, by the batch container, responsive to a failure to commit.

19. The computer-implemented method of claim 12, further comprising:

upon completion of the checkpoint interval duration, determining by the batch container the size of the checkpoint interval for a next checkpoint interval.

20. The computer-implemented method of claim 12, wherein a duration of the checkpoint interval is determined by the batch container based on a number of records in an input data stream.

21. The computer-implemented method of claim 12, wherein the checkpoint interval is determined by the batch container based on actual workload conditions of the distributed object environment.

22. A computer-implemented method of batch processing a batch job in a batch component model using a batch container, the computer-implemented method comprising:

dividing, by the batch container on a processor of a data processing system, an input stream into a plurality of batch job partitions of the input stream, to form a contractual relationship between each batch component and the batch container;

wrapping the contractual relationship between each batch component and the batch container to form an adapter, wherein the adapter isolates each batch component from different implementations of the batch container;

computing, by the batch container, on the processor, for each use of a checkpoint interval, a size of the checkpoint interval for a batch component of a batch job using a set of deployment descriptors and other processing workloads, wherein the set of deployment descriptors is a set of declarative policies for the batch component, and wherein context information, including the size of the checkpoint interval and resource dependencies, is persisted and passed to downstream batch containers;

running, through a distribution of a scheduler, a plurality of instances of a batch component, by the batch container, within the checkpoint interval, wherein each instance of the batch component within the plurality of instances of the batch component operates on a respective batch job partition of the input stream to produce a separate output stream; and

assembling the separate output streams produced by each respective batch job partition.

23. The computer-implemented method of claim 22, wherein the plurality of instances of the batch component operate in parallel.

24. The computer-implemented method of claim 22, wherein a second instance of the batch component begins operation at completion of operation of a first instance of the batch component.

25. The computer-implemented method of claim 22, wherein initiation of the plurality of instances of the batch component is staggered by the batch container.

26. An apparatus for batch processing in a batch component model within a distributed object environment, the apparatus comprising:

a system bus;

a local memory in communication with the system bus, wherein the local memory contains computer executable instructions;

a processor connected to the system bus, wherein the processor executes the computer executable instructions to;

instantiate a batch component on a processor of a data processing system, for use with a batch job within the distributed object;

initialize the batch component with a set of deployment descriptors and an instance of a batch container environment to form a contractual relationship between the batch component and the batch container, wherein the set of deployment descriptors is a set of declarative policies for the batch component;

wrap the contractual relationship between the batch component and the batch container to form an adapter, wherein the adapter isolates the batch component from different implementations of the batch container;

dynamically compute by the batch container a size of a checkpoint interval for the batch job using the set of deployment descriptors and other processing workloads, for each use of the checkpoint interval;

manage operation of the batch component in the batch component model within the distributed object environment in accordance with the set of deployment descriptors and the other processing workloads by the batch container; and

commit by the batch container checkpoint cursors and data of the batch job that are updated during the batch processing to a storage of the a data processing system on completion of the checkpoint interval, wherein context information, including the size of the checkpoint interval and resource dependencies is persisted and passed to downstream batch containers.

27. The apparatus of claim 26, wherein the set of deployment descriptors identify one or more input stream components.

28. The apparatus of claim 27, wherein the one or more input stream components are abstractions of actual underlying data technology from which the one or more input stream components receive data.

29. The apparatus of claim 26, wherein the set of deployment descriptors identify one or more output stream components.

30. The apparatus of claim 27, wherein the one or more output stream components are abstractions of actual underlying data technology to which the one or more output stream components send data.

31. The apparatus of claim 26, wherein the set of deployment descriptors identify at least one business function component that the batch component uses.

32. The apparatus of claim 26, wherein the set of deployment descriptors identify resources that are needed for the batch job and include descriptors for identifying business function components, symbolic references resolved at deployment time, for conditioning a checkpoint interval and enumerating resource dependencies.

33. The apparatus of claim 32, wherein the processor further executes the computer executable instructions to allocate by the batch container at least a portion of the resources for at least a portion of the batch job.

34. The apparatus of claim 33, wherein the processor further executes the computer executable instructions to lock by the batch container at least a portion of the resources required by the batch job.

35. The apparatus of claim 26, wherein the processor further executes the computer executable instructions to checkpoint progress of the batch job periodically by the batch container.

36. The apparatus of claim 26, wherein the batch component shares business logic and data components within the distributed object environment with other batch jobs or online transaction processing components.

37. The apparatus of claim 26, wherein the processor further executes the computer executable instructions to:

determine a duration of the checkpoint interval for a batch component by the batch container;

execute the batch component within the checkpoint interval by the batch container; and

store at least one checkpoint cursor upon completion of the checkpoint interval duration by the batch container.

38. The apparatus of claim 37, wherein the processor further executes the computer executable instructions to:

commit the at least one checkpoint cursor by the batch container upon completion of the duration of the checkpoint interval.

39. The apparatus of claim 38, wherein the processor further executes the computer executable instructions to:

determine a starting point based on the at least one checkpoint cursor and restart the batch component at the starting point by the batch container, responsive to the batch component stopping during a given checkpoint interval.

40. The apparatus of claim 37, wherein the processor further executes the computer executable instructions to:

commit, by the batch container, transactional updates that occurred during the checkpoint interval upon completion of the duration of the checkpoint interval.

41. The apparatus of claim 37, wherein the processor further executes the computer executable instructions to:

allocate resources by the batch container to be used by the batch component during the checkpoint interval; and

lock the resources by the batch container before running the batch component within the checkpoint interval.

42. The apparatus of claim 41, wherein the processor further executes the computer executable instructions to:

unlock the resources by the batch container upon completion of the duration of the checkpoint interval.

43. The apparatus of claim 37, wherein the processor further executes the computer executable instructions to:

receive a set of deployment descriptors for the batch component by the batch container; and

determine the duration of the checkpoint interval by the batch container using the deployment descriptors and the other processing workloads.

44. The apparatus of claim 37, wherein the processor further executes the computer executable instructions to:

determine the size of the checkpoint interval by the batch container for a next checkpoint interval upon completion of the duration of the checkpoint interval.

45. The apparatus of claim 37, wherein the duration of the checkpoint interval is one of a period of time and a number of records in an input data stream.

46. The apparatus of claim 37, wherein the checkpoint interval is determined by the batch container based on actual processing workloads of the distributed object environment.

47. The apparatus of claim 26, wherein the processor executes the computer executable instructions to:

partition a batch job by the batch container;

divide an input stream into a plurality of partitions of the input stream by the batch container; and

execute a plurality of instances of the batch component, wherein each instance of the batch component within the plurality of instances of the batch component operates on a respective partition of the input stream.

48. The apparatus of claim 47, wherein the plurality of instances of the batch component operate in parallel.

49. The apparatus of claim 47, wherein a second instance of the batch component begins operation at completion of operation of a first instance of the batch component.

50. The apparatus of claim 47, wherein initiation of the plurality of instances of the batch component by the batch container is staggered.

51. The apparatus of claim 26, wherein the processor executes the computer executable instructions to receive a query in the batch container from an administrator and return information by the batch container about progress of the batch job to the administrator.

52. The apparatus of claim 26, further comprising:

business function components; and

business object components.

53. The apparatus of claim 52, wherein the business function components are stateless session beans and wherein the business object components are entity beans.

54. The apparatus of claim 52, further comprising:

at least one online processing client that performs operations using the business function components and the business object components in real time.

55. The apparatus of claim 54, wherein the batch component shares the business function components and the business object components with the at least one online processing client.

56. The apparatus of claim 54, wherein the at least one online processing client includes at least one of a terminal services client, a Web services client, a Web application client, and a workflow.

57. A computer program product, for batch processing in a batch component model within a distributed object environment, the computer program product comprising:

a computer readable recordable type media having computer executable instructions stored thereon, the computer executable instructions comprising;

computer executable instructions for instantiating a batch component for use with a batch job within the distributed object environment;

computer executable instructions for initializing the batch component with a set of deployment descriptors and an instance of a batch container to form a contractual relationship between the batch component and the batch container, wherein the set of deployment descriptors is a set of declarative policies for the batch component;

computer executable instructions for wrapping the contractual relationship between the batch component and the batch container to form an adapter, wherein the adapter isolates the batch component from different implementations of the batch container;

computer executable instructions for dynamically computing, by the batch container, a size of a checkpoint interval for the batch job using the set of deployment descriptors and other processing workloads, for each use of the checkpoint interval;

computer executable instructions for managing operation of the batch component within the distributed object environment by the batch container in accordance with the set of deployment descriptors and the other processing workloads; and

computer executable instructions for committing by the batch container on completion of the checkpoint interval, checkpoint cursors and data of the batch job that are updated during the batch processing to a storage of a data processing system, wherein context information including the size of the checkpoint interval and resource dependencies is persisted and passed to downstream batch containers.

58. The computer program product, of claim 57, wherein computer executable instructions for managing operation of the batch component further comprises:

computer executable instructions for checkpointing a batch job in a distributed object environment by the batch container;

computer executable instructions for determining a duration of a checkpoint interval for the batch component by the batch container;

computer executable instructions for running the batch component within the checkpoint interval by the batch container; and

computer executable instructions for storing at least one checkpoint cursor upon completion of the checkpoint interval duration by the batch container.

59. The computer program product of claim 57, wherein computer executable instructions for instantiating a batch component further comprises:

computer executable instructions for partitioning the batch job by the batch container;

computer executable instructions for dividing an input stream into a plurality of partitions of the input stream by the batch container; and

computer executable instructions for running a plurality of instances of the batch component by the batch container, wherein each instance of the batch component within the plurality of instances of the batch component operates on a respective partition of the input stream.