Component model for batch computing in a distributed object environment

• US 20050256826A1
• Filed: 05/13/2004
• Published: 11/17/2005
• Est. Priority Date: 05/13/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Little, Rodney Alan, High, Robert Howard Jr., Hambrick, Geoffrey Martin
• Granted Patent: US 8,024,733 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/5013 Request control
• CPC Class Codes: G06F 2209/5014 Reservation
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 2209/503 Resource availability
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups

Abstract

A batch component model is provided within a distributed object environment. The batch component is designed to capture the iterative logic of a batch program as it reads from one or more input streams, invokes operations on other business component functions, and generates output to one or more output streams. Deployment descriptors express declarative policies for the component that will influence how the component is managed including the streams it uses, business components it depends on, how processing costs are accounted for, the resource demands the job will put on the system, and the like. Input streams and output streams are encapsulated in objects that hide the actual source of input and output data so that the component can be redeployed in different execution environments to different physical data sources without requiring the program to be changed. A batch container enforces the deployment policies declared for the batch component.

Claims

1. A method of batch processing in a distributed object environment, the method comprising:

providing a batch component for a batch job within the distributed object environment;

initializing the batch component with a set of deployment descriptors; and

managing operation of the batch component within the distributed object environment in accordance with the set of deployment descriptors.

2. The method of claim 1, wherein the set of deployment descriptors identify one or more input stream components.

3. The method of claim 2, wherein the one or more input stream components are abstractions of actual underlying data technology from which the one or more input stream components receive data.

4. The method of claim 1, wherein the set of deployment descriptors identify one or more output stream components.

5. The method of claim 2, wherein the one or more output stream components are abstractions of actual underlying data technology to which the one or more output stream components send data.

6. The method of claim 1, wherein the set of deployment descriptors identify at least one business function component that the batch component uses.

7. The method of claim 1, wherein the set of deployment descriptors identify resources that are needed for the batch job.

8. The method of claim 7, wherein managing operation of the batch component includes allocating at least a portion of the resources for at least a portion of the batch job.

9. The method of claim 8, wherein managing operation of the batch component further includes locking at least a portion of the resources required by the batch job.

10. The method of claim 1, wherein managing operation of the batch component includes checkpointing progress of the batch job periodically.

11. The method of claim 1, wherein the batch component shares business logic and data components within the distributed object environment with other batch jobs or online transaction processing components.

12. A method of checkpointing a batch job in a distributed object environment, the method comprising:

determining a checkpoint interval duration of a checkpoint interval for a batch component;

running the batch component within the checkpoint interval; and

upon completion of the checkpoint interval duration, storing at least one checkpoint cursor.

13. The method of claim 12, further comprising:

upon completion of the checkpoint interval duration, committing the at least one checkpoint cursor.

14. The method of claim 13, further comprising:

responsive to the batch component,stopping during a given checkpoint interval, determining a starting point based on the at least one checkpoint cursor and restarting the batch component at the starting point.

15. The method of claim 12, further comprising:

upon completion of the checkpoint interval duration, committing transactional updates that occurred during the checkpoint interval.

16. The method of claim 12, further comprising:

before running the batch component within the checkpoint interval, allocating resources to be used by the batch component during the checkpoint interval and locking the resources.

17. The method of claim 16, further comprising:

upon completion of the checkpoint interval duration, unlocking the resources.

18. The method of claim 12, further comprising:

receiving a set of deployment descriptors for the batch component,

wherein determining a checkpoint interval duration for the checkpoint interval includes determining the checkpoint interval duration based on the deployment descriptors.

19. The method of claim 12, further comprising:

upon completion of the checkpoint interval duration, determining a size of a checkpoint interval for a next checkpoint interval.

20. The method of claim 12, wherein the checkpoint interval duration is defined as one of a period of time and a number of records in an input data stream.

21. The method of claim 12, wherein the checkpoint interval is determined based on actual workload conditions of the distributed object environment.

22. A method of partitioning a batch job, the method comprising:

dividing an input stream into a plurality of partitions of the input stream; and

running a plurality of instances of a batch component, wherein each instance of the batch component within the plurality of instances of the batch component operates on a respective partition of the input stream.

23. The method of claim 22, wherein the plurality of instances of the batch component operate in parallel.

24. The method of claim 22, wherein a second instance of the batch component begins operation at completion of operation of a first instance of the batch component.

25. The method of claim 22, wherein initiation of the plurality of instances of the batch component is staggered.

26. An apparatus for batch processing in a distributed object environment, the apparatus comprising:

means for providing a batch component for a batch job within the distributed object environment;

means for initializing the batch component with a set of deployment descriptors; and

means for managing operation of the batch component within the distributed object environment in accordance with the set of deployment descriptors.

27. The apparatus of claim 26, wherein the set of deployment descriptors identify one or more input stream components.

28. The apparatus of claim 27, wherein the one or more input stream components are abstractions of actual underlying data technology from which the one or more input stream components receive data.

29. The apparatus of claim 26, wherein the set of deployment descriptors identify one or more output stream components.

30. The apparatus of claim 27, wherein the one or more output stream components are abstractions of actual underlying data technology to which the one or more output stream components send data.

31. The apparatus of claim 26, wherein the set of deployment descriptors identify at least one business function component that the batch component uses.

32. The apparatus of claim 26, wherein the set of deployment descriptors identify resources that are needed for the batch job.

33. The apparatus of claim 32, wherein the means for managing operation of the batch component includes means for allocating at least a portion of the resources for at least a portion of the batch job.

34. The apparatus of claim 33, wherein the means for managing operation of the batch component further includes means for locking at least a portion of the resources required by the batch job.

35. The apparatus of claim 26, wherein the means for managing operation of the batch component includes means for checkpointing progress of the batch job periodically.

36. The apparatus of claim 26, wherein the batch component shares business logic and data components within the distributed object environment with other batch jobs or online transaction processing components.

37. An apparatus for checkpointing a batch job in a distributed object environment, the apparatus comprising:

means for determining a checkpoint interval duration of a checkpoint interval for a batch component;

means for running the batch component within the checkpoint interval; and

means for storing at least one checkpoint cursor upon completion of the checkpoint interval duration.

38. The apparatus of claim 37, further comprising:

means for committing the at least one checkpoint cursor upon completion of the checkpoint interval duration.

39. The apparatus of claim 38, further comprising:

means, responsive to the batch component stopping during a given checkpoint interval, for determining a starting point based on the at least one checkpoint cursor and restarting the batch component at the starting point.

40. The apparatus of claim 37, further comprising:

means for committing transactional updates that occurred during the checkpoint interval upon completion of the checkpoint interval duration.

41. The apparatus of claim 37, further comprising:

means for allocating resources to be used by the batch component during the checkpoint interval and locking the resources before running the batch component within the checkpoint interval.

42. The apparatus of claim 41, further comprising:

means for unlocking the resources upon completion of the checkpoint interval duration.

43. The apparatus of claim 37, further comprising:

means for receiving a set of deployment descriptors for the batch component,

wherein the means for determining a checkpoint interval duration for the checkpoint interval includes means for determining the checkpoint interval duration based on the deployment descriptors.

44. The apparatus of claim 37, further comprising:

means for determining a size of a checkpoint interval for a next checkpoint interval upon completion of the checkpoint interval duration.

45. The apparatus of claim 37, wherein the checkpoint interval duration is defined as one of a period of time and a number of records in an input data stream.

46. The apparatus of claim 37, wherein the checkpoint interval is determined based on actual workload conditions of the distributed object environment.

47. An apparatus for partitioning a batch job, the apparatus comprising:

means for dividing an input stream into a plurality of partitions of the input stream; and

means for running a plurality of instances of a batch component, wherein each instance of the batch component within the plurality of instances of the batch component operates on a respective partition of the input stream.

48. The apparatus of claim 47, wherein the plurality of instances of the batch component operate in parallel.

49. The apparatus of claim 47, wherein a second instance of the batch component begins operation at completion of operation of a first instance of the batch component.

50. The apparatus of claim 47, wherein initiation of the plurality of instances of the batch component is staggered.

51. A distributed object environment, comprising:

a batch component; and

a batch container, wherein the batch container receives a set of deployment descriptors for the batch component and manages the batch component in accordance with the set of deployment descriptors.

52. The distributed object environment of claim 51, wherein the batch container allocates resources for at lest a portion of the batch job.

53. The distributed object environment of claim 52, wherein the batch container locks the resources.

54. The distributed object environment of claim 51, wherein the batch container receives a query from an administrator and returns information about progress of the batch job.

55. The distributed object environment of claim 51, further comprising:

business function components; and

business object components.

56. The distributed object environment of claim 55, wherein the business function components are stateless session beans and wherein the business object components are entity beans.

57. The distributed object environment of claim 55, further comprising:

at least one online processing client that performs operations using the business function components and the business object components in real time.

58. The distributed object environment of claim 57, wherein the batch component shares the business function components and the business object components with the at least one online processing client.

59. The distributed object environment of claim 57, wherein the at least one online processing client includes at least one of a terminal services client, a Web services client, a Web application client, and a workflow.

60. A computer program product, in a computer program product, for batch processing in a distributed object environment, the computer program product comprising:

instructions for providing a batch component for a batch job within the distributed object environment;

instructions for initializing the batch component with a set of deployment descriptors; and

instructions for managing operation of the batch component within the distributed object environment in accordance with the set of deployment descriptors.

61. A computer program product, in a computer readable medium, for checkpointing a batch job in a distributed object environment, the computer program product comprising:

instructions for determining a checkpoint interval duration of a checkpoint interval for a batch component;

instructions for running the batch component within the checkpoint interval; and

instructions for upon completion of the checkpoint interval duration, storing at least one checkpoint cursor.

62. A computer program product, in a computer readable medium, for partitioning a batch job, the computer program product comprising:

instructions for dividing an input stream into a plurality of partitions of the input stream; and

instructions for running a plurality of instances of a batch component, wherein each instance of the batch component within the plurality of instances of the batch component operates on a respective partition of the input stream.