Application resource model composition from constituent components
First Claim
1. A method for composing an application resource model for a data stream processing system, the method comprising steps of:
- obtaining one or more operator-level metrics from an execution of a data stream processing application in accordance with a first configuration by learning a communication overhead function associated with communication between two or more processing elements, said communication overhead function based on a maximum rate and a central processing unit usage measured from a benchmark configuration scaled according to a rate and a tuple size for which the overhead is to be computed, wherein the application is executed by one or more nodes of the data stream processing system and the application is comprised of one or more processing elements that are comprised of multiple operators, wherein two or more of the multiple operators are combined in a first combination to form a processing element, and wherein the first combination of operators is in accordance with the first configuration;
generating one or more operator-level resource functions from the first combination of operators in accordance with the first configuration based on the obtained one or more operator-level metrics; and
generating a processing element-level resource function using the one or more generated operator-level resource functions to predict a model for the processing element formed by a second combination of two or more of the multiple operators, the processing element-level resource function representing an application resource model usable for predicting one or more characteristics of the application executed in accordance with a second configuration;
wherein the generating steps are executed on a computing system comprising a processor and a memory, wherein one or more software modules for implementing the generating steps are loaded from the memory and executed by the processor resulting in the generation of the application resource model.
0 Assignments
0 Petitions
Accused Products
Abstract
Techniques for composing an application resource model are disclosed. The techniques include obtaining operator-level metrics from an execution of a data stream processing application according to a first configuration, wherein the application is executed by nodes of the data stream processing system and the application includes processing elements comprised of multiple operators, wherein two or more of the operators are combined in a first combination to form a processing element according to the first configuration, generating operator-level resource functions from the first combination of operators based on the obtained operator-level metrics, and generating a processing element-level resource function using the generated operator-level resource functions to predict a model for the processing element formed by a second combination of operators, the processing element-level resource function representing an application resource model usable for predicting characteristics of the application executed according to a second configuration.
12 Citations
19 Claims
-
1. A method for composing an application resource model for a data stream processing system, the method comprising steps of:
-
obtaining one or more operator-level metrics from an execution of a data stream processing application in accordance with a first configuration by learning a communication overhead function associated with communication between two or more processing elements, said communication overhead function based on a maximum rate and a central processing unit usage measured from a benchmark configuration scaled according to a rate and a tuple size for which the overhead is to be computed, wherein the application is executed by one or more nodes of the data stream processing system and the application is comprised of one or more processing elements that are comprised of multiple operators, wherein two or more of the multiple operators are combined in a first combination to form a processing element, and wherein the first combination of operators is in accordance with the first configuration; generating one or more operator-level resource functions from the first combination of operators in accordance with the first configuration based on the obtained one or more operator-level metrics; and generating a processing element-level resource function using the one or more generated operator-level resource functions to predict a model for the processing element formed by a second combination of two or more of the multiple operators, the processing element-level resource function representing an application resource model usable for predicting one or more characteristics of the application executed in accordance with a second configuration; wherein the generating steps are executed on a computing system comprising a processor and a memory, wherein one or more software modules for implementing the generating steps are loaded from the memory and executed by the processor resulting in the generation of the application resource model. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. Apparatus for composing an application resource model for a data stream processing system, the apparatus comprising:
-
a memory; and a processor coupled to the memory and configured to; (i) obtain one or more operator-level metrics from an execution of a data stream processing application in accordance with a first configuration by learning a communication overhead function associated with communication between two or more processing elements, said communication overhead function based on a maximum rate and a central processing unit usage measured from a benchmark configuration scaled according to a rate and a tuple size for which the overhead is to be computed, wherein the application is executed by one or more nodes of the data stream processing system and the application is comprised of one or more processing elements that are comprised of multiple operators, wherein two or more of the multiple operators are combined in a first combination to form a processing element, and wherein the first combination of operators is in accordance with the first configuration; (ii) generate one or more operator-level resource functions from the first combination of operators in accordance with the first configuration based on the obtained one or more operator-level metrics; and (iii) generate a processing element-level resource function using the one or more generated operator-level resource functions to predict a model for the processing element formed by a second combination of two or more of the multiple operators, the processing element-level resource function representing an application resource model usable for predicting one or more characteristics of the application executed in accordance with a second configuration. - View Dependent Claims (14, 15, 16)
-
-
17. A computer program product comprising a tangible non-transitory computer readable recordable storage medium including computer usable program code for composing an application resource model for a data stream processing system, the computer program product comprising:
-
computer usable program code for obtaining one or more operator-level metrics from an execution of a data stream processing application in accordance with a first configuration by learning a communication overhead function associated with communication between two or more processing elements, said communication overhead function based on a maximum rate and a central processing unit usage measured from a benchmark configuration scaled according to a rate and a tuple size for which the overhead is to be computed, wherein the application is executed by one or more nodes of the data stream processing system and the application is comprised of one or more processing elements that are comprised of multiple operators, wherein two or more of the multiple operators are combined in a first combination to form a processing element, and wherein the first combination of operators is in accordance with the first configuration; computer usable program code for generating one or more operator-level resource functions from the first combination of operators in accordance with the first configuration based on the obtained one or more operator-level metrics; and computer usable program code for generating a processing element-level resource function using the one or more generated operator-level resource functions to predict a model for the processing element formed by a second combination of two or more of the multiple operators, the processing element-level resource function representing an application resource model usable for predicting one or more characteristics of the application executed in accordance with a second configuration. - View Dependent Claims (18, 19)
-
Specification