System and method for creating, tracking, and maintaining big data use cases
First Claim
Patent Images
1. A method for tracking and maintaining big data use in a distributed file system having a plurality of processors, the method comprising:
- executing, by a processor of the plurality of processors, instructions stored in a memory to cause an application development framework stored in the memory to perform operations of;
registering, by the application development framework, a use case identifying a requesting party, a purpose of the use case, a group of users for the use case, requested data for the use case and an application with an application identifier,the application running by using the requested data and access permissions of the use case, andthe application development framework being a unified environment for development and deployment of the use case;
creating, by the application development framework, a process to democratize application development tasks via a framework of support libraries;
generating, by the application development framework, a transformation associated with the use case;
receiving, by the application development framework, a request to derive an output variable, wherein the request includes parameters having proposed logic for application to a source variable;
generating, by the application development framework, metadata describing the proposed logic;
comparing, by the application development framework, the metadata describing the proposed logic to metadata for existing logic to determine that the proposed logic is new;
generating, by the application development framework, transformation code, in response to the proposed logic being new;
deriving, by the application development framework and using the transformation, the output variable from the source variable;
executing, by the application development framework, the transformation to derive output data for the output variable from source data of the source variable by executing dynamically generated code that applies logic to the source data;
outputting, by the application development framework, the output data of the output variable to at least one of a big data management system or a batch consumer system in response to the executing the transformation;
cataloging, by the application development framework, an access permission at at least one of a file, table, variable, column or data type level,the access permission including the application identifier associated with the registered application, data about the application having at least one of read or write access to the requested data, and data about the application being authorized to view information contained in the source variable;
indexing, by the application development framework, the access permission by the application identifier for monitoring and restricting data access;
looking up, by the application development framework and in response to an access request by the application, the access permission for the application using the application identifier; and
deciding, by the application development framework, on the access request based on the access permission, wherein the deciding includes denying the access request in response to restrictions for the application based on a review of metadata describing the source variable and the output variable.
1 Assignment
0 Petitions
Accused Products
Abstract
A system may register a use case with the use case including an application. An application identifier may be assigned to the application. The system may generate a transformation associated with the use case. The transformation may include logic to derive an output variable from a source variable. The system may also execute the transformation to derive output data for the output variable from source data of the source variable. The system may further lookup an access permission for the application using the application identifier in response to an access request.
83 Citations
15 Claims
-
1. A method for tracking and maintaining big data use in a distributed file system having a plurality of processors, the method comprising:
-
executing, by a processor of the plurality of processors, instructions stored in a memory to cause an application development framework stored in the memory to perform operations of; registering, by the application development framework, a use case identifying a requesting party, a purpose of the use case, a group of users for the use case, requested data for the use case and an application with an application identifier, the application running by using the requested data and access permissions of the use case, and the application development framework being a unified environment for development and deployment of the use case; creating, by the application development framework, a process to democratize application development tasks via a framework of support libraries; generating, by the application development framework, a transformation associated with the use case; receiving, by the application development framework, a request to derive an output variable, wherein the request includes parameters having proposed logic for application to a source variable; generating, by the application development framework, metadata describing the proposed logic; comparing, by the application development framework, the metadata describing the proposed logic to metadata for existing logic to determine that the proposed logic is new; generating, by the application development framework, transformation code, in response to the proposed logic being new; deriving, by the application development framework and using the transformation, the output variable from the source variable; executing, by the application development framework, the transformation to derive output data for the output variable from source data of the source variable by executing dynamically generated code that applies logic to the source data; outputting, by the application development framework, the output data of the output variable to at least one of a big data management system or a batch consumer system in response to the executing the transformation; cataloging, by the application development framework, an access permission at at least one of a file, table, variable, column or data type level, the access permission including the application identifier associated with the registered application, data about the application having at least one of read or write access to the requested data, and data about the application being authorized to view information contained in the source variable; indexing, by the application development framework, the access permission by the application identifier for monitoring and restricting data access; looking up, by the application development framework and in response to an access request by the application, the access permission for the application using the application identifier; and deciding, by the application development framework, on the access request based on the access permission, wherein the deciding includes denying the access request in response to restrictions for the application based on a review of metadata describing the source variable and the output variable. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A computer-based system for tracking and maintaining big data use in a distributed file system, comprising:
-
a plurality of processors; and a tangible, non-transitory memory configured to communicate with a processor of the plurality of processors, the tangible, non-transitory memory having instructions stored thereon that, in response to execution by the processor, cause an application development framework to perform operations comprising; registering, by the application development framework, a use case identifying a requesting party, a purpose of the use case, a group of users for the use case, requested data for the use case and an application with an application identifier, the application running by using the requested data and access permissions of the use case, and the application development framework being a unified environment for development and deployment of the use case; creating, by the application development framework, a process to democratize application development tasks via a framework of support libraries; generating, by the application development framework, a transformation associated with the use case; receiving, by the application development framework, a request to derive an output variable, wherein the request includes parameters having proposed logic for application to a source variable; generating, by the application development framework, metadata describing the proposed logic; comparing, by the application development framework, the metadata describing the proposed logic to metadata for existing logic to determine that the proposed logic is new; generating, by the application development framework, transformation code, in response to the proposed logic being new; deriving, by the application development framework and using the transformation, the output variable from the source variable; executing, by the application development framework, the transformation to derive output data for the output variable from source data of the source variable by executing dynamically generated code that applies logic to the source data; outputting, by the application development framework, the output data of the output variable to at least one of a big data management system or a batch consumer system in response to the executing the transformation; cataloging, by the application development framework, an access permission at at least one of a file, table, variable, column or data type level, the access permission including the application identifier associated with the registered application, data about the application having at least one of read or write access to the requested data, and data about the application being authorized to view information contained in the source variable; indexing, by the application development framework, the access permission by the application identifier for monitoring and restricting data access; looking up, by the application development framework and in response to an access request by the application, the access permission for the application using the application identifier; and deciding, by the application development framework, on the access request based on the access permission, wherein the deciding includes denying the access request in response to restrictions for the application based on a review of metadata describing the source variable and the output variable. - View Dependent Claims (7, 8, 9, 10)
-
-
11. An article of manufacture including a non-transitory, tangible computer readable storage medium having instructions stored thereon that cause an application development framework to perform operations for tracking and maintaining big data use in a distributed file system comprising:
-
registering, by the application development framework, a use case identifying a requesting party, a purpose of the use case, a group of users for the use case, requested data for the use case and an application with an application identifier, the application running by using the requested data and access permissions of the use case, and the application development framework being a unified environment for development and deployment of the use case; creating, by the application development framework, a process to democratize application development tasks via a framework of support libraries; generating, by the application development framework, a transformation associated with the use case; receiving, by the application development framework, a request to derive an output variable, wherein the request includes parameters having proposed logic for application to a source variable; generating, by the application development framework, metadata describing the proposed logic; comparing, by the application development framework, the metadata describing the proposed logic to metadata for existing logic to determine that the proposed logic is new; generating, by the application development framework, transformation code, in response to the proposed logic being new; deriving, by the application development framework and using the transformation, the output variable from the source variable; executing, by the application development framework, the transformation to derive output data for the output variable from source data of the source variable by executing dynamically generated code that applies logic to the source data; outputting, by the application development framework, the output data of the output variable to at least one of a big data management system or a batch consumer system in response to the executing the transformation; cataloging, by the application development framework, an access permission at at least one of a file, table, variable, column or data type level, the access permission including the application identifier associated with the registered application, data about the application having at least one of read or write access to the requested data, and data about the application being authorized to view information contained in the source variable; indexing, by the application development framework, the access permission by the application identifier for monitoring and restricting data access; looking up, by the application development framework and in response to an access request by the application, the access permission for the application using the application identifier; and deciding, by the application development framework, on the access request based on the access permission, wherein the deciding includes denying the access request in response to restrictions for the application based on a review of metadata describing the source variable and the output variable. - View Dependent Claims (12, 13, 14, 15)
-
Specification