Method of processing big data, including arranging icons in a workflow GUI by a user, checking process availability and syntax, converting the workflow into execution code, monitoring the workflow, and displaying associated information

• US 10,175,954 B2
• Filed: 09/24/2014
• Issued: 01/08/2019
• Est. Priority Date: 09/24/2013
• Status: Active Grant

• Current Assignee: LG CNS Company Limited (LG Corporation)
• Original Assignee: LG CNS Company Limited (LG Corporation)
• Inventors: Lee, Joo Youl, Kang, Hyun Sang, Kim, Ki Do, Oh, Seok Keun, Park, Jong Won, Choi, Don Jung
• Primary Examiner(s): Kawsar, Abdullah Al
• Assistant Examiner(s): Stanley, Jeremy L
• Application Number:
  US14/495,455
• Publication Number:
  US 20150089415A1
• Time in Patent Office: 1,567 Days
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06Q 10/0633 Workflow analysis

Abstract

Provided is a method of processing a big data which may include arranging a plurality of action icons from a palette and a plurality of flow icons representing a non-cyclic order for the plurality of the action icons to generate a workflow, checking a syntax for the plurality of the arranged action icons and the plurality of the flow icons, the syntax being dependent on a corresponding action icon and graphically generating an alarm badge indicating a syntax error for a specific icon when the syntax error for the specific icon is found. Therefore, the method and apparatus may use a Hadoop framework to effectively manage a workflow for processing irregular big data.

Claims

1. A method of processing big data, the method comprising:

displaying, on a display, a plurality of action icons and a plurality of flow icons, the plurality of action icons and the plurality of flow icons being graphically arranged to represent a non-cyclic order for the plurality of the action icons to generate a workflow, the plurality of action icons being associated with executable functions for a framework for distributed storage and processing of big data including at least one of a distributed file system (DSF) input function, a DSF output function, an expression function, or an aggregator function, and checking a sequential process availability for a pair of action icons when a specific flow icon between the pair of action icons is arranged,

wherein checking a sequential process availability further includes sequentially processing the workflow when all requisite action icons are arranged in the workflow between the pair of action icons, and refusing sequential process availability when all requisite action icons are not arranged in the workflow between the pair of action icons;

checking, by a controller, a syntax for each of the plurality of the arranged action icons and the plurality of the flow icons, the syntax being dependent on an action icon for which the syntax is checked or an action icon corresponding to a flow icon for which the syntax is checked;

generating, by the controller, an alarm badge for display on a corresponding action icon for which the alarm badge is generated, the alarm badge indicating a syntax error for the corresponding action icon for which the alarm badge is generated when the syntax error is found;

converting the workflow into a workflow definition document when a syntax check for the generated workflow is completed and storing the workflow definition document in a workflow database;

converting the workflow definition document into an execution code and analyzing the execution code to determine an action icon;

transferring the execution code for the determined action icon to a job tracker module to automatically receive a processing progress status including a completion time of the determined action icon; and

providing process work information for a plurality of the generated workflow through a work monitoring menu included in a workflow work environment, wherein the process work information includes an ID, a status whether a process succeeded or failed, a start time, an end time, a generation time, and a latest change time for the generated workflow,

wherein displaying the plurality of the action icons and the plurality of the flow icons further includes receiving a setting parameter for the framework for distributed storage and processing of big data associated with a map-reduce process, and

wherein at least one of the plurality of the action icons is associated with a big data process and at least one of the plurality of the action icons is associated with a flow process,

wherein displaying the plurality of the action icons and the plurality of the flow icons further includes providing metadata for a specific action icon from an icon window when the specific action icon in the workflow is selected, and storing a sequential execution procedure for the metadata,

wherein transparency for the icon window is adjusted according to a movement speed in which an icon window is initially popped up for display based on selection of the specific action icon, wherein when the movement speed is slow the transparency is increased and when the movement speed is fast the transparency is decreased, and

wherein the icon window includes

a first window having an input metadata region and an output metadata region, the input metadata region displaying a plurality of map-reduce input items for the specific action icon associated with input metadata and the output metadata region displays a plurality of items associated with output metadata for an operation, expression and execution sequence,

a second window that displays a configuration region for the framework including at least one key and value associated with an operation mode of the map-reduce process, and

a third window that displays an input-output location of distribution storage, wherein displaying the plurality of the action icons and the plurality of the flow icons further includes setting the input-output location of the distribution storage associated with the map-reduce process in the third window.

2. The method of claim 1, wherein checking the syntax includes checking whether an execution procedure being used in the plurality of the arranged action icons and dependent flow icons thereof are defined or available.

3. The method of claim 1, wherein generating the alarm badge further includes determining a type of the syntax error to determine the alarm badge corresponding to a type of the syntax error.

4. The method of claim 1, further comprising:

defining metadata to be added or deleted for generating the workflow for the plurality of the action icons through a file metadata menu included in the workflow work environment, wherein the file metadata menu includes a list of a plurality of defined metadata for big data, each of the plurality of defined metadata including a name, a column explanation, a unique type, and whether a primary key is included.

5. The method of claim 4, further comprising:

checking whether the metadata defined on the file metadata menu is changed; and

providing a workflow list related to the changed metadata through an effect analysis screen for the metadata included in the workflow work environment when the metadata is determined to have changed, wherein each changed metadata listed in the workflow list includes a workflow name, a workflow path, an input metadata, and an output metadata.

6. The method of claim 1, further comprising:

storing a result for a corresponding map-reduce process in a map-reduce database when a map-reduce process for the action icon is completed.

7. The method of claim 1, wherein the icon window includes a first tab for displaying the first window, a second tab for displaying the second window, and a third tab for displaying the third window.

8. The method of claim 1, wherein generating the alarm badge for display on a corresponding action icon includes displaying a first badge for a first type of syntax error and a second badge for a second type of syntax error, wherein the first type of syntax error is different than the second type of syntax error.

9. The method of claim 8, wherein the first type of syntax error is found when a typing error or an error in value is detected and the second type of syntax error is found when a syntax is not inputted at a syntax input item of a corresponding action icon.

10. An apparatus of processing big data, the apparatus comprising:

a workflow writing unit configured to arrange a plurality of action icons and a plurality of flow icons representing a non-cyclic order for the plurality of the action icons to generate a workflow, the plurality of action icons being associated with executable functions for a framework for distributed storage and processing of big data including at least one of a distributed file system (DSF) input function, a DSF output function, an expression function, or an aggregator function, and check a sequential process availability for a pair of action icons when a specific flow icon between the pair of action icons is arranged,

wherein checking a sequential process availability further includes sequentially processing the workflow when all requisite action icons are arranged in the workflow between the pair of action icons, and refusing sequential process availability when all requisite action icons are not arranged in the workflow between the pair of action icons;

an icon syntax checking unit configured to check a syntax for each of the plurality of the arranged action icons and the plurality of the flow icons, the syntax being dependent on an action icon for which the syntax is checked or an action icon corresponding to a flow icon for which the syntax is checked;

an alarm badge generating unit configured to graphically generate an alarm badge that indicates a syntax error for the corresponding action icon for which the alarm badge is generated when the syntax error is found; and

an execution completion transferring unit configured to convert the workflow definition document into an execution code, to analyze the execution code to determine an action icon and to transfer the execution code for the determined action icon to a job tracker module to automatically receive a processing progress status including a completion time of the determined action icon,

wherein displaying the plurality of the action icons and the plurality of the flow icons further includes receiving a setting parameter for the framework for distributed storage and processing of big data associated with a map-reduce process,

wherein at least one of the plurality of the action icons is associated with a big data process and at least one of the plurality of the action icons is associated with a flow process,

wherein displaying the plurality of the action icons and the plurality of the flow icons further includes providing metadata for a specific action icon from an icon window when the specific action icon in the workflow is selected, and storing a sequential execution procedure for the metadata,

wherein transparency for the icon window is adjusted according to a movement speed in which an icon window is initially popped up for display based on selection of the specific action icon, wherein when the movement speed is slow the transparency is increased and when the movement speed is fast the transparency is decreased, and

wherein the icon window includes

a first window having an input metadata region and an output metadata region, the input metadata region including a plurality of map-reduce input items for the specific action icon to set input metadata and the output metadata region including a plurality of items to set output metadata for an operation, expression and execution sequence,

a second window that displays a configuration region for the framework including at least one key and value associated with an operation mode of the map-reduce process, and

a third window that displays an input-output location of distribution storage, wherein displaying the plurality of the action icons and the plurality of the flow icons further includes setting the input-output location of the distribution storage associated with the map-reduce process in the third window, and

wherein a work monitoring menu included in a workflow work environment displays process work information for a plurality of the generated workflow, wherein the process work information includes an ID, a status whether a process succeeded or failed, a start time, an end time, a generation time, and a latest change time for the generated workflow.

11. The apparatus of claim 10, further comprising:

a workflow database storing unit configured to convert the workflow into a workflow definition document when a syntax check for the generated workflow is completed and store the workflow definition document in a workflow database.

12. The apparatus of claim 11, further comprising:

a map-reduce database storing unit configured to store a result for a corresponding map-reduce process in a map-reduce database when a map-reduce process for the action icon is completed.

13. A machine-readable non-transitory medium storing a computer program for a method of processing a big data being performed in an apparatus of processing the big data, the machine-readable non-transitory medium having stored thereon machine-executable instructions for:

displaying, on a display, a plurality of action icons and a plurality of flow icons, the plurality of action icons and the plurality of flow icons being graphically arranged to represent a non-cyclic order for the plurality of the action icons to generate a workflow, the plurality of action icons being associated with executable functions for a framework for distributed storage and processing of big data including at least one of a distributed file system (DSF) input function, a DSF output function, an expression function, or an aggregator function, and checking a sequential process availability for a pair of action icons when a specific flow icon between the pair of action icons is arranged,

wherein checking a sequential process availability further includes sequentially processing the workflow when all requisite action icons are arranged in the workflow between the pair of action icons, and refusing sequential process availability when all requisite action icons are not arranged in the workflow between the pair of action icons;

checking, by a controller, a syntax for each of the plurality of the arranged action icons and the plurality of the flow icons, the syntax being dependent on an action icon for which the syntax is checked or an action icon corresponding to a flow icon for which the syntax is checked;

generating, by the controller, an alarm badge for display on a corresponding action icon for which the alarm badge is generated, the alarm badge indicating a syntax error for the corresponding action icon for which the alarm badge is generated when the syntax error is found;

converting the workflow into a workflow definition document when a syntax check for the generated workflow is completed and storing the workflow definition document in a workflow database;

converting the workflow definition document into an execution code and analyzing the execution code to determine an action icon;

transferring the execution code for the determined action icon to a job tracker module to automatically receive a processing progress status including a completion time of the determined action icon; and

providing process work information for a plurality of the generated workflow through a work monitoring menu included in a workflow work environment, wherein the process work information includes an ID, a status whether a process succeeded or failed, a start time, an end time, a generation time, and a latest change time for the generated workflow,

wherein displaying the plurality of the action icons and the plurality of the flow icons further includes receiving a setting parameter for the framework for distributed storage and processing of big data associated with a map-reduce process,

wherein at least one of the plurality of the action icons is associated with a big data process and at least one of the plurality of the action icons is associated with a flow process,

wherein displaying the plurality of the action icons and the plurality of the flow icons further includes providing metadata for a specific action icon from an icon window when the specific action icon in the workflow is selected, and storing a sequential execution procedure for the metadata,

wherein transparency for the icon window is adjusted according to a movement speed in which an icon window is initially popped up for display based on selection of the specific action icon, wherein when the movement speed is slow the transparency is increased and when the movement speed is fast the transparency is decreased, and

wherein the icon window includes

a first window having an input metadata region and an output metadata region, the input metadata region including a plurality of map-reduce input items for the specific action icon associated with input metadata and the output metadata region including a plurality of items associated with output metadata for an operation, expression and execution sequence,

a second window that displays a configuration region for the framework including at least one key and value associated with an operation mode of the map-reduce process, and

a third window that displays an input-output location of distribution storage, wherein displaying the plurality of the action icons and the plurality of the flow icons further includes setting the input-output location of the distribution storage associated with the map-reduce process in the third window.