Workflow monitoring and analysis system and method thereof

• US 9,613,328 B2
• Filed: 05/23/2013
• Issued: 04/04/2017
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lin, Hung-Wei, Wang, Hau-Wei, Dong, Shu-Ping
• Primary Examiner(s): MANSFIELD, THOMAS L
• Application Number:
  US13/900,905
• Publication Number:
  US 20140180743A1
• Time in Patent Office: 1,412 Days
• Field of Search: 705 728
• CPC Class Codes: G06Q 10/063114 Status monitoring or status determination for a person or group

Abstract

A method of workflow monitoring and analysis includes: according to an image to generate at least one three-dimensional joint coordinate, and according to the three dimensional joint coordinate to generate at least one task posture information; according to a movement information to generate at least one three-dimensional track information, and according to the three dimensional track to generate at least one task track information; and according to a workpiece posture information, the task posture information, a workpiece movement information and the task track information to generate a task semanticist.

Claims

1. A workflow monitoring and analysis method, comprising the steps of:

detecting, by a first image capturing module, at least one feature of at least one detection zone of a detection material, wherein the first image capturing module is a charge-coupled device (CCD);

generating, by the first image capturing module, an image information according to the at least one feature being detected;

generating, by a task model database, at least one three-dimensional joint coordinate according to the image information;

generating, by the task model database, at least one task posture information according to the at least one three dimensional joint coordinate;

generating, by a workpiece posture module, a workpiece posture information according to the image information;

capturing, by a second image capturing module, movements of the detection material, wherein the second image capturing module is a depth camera;

generating, by the second image capturing module, a movement information according to the movements of the detection material;

generating, by a detection zone movement module, at least one three-dimensional track information according to the movement information of the detection material;

generating, by a workpiece movement module, at least one workpiece track information according to the movement information of the detection material;

generating, by the task model database, a task track information according to the three-dimensional track information;

generating, by the task model database, a task semantics according to the task posture information and the task track information; and

outputting, by an output module, the task semantics for use in a hybrid automation device.

2. The method of claim 1, wherein the image information is a color image information.

3. The method of claim 1, wherein the movement information is acquired using a depth camera.

4. The method of claim 1, wherein the three-dimensional joint coordinate and the task posture information are generated by the step of:

inputting, by an image recognition module, the image information to the detection zone posture module;

performing, by the detection zone posture module, a measurement upon a region according to the image information;

establishing, by the detection zone posture module, at least one feature point according to the image information;

comparing, by the detection zone posture module, the at least one feature point with a primitive data and outputting, by the detection zone posture module, a comparison result to the task model database;

acquiring a mid-level shape information, by the task model database, according to the comparison result in a successive approximation approach manner;

performing, by the task model database, an angle calibration procedure upon the comparison result according to the mid-level shape information and generating an angle calibration result;

generating, by the task model database, the at least one three-dimensional joint coordinate according to the angle calibration result; and

generating, by the task model database, the at least one task posture information according to the at least one three-dimensional joint coordinate.

5. The method of claim 4, wherein the at least one feature point is established by determining, through a color distribution analysis, the feature point is in a color histogram and the feature point is a target region.

6. The method of claim 4, wherein the comparison result is determined based upon a geometric distance between the feature point and an image of the primitive data.

7. The method of claim 1, wherein the three-dimensional track information and the task track information are generated by the steps of:

receiving, by the detection zone movement module, the movement information of the detection material;

performing, by the detection zone movement module, a measurement upon a region according to the movement information and consequently generating a measurement result;

fetching, by the detection zone movement module, the three-dimensional joint coordinate according to the measurement result;

according to the three-dimensional joint coordinate, in response to determining, by the detection zone movement module, the movement information is one of a task starting coordinate and a task terminating coordinate, generating a three-dimensional track information, and

in response to determining, by the detection zone movement module, the movement information is not one of the task starting coordinate and the task terminating coordinate, proceeding back to the fetching of the three-dimensional joint coordinate;

updating, by the detection zone movement module, the three-dimensional joint coordinate and then repeating the fetching step;

making an evaluation to determine whether the process is ended; and

in response to determining that the process is ended, generating a task track information according to the three-dimensional track information.

8. The method of claim 1, further comprising:

according to the workpiece track information,

in response to determining, by the task model database, that a task track is

definable, continuing to determine, by the task model database, the definability of a task posture, and

in response to determining, by the task model database, the task track is not definable, issuing a warning and proceeding back to the generating of at least one workpiece track information;

according to the task track information and the workpiece posture information,

in response to determining, by the task model database, the task posture is definable, issuing, by the task model database, a warning and proceeding back to the generating of at least one workpiece posture information, and

in response to determining, by the task model database, that a task posture is not definable, generating, by the task model database, the task semantics;

generating, by the task model database, at least one task model data according to the task posture, the workpiece posture information and the workpiece track information; and

generating, by the task model database, the task semantics according to the at least one task model data.

9. A workflow monitoring and analysis system, comprising:

a detection material;

at least one first image capturing module;

at least one second image capturing module; and

a workflow analysis unit, electrically and respectively coupled to the at least one first image capturing module and the at least one second image capturing module, and comprises

an image recognition module,

a detection zone posture module,

a detection zone movement module,

an output module, and

a task model database;

wherein the at least one first image capturing module is configured to

detect at least one feature of at least one detection zone of the detection material,

generate an image information according to the detected at least one feature, and

send the image information to the image recognition module;

wherein the at least one second image capturing module is configured to

capture movements of the detection material,

generate a movement information according to the movements of the detection material, and

send the movement information to the image recognition module;

wherein the image recognition module is configured to

receive the image information from the first image capturing module and the movement information from the second image capturing module,

recognize the image information and the movement information,

send the recognized image information to the detection zone posture module, and

send movement information to the detection zone movement module;

wherein the detection zone posture module is configured to

receive the image information from the image recognition module,

generate at least one feature point according to the image information, and

send the at least one feature point to the task model database;

wherein the detection zone movement module is configured to

receive the movement information from the image recognition module,

generate at least one three-dimensional track information according to the movement information, and

send the at least one three-dimensional track information to the task model database;

wherein the task model database is configured to

receive the at least one feature point from the detection zone posture module and the at least one three-dimensional track information from the detection zone movement module,

generate at least one task posture information according to the at least one feature point,

generate at least one task track information according to the at least one three-dimensional track information, and

generate a task semantics according to the at least one task posture information and the at least one task track information, and

wherein the output module is configured to output the task semantics for use in a hybrid automation device; and

wherein the first image capturing module is a charge-coupled device (CCD) and the second image capturing module is a depth camera.

10. The system of claim 9, wherein the workflow analysis unit includes a workpiece posture module, and a workpiece movement module.

11. The system of claim 9, wherein each of the detection zones is an area filled with components selected from the group consisting of:

colored regions, characters, patches, color blocks, blight spots, light-emitting units and reflective units, and the combination thereof.

12. The system of claim 9, wherein the detection material is a glove.