Computer-vision based process recognition of activity workflow of human performer

• US 10,110,858 B2
• Filed: 04/16/2015
• Issued: 10/23/2018
• Est. Priority Date: 02/06/2015
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Loce, Robert P., Xu, Beilei, Bernal, Edgar A., Prabhat, Saurabh, Wu, Wencheng
• Primary Examiner(s): Harvey, David
• Application Number:
  US14/688,230
• Publication Number:
  US 20160234464A1
• Time in Patent Office: 1,286 Days
• Field of Search: 348143
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A computer-vision based method for validating an activity workflow of a human performer includes identifying a target activity. The method includes determining an expected sequence of actions associated with the target activity. The method includes receiving a video stream from an image capture device monitoring an activity performed by an associated human performer. The method includes determining an external cue in the video stream. The method includes associating a frame capturing the external cue as a first frame in a key frame sequence. The method includes determining an action being performed by the associated human performer in the key frame sequence. In response to determining the action in the key frame sequence matching an expected action in the target activity, the method includes verifying the action as being performed in the monitored activity. In response to not determining the action in the key frame sequence, the method includes generating an alert indicating an error in the monitored activity.

Claims

1. A computer-vision based method for validating an activity workflow of a human performer, the method comprising:

acquire from a storage device in communication with a computer an expected sequence of actions associated with a target activity;

receiving at a computer a video stream from an image capture device monitoring an activity performed by an associated human performer;

using a processor, processing the video stream to detect an external cue in the video stream;

associating a frame capturing the external cue as a first frame in a key frame sequence;

using the processor, extracting features from the key frame sequence;

applying the extracted features to at least one classifier to determine an action being performed by the associated human performer in the key frame sequence;

in response to determining the action in the key frame sequence matching an expected action in the target activity, verifying the action as being performed in the monitored activity;

using the processor, comparing an ordered sequence of the verified actions against the expected sequence; and

,

in response to the ordered sequence not matching the key frame sequence, generating an alert indicating an error in the monitored activity;

in response to the ordered sequence matching the key frame sequence, performing a quantification about an object detected in the key frame sequence to determine a measure associated with the object and embedding the measure in the video stream as metadata.

2. The method of claim 1, wherein the external cue is selected from a group consisting of:

an appearance of a certain object via object detection;

a video analysis to detect a segment that includes lack of motion;

a gesture made by the human performer being monitored;

audio feedback;

voice recognition;

an input made at a GUI by the associated human performer; and

,

a combination of the above.

3. The method of claim 1, wherein the quantification includes:

generating a binary image of the object located in an image frame; and

,

summing pixels in the binary image to compute the measure.

4. The method of claim 1, wherein in response to an error being associated with the activity workflow, the method further comprising:

extracting the embedded measure from the video stream;

comparing the extracted measure against a predetermined value; and

,

in response to a difference between the extracted measure and the predetermined value meeting and exceeding a predetermined threshold, generating an alert.

5. The method of claim 1, wherein the object is a syringe and the measure is a liquid fill-level of the syringe.

6. The method of claim 1, wherein the object is one of a bottle, container, and vial, and the measure is a number of items for delivery in the object.

7. The method of claim 1, where the target activity is a production of a pharmaceutical, and the action is a step of or relating to the target activity, wherein at least one action forms the target activity.

8. The method of claim 1, in response to generating the alert, providing the video stream to a user for review.

9. The method of claim 1, wherein the determining the action includes:

performing gesture recognition on the video stream;

in response to detecting a gesture, segmenting a sequence of frames in the video stream proximate the detected gesture; and

,

searching the sequence of frames for the expected action.

10. The method of claim 1, wherein the determining the action includes:

searching for visual representations in the video sequence and determining if the representations correspond to a select object.

11. A system for validating an activity workflow of a human performer, the system comprising a computer device including a memory in communication with a processor configured to:

acquire from a storage device an expected sequence of actions associated with a target activity;

receive at a computer a video stream from an image capture device monitoring an activity performed by an associated human performer;

using a processor, determining an external cue in the video stream;

associating a frame capturing the external cue as a first frame in a key frame sequence;

applying extracted features from select frames to a classifier to determine at least two actions being performed by the associated human performer in the key frame sequence;

in response to determining a sequence of the at least two actions in the key frame sequence matching the expected sequence of the target activity, verifying a quality of an associated product prepared by the monitored activity, the verifying including;

performing a quantification about an object detected in the key frame sequence to determine a measure associated with the object, and

embedding the measure in the video stream as metadata; and

,

in response to not determining an action in the key frame sequence, generate an alert indicating an error in the monitored activity.

12. The system, of claim 11, wherein the external cue is selected from a group consisting of:

an appearance of a certain object via object detection;

a video analysis to detect a segment that includes lack of motion;

a gesture made by the human performer being monitored;

audio feedback;

voice recognition;

an input made at a GUI by the associated human performer; and

,

a combination of the above.

13. The system, of claim 11, wherein the processor is further configured to:

generate a binary image of the object located in an image frame; and

,

sum pixels in the binary image to compute the measure.

14. The system, of claim 11, wherein the processor is further configured to:

in response to an error being associated with the activity workflow, extract the embedded measure from the video stream;

compare the extracted measure against a predetermined value; and

,

in response to a difference between the extracted measure and the predetermined value meeting and exceeding a predetermined threshold, generate an alert.

15. The system, of claim 11, wherein the processor is further configured to:

in response to generating the alert, provide the video stream to a user for review.

16. The system, of claim 11, wherein the processor is further configured to:

perform gesture recognition on the video stream;

in response to detecting a gesture, segment a sequence of frames in the video stream proximate the detected gesture; and

,

search the sequence of frames for the expected action.

17. The system, of claim 11, wherein the processor is further configured to:

search for visual representations in the video sequence and determining if the representations correspond to a select object.

18. A computer-vision based method for validating an activity workflow of a human performer, the method comprising:

determining an expected sequence of actions associated with a target activity;

receiving at a computer a video stream from an image capture device monitoring an activity performed by an associated human performer;

applying features extracted from select frames to a classifier for determining at least two actions performed by the associated human performer in the monitored video stream;

in response to determining a sequence of the at least two actions matching the expected sequence in the monitored video stream, verifying a quality of an associated product being prepared by the monitored activity, the verifying including;

detecting an object in a key frame sequence of the video stream,

performing a quantification about the object to determine a measure associated with the object, and

embedding the measure in the video stream as metadata; and

,

in response to not determining an action in the monitored video stream, generating an alert indicating an error in the monitored activity.