CONTROL SYSTEM USING AESTHETICALLY GUIDED GESTURE RECOGNITION

• US 20170285757A1
• Filed: 03/31/2016
• Published: 10/05/2017
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: ROBERTSON, JAMES R., SCANLON, RAYMOND J., HONECK, MICHAEL R., JAY, ANGELA, ROUSE, CORY J., IRMLER, HOLGER
• Granted Patent: US 10,338,686 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method for facilitating and enhancing computer-based authoring of gesture definitions that are useful in controlling a walk-around character and other systems using gesture controls. The method includes, during performance of a gesture by a performer, collecting sets of raw sensor data each corresponding to differing parameters of the performance of the gesture. The method includes displaying a graphical user interface with a graphical plot of each of the sets of raw sensor data. The method includes receiving user input identifying which of the parameters to include in a gesture definition. The method includes, for the graphical plots associated with the chosen parameters receiving user input defining a starting position, an ending position, a maximum value, and a minimum value. The gesture may involve movement of a performer'"'"'s arms, legs, hands, head, eyes, and so on in a particular manner.

Claims

1. A method for authoring gesture definitions for use in gesture-based control systems, comprising:

during performance of a gesture by a performer, collecting a plurality of sets of raw sensor data each corresponding to differing parameters of the performance of the gesture;

displaying a graphical user interface (GUI) on a display device, wherein the GUI includes a graphical plot of each of the sets of raw sensor data;

receiving and storing in memory user input identifying a subset of the parameters to include in a gesture definition; and

for each of the graphical plots for the subset of the parameters in the gesture definition, receiving and storing, in the memory in the gesture definition, user input defining at least one of a starting position, an ending position, a maximum value, and a minimum value.

2. The method of claim 1, wherein the user input defining the starting position, the ending position, the maximum value, and the minimum value includes positioning a set of bounding boxes in each of the graphical plots of the sets of the raw sensor data for the subset of the parameters and sizing each of the bounding boxes.

3. The method of claim 1, further including receiving user input defining for one of the graphical plots associated with the subset of the parameters used in the gesture definition a must pass window defining an area a plot of sensor data must pass for gesture recognition.

4. The method of claim 1, further including receiving user input defining for one of the graphical plots associated with the subset of the parameters used in the gesture definition a must not pass window defining an area a plot of sensor data must not pass for gesture recognition.

5. The method of claim 1, further including receiving additional user input modifying the stored definition of at least one of the starting position, the ending position, the maximum value, and the minimum value, whereby the additional user input adjusts the gesture definition based on aesthetic or other guidelines.

6. The method of claim 1, further comprising performing training of an additional performer comprising:

receiving an additional sets of raw sensor data each corresponding to the differing parameters of the performance of the gesture by the additional performer; and

comparing the additional sets of the raw sensor data with each of the graphical plots for the subset of the parameters in the gesture definition to verify at least one of the starting position, the ending position, the maximum value, and the minimum value are matched.

7. The method of claim 6, wherein the performing training of an additional performer further comprises providing visual or audio feedback to the additional performer indicating whether the performance of the gesture satisfied the gesture definition.

8. The method of claim 7, wherein the providing of the visual or audio feedback comprises a plotting of the additional sets of raw sensor data relative to the graphical plots of each of the sets of raw sensor data used to create the gesture definition.

9. The method of claim 1, further comprising defining a length of time for performing the gesture and wherein the length of time is stored with the gesture definition.

10. The method of claim 1, wherein the sets of raw sensor data include data in a number of channels ranging from 1 to 13 channels, whereby the differing parameters of the performance of the gesture number from 1 to 13.

11. A system for authoring gestures for use by a gesture-based controller, comprising:

a computer with a processor managing a display and a memory device;

a gesture monitoring assembly including a set of sensors sampling data during performance of a gesture by a performer; and

a gesture authoring tool run by the processor to;

during the performance of the gesture by the performer, collect sets of sensor data from each of the sensors;

display a graphical user interface (GUI) on a display device, wherein the GUI includes a graphical plot of each of the sets of sensor data;

receiving and storing in the memory device user input identifying a subset of the sensors to include in a gesture definition; and

for each of the graphical plots for the subset of the sensors included in the gesture definition, receiving and storing, in the memory device as part of the gesture definition, user input defining at least one of a starting position, an ending position, a maximum value, and a minimum value.

12. The system of claim 11, wherein the user input defining the starting position, the ending position, the maximum value, and the minimum value includes positioning a set of bounding boxes in each of the graphical plots of the sets of the raw sensor data for the subset of the parameters and sizing each of the bounding boxes.

13. The system of claim 11, wherein the gesture authoring tool further acts to receive user input defining for one of the graphical plots associated with the subset of the parameters used in the gesture definition at least one of a must pass window and a must not pass window defining, respectively, an area a plot of sensor data must pass for gesture recognition and an area of a plot of sensor data must not pass for gesture recognition.

14. The system of claim 11, wherein the gesture authoring tool receives additional user input modifying the stored definition of at least one of the starting position, the ending position, the maximum value, and the minimum value, whereby the additional user input adjusts the gesture definition based on aesthetic or other guidelines.

15. The system of claim 11, wherein the gesture authoring tool performs regression analysis including comparing the gesture definition with at least one previously generated gesture definition to determine whether the gesturing monitoring assembly can accurately distinguish performances of a corresponding pair of gestures and, when not, to flag the gesture definition and the at least one previously generated gesture definition for modification.

16. A gesture authoring method, comprising:

collecting a set of data from a sensor during a data collection window;

receiving first user input defining a gesture performance time period with a start time and an end time for performance of gesture by a performer during the collecting;

creating a first gesture definition including the set of data from the sensor during the gesture performance time period;

receiving second user input marking in the first gesture definition at least one of a start position, an end position, a maximum value, and a minimum value; and

creating a second gesture definition comprising a modified version of the first gesture definition including the marking from the second user input.

17. The method of claim 16, further comprising, prior to the receiving of the second user input, generating a graphical user input including a graphical plot of the set of data from the sensor and displaying the graphical user input on a monitor of a computer.

18. The method of claim 17, wherein the receiving of second user input providing the marking includes displaying bounding boxes for each of the start position, the end position, the maximum value and the minimum value and receiving user input placing each of the bounding boxes relative to the graphical plot and sizing each of the bounding boxes.

19. The method of claim 17, further comprising receiving third user input modifying the marking from the second user input and, in response, creating a third gesture definition comprising a modified version of the second gesture definition.

20. The method of claim 16, further comprising using the second gesture definition for training a performer including collecting additional data from the sensor and comparing the additional data to the graphical plot to verify at least the marked subset of the start position, the end position, the maximum value and the minimum value are matched.