Method of training neural networks for hand pose detection

• US 10,503,270 B2
• Filed: 06/10/2019
• Issued: 12/10/2019
• Est. Priority Date: 12/15/2015
• Status: Active Grant

• Current Assignee: Purdue Research Foundation (The Trustees of Purdue University (d/b/a Purdue University))
• Original Assignee: Purdue Research Foundation (The Trustees of Purdue University (d/b/a Purdue University))
• Inventors: Sinha, Ayan, Choi, Chiho, Choi, Joon Hee, Ramani, Karthik
• Primary Examiner(s): Lee, Laurence J
• Application Number:
  US16/436,588
• Publication Number:
  US 20190310716A1
• Time in Patent Office: 183 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A method for training a hierarchy of trained neural networks for hand pose detection includes training a first neural network to generate a first plurality of activation features that classify an input depth map data corresponding to a hand based on a wrist angle of the hand, the training using a plurality of depth maps of a hand with predetermined wrist angles as inputs to the first neural network during the training, and storing the first neural network in a memory after the training for use in classifying an additional depth map corresponding to a hand based on an angle of a wrist of the hand in the additional depth map.

Claims

1. A method for training a hierarchy of trained neural networks for hand pose detection comprising:

training, with a processor, a first neural network to generate a first plurality of activation features that classify an input depth map data corresponding to a hand based on a wrist angle of the hand, the training using a plurality of depth maps of a hand with predetermined wrist angles as inputs to the first neural network during the training; and

storing, with the processor, the first neural network in a memory after the training for use in classifying an additional depth map corresponding to a hand based on an angle of a wrist of the hand in the additional depth map.

2. The method of claim 1 further comprising:

generating, with the processor, the plurality of training depth maps prior to training of the first neural network, the generating comprising;

generating, with the processor, a synthetic hand model in a predetermined pose based on hand model data stored in the memory and a predetermined plurality of constraints for an angle of at least one joint in the synthetic hand model;

generating, with the processor, a depth map of the hand model in the predetermined pose; and

storing, with the processor, the depth map with the plurality of depth maps in the memory.

3. The method of claim 2 further comprising:

applying, with the processor, Gaussian noise to the depth map of the hand model in the predetermined pose prior to storing the depth map in the memory.

4. The method of claim 1 further comprising:

providing, with the processor, one depth map in the plurality of depth maps in the training data to the first neural network after the training to generate a first plurality of activation features corresponding to the one depth map; and

storing, with the processor, the first plurality of activation features and a hand pose parameter of the hand in the one depth map in a database in the memory to map the first plurality of activation features to the hand pose parameter.

5. The method of claim 4, wherein the hand pose parameter is a wrist angle parameter for the hand in the one depth map.

6. The method of claim 1 further comprising:

identifying, with the processor, a first subset of the plurality of depth maps that correspond to a first predetermined range of wrist angles;

training, with the processor, a second neural network to generate a second plurality of activation features that classify the input depth map data corresponding to the hand based on an angle of at least one joint in a finger of the hand, the training using the first subset of the plurality of depth maps of the hand as inputs to the second neural network during the training; and

storing, with the processor, the second neural network in the memory after the training for use in classifying the additional depth map corresponding to the hand based on an angle of the at least one joint in the finger of the hand in the additional depth map.

7. The method of claim 6 further comprising:

identifying, with the processor, a second subset of the plurality of depth maps that correspond to a second predetermined range of wrist angles, the second predetermined range of wrist angles being different than the first predetermined range of wrist angles;

training, with the processor, a third neural network to generate a third plurality of activation features that classify the input depth map data corresponding to the hand based on an angle of the at least one joint in the finger of the hand, the training using the second subset of the plurality of depth maps of the hand as inputs to the third neural network during the training; and

storing, with the processor, the third neural network in the memory after the training for use in classifying the additional depth map corresponding to the hand based on an angle of the at least one joint in the finger of the hand in the additional depth map.