Method and system for hand pose detection

• US 10,852,840 B2
• Filed: 12/09/2019
• Issued: 12/01/2020
• Est. Priority Date: 12/15/2015
• Status: Active Grant

• Current Assignee: Purdue Research Foundation (The Trustees of Purdue University (d/b/a Purdue University))
• Original Assignee: Purdue Research Foundation (The Trustees of Purdue University (d/b/a Purdue University))
• Inventors: Sinha, Ayan, Choi, Chiho, Choi, Joon Hee, Ramani, Karthik
• Primary Examiner(s): Lee, Laurence J
• Application Number:
  US16/707,936
• Publication Number:
  US 20200225761A1
• Time in Patent Office: 358 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A method for hand pose identification in an automated system includes providing depth map data of a hand of a user to a first neural network trained to classify features corresponding to a joint angle of a wrist in the hand to generate a first plurality of activation features and performing a first search in a predetermined plurality of activation features stored in a database in the memory to identify a first plurality of hand pose parameters for the wrist associated with predetermined activation features in the database that are nearest neighbors to the first plurality of activation features. The method further includes generating a hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters and performing an operation in the automated system in response to input from the user based on the hand pose model.

Claims

1. A system for computer human interaction comprising:

a depth camera configured to generate depth map data of a hand of a user;

an output device;

a memory storing at least a first neural network, and a recommendation engine; and

a processor operatively connected to the depth camera, the output device, and the memory, the processor being configured to;

receive depth map data of a hand of a user from the depth camera;

generate, using the first neural network, a first plurality of activation features based at least in part on the depth map data;

use the recommendation engine to perform a first search in a predetermined plurality of activation features stored in a database of the memory to identify a first plurality of hand pose parameters for the wrist associated with predetermined activation features in the database that are nearest neighbors to the first plurality of activation features;

generate a hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters; and

generate an output with the output device in response to input from the user based at least in part on the hand pose model.

2. The system of claim 1, wherein the processor is further configured to:

identify a second neural network stored in the memory based upon the first plurality of hand pose parameters, the second neural network being one neural network in a plurality of neural networks stored in the memory trained to classify features corresponding to joint angles of a first finger of the hand; and

generate a second plurality of activation features using the depth map data of the hand of the user and the second neural network;

perform a second search, using the recommendation engine, in the predetermined plurality of activation features stored in the database to identify a second plurality of hand pose parameters for the one finger associated with predetermined activation features in the database that are nearest neighbors to the second plurality of activation features; and

generate the hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters and the second plurality of hand pose parameters.