Hand tracking using a passive camera system

• US 10,706,584 B1
• Filed: 05/18/2018
• Issued: 07/07/2020
• Est. Priority Date: 05/18/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Ye, Yuting, Wang, Robert Y., Twigg, Christopher David, Han, Shangchen
• Primary Examiner(s): Lemieux, Ian L
• Application Number:
  US15/984,276
• Time in Patent Office: 781 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system tracks a user'"'"'s hands by processing image data captured using one or more passive cameras. The system includes one or more passive cameras, such as color or monochrome cameras, and a hand tracking unit. The hand tracking unit receives the image data of the user'"'"'s hand from the one or more passive cameras. The hand tracking unit determines, based on applying the image data to a neural network model, heat maps indicating locations of hand features of a hand shape model. The hand tracking unit may include circuitry that implements the neural network model. The neural network model is trained using image data from passive cameras, depth cameras, or both. The hand tracking unit determines a hand pose of the user'"'"'s hand based on the locations of the hand features of the hand shape model. The hand pose may be used as a user input, or to render the hand for a display, such as in a head-mounted display.

Claims

1. A system comprising:

one or more passive cameras configured to generate image data of a user'"'"'s hand; and

a circuitry configured to;

receive the image data of the user'"'"'s hand from the one or more passive cameras;

determine, based on applying the image data to a neural network model, heat maps indicating locations of hand features of a hand shape model, each of the hand features being associated with one or more degrees of freedom defining a range of motion, each heat map indicating a location of a hand feature; and

determine a hand pose of the user'"'"'s hand by fitting the locations of the hand features determined from the heat maps to a state of the hand shape model as constrained by the one or more degrees of freedom of each of the hand features.

2. The system of claim 1, wherein the one or more passive cameras include at least one of a color camera and a monochrome camera.

3. The system of claim 1, wherein the one or more passive cameras include a plurality of passive cameras at different locations and the image data includes image data sets generated by the plurality of passive cameras.

4. The system of claim 1, wherein the neural network model is trained using depth image data generated by a depth camera.

5. The system of claim 1, wherein the neural network model is trained using image data sets generated by a plurality of passive cameras.

6. The system of claim 1, wherein the circuitry is further configured to determine a location of at least one hand feature that is occluded in the image data using a kinematic constraint, and wherein the hand pose is determined based on the location of the at least one hand feature.

7. The system of claim 1, wherein the hand shape model is a user hand shape model generated by fitting vertices of a base hand shape model with corresponding vertices of input hand shapes of poses of the user'"'"'s hand.

8. The system of claim 1, wherein the hand shape model is a user hand shape model generated by fitting one or more input hand shapes to a set of principal component hand shape models defining a hand shape variation space.

9. The system of claim 1, further comprising a head-mounted display (HMD) configured to present a rendered image of the user'"'"'s hand, the rendered image generated based on the hand pose.

10. A method, comprising:

receiving image data of a user'"'"'s hand generated by one or more passive cameras;

determining, based on applying the image data to a neural network model, heat maps indicating locations of hand features of a hand shape model, each of the hand features being associated with one or more degrees of freedom defining a range of motion, each heat map indicating a location of a hand feature; and

determining a hand pose by fitting the locations of the hand features determined from the heat maps to a state of the hand shape model as constrained by the one or more degrees of freedom of each of the hand features.

11. The method of claim 10, wherein the one or more passive cameras include at least one of a color camera and a monochrome camera.

12. The method of claim 10, wherein the one or more passive cameras include a plurality of passive cameras at different locations and the image data includes image data sets generated by the plurality of passive cameras.

13. The method of claim 10, wherein the neural network model is trained using depth image data generated by a depth camera.

14. The method of claim 10, wherein the neural network model is trained using image data sets generated by a plurality of passive cameras.

15. The method of claim 10, further comprising determining a location of at least one hand feature that is occluded in the image data using a kinematic constraint, and wherein the hand pose is determined based on the location of the at least one hand feature.

16. The method of claim 10, wherein the hand shape model is a user hand shape model generated by fitting vertices of a base hand shape model with corresponding vertices of input hand shapes of poses of the user'"'"'s hand.

17. The method of claim 10, wherein the hand shape model is a user hand shape model generated by fitting one or more input hand shapes to a set of principal component hand shape models defining a hand shape variation space.

18. The method of claim 10, further including presenting a rendered image of the user'"'"'s hand on a head-mounted display (HMD), the rendered image being generated based on the hand pose.

19. A head-mounted display (HMD), comprising:

a circuitry configured to;

receive image data of a user'"'"'s hand generated by one or more passive cameras;

determine, based on applying the image data to a neural network model, heat maps indicating locations of hand features of a hand shape model, each of the hand features being associated with one or more degrees of freedom defining a range of motion, each heat map indicating a location of a hand feature; and

determine a hand pose of the user'"'"'s hand by fitting the locations of the hand features determined from the heat maps to a state of the hand shape model as constrained by the one or more degrees of freedom of each of the hand features.

20. The HMD of claim 19, wherein the one or more passive cameras include at least one of a color camera and a monochrome camera.