Scenario-specific body-part tracking

• US 9,489,042 B2
• Filed: 10/13/2014
• Issued: 11/08/2016
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Craig, Robert Matthew
• Primary Examiner(s): Shah, Uptal
• Application Number:
  US14/513,128
• Publication Number:
  US 20150029097A1
• Time in Patent Office: 757 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/241 relating to the classification model, e.g. parametric or non-parametric approaches
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

A human subject is tracked within a scene of an observed depth image supplied to a general-purpose body-part tracker. The general-purpose body-part tracker is retrained for a specific scenario. The general-purpose body-part tracker was previously trained using supervised machine learning to identify one or more general-purpose parameters to be used by the general-purpose body-part tracker to track a human subject. During a retraining phase, scenario data is received that represents a human training-subject performing an action specific to a particular scenario. One or more special-purpose parameters are identified from the processed scenario data. The special-purpose parameters are selectively used to augment or replace one or more general-purpose parameters if the general-purpose body-part tracker is used to track a human subject performing the action specific to the particular scenario.

Claims

1. A computing system, comprising:

a camera input subsystem to receive depth-camera recordings;

one or more memory devices holding instructions to retrain a general-purpose body part tracker;

one or more processors configured to execute the instructions to;

receive via the camera input subsystem, a set of different instances of scenario data, each instance of scenario data including one or more depth images representing a depth-camera recording of a human training-subject performing an action specific to a particular scenario;

iterate over the set of different instances of scenario data with the general-purpose body-part tracker, the general-purpose body-part tracker previously trained using supervised machine learning to identify one or more general-purpose parameters to be used by the general-purpose body-part tracker to track a human subject;

identify one or more special-purpose parameters to be selectively used to augment or replace the one or more general-purpose parameters if the general-purpose body-part tracker is used to track a human subject performing the action specific to the particular scenario; and

associate the one or more special-purpose parameters with a scenario identifier that identifies the particular scenario for which the special-purpose parameters are to be selectively used by the general-purpose body-part tracker to track a human subject.

2. The computing system of claim 1, wherein the instructions are further executable by the one or more processors to:

receive via the camera input subsystem, input data representing a human subject performing an action;

receive the scenario identifier indicated as being applicable to the input data; and

analyze the input data with the general-purpose body-part tracker using the one or more special-purpose parameters associated with the scenario identifier indicated as being applicable to the input data to identify a body model representing the human subject.

3. The computing system of claim 2, wherein the one or more special-purpose parameters influence one or more of:

a weighting of one or more classifier functions used to identify the body model of the human subject;

a selection and/or branching of one or more classifier functions used to identify the body model of the human subject;

a weighting of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space; and

/or

a selection and/or branching of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space.

4. The computing system of claim 2, wherein the one or more special-purpose parameters influence selection of a three-dimensional position of one or more points defining the body model of the human subject.

5. The computing system of claim 2, wherein the instructions are further executable by the one or more processors to:

retrieve the one or more special-purpose parameters associated with the scenario identifier from a data store responsive to receiving the scenario identifier, the data store forming an element of an application program.

6. The computing system of claim 1, wherein the instructions are further executable by the one or more processors to:

receive via the camera input subsystem, input data representing a human subject performing an action;

analyze the input data with the general-purpose body-part tracker using the one or more special-purpose parameters associated with the scenario identifier; and

output an indication that the input data is applicable to the particular scenario, the indication including the scenario identifier.

7. The computing system of claim 1, wherein the instructions to retrain the general-purpose body-part tracker form part of the general-purpose body-part tracker or form a separate retraining shell component interfacing with the general-purpose body-part tracker at runtime.

8. The computing system of claim 1, wherein the general-purpose body part tracker is retrained to identify the one or more special-purpose parameters without supervision, and the set of different instances of scenario data is received without supervised ground-truth annotations.

9. The computing system of claim 1, wherein the instructions are further executable by the one or more processors to:

output an indication that one or more additional instances of scenario data are needed to retrain the general-purpose body-part tracker based on convergence and/or divergence of said iterating.

10. The computing system of claim 1, wherein the instructions are further executable by the one or more processors to:

receive from a currently-active application program, an indication that a currently-observed human subject is performing the action specific to the particular scenario, the indication representative of the scenario identifier; and

use the one or more special-purpose parameters associated with the scenario identifier to enhance body-part tracking of the general-purpose body-part tracker if the general-purpose body-part tracker tracks the currently-observed human performing the action specific to the particular scenario.

11. A method of retraining a body-part tracker, comprising:

receiving, from a currently-active application program, an indication of a particular scenario;

receiving a set of different instances of scenario data, each instance of scenario data including one or more depth images of a human subject performing an action specific to the particular scenario while the currently-active application program is running;

iterating over the set of different instances of scenario data with a general-purpose body-part tracker, the general-purpose body-part tracker previously trained using supervised machine learning to identify one or more general-purpose parameters to be used by the general-purpose body-part tracker to track a human subject; and

identifying one or more special-purpose parameters to be subsequently used by the currently-active application program to augment or replace the one or more general-purpose parameters, the one or more special-purpose parameters based on iteration over the set of different instances of scenario data received while the currently-active application program was running.

12. The method of claim 11, further comprising:

outputting an indication that one or more additional instances of scenario data are needed based on convergence and/or divergence of the iteration.

13. The method of claim 11, further comprising:

associate the one or more special-purpose parameters with a scenario identifier that identifies the particular scenario for which the special-purpose parameters are to be selectively used by the general-purpose body-part tracker to track a human subject;

wherein during runtime of the currently-active application program, the currently-active application program indicates applicability of the one or more special-purpose parameters by the scenario identifier.

14. The method of claim 11, wherein the one or more special-purpose parameters influence one or more of:

a weighting of one or more classifier functions used to identify a body model of the human subject; and

/or

a selection and/or branching of one or more classifier functions used to identify the body model of the human subject;

a weighting of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space; and

/or

a selection and/or branching of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space.

15. The method of claim 11, wherein the one or more special-purpose parameters influence selection of a three-dimensional position of one or more points defining a body model of the human subject.

16. The method of claim 11, further comprising:

performing said iterating and said identifying of the one or more special-purpose parameters within a runtime of the general-purpose body-part tracker or within a separate retraining shell component interfacing with the general-purpose body-part tracker at runtime.

17. The method of claim 11, further comprising:

receiving the set of different instances of scenario data without supervised ground-truth annotations; and

identifying the one or more special-purpose parameters without supervised ground-truth annotations.

18. A computing system for tracking a human subject, comprising:

a computing device configured to;

receive via a camera input subsystem of the computing device, input data including a depth image representing a depth-camera recording of a human subject performing an action while a currently-active application program is running;

receive a scenario identifier from the currently-active application program indicated as being applicable to the input data;

retrieve one or more special-purpose parameters associated with the scenario identifier from a data store; and

analyze the input data with a general-purpose body-part tracker using the one or more special-purpose parameters that augment or replace one or more general-purpose parameters to identify a body model representing the human subject performing the action for use by the currently-active application program;

the general-purpose body-part tracker previously trained using unsupervised machine learning to identify the one or more special-purpose parameters after being initially trained using supervised machine learning to identify the one or more general-purpose parameters.

19. The computing system of claim 18, wherein the computing device is further configured to, during previous training using unsupervised machine learning:

receive via the camera input subsystem, a set of different instances of scenario data, each instance of scenario data representing a human training-subject performing an action specific to a particular scenario;

identify, based on the set of different instances of scenario data, one or more special-purpose parameters to be selectively used to augment or replace the one or more general-purpose parameters if the general-purpose body-part tracker is used to track the human subject performing the action specific to the particular scenario; and

associate the one or more special-purpose parameters with the scenario identifier that identifies the particular scenario for which the special-purpose parameters are to be selectively used by the general-purpose body-part tracker to track the human subject.

20. The computing system claim 18, wherein the one or more special-purpose parameters influence one or more of:

a weighting of one or more classifier functions used to identify the body model of the human subject;

a selection and/or branching of one or more classifier functions used to identify the body model of the human subject;

a weighting of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space; and

/or

a selection and/or branching of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space.