SCENARIO-SPECIFIC BODY-PART TRACKING

• US 20140119640A1
• Filed: 10/31/2012
• Published: 05/01/2014
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Craig, Robert Matthew
• Granted Patent: US 8,867,786 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/241 relating to the classification model, e.g. parametric or non-parametric approaches
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

A human subject is tracked within a scene of an observed depth image supplied to a general-purpose body-part tracker. The general-purpose body-part tracker is retrained for a specific scenario. The general-purpose body-part tracker was previously trained using supervised machine learning to identify one or more general-purpose parameters to be used by the general-purpose body-part tracker to track a human subject. During a retraining phase, scenario data is received that represents a human training-subject performing an action specific to a particular scenario. One or more special-purpose parameters are identified from the processed scenario data. The special-purpose parameters are selectively used to augment or replace one or more general-purpose parameters if the general-purpose body-part tracker is used to track a human subject performing the action specific to the particular scenario.

Claims

1. A method of retraining a general-purpose body-part tracker, the method comprising:

receiving a set of different instances of scenario data, each instance of scenario data representing a human training-subject performing an action specific to a particular scenario;

iterating over the set of different instances of scenario data with the general-purpose body-part tracker, the general-purpose body-part tracker previously trained using supervised machine learning to identify one or more general-purpose parameters to be used by the general-purpose body-part tracker to track a human subject; and

identifying one or more special-purpose parameters to be selectively used to augment or replace the one or more general-purpose parameters if the general-purpose body-part tracker is used to track a human subject performing the action specific to the particular scenario; and

associating the one or more special-purpose parameters with a scenario identifier that identifies the particular scenario for which the special-purpose parameters are to be selectively used by the general-purpose body-part tracker to track a human subject.

2. The method of claim 1, further comprising:

receiving input data representing a human subject performing an action;

receiving the scenario identifier indicated as being applicable to the input data; and

analyzing the input data with the general-purpose body-part tracker using the one or more special-purpose parameters associated with the scenario identifier indicated as being applicable to the input data to identify a body model representing the human subject.

3. The method of claim 2, wherein the one or more special-purpose parameters influence one or more of:

a weighting of one or more classifier functions used to identify the body model of the human subject;

a selection and/or branching of one or more classifier functions used to identify the body model of the human subject;

a weighting of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space; and

/or

a selection and/or branching of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space.

4. The method of claim 2, wherein the one or more special-purpose parameters influence selection of a three-dimensional position of one or more points defining the body model of the human subject.

5. The method of claim 2, further comprising:

retrieving the one or more special-purpose parameters associated with the scenario identifier from a data store responsive to receiving the scenario identifier, the data store forming an element of an application program.

6. The method of claim 1, further comprising:

receiving input data representing a human subject performing an action;

analyzing the input data with the general-purpose body-part tracker using the one or more special-purpose parameters associated with the scenario identifier; and

outputting an indication that the input data is applicable to the particular scenario, the indication including the scenario identifier.

7. The method of claim 1, wherein each instance of scenario data includes one or more depth images representing a depth-camera recording of the human training-subject performing the action.

8. The method of claim 1, wherein retraining to identify the one or more special-purpose parameters is unsupervised, and the set of different instances of scenario data is received without supervised ground-truth annotations.

9. The method of claim 1, further comprising:

outputting an indication that one or more additional instances of scenario data are needed based on convergence and/or divergence of said iterating.

10. The method of claim 1, further comprising:

receiving from a currently-active application program, an indication that a currently-observed human subject is performing the action specific to the particular scenario, the indication representative of the scenario identifier; and

using the one or more special-purpose parameters associated with the scenario identifier to enhance body-part tracking of the general-purpose body-part tracker if the general-purpose body-part tracker tracks the currently-observed human performing the action specific to the particular scenario.

11. A storage subsystem holding instructions, the instructions defining a body-part tracker, the instructions executable by a logic subsystem to:

receive a set of different instances of scenario data, each instance of scenario data representing a human training-subject performing an action specific to a particular scenario;

iterate over the set of different instances of scenario data with the general-purpose body-part tracker, the general-purpose body-part tracker previously trained using supervised machine learning to identify one or more general-purpose parameters to be used by the general-purpose body-part tracker to track a human subject; and

identify one or more special-purpose parameters to be selectively used to augment or replace the one or more general-purpose parameters if the general-purpose body-part tracker is used to track a human subject performing the action specific to the particular scenario, the one or more special-purpose parameters based on iteration over the set of different instances of scenario data;

receive input data representing a human subject performing the action specific to the particular scenario;

receive an indication of applicability of the one or more special-purpose parameters to the input data;

analyze the input data with the general-purpose body-part tracker using the one or more special-purpose parameters indicated as being applicable to the input data to identify a body model of the human subject performing the action.

12. The storage subsystem of claim 11, further executable by the logic subsystem to:

output an indication that one or more additional instances of scenario data are needed based on convergence and/or divergence of the iteration.

13. The storage subsystem of claim 11, further executable by the logic subsystem to:

associate the one or more special-purpose parameters with a scenario identifier that identifies the particular scenario for which the special-purpose parameters are to be selectively used by the general-purpose body-part tracker to track a human subject; and

wherein the indication of applicability includes the scenario identifier.

14. The storage subsystem of claim 11, wherein the one or more special-purpose parameters influence one or more of:

a weighting of one or more classifier functions used to identify the body model of the human subject; and

/or

a selection and/or branching of one or more classifier functions used to identify the body model of the human subject;

a weighting of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space; and

/or

a selection and/or branching of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space.

15. The storage subsystem of claim 11, wherein the one or more special-purpose parameters influence selection of a three-dimensional position of one or more points defining the body model of the human subject.

16. The storage subsystem of claim 11, wherein each instance of scenario data includes one or more depth images representing a depth-camera recording of the human training-subject performing the action;

and

wherein the input data representing a human subject includes one or more depth images representing a depth-camera recording of the human subject performing the action.

17. The storage subsystem of claim 11, further executable by the logic subsystem to:

receive the set of different instances of scenario data without supervised ground-truth annotations; and

identify the one or more special-purpose parameters without supervised ground-truth annotations.

18. A method of tracking a human subject, the method comprising:

receiving input data including a depth image representing a depth-camera recording of a human subject performing an action;

receiving a scenario identifier indicated as being applicable to the input data;

retrieving one or more special-purpose parameters associated with the scenario identifier from a data store; and

analyzing the input data with a general-purpose body-part tracker using the one or more special-purpose parameters that augment or replace one or more general-purpose parameters to identify a body model representing the human subject performing the action;

the general-purpose body-part tracker previously trained using unsupervised machine learning to identify the one or more special-purpose parameters after being initially trained using supervised machine learning to identify the one or more general-purpose parameters.

19. The method of claim 18, further comprising, during previous training using unsupervised machine learning:

receiving a set of different instances of scenario data, each instance of scenario data representing a human training-subject performing an action specific to a particular scenario;

identifying, based on the set of different instances of scenario data, one or more special-purpose parameters to be selectively used to augment or replace the one or more general-purpose parameters if the general-purpose body-part tracker is used to track the human subject performing the action specific to the particular scenario; and

associating the one or more special-purpose parameters with the scenario identifier that identifies the particular scenario for which the special-purpose parameters are to be selectively used by the general-purpose body-part tracker to track the human subject.

20. The method of claim 18, wherein the one or more special-purpose parameters influence one or more of:

a weighting of one or more classifier functions used to identify the body model of the human subject;

a selection and/or branching of one or more classifier functions used to identify the body model of the human subject;

a weighting of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space; and

/or

a selection and/or branching of one or more regression functions used to identify a position of at least a portion of the body model of the human subject in two-dimensional or three-dimensional space.