Using high-level attributes to guide image processing

• US 8,879,831 B2
• Filed: 12/15/2011
• Issued: 11/04/2014
• Est. Priority Date: 12/15/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kohli, Pushmeet, Shotton, Jamie Daniel Joseph, Sun, Min
• Primary Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US13/327,464
• Publication Number:
  US 20130156298A1
• Time in Patent Office: 1,055 Days
• Field of Search: 382/159, 382/153, 382/226
• US Class Current: 382/159
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Using high-level attributes to guide image processing is described. In an embodiment high-level attributes of images of people such as height, torso orientation, body shape, gender are used to guide processing of the images for various tasks including but not limited to joint position detection, body part classification, medical image analysis and others. In various embodiments one or more random decision forests are trained using images where global variable values such as player height are known in addition to ground-truth data appropriate for the image processing task concerned. In some examples sequences of images are used where global variables are static or vary smoothly over the sequence. In some examples one or more trained random decision forests are used to find global variable values as well as output values for the task concerned such as joint positions or body part classes.

Claims

1. A computer-implemented method of processing an image having at least one high-level attribute, the method comprising:

accessing at least one random decision forest arranged to take into account the high-level attribute by virtue of having been trained using training images for which values of at least one global variable expressing the high-level attribute are known;

for each of a plurality of image elements of the image, pushing the image element through the random decision forest to obtain candidate output values and, information about the certainty of the candidate output values;

using the candidate output values and certainty information to select a set of candidate output values as either;

a most confident set of candidate output values over the global variable and the candidate output values jointly;

or a most confident set of candidate output values given an estimate of the global variable;

or a weighted combination of the candidate output values given an estimate of the global variable;

the method further comprising;

receiving the image as part of a stream of images, calculating a running average value of the global variable of the image and using the calculated value to influence how the candidate output values are obtained, receiving subsequent images from the stream of images and repeating the calculating and influencing for each subsequent image;

or

receiving a sequence of images comprising the image and wherein receiving the random decision forest comprises receiving a single random decision forest trained to jointly estimate the global variable and the candidate output values over all images in the sequence.

2. A method as claimed in claim 1 comprising accessing a plurality of random decision forests each having been trained using training images for which the global variable is in a different range;

and either selecting one of the random decision forests for pushing the image elements through using knowledge of a value of the global variable of the image, or pushing the image elements through all the random decision forests and making a weighted combination of candidate output values from each of the random decision forests.

3. A method as claimed in claim 2 comprising calculating a value of the global variable of the image.

4. A method as claimed in claim 3 comprising obtaining the candidate output values by either selecting one of a plurality of bins according to knowledge of a value of the global variable of the image;

or making a weighted combination of candidate output values from each of the bins.

5. A method as claimed in claim 4 comprising pushing the image elements through the random decision forest to obtain both candidate output values and a global variable value.

6. A method as claimed in claim 1 wherein accessing the random decision forest comprises accessing a plurality of bins of candidate output values associated with a leaf node of the random decision forest, each bin holding candidate output values associated with a specified range of values of the global variable.

7. A method as claimed in claim 1 wherein accessing the random decision forest comprises accessing a random decision forest which has been trained to jointly estimate the global variable and the candidate output values.

8. A method as claimed in claim 1 comprising receiving the image as part of a stream of images;

calculating a running average value of the global variable of the image and using the calculated value to influence how the candidate output values are obtained;

receiving subsequent images from the stream of images and repeating the calculating and influencing for each subsequent image.

9. A method as claimed in claim 1 comprising receiving a sequence of images comprising the image and wherein receiving the random decision forest comprises receiving a single random decision forest trained to jointly estimate the global variable and the candidate output values over all images in the sequence.

10. A method as claimed in claim 1, selecting a set of candidate output values comprising aggregating a sample of candidate output values to form an overall candidate value for an image element.

11. A computer-implemented method of processing an image having at least one high-level attribute, the method comprising:

training at least one random decision forest using training images for which values of at least one global variable expressing the high-level attribute are known, the training being such that the random decision forest is able to take into account the high-level attribute, each of the at least one random decision forests having been trained using images for which the global variable is in a different range;

for each of a plurality of image elements of the image;

pushing the image element through one of the at least one random decision forests, the random decision forest being selected using knowledge of a value of the global variable of the image, to obtain candidate output values and information about the certainty of the candidate output values;

or

pushing the image element through all of the at least one random decision forests and making a weighted combination of candidate output values from each of the random decision forests;

using the candidate output values and certainty information to select a set of candidate output values as either;

a most confident set of candidate output values over the global variable and the candidate output values jointly;

or a most confident set of candidate output values given an estimate of the global variable;

or a weighted combination of the candidate output values given an estimate of the global variable; and

learning weights for the weighted combination of candidate output values from a validation set of training images.

12. A method as claimed in claim 11 comprising training the random decision forest by at least populating a plurality of bins of candidate output values associated with each leaf node of the random decision forest, each bin holding candidate output values associated with a specified range of values of the global variable.

13. A method as claimed in claim 11 comprising training the random decision forest to jointly estimate the global variable and the candidate output values.

14. A method as claimed in claim 11 comprising training the random decision forest to jointly estimate the global variable and the candidate output values over all images in a sequence of images over which the global variable is static or varies smoothly.

15. A method as claimed in claim 11, training the random decision forest further comprising filtering candidate output values to remove outliers using threshold values learned using a validation set of data.

16. An image processing system comprising:

at least one random decision forest arranged to take into account a high-level attribute of an image by virtue of having been trained using training images for which values of at least one global variable expressing the high-level attribute are known;

a processor, arranged to push each of a plurality of image elements of the image through the random decision forest to obtain candidate output values and information about the certainty of the candidate output values;

the processor being arranged to use the candidate output values and certainty information to select a set of candidate output values as either;

a most confident set of candidate output values over the global variable and the candidate output values jointly;

or a most confident set of candidate output values given an estimate of the global variable;

or a weighted combination of the candidate output values given an estimate of the global variable;

the processor being further arranged to;

receive the image as part of a stream of images;

calculate a running average value of the global variable of the image and use the calculated value to influence how the candidate output values are obtained;

receive subsequent images from the stream of images and repeat the calculating and influencing for each subsequent image;

or

receive a sequence of images comprising the image and wherein the random decision forest comprises a single random decision forest trained to jointly estimate the global variable and the candidate output values over all images in the sequence.

17. An image processing system as claimed in claim 16 comprising a plurality of random decision forests each having been trained using training images for which the global variable is in a different range;

the processor being arranged to either select one of the random decision forests for pushing the image elements through using knowledge of a value of the global variable of the image, or to push the image elements through all the random decision forests and make a weighted combination of candidate output values from each of the random decision forests.

18. A system as claimed in claim 16 wherein the random decision forest comprises a plurality of bins of candidate output values associated with each leaf node of the random decision forest, each bin holding candidate output values associated with a specified range of values of the global variable.

19. A system as claimed in claim 16 wherein the random decision forest has been trained to jointly estimate the global variable and the candidate output values.

20. A system as claimed in claim 16 wherein the random decision forest has been trained to jointly estimate the global variable and the candidate output values over each frame of a sequence of images.