Object Recognition Using Textons and Shape Filters

• US 20080075361A1
• Filed: 09/21/2006
• Published: 03/27/2008
• Est. Priority Date: 09/21/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Shotton, Jamie, Winn, John, Rother, Carsten
• Granted Patent: US 7,840,059 B2
• Time in Patent Office: Days
• US Class Current: 382/155
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components

Abstract

Given an image of structured and/or unstructured objects we automatically partition it into semantically meaningful areas each labeled with a specific object class. We use a novel type of feature which we refer to as a shape filter. Shape filters enable us to capture some or all of shape, texture and appearance context information. A shape filter comprises one or more regions of arbitrary shape, size and position within a bounding area of an image, paired with a specified texton. A texton comprises information describing the texture of a patch of surface of an object. In a training process we select a sub-set of possible shape filters and incorporate those into a conditional random field model of object classes. That model is then used for object detection and recognition.

Claims

1. A method comprising;

(i) receiving a plurality of training images of objects;

(ii) receiving an object label map for each training image, each object label map comprising a label for each image element specifying one of a plurality of object classes;

(iii) accessing a dictionary of textons, each texton comprising information describing the texture of a patch of surface of an object;

(iv) forming a texton map for each training image using the dictionary of textons, each texton map comprising, for each image element a label indicating a texton;

(v) for each texton map computing a plurality of feature responses by applying a different shape filter for each feature response;

(vi) selecting a sub-set of the shape filters used in computing the feature responses by forming a multi-class classifier to classify image elements into the object classes on the basis of at least some of the feature responses; and

(vii) forming an object detection and recognition system using the selected shape filters.

2. A method as claimed in claim 1 wherein each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied, that bounding area being movable within the image and having an area about ¼

to ¾

of the image area.

3. A method as claimed in claim 1 wherein each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied and a plurality of substantially randomly sized and positioned rectangular regions within that bounding area.

4. A method as claimed in claim 1 wherein the step of accessing the dictionary of textons comprises forming the dictionary of textons using the training images.

5. A method as claimed in claim 1 wherein the multi-class classifier is formed using a joint boosting process.

6. A method as claimed in claim 5 wherein the joint boosting process comprises iteratively building the multi-class classifier as a sum of decision stumps comprising thresholds applied to the feature responses, each decision stump being shared between a plurality of object classes.

7. A method as claimed in claim 1 wherein the step of forming the object detection and recognition system comprises forming a conditional random field model of object classes that model comprising definitions of the conditional probability of object class labels given an image on the basis of a plurality of potentials comprising at least shape-texture potentials using the shape filter, texton pairs.

8. A method as claimed in claim 7 which further comprises learning parameters for the conditional random field model by dividing the model into pieces and training each piece independently using a training method incorporating fixed powers.

9. A method as claimed in claim 6 wherein the conditional random field model is also formed on the basis of color potentials arranged to represent the color distribution of the instances of an object class in a particular image.

10. A method as claimed in claim 6 wherein the conditional random field model is also formed on the basis of location potentials.

11. A method as claimed in claim 6 wherein the conditional random field model is also formed on the basis of edge potentials.

12. A method as claimed in claim 6 which further comprises using the conditional random field model to agree an overall object labeling for a previously unseen image and using an inference process to infer an object label map from the agreed overall object labeling.

13. A method comprising:

(i) receiving a plurality of training images of objects;

(ii) receiving an object label map for each training image, each object label map comprising a label for each image element specifying one of a plurality of object classes;

(iii) accessing a dictionary of textons, each texton comprising information describing the texture of a patch of surface of an object;

(iv) forming a texton map for each training image using the dictionary of textons, each texton map comprising, for each image element a label indicating a texton;

(v) for each texton map computing a plurality of feature responses by applying a different shape filter for each feature response;

(vi) selecting a sub-set of the shape filters used in computing the feature responses by forming a multi-class classifier to classify image elements into the object classes on the basis of at least some of the feature responses; and

(vii) forming an object label map for a previously unseen image using the selected shape filters.

14. A method as claimed in claim 13 wherein the step of forming the object label map for a previously unseen image comprises forming a conditional random field model having shape-texture potentials, edge potentials, color potentials and location potentials.

15. A method as claimed in claim 14 which further comprises learning parameters for the shape-texture potentials using a joint boosting process with substantially random selection of shape filters.

16. A method as claimed in claim 14 which further comprises learning parameters for the color potentials using an iterative conditional mode method.

17. One or more computer readable media having computer executable instructions for performing steps comprising:

(i) receiving a plurality of training images of objects;

(ii) receiving an object label map for each training image, each object label map comprising a label for each image element specifying one of a plurality of object classes;

(iii) accessing a dictionary of textons, each texton comprising information describing the texture of a patch of surface of an object;

(iv) forming a texton map for each training image using the dictionary of textons, each texton map comprising, for each image element a label indicating a texton;

(v) for each texton map computing a plurality of feature responses by applying a different shape filter and texton pair for each feature response;

(vi) selecting a sub-set of the shape filters used in computing the feature responses by forming a multi-class classifier to classify image elements into the object classes on the basis of at least some of the feature responses; and

(vii) forming an object label map for a previously unseen image using the selected shape filters.

18. The computer readable media of claim 17 wherein the computer executable instructions are arranged to apply the shape filters such that each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied, that bounding area being movable within the image and having an area about ¼

to ¾

of the image area.

19. The computer readable media of claim 17 wherein the computer executable instructions are arranged to apply the shape filters such that each shape filter comprises a bounding area having an area of about ½

the image area.

20. The computer readable media of claim 17 wherein the computer executable instructions are arranged to apply the shape filters such that each shape filter comprises a bounding area defining an area of an image within which the shape filter is applied and a plurality of substantially randomly sized and positioned rectangular regions within that bounding area