Object recognition system and process for identifying people and objects in an image of a scene

• US 7,092,566 B2
• Filed: 11/18/2004
• Issued: 08/15/2006
• Est. Priority Date: 11/23/1999
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Krumm, John
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US10/992,936
• Publication Number:
  US 20050089223A1
• Time in Patent Office: 635 Days
• Field of Search: 382162-173, 382/180, 382/103, 382/305, 382/118, 382/203, 382/288, 382217-219, 382224-225, 348 141- 1416, 348 77- 78, 348169-172, 348/152, 340 582- 583, 707 1- 7
• US Class Current: 382/170
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

An object recognition system and process that identifies people and objects depicted in an image of a scene. In general, this system and process entails first creating model histograms of the people and objects that it is desired to identify in the image. Then, the image is segmented to extract regions which likely correspond to the people and objects being identified. A histogram is computed for each of the extracted regions, and the degree of similarity between each extracted region histogram and each of the model histograms is assessed. The extracted regions having a histogram that exhibits a degree of similarity to one of the model histograms which exceeds a prescribed threshold is designated as corresponding to the person or object associated with that model histogram. Finally, the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram can be stored as an additional model histogram associated with that person or object. Preferably, the foregoing general system and process is repeated for subsequently generated images of the scene, so that the identity of people and objects can be monitored over time as they move into and about the scene. In addition, preferably color images of the scene and color histograms are employed in the object recognition system and process.

Claims

1. A computer-implemented process for identifying a person or object in an image of a scene, comprising:

a creating step for creating model histograms of people and objects that it is desired to identify in said image of the scene;

a segmenting step for segmenting said image to extract regions which correspond to at least one person or object whose identity it is desired to determine;

a computing step for computing a histogram for each of region extracted from the image;

a producing step for respectively producing an indicator of the degree of similarity between each extracted region histogram and each of said model histograms;

a forming step for forming exclusive combinations of said degree of similarity indicators wherein each combination is made up of one indicator associated with each extracted region of the image and each indicator in the combination is derived from a different model histogram;

a second computing step for computing a combined degree of similarity value for each of said indicator combinations;

an identifying step for identifying the largest combined degree of similarity value; and

a designating step for designating each extracted region having a histogram associated with an individual one of the indicators used to compute the identified largest combined degree of similarity value which exceeds a prescribed threshold as corresponding to the person or object associated with the model histogram used in part to compute the individual one of the indicators.

2. The process of claim 1, further comprising the process action of repeating said segmenting, first computing, producing, forming, second computing, identifying and designating steps for successive images of the scene so as to track the identity of persons and objects in the scene over time.

3. The process of claim 1, wherein the image is a color image of the scene, and the model histograms and histograms computed for each of region extracted from the color image are color histograms.

4. The process of claim 3, wherein the creating step for creating histograms of people and objects that it is desired to identify in said image of the scene, comprises:

a capturing step for capturing one or more model images of the people and objects that it is desired to identify;

a second segmenting step for segmenting said model image to extract model regions which correspond to each of said people and objects whose identity it is desired to determine; and

performing for each model region,

a determining step for determining the actual colors exhibited by the pixels of the model region;

a dividing step for dividing the overall gamut of actual colors exhibited by the pixels of the extracted model region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

an assigning step for respectively assigning each pixel of the extracted model region to the quantized color category into which the actual color of the pixel falls, and

an establishing step for establishing a count of the number of pixels of the extracted model region assigned to the same quantized color category.

5. The process of claim 4, wherein the computing step for computing a histogram for each of region extracted from the image, comprises:

a second determining step for determining the actual colors exhibited by the pixels of the extracted region;

a second dividing step for dividing the overall gamut of actual colors exhibited by the pixels of the extracted region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

a second assigning step for respectively assigning each pixel of the extracted region to the quantized color category into which the actual color of the pixel falls; and

a second establishing step for establishing a count of the number of pixels of the extracted region assigned to the same quantized color category.

6. The process of claim 5, wherein the first and second dividing steps for dividing the overall gamut of actual colors exhibited by the pixels of the extracted model regions and extracted image regions into a series of discrete color ranges, comprise employing the same quantized color categories for each.

7. The process of claim 6, wherein the producing step for respectively producing an indicator of the degree of similarity between the extracted region histogram and each of said model histograms, comprises:

a comparing step for respectively comparing the pixel count from each quantized color category of the histogram derived from the extracted region to the pixel count from the corresponding quantized color category of each model histograms;

an identifying step for identifying the smaller of the two counts in each quantized color category for each pair of histograms compared;

a summing step for summing the smaller counts from each quantized color category to produce a separate similarity value for each pair of histograms compared; and

a normalizing step for normalizing the similarity value for each pair of histograms compared by dividing it by a maximum possible similarity value to produce a match quality indicator.

8. The process of claim 7, wherein the second computing step for computing a combined degree of similarity value for each of said indicator combinations, comprises a second summing step for summing the match quality indicators in each combination to produce a combined indicator for each combination.

9. The process of claim 1, further comprising a storing step for storing the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram as an additional model histogram associated with that person or object.

10. The process of claim 1, further comprising a second designating step for designating each extracted region having a histogram associated with an individual one of the indicators used to compute the identified largest combined degree of similarity value which does not exceed the prescribed threshold as corresponding to a person or object of unknown identity.

11. A computer-implemented process for identifying a person or object in an image of a scene, comprising:

a creating step for creating model histograms of people and objects that it is desired to identify in said image of the scene;

a dividing step for dividing the image into a plurality of cells;

an assigning step for assigning each model histogram to one of the image cells;

a segmenting step for segmenting said image to extract regions which correspond to at least one person or object whose identity it is desired to determine;

performing for each region extracted from the image,

a computing step for computing a histogram for the extracted region,

a determining step for determining the centroid of the extracted region and identifying the cell in which it resides,

for each of a set of one or more model histograms associated with the same person or object, an ascertaining step for ascertaining the closest image cell to the identified cell, including the identified cell itself, that has a histogram associated with that person or object assigned thereto,

an assessing step for respectively assessing the degree of similarity between the histogram computed for the extracted region and each of the model histograms previously ascertained to be in a cell closest to the identified cell of the extracted region,

a second determining step for determining whether the extracted region'"'"'s histogram exhibits a degree of similarity to one of the model histograms previously ascertained to be in a cell closest to the identified cell of the extracted region which exceeds a prescribed threshold, and

whenever the extracted region'"'"'s histogram exhibits a degree of similarity to one of said previously ascertained model histograms which exceeds the prescribed threshold, performing a designating step for designating the extracted region as corresponding to the person or object associated with that model histogram.

12. The process of claim 11, wherein the creating step for creating model histograms, comprises:

an obtaining step for obtaining at least one prefatory image of the scene which depict the people and objects that it is desired to identify in a subsequent image of the scene;

a dividing step for dividing each prefatory image into a plurality of cells;

a second segmenting step for segmenting each of the prefatory images to extract regions which correspond to at least one person or object whose identity is known; and

performing for each region extracted from the prefatory images,

a second computing step for computing a histogram for the extracted region to produce a model histogram associated with the person or object represented by the extracted region, and

a third determining step for determining the centroid of the extracted region and identifying the cell in which it resides; and

wherein

the assigning step for assigning each model histogram to one of the image cells comprises a step for respectively assigning each model histogram to the cell which corresponds to the cell of the prefatory image where the centroid of the extracted region associated with each model histogram was determined to reside.

13. The process of claim 11, wherein the creating step for creating model histograms, comprises:

an obtaining step for obtaining at least one model image which depict the people and objects that it is desired to identify in said image of the scene;

a second segmenting step for segmenting each of the model images to extract regions which correspond to at least one person or object whose identity is known; and

performing for each region extracted from the model images, a second computing step for computing a histogram for the extracted region to produce a model histogram associated with the person or object represented by the extracted region.

14. The process of claim 11, further comprising a second designating step for designating the selected region as corresponding to a person or object of unknown identity whenever the selected region'"'"'s histogram does not exhibit a degree of similarity to any of said previously ascertained model histograms which exceeds the prescribed threshold.

15. The process of claim 11, further comprising repeating said segmenting, dividing, assigning, segmenting, computing, determining, ascertaining, assessing, second determining, and designating steps for successive images of the scene so as to track the identity of persons and objects in the scene over time.

16. The process of claim 11, wherein the image is a color image of the scene, and the model histograms and histograms computed for each of region extracted from the color image are color histograms.

17. The process of claim 11, further comprising a storing step for storing the histogram computed for the selected region of the image that is designated as corresponding to a person or object associated with one of the model histograms as an additional model histogram associated with that person or object and assigning the newly stored histogram to the cell in which the centroid of the corresponding extracted region resides.

18. The process of claim 17, wherein the storing step for storing the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram as an additional model histogram associated with that person or object, comprises:

performing for each region extracted from the image and designated as corresponding to a person or object associated with a model histogram,

a third determining step for determining whether a histogram, associated with the person or object corresponding to the histogram derived from the extracted region, was previously stored and assigned to the cell containing the centroid of the extracted region, and

a second storing step for storing the histogram derived from the extracted region as an additional model histogram and assigning the newly stored histogram to the cell containing the centroid of the extracted region whenever it is determined that a histogram associated with the person or object corresponding to the histogram derived from the extracted region was not previously stored and assigned to the cell containing the centroid of the extracted region.

19. The process of claim 17, wherein the storing step for storing the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram as an additional model histogram associated with that person or object, comprises:

performing for each region extracted from the image and designated as corresponding to a person or object associated with a model histogram,

a third determining step for determining whether a histogram, associated with the person or object corresponding to the histogram derived from the extracted region, was previously stored and assigned to the cell containing the centroid of the extracted region,

whenever it is determined that a histogram associated with the person or object corresponding to the histogram derived from the extracted region was previously stored and assigned to the cell containing the centroid of the extracted region, performing an identifying step for identifying the time when said previously stored histogram was stored and assigned,

a second ascertaining step for ascertaining whether the previously stored histogram was stored within a prescribed threshold time frame from the current time, and

a second storing step for storing the histogram derived from the extracted region as an additional model histogram and assigning the newly stored histogram to the cell containing the centroid of the extracted region whenever it is determined that the previously stored histogram was not stored within the prescribed threshold time frame from the current time.