Object recognition system and process for identifying people and objects in an image of a scene

• US 6,611,622 B1
• Filed: 11/23/1999
• Issued: 08/26/2003
• Est. Priority Date: 11/23/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Krumm, John
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US09/448,616
• Time in Patent Office: 1,372 Days
• Field of Search: 382/162-180
• US Class Current: 382/170
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

An object recognition system and process that identifies people and objects depicted in an image of a scene. In general, this system and process entails first creating model histograms of the people and objects that it is desired to identify in the image. Then, the image is segmented to extract regions which likely correspond to the people and objects being identified. A histogram is computed for each of the extracted regions, and the degree of similarity between each extracted region histogram and each of the model histograms is assessed. The extracted regions having a histogram that exhibits a degree of similarity to one of the model histograms which exceeds a prescribed threshold is designated as corresponding to the person or object associated with that model histogram. Finally, the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram can be stored as an additional model histogram associated with that person or object. Preferably, the foregoing general system and process is repeated for subsequently generated images of the scene, so that the identity of people and objects can be monitored over time as they move into and about the scene. In addition, preferably color images of the scene and color histograms are employed in the object recognition system and process.

Claims

1. A computer-implemented process for identifying a person or object in an image of a scene, said process comprising using a computer to perform the following process actions:

creating model histograms of people and objects that it is desired to identify in said image of the scene;

segmenting said image to extract regions which correspond to at least one person or object whose identity it is desired to determine;

for each region extracted from the image,

computing a histogram for the extracted region,

respectively assessing the degree of similarity between the extracted region histogram and each of said model histograms, and

designating each extracted region having a histogram that exhibits a degree of similarity to a one of the model histograms which exceeds a prescribed threshold as corresponding to the person or object associated with that model histogram; and

storing the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram as an additional model histogram associated with that person or object.

2. The process of claim 1, further comprising the process actions of:

designating each extracted region having a histogram that does not exhibit a degree of similarity to any of the model histograms which exceeds the prescribed threshold as corresponding to the person or object of unknown identity.

3. The process of claim 1, further comprising the process action of repeating said segmenting, computing, assessing, designating and storing actions for successive images of the scene so as to track the identity of persons and objects in the scene over time.

4. The process of claim 1, wherein the image is a color image of the scene, and the model histograms and histograms computed for each of region extracted from the color image are color histograms.

5. The process of claim 4, wherein the process action of creating model histograms of people and objects that it is desired to identify in said image of the scene, comprises the actions of:

capturing one or more model images of the people and objects that it is desired to identify;

segmenting said model image to extract model regions which correspond to each of said people and objects whose identity it is desired to determine; and

for each model region,

determining the actual colors exhibited by the pixels of the model region;

dividing the overall gamut of actual colors exhibited by the pixels of the extracted model region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

respectively assigning each pixel of the extracted model region to the quantized color category into which the actual color of the pixel falls, and

establishing a count of the number of pixels of the extracted model region assigned to the same quantized color category.

6. The process of claim 5, wherein the process action of computing a histogram for each of region extracted from the image, comprises the actions of:

determining the actual colors exhibited by the pixels of the extracted region;

dividing the overall gamut of actual colors exhibited by the pixels of the extracted region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

respectively assigning each pixel of the extracted region to the quantized color category into which the actual color of the pixel falls; and

establishing a count of the number of pixels of the extracted region assigned to the same quantized color category.

7. The process of claim 6, wherein the process actions of dividing the overall gamut of actual colors exhibited by the pixels of the extracted model regions and extracted image regions into a series of discrete color ranges, comprises the action of employing the same quantized color categories for each.

8. The process of claim 7, wherein the process action of respectively assessing the degree of similarity between the extracted region histogram and each of the model histograms for each histogram associated with an extracted region, comprises the actions of:

respectively comparing the pixel count from each quantized color category of the histogram derived from the extracted region to the pixel count from the corresponding quantized color category of each model histograms;

identifying the smaller of the two counts in each quantized color category for each pair of histograms compared; and

summing the smaller counts from each quantized color category to produce a separate similarity value for each pair of histograms compared.

9. The process of claim 8, wherein the process action of designating each extracted region having a histogram that exhibits a degree of similarity to one of the model histograms which exceeds a prescribed threshold as corresponding to the person or object associated with that model histogram, comprises the actions of:

for each extracted region, normalizing the similarity value associated with each pair of compared histograms by dividing it by a maximum possible similarity value to produce a match quality indicator;

identify the largest match quality indicator associated with each extracted region;

for each extracted region, designating the extracted region to be the person or object associated with the model histogram used in part to compute the largest match quality indicator for that region whenever the largest match quality indicator exceeds the prescribed threshold.

10. The process of claim 2, wherein the process action of designating each extracted region having a histogram that does not exhibit a degree of similarity to one of the model histograms which exceeds the prescribed threshold as corresponding to the person or object of unknown identity, comprises the actions of:

for each extracted region, normalizing the similarity value associated with each pair of compared histograms by dividing it by a maximum possible similarity value to produce a match quality indicator;

identify the largest match quality indicator associated with each extracted region;

for each extracted region, designating the extracted region to be an unknown person or object whenever the largest match quality indicator does not exceeds the prescribed threshold.

11. An object recognition system for identifying a person or object in an image of a scene, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

(a) create model histograms of people and objects that it is desired to identify in said image of the scene,

(b) segment said image to extract regions which correspond to at least one person or object whose identity it is desired to determine,

(c) compute a histogram for each of region extracted from the image,

(d) for each histogram associated with an extracted region, respectively assess the degree of similarity between the extracted region histogram and each of said model histograms,

(e) designate each extracted region having a histogram that exhibits a degree of similarity to one of the model histograms which exceeds a prescribed threshold as corresponding to the person or object associated with that model histogram, and

(f) store the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram as an additional model histogram associated with that person or object.

12. The system of claim 11, further comprising the program modules for:

designating each extracted region having a histogram that does not exhibit a degree of similarity to any of the model histograms which exceeds the prescribed threshold as corresponding to the person or object of unknown identity.

13. The system of claim 11, further comprising a program module for repeating actions (b) through (f) for successive images of the scene so as to track the identity of persons and objects in the scene over time.

14. The system of claim 11, wherein the image is a color image of the scene, and the model histograms and histograms computed for each of region extracted from the color image are color histograms.

15. The system of claim 14, wherein the program module for creating model histograms of people and objects that it is desired to identify in said image of the scene, comprises sub-modules for:

capturing one or more model images of the people and objects that it is desired to identify;

segmenting said model image to extract model regions which correspond to each of said people and objects whose identity it is desired to determine; and

for each model region,

determining the actual colors exhibited by the pixels of the model region;

dividing the overall gamut of actual colors exhibited by the pixels of the extracted model region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

respectively assigning each pixel of the extracted model region to the quantized color category into which the actual color of the pixel falls, and

establishing a count of the number of pixels of the extracted model region assigned to the same quantized color category.

16. The system of claim 15, wherein the program module for computing a histogram for each of region extracted from the image, comprises sub-modules for:

determining the actual colors exhibited by the pixels of the extracted region;

dividing the overall gamut of actual colors exhibited by the pixels of the extracted region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

respectively assigning each pixel of the extracted region to the quantized color category into which the actual color of the pixel falls; and

establishing a count of the number of pixels of the extracted region assigned to the same quantized color category.

17. The system of claim 16, wherein the sub-modules for dividing the overall gamut of actual colors exhibited by the pixels of the extracted model regions and extracted image regions into a series of discrete color ranges, comprises a sub-module for employing the same quantized color categories for each.

18. The system of claim 17, wherein the program module for respectively assessing the degree of similarity between the extracted region histogram and each of the model histograms for each histogram associated with an extracted region, comprises sub-modules for:

respectively comparing the pixel count from each quantized color category of the histogram derived from the extracted region to the pixel count from the corresponding quantized color category of each model histograms;

identifying the smaller of the two counts in each quantized color category for each pair of histograms compared; and

summing the smaller counts from each quantized color category to produce a separate similarity value for each pair of histograms compared.

19. The system of claim 18, wherein the program module for designating each extracted region having a histogram that exhibits a degree of similarity to one of the model histograms which exceeds a prescribed threshold as corresponding to the person or object associated with that model histogram, comprises sub-modules for:

for each extracted region, normalizing the similarity value associated with each pair of compared histograms by dividing it by a maximum possible similarity value to produce a match quality indicator;

identify the largest match quality indicator associated with each extracted region;

for each extracted region, designating the extracted region to be the person or object associated with the model histogram used in part to compute the largest match quality indicator for that region whenever the largest match quality indicator exceeds the prescribed threshold.

20. The system of claim 12, wherein the program module for designating each extracted region having a histogram that does not exhibit a degree of similarity to one of the model histograms which exceeds the prescribed threshold as corresponding to the person or object of unknown identity, comprises sub-modules for:

for each extracted region, normalizing the similarity value associated with each pair of compared histograms by dividing it by a maximum possible similarity value to produce a match quality indicator;

identify the largest match quality indicator associated with each extracted region;

for each extracted region, designating the extracted region to be an unknown person or object whenever the largest match quality indicator does not exceeds the prescribed threshold.

21. A computer-readable memory for causing a computer to perform an object recognition procedure for identifying a person or object in an image of a scene, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

(a) create model histograms of people and objects that it is desired to identify in said image of the scene,

(b) segment said image to extract regions which correspond to at least one person or object whose identity it is desired to determine,

(c) compute a histogram for each of region extracted from the image,

(d) for each histogram associated with an extracted region, respectively assess the degree of similarity between the extracted region histogram and each of said model histograms,

(e) designate each extracted region having a histogram that exhibits a degree of similarity to one of the model histograms which exceeds a prescribed threshold as corresponding to the person or object associated with that model histogram, and

(f) store the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram as an additional model histogram associated with that person or object.

22. The computer-readable memory of claim 21, further comprising the program modules for:

designating each extracted region having a histogram that does not exhibit a degree of similarity to any of the model histograms which exceeds the prescribed threshold as corresponding to the person or object of unknown identity.

23. The computer-readable memory of claim 21, further comprising a program module for repeating actions (b) through (f) for successive images of the scene so as to track the identity of persons and objects in the scene over time.

24. The computer-readable memory of claim 21, wherein the image is a color image of the scene, and the model histograms and histograms computed for each of region extracted from the color image are color histograms.

25. The computer-readable memory of claim 24, wherein the program module for creating model histograms of people and objects that it is desired to identify in said image of the scene, comprises sub-modules for:

capturing one or more model images of the people and objects that it is desired to identify;

segmenting said model image to extract model regions which correspond to each of said people and objects whose identity it is desired to determine; and

for each model region,

determining the actual colors exhibited by the pixels of the model region;

dividing the overall gamut of actual colors exhibited by the pixels of the extracted model region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

respectively assigning each pixel of the extracted model region to the quantized color category into which the actual color of the pixel falls, and

establishing a count of the number of pixels of the extracted model region assigned to the same quantized color category.

26. The computer-readable memory of claim 25, wherein the program module for computing a histogram for each of region extracted from the image, comprises sub-modules for:

determining the actual colors exhibited by the pixels of the extracted region;

dividing the overall gamut of actual colors exhibited by the pixels of the extracted region into a series of discrete color ranges, hereinafter referred to as quantized color categories;

respectively assigning each pixel of the extracted region to the quantized color category into which the actual color of the pixel falls; and

establishing a count of the number of pixels of the extracted region assigned to the same quantized color category.

27. The computer-readable memory of claim 26, wherein the sub-modules for dividing the overall gamut of actual colors exhibited by the pixels of the extracted model regions and extracted image regions into a series of discrete color ranges, comprises a sub-module for employing the same quantized color categories for each.

28. The computer-readable memory of claim 27, wherein the program module for respectively assessing the degree of similarity between the extracted region histogram and each of the model histograms for each histogram associated with an extracted region, comprises sub-modules for:

respectively comparing the pixel count from each quantized color category of the histogram derived from the extracted region to the pixel count from the corresponding quantized color category of each model histograms;

identifying the smaller of the two counts in each quantized color category for each pair of histograms compared; and

summing the smaller counts from each quantized color category to produce a separate similarity value for each pair of histograms compared.

29. The computer-readable memory of claim 28, wherein the program module for designating each extracted region having a histogram that exhibits a degree of similarity to one of the model histograms which exceeds a prescribed threshold as corresponding to the person or object associated with that model histogram, comprises sub-modules for:

for each extracted region, normalizing the similarity value associated with each pair of compared histograms by dividing it by a maximum possible similarity value to produce a match quality indicator;

identify the largest match quality indicator associated with each extracted region;

for each extracted region, designating the extracted region to be the person or object associated with the model histogram used in part to compute the largest match quality indicator for that region whenever the largest match quality indicator exceeds the prescribed threshold.

30. The computer-readable memory of claim 22, wherein the program module for designating each extracted region having a histogram that does not exhibit a degree of similarity to one of the model histograms which exceeds the prescribed threshold as corresponding to the person or object of unknown identity, comprises sub-modules for:

for each extracted region, normalizing the similarity value associated with each pair of compared histograms by dividing it by a maximum possible similarity value to produce a match quality indicator;

identify the largest match quality indicator associated with each extracted region;

for each extracted region, designating the extracted region to be an unknown person or object whenever the largest match quality indicator does not exceeds the prescribed threshold.

31. A computer-implemented process for identifying a person or object in an image of a scene, said process comprising using a computer to perform the following process actions:

creating model histograms of people and objects that it is desired to identify in said image of the scene;

segmenting said image to extract regions which correspond to at least one person or object whose identity it is desired to determine;

for each region extracted from the image,

computing a histogram for the extracted region,

respectively assessing the degree of similarity between the extracted region histogram and at least some of said model histograms, and

designating each extracted region having a histogram that exhibits a degree of similarity to a one of the model histograms which exceeds a prescribed threshold as corresponding to the person or object associated with that model histogram; and

storing the histogram computed for any extracted region of the image that is designated as corresponding to a person or object associated with a model histogram as an additional model histogram associated with that person or object.