Audience measurement system and method

• US 5,771,307 A
• Filed: 12/21/1995
• Issued: 06/23/1998
• Est. Priority Date: 12/15/1992
• Status: Expired due to Term

• Current Assignee: Nielsen Company (US) LLC (RR Donnelley & Sons Company)
• Original Assignee: Nielsen Media Research, Inc. (Nielsen Holdings plc)
• Inventors: Lee, Morris, Thomas, William L., Bernard, Bruce, Lu, Daozheng, Zhang, Jia, Shagrin, Ceril T.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): MEHTA, BHAVESH M
• Application Number:
  US08/576,467
• Time in Patent Office: 915 Days
• Field of Search: 382/115-119, 382/103, 340/825.34, 235/382, 902/3-6, 902/23-25, 348/1, 348/2, 348/143, 348/150, 348/156, 348/161
• US Class Current: 382/116
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: H04H 60/45 for identifying users
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015

Abstract

In a passive identification apparatus for identifying a predetermined individual member of a television viewing audience in a monitored viewing area, a video image of a monitored viewing area is captured. A template matching score is provided for an object in the video image. A Eigenface recognition score is provided for an object in the video image. These scores may be provided by comparing objects in the video image to reference files. The template matching score and the Eigenface recognition score are fused to form a composite identification record from which a viewer may be identified. Body shape matching, viewer tracking, viewer sensing, and/or historical data may be used to assist in viewer identification. The reference files may be updated as recognition scores decline.

Claims

1. An apparatus for passively identifying a person in a monitored area comprising:

image capturing means for capturing video images of the monitored area;

first means for processing the video images in order to provide therefrom a first identity-indicating score relative to a person in the video images, wherein the first identity-indicating score distinguishes the person from other persons in the monitored area, and wherein the first means relies upon a first recognition methodology to derive the first identity-indicating score;

second means for processing the video images in order to provide therefrom a second identity-indicating score relative to the person in the video images, wherein the second identity-indicating score distinguishes the person from other persons in the monitored area, wherein the second means relies upon a second recognition methodology to derive the second identity-indicating score, and wherein the second recognition methodology is different from the first recognition methodology; and

,

fusing means for fusing the first and second identity-indicating scores to form a composite identification record from which the person in the video image may be identified.

2. The apparatus of claim 1 wherein the first means provides the first identity-indicating score based upon at least one face recognition methodology.

3. The apparatus of claim 1 wherein the first means comprises means for providing at least first and second face recognition scores relative to the person in the video image, the at least first and second face recognition scores being produced by corresponding different face recognition methodologies, wherein the fusing means is first fusing means, and wherein the first means further comprises second fusing means for fusing the at least first and second face recognition scores to form the first identity-indicating score.

4. The apparatus of claim 1 wherein the second means comprises body shape recognition means for providing a body shape recognition score as the second identity-indicating score, and wherein the body shape recognition score distinguishes the person from other persons in the monitored area.

5. The apparatus of claim 1 wherein the image capturing means comprises means for capturing a plurality of images and wherein the second means comprises means for providing, as the second identity-indicating score, a tracking recognition score based upon tracking of the person from image to image.

6. The apparatus of claim 1 further comprising sensing means for sensing the presence of a person in the monitored area and wherein the second means comprises means responsive to the sensing means for estimating the number of persons in the monitored area.

7. The apparatus of claim 1 wherein the person has viewing or listening habits from past times and further comprising memory means for maintaining an historical record of the viewing or listening habits of the person during selected past times and wherein the second means comprises means responsive to the memory means for estimating, from the historical record, the probability that the person is in a monitored area.

8. The apparatus of claim 1 wherein the apparatus further comprises third means for generating a third identity-indicating score based upon a statistical identity estimate derived from an historical record indicating that predetermined individuals were present in the monitored area during selected past times, and wherein the fusing means fuses the first, second, and third identity-indicating scores to form a composite identification record from which the person in the video image may be identified.

9. The apparatus of claim 1 wherein the apparatus further comprises third means for generating a third identity-indicating score based upon a body shape identity estimate resulting from a comparison of a body shape of objects in the images to reference body shapes of predetermined individuals, and wherein the fusing means fuses the first, second, and third identity-indicating scores to form a composite identification record from which the person in the video image may be identified.

10. The apparatus of claim 1 wherein the apparatus further comprises third means for generating a third identity-indicating score based upon an estimate of a number of persons present in the monitored area at the instant that the video images are captured, and wherein the fusing means fuses the first, second, and third identity-indicating scores to form a composite identification record from which the person in the video images may be identified.

11. The apparatus of claim 1 wherein the apparatus further comprises third means for generating a third identity-indicating score based upon a tracking record obtained by tracking a person from one video image to a subsequent video image, and wherein the fusing means fuses the first, second, and third identity-indicating scores to form a composite identification record from which the person in the video image may be identified.

12. A system for identifying a predetermined individual in a monitored area comprising:

means for capturing first and second current images of the monitored area at different times;

means for storing a reference facial image signature corresponding to the predetermined individual;

means for extracting, from the first and second current images of the monitored area, corresponding first and second current facial image signatures and for comparing the first and second current facial image signatures with the reference facial image signature to form corresponding first and second facial image identification records corresponding to the predetermined individual;

means for tracking a tracked individual from the first current image to the second current image; and

,

means for identifying the predetermined individual based on the first and second facial image identification records and based upon tracking the tracked individual from the first current image to the second current image.

13. The system of claim 12 wherein the means for storing a reference facial image signature corresponding to the predetermined individual comprises means for storing a first reference template facial image signature and wherein the means for extracting a current facial image signature comprises means for extracting a current image template signature and for providing a first facial identification record by comparing the current image template signature to the reference template facial template signature.

14. The system of claim 12 wherein the means for storing a reference facial image signature corresponding to the predetermined individual also comprises means for storing a reference Eigenface parameter signature and wherein the means for extracting a current facial image signature also comprises means for extracting a current Eigenface parameter signature and for providing a second facial identification record by comparing the current Eigenface parameter signature to the reference Eigenface parameter signature.

15. The system of claim 12 wherein the means for extracting comprises means for employing a polynomial discriminant function to form a composite identity-indicating record by fusing the first and second facial identification records.

16. The system of claim 12 wherein the means for tracking comprises means for locating the predetermined individual in the second current image and means for recognizing the predetermined individual from the facial identification record resulting from the first current image.

17. The system of claim 16 wherein the means for storing a reference facial image signature corresponding to the predetermined individual comprises means for storing a first reference template facial image signature, wherein the means for extracting a current facial image signature comprises means for extracting a current image template signature and for providing a first facial identification record by comparing the current image template signature to the reference template facial template signature, and wherein the facial image identification record includes the first facial image identification record.

18. The system of claim 17 wherein the means for storing a reference facial image signature corresponding to the predetermined individual also comprises means for storing a reference Eigenface parameter signature, wherein the means for extracting a current facial image signature also comprises means for extracting a current Eigenface parameter signature and for providing a second facial identification record by comparing the current Eigenface parameter signature to the reference Eigenface parameter signature, and wherein the facial image identification record includes the first and second facial image identification record.

19. The system of claim 12 further comprising:

means for storing a reference body shape image signature corresponding to the predetermined individual;

means for extracting a current body shape signature from the current image and for comparing the current body shape signature with the reference body shape image signature to form a body shape identification record corresponding to the predetermined individual; and

,

means for determining an identity of the predetermined individual from the facial image identification record and the body shape identification record.

20. The system of claim 12 wherein the predetermined individual has viewing or listening habits from past times and further comprising:

an historical record related to the viewing or listening habits of the predetermined individual during selected past times;

means for estimating, from the historical record, a probability that the predetermined individual is present in the monitored area; and

,

means for determining an identity of the predetermined individual from the facial image identification record and from the probability that the predetermined individual is present in the monitored area.

21. The system of claim 12 further comprising:

sensing means for sensing the presence of an individual in the monitored area;

means responsive to the sensing means for estimating the number of individuals present in the monitored area; and

,

means for determining an identity of the predetermined individual from the facial image identification record, from tracking the tracked individual, and from the estimated number of individuals present in the monitored area.

22. A method for identifying individuals within a monitored area comprising the following steps:

a) constructing reference facial image signatures, each of the reference facial image signatures corresponding to individuals who may be in the monitored area;

b) counting the individuals within the monitored area;

c) locating an individual in the monitored area;

d) computing a quantitative estimate that the located individual is one of the individuals who may be in the monitored area;

e) performing steps c) and d) a number of times equal to the counted individuals in the monitored area to thereby form a set of quantitative estimates;

f) determining a maximum quantitative estimate of the set of quantitative estimates;

g) comparing the maximum quantitative estimate with a predetermined threshold;

h) assigning an identity label of "guest" to an individual in the monitored area who has a corresponding quantitative estimate which is less than the threshold value; and

,

i) identifying an individual in the monitored area who has a corresponding quantitative estimate which is greater than the threshold value;

j) repeating steps f) through i) until all quantitative estimates in the set of quantitative estimates have been so processed.

23. A method of adaptively identifying a predetermined individual whose appearance changes between a first time and a second time that the predetermined individual is in a monitored area, the method comprising the following steps:

a) storing a reference facial image signature wherein the reference facial image signature corresponds to the predetermined individual;

b) capturing a first current image of the predetermined individual in the monitored area at a first time;

c) extracting a first current facial image signature from the first current image;

d) comparing the first current facial image signature with the reference facial image signature to thereby generate a first score representing a degree of agreement between the first current facial image signature and the reference facial image signature;

e) capturing a second current image of the predetermined individual in the monitored area at a second time;

f) extracting a second current facial image signature from the second current image;

g) comparing the second current facial image signature with the reference facial image signature to thereby generate a second score representing a degree of agreement between the second current facial image signature and the reference facial image signature;

h) comparing the first and second scores to determine a difference therebetween; and

,

i) replacing the reference facial image signature corresponding to the predetermined individual with the second current facial image signature if the difference between the first and second scores exceeds a predetermined value.

24. The method of claim 23 further comprising the following steps:

a) storing a reference body shape signature wherein the reference body shape signature corresponds to the predetermined individual;

b) extracting a first current body shape signature from the first current image;

c) comparing the first current body shape signature with the reference body shape signature to thereby generate a third score representing a degree of agreement between the first current body shape signature and the reference body shape signature;

d) selecting the larger of the first and third scores as a first maximum score, the first maximum score corresponding to the predetermined individual;

e) extracting a second current body shape signature from the second current image;

f) comparing the second current body shape signature with the reference body shape signature to thereby generate a fourth score representing a degree of agreement between the second current body shape signature and the reference body shape signature; and

,

g) selecting the larger of the second and fourth scores as a second maximum score.

25. The method of claim 24 wherein the step of comparing the first and second scores comprises the step of comparing the first and second maximum scores to determine a difference therebetween, and wherein the step of replacing the reference facial image signature comprises the steps of (i) replacing the reference facial image signature corresponding to the predetermined individual with the second current facial image signature if the difference between the first and second maximum scores exceeds a predetermined value, and (ii) replacing the reference body shape signature corresponding to the predetermined individual with the second current body shape signature if the difference between the first and second maximum scores exceeds a predetermined value.

26. An image recognition apparatus for identifying a predetermined individual from a set of unknown individuals who may be in a monitored area comprising:

means for storing a first library of image signatures formed from a first set of images, each image signature of the first library of image signatures relating to a set of known individuals;

means for generating reference identification parameters from the first library and from a set of initial video images, each of the reference identification parameters corresponding respectively to a member of the set of known individuals, the reference identification parameters being generated according to a first methodology;

means for generating reference facial image signatures from the set of initial video images, each of the reference facial image signatures corresponding respectively to a member of the set of known individuals, the reference facial image signatures being generated according to a second methodology;

means for storing the reference identification parameters and the reference facial image signatures;

means for capturing a current image of unknown individuals in the monitored area;

means for generating a current identification parameter related to an unknown individual in the current image, the current identification parameter being generated according to the first methodology;

means for comparing the current identification parameter with the reference identification parameters to thereby generate a first set of scores wherein each score of the first set of scores represents a degree of agreement between the current identification parameter and a corresponding one of the reference identification parameters;

means for generating a current facial image signature related to the unknown individual in the current image, the current facial image signature being generated according to the second methodology;

means for comparing the current facial image signature with the reference facial image signatures to thereby generate a second set of scores wherein each score of the second set of scores represents a degree of agreement between the current facial image signature and a corresponding one of the reference facial image signatures;

means for forming a set of composite scores from the first and second sets of scores wherein each composite score is a single score derived from a score of the first set of scores and a corresponding score of the second set of scores; and

means for selecting which of the composite scores has a maximum value to identify the predetermined individual.

27. The image recognition apparatus of claim 26 further comprising;

means for storing a library of reference body shape signatures, each of the reference body shape signatures corresponding to a known individual;

means for generating a current body shape signature from the current image; and

,

means for comparing the current body shape signature with the reference body shape signatures to thereby generate a set of body shape scores wherein each score of the set of body shape scores represents a degree of agreement between the current body shape signature and a corresponding one of the reference body shape signatures.

28. An apparatus for identifying an individual in a monitored area comprising:

image capturing means for capturing a video image of the monitored area;

first means for providing a first identity-indicating score relative to the individual in the video image, the first identity-indicating score being based upon a face recognition methodology;

means for interrogating the individual and for requiring the individual to supply a manually supplied identity datum;

manual input means for manually supplying the manually supplied identity datum;

first fusing means for fusing the identity-indicating score and the manually supplied identity datum;

second means for providing a second identity-indicating score relative to the individual, the second identity-indicating score being based upon a second recognition methodology which is different from the face recognition methodology; and

second fusing means for fusing the first and second identity-indicating scores to form a composite passive identity-indicating score to be operated on by the first fusing means.