Automatic detection and aggregation of demographics and behavior of people

• US 20080109397A1
• Filed: 12/17/2007
• Published: 05/08/2008
• Est. Priority Date: 07/29/2002
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Jung, Namsoon, Sharma, Rajeev, Moon, Hankyu
• Granted Patent: US 8,351,647 B2
• Time in Patent Office: Days
• US Class Current: 707/1
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

The present invention is a system and framework for automatically measuring and correlating visual characteristics of people and accumulating the data for the purpose of demographic and behavior analysis. The demographic and behavior characteristics of people are extracted from a sequence of images using techniques from computer vision. The demographic and behavior characteristics are combined with a timestamp and a location marker to provide a feature vector of a person at a particular time at a particular location. These feature vectors are then accumulated and aggregated automatically in order to generate a data set that can be statistically analyzed, data mined and/or queried.

Claims

1. A method for detecting and aggregating demographics and behavior of a person or a plurality of people, comprising the following steps of:

a) detecting a plurality of images for a person or a plurality of people in the vicinity of a fixture with a single or a plurality of visual sensing devices,

b) processing at least an image from the detected plurality of images in order to obtain demographic information and behavior features of the person or the plurality of people automatically,

c) computing a timestamp and a location marker for the obtained demographic information and behavior features, and

d) combining the demographic information and behavior features with the timestamp and the location marker into a feature vector.

2. The method according to claim 1, wherein the method further comprises a step of computing gender, age, ethnicity, height, skin color, hair color, time in the field of view, and time looking at the fixture.

3. The method according to claim 1, wherein the method further comprises a step of accumulating the feature vectors into a plurality of feature vectors for the purposes of statistical analysis, data mining, and querying based on time, location, and the demographic information and behavior features.

4. The method according to claim 1, wherein the method further comprises a step of generating a data set, wherein queries for the data set are based on a digitized interval of collected timestamps,

whereby the digitized interval comprises hour, day, week, month, and year.

5. The method according to claim 1, wherein the method further comprises a step of generating a data set, wherein queries for the data set are based on actual location marker or an aggregate of the location,

whereby the location marker is a marker that represents the actual address of a location in question.

6. The method according to claim 1, wherein the method further comprises a step of estimating the pose of a person among the plurality of people by using a plurality of learning machines, each of which is specialized to a given pose range.

7. The method according to claim 1, wherein the method further comprises a step of detecting the people'"'"' faces in the plurality of input images, tracking the detected faces by keeping their assigned identities, and recording the appearance and disappearance times and the duration of the viewing.

8. The method according to claim 7, wherein the method further comprises a step of building an appearance model of a person across image frames containing the same person'"'"'s face, by averaging the pixel values of the facial images from the same person, aligned using the estimates from the step of estimating the position, size, and orientation of the detected faces.

9. The method according to claim 7, wherein the method further comprises a step of segmenting out the regions having skin-like color pixel values in the plurality of input images.

10. The method according to claim 1, wherein the method further comprises a step of managing face tracks to find a correct match between the face tracks history and a new input face, using both the geometric, temporal, and appearance measurements.

11. The method according to claim 1, wherein the method further comprises a step of storing the people'"'"' appearance and disappearance times, the viewing duration, the number of detected faces, and the number of attentive faces as a table, and converting the data to appropriate statistics.

12. The method according to claim 1, wherein the method further comprises a step of counting a statistics of demographic information and measuring the demographic composition of the plurality of people during a predefined window of time.

13. The method according to claim 1, wherein the method further comprises a step of aligning the facial image detected from the face detection step using facial geometry estimation,

wherein the method further comprises a step of detecting the locations of the facial features on the detected faces and estimating the position, size, and orientation of the detected faces.

14. The method according to claim 13, wherein the method further comprises a step of automatically classifying the demographics of a person in the plurality of people based on the demographics score of the aligned face image.

15. An apparatus for detecting and aggregating demographics and behavior of a person or a plurality of people, comprising:

a) means for detecting a plurality of images for a person or a plurality of people in the vicinity of a fixture with a single or a plurality of visual sensing devices,

b) means for processing at least an image from the detected plurality of images in order to obtain demographic information and behavior features of the person or the plurality of people automatically,

c) means for computing a timestamp and a location marker for the obtained demographic information and behavior features, and

d) means for combining the demographic information and behavior features with the timestamp and the location marker into a feature vector.

16. The apparatus according to claim 15, wherein the apparatus further comprises means for computing gender, age, ethnicity, height, skin color, hair color, time in the field of view, and time looking at the fixture.

17. The apparatus according to claim 15, wherein the apparatus further comprises means for accumulating the feature vectors into a plurality of feature vectors for the purposes of statistical analysis, data mining, and querying based on time, location, and the demographic information and behavior features.

18. The apparatus according to claim 15, wherein the apparatus further comprises means for generating a data set, wherein queries for the data set are based on a digitized interval of collected timestamps,

whereby the digitized interval comprises hour, day, week, month, and year.

19. The apparatus according to claim 15, wherein the apparatus further comprises means for generating a data set, wherein queries for the data set are based on actual location marker or an aggregate of the location,

whereby the location marker is a marker that represents the actual address of a location in question.

20. The apparatus according to claim 15, wherein the apparatus further comprises means for estimating the pose of a person among the plurality of persons by using a plurality of learning machines, each of which is specialized to a given pose range.

21. The apparatus according to claim 15, wherein the apparatus further comprises means for detecting the people'"'"' faces in the plurality of input images, tracking the detected faces by keeping their assigned identities, and recording the appearance and disappearance times and the duration of the viewing.

22. The apparatus according to claim 21, wherein the apparatus further comprises means for building an appearance model of a person across image frames containing the same person'"'"'s face, by averaging the pixel values of the facial images from the same person, aligned using the estimates calculated by the usage of the means for estimating the position, size, and orientation of the detected faces.

23. The apparatus according to claim 21, wherein the apparatus further comprises means for segmenting out the regions having skin-like color pixel values in the plurality of input images.

24. The apparatus according to claim 15, wherein the apparatus further comprises means for managing face tracks to find a correct match between the face tracks history and a new input face, using both the geometric, temporal, and appearance measurements.

25. The apparatus according to claim 15, wherein the apparatus further comprises means for storing the people'"'"' appearance and disappearance times, the viewing duration, the number of detected faces, and the number of attentive faces as a table, and converting the data to appropriate statistics.

26. The apparatus according to claim 15, wherein the apparatus further comprises means for counting a statistics of demographic information and measuring the demographic composition of the plurality of persons during a predefined window of time.

27. The apparatus according to claim 15, wherein the apparatus further comprises means for aligning the facial image using facial geometry estimation, wherein the apparatus further comprises means for detecting the locations of the facial features on the detected faces and estimating the position, size, and orientation of the detected faces.

28. The apparatus according to claim 27, wherein the apparatus further comprises means for automatically classifying the demographics of a person in the plurality of persons based on the demographics score of the aligned face image.