Method and system for automatically measuring and forecasting the demographic characterization of customers to help customize programming contents in a media network

• US 8,706,544 B1
• Filed: 05/23/2007
• Issued: 04/22/2014
• Est. Priority Date: 05/25/2006
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Videomining Corporation
• Inventors: Sharma, Rajeev, Mummareddy, Satish, Hershey, Jeff, Moon, Hankyu
• Primary Examiner(s): Loftis, Johnna
• Application Number:
  US11/805,321
• Time in Patent Office: 2,526 Days
• Field of Search: 705/7.31
• US Class Current: 705/7.31
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0202 Market predictions or forecasting for commercial activities

Abstract

The present invention is a method and system for forecasting the demographic characterization of customers to help customize programming contents on each means for playing output of each site of a plurality of sites in a media network through automatically measuring, characterizing, and estimating the demographic information of customers that appear in the vicinity of each means for playing output. The analysis of demographic information of customers is performed automatically based on the visual information of the customers, using a plurality of means for capturing images and a plurality of computer vision technologies on the visual information. The measurement of the demographic information is performed in each measured node, where the node is defined as means for playing output. Extrapolation of the measurement characterizes the demographic information per each node of a plurality of nodes in a site of a plurality of sites of a media network. The forecasting and customization of the programming contents is based on the characterization of the demographic information.

Claims

1. A method for forecasting demographic characterization of customers on each means for playing output of each site of a plurality of sites in a media network through automatically measuring, characterizing, and estimating demographic information of customers that appear in the vicinity of each means for playing output, comprising the following steps of:

a) capturing a plurality of input images of the customers in the vicinity of each measured means for playing output over a period of time by means for capturing images,

b) processing the plurality of input images in order to measure the demographic information of the customers using a computer vision based demographic segmentation technology in a computer as an analysis of the customers,

c) transferring the demographic information measurement data from each computer to a central server,

d) extrapolating the measurement of the demographic information in order to characterize the demographic information for each means for playing output of each site,

e) forecasting the demographic characterization of customers individually for each means for playing output of each site,

f) querying the demographic characterization and the forecasting of demographic characterization and representing the demographic information measurement from the plurality of sites in the media network using a data representation means, and

g) customizing programming contents of each means for playing output in the media network, wherein the extrapolation, the characterization, and the forecasting of demographic characterization for said each means for playing output are processed in the central server, whereby said demographic characterization is used to help customize programming contents in said media network, and

whereby the demographic information comprises age, gender, and ethnicity information.

2. The method according to claim 1, wherein the method further comprises a step of sampling a predefined number of means for playing output in said media network,

wherein the measurement of demographic information of said customers is performed for the sample means for playing output, and the sample measurement is extrapolated to other means for playing output.

3. The method according to claim 1, wherein the method further comprises a step of measuring the demographic information through the Support Vector Machine (SVM) based image classifier,

wherein the SVM is utilized as a comprehensive solution that captures video frames, detects customer faces in the frames, tracks the faces individually, corrects the pose of the faces, and classifies the demographics profiles of the customers, including gender and ethnicity.

4. The method according to claim 1, wherein the method further comprises a step of reporting the characterization and forecasting results to a decision maker in said each site or in said media network through queries,

whereby the step helps said decision maker to better understand said customers in the vicinity of said means for playing output.

5. The method according to claim 1, wherein the method further comprises a step of utilizing the historical data and additional input information for forecasting said programming contents.

6. The method according to claim 1, wherein the method further comprises steps of detecting the customers'"'"' faces in said plurality of input images, individually tracking the detected faces by keeping their assigned identities, and measuring the demographic composition of the audience,

wherein the demographic composition measurement is used for forecasting the demographic characterization of customers.

7. The method according to claim 1, wherein the method further comprises a step of detecting the locations of the facial features on detected faces in said plurality of input images, and thereby estimating the position, size, and orientation of said detected faces,

wherein the position, size, and orientation are used to correct the face image so that the facial features are placed on predefined locations.

8. The method according to claim 7, wherein the method further comprises a step of building an appearance model of a customer across image frames containing the same customer'"'"'s face, aligned using the estimates from the step of estimating the position, size, and orientation of said detected faces.

9. The method according to claim 1, wherein the method further comprises a step of segmenting out the regions having skin-like color pixel values in said plurality of input images.

10. The method according to claim 1, wherein the method further comprises a step of managing face tracks to find a correct match between the face tracks history and a new input face, using geometric match score and appearance match score, wherein the geometric match score is based on the difference in the position, size, and the time between the new face and the last face in the track, and

wherein the appearance match score measures a difference between the model face appearance stored in the track and the new face.

11. The method according to claim 1, wherein the method further comprises a step of automatically classifying the gender of a customer based on the accumulated evidence from the gender scores of all the faces in the customer track.

12. The method according to claim 1, wherein the method further comprises a step of automatically classifying the ethnicity of a customer among multiple designated classes of ethnicity based on the accumulated evidence from the ethnicity scores of all the faces in the customer track,

wherein the ethnicity scores are calculated using the same number of learning machines as the number of ethnic groups, and

wherein each learning machine is tuned to a given ethnic group.

13. The method according to claim 1, wherein the method further comprises a step of extrapolating the customers'"'"' demographic measurement from sampled nodes to all the other nodes within the same node type category based on the node types and the clustering of sites based on POS data and store layout,

wherein attribute information for the node types comprises node cluster category, site, and location of the node in the site.

14. The method according to claim 1, wherein the method further comprises a step of forecasting the customers'"'"' demographics based on growth rate patterns in various levels of detail and a historical measurement of the customers'"'"' demographics, whereby the growth rate patterns comprise month of the year patterns, week of the month patterns, day of the week patterns, and hour of the day patterns.

15. An apparatus for forecasting demographic characterization of customers on each means for playing output of each site of a plurality of sites in a media network through automatically measuring, characterizing, and estimating demographic information of customers that appear in the vicinity of each means for playing output, comprising:

a) means for capturing images that captures a plurality of input images of the customers in the vicinity of each measured means for playing output over a period of time,

b) a computer for processing the plurality of input images in order to measure the demographic information of the customers as an analysis of the customers,

c) a central server that performs the following steps of;

extrapolating the measurement of the demographic information in order to characterize the demographic information for each means for playing output of each site,

forecasting the demographic characterization of customers individually for each means for playing output of each site,

querying the demographic characterization and the forecasting of demographic characterization and representing the demographic information measurement from the plurality of sites in the media network using a data representation means, and

customizing programming contents of each means for playing output in the media network,

wherein the demographic information measurement data is transferred from each computer to the central server,

whereby said demographic characterization is used to help customize programming contents in said media network, and

whereby the demographic information comprises age, gender, and ethnicity information.

16. The apparatus according to claim 15, wherein the apparatus further comprises a computer for sampling a predefined number of means for playing output in said media network,

wherein the measurement of demographic information of said customers is performed for the sample means for playing output, and the sample measurement is extrapolated to other means for playing output.

17. The apparatus according to claim 15, wherein the apparatus further comprises a computer for measuring the demographic information through the Support Vector Machine (SVM) based image classifier,

wherein the SVM is utilized as a comprehensive solution that captures video frames, detects customer faces in the frames, tracks the faces individually, corrects the pose of the faces, and classifies the demographics profiles of the customers, including gender and ethnicity.

18. The apparatus according to claim 15, wherein the apparatus further comprises a computer for reporting the characterization and forecasting results to a decision maker in said each site or in said media network through queries,

whereby the measurement and estimation information help said decision maker to better understand said customers in the vicinity of said means for playing output.

19. The apparatus according to claim 15, wherein the apparatus further comprises a computer for utilizing the historical data and additional input information for forecasting said programming contents.

20. The apparatus according to claim 15, wherein the apparatus further comprises a computer for detecting the customers'"'"' faces in said plurality of input images, individually tracking the detected faces by keeping their assigned identities, and measuring the demographic composition of the audience,

wherein the demographic composition measurement is used for forecasting the demographic characterization of customers.

21. The apparatus according to claim 15, wherein the apparatus further comprises a computer for detecting the locations of the facial features on detected faces in said plurality of input images, and thereby estimating the position, size, and orientation of said detected faces,

wherein the position, size, and orientation are used to correct the face image so that the facial features are placed on predefined locations.

22. The apparatus according to claim 21, wherein the apparatus further comprises a computer for building an appearance model of a customer across image frames containing the same customer'"'"'s face, aligned using the estimates from the step of estimating the position, size, and orientation of said detected faces.

23. The apparatus according to claim 15, wherein the apparatus further comprises means for segmenting out the regions having skin-like color pixel values in said plurality of input images.

24. The apparatus according to claim 15, wherein the apparatus further comprises a computer for managing face tracks to find a correct match between the face tracks history and a new input face, using geometric match score and appearance match score,

wherein the geometric match score can be based on the difference in the position, size, and the time between the new face and the last face in the track, and

wherein the appearance match score measures the difference between the model face appearance stored in the track and the new face.

25. The apparatus according to claim 15, wherein the apparatus further comprises a computer for automatically classifying the gender of a customer based on the accumulated evidence from the gender scores of all the faces in the customer track.

26. The apparatus according to claim 15, wherein the apparatus further comprises a computer for automatically classifying the ethnicity of a customer among multiple designated classes of ethnicity based on the accumulated evidence from the ethnicity scores of all the faces in the customer track,

wherein the ethnicity scores are calculated using the same number of learning machines as the number of ethnic groups, and

wherein each learning machine is tuned to a given ethnic group.

27. The apparatus according to claim 15, wherein the apparatus further comprises a computer for extrapolating the customers'"'"' demographic measurement from sampled nodes to all the other nodes within the same node type category based on the node types and the clustering of sites based on POS data and store layout,

wherein attribute information for the node types comprises node cluster category, site, and location of the node in the site.

28. The apparatus according to claim 15, wherein the apparatus further comprises a computer for forecasting the customers'"'"' demographics based on growth rate patterns in various levels of detail and a historical measurement of the customers'"'"' demographics,

whereby the growth rate patterns comprise month of the year patterns, week of the month patterns, day of the week patterns, and hour of the day patterns.