Method and system for characterizing physical retail spaces by determining the demographic composition of people in the physical retail spaces utilizing video image analysis

• US 7,987,111 B1
• Filed: 10/26/2007
• Issued: 07/26/2011
• Est. Priority Date: 10/30/2006
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Sharma, Rajeev, Mummareddy, Satish, Hershey, Jeff, Moon, Hankyu
• Primary Examiner(s): Jarrett, Scott L
• Application Number:
  US11/978,021
• Time in Patent Office: 1,369 Days
• Field of Search: 705/7.29
• US Class Current: 705/7.29
• CPC Class Codes: A61B 2503/20 Workers
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data

Abstract

The present invention is a method and system for characterizing physical space based on automatic demographics measurement, using a plurality of means for capturing images and a plurality of computer vision technologies. The present invention is called demographic-based retail space characterization (DBR). Although the disclosed method is described in the context of retail space, the present invention can be applied to any physical space that has a restricted boundary. In the present invention, the physical space characterization can comprise various types of characterization depending on the objective of the physical space, and it is one of the objectives of the present invention to provide the automatic demographic composition measurement to facilitate the physical space characterization. The demographic classification and composition measurement of people in the physical space is performed automatically based on a novel usage of a plurality of means for capturing images and a plurality of computer vision technologies on the captured visual information of the people in the physical space. The plurality of computer vision technologies can comprise face detection, person tracking, body parts detection, and demographic classification of the people, on the captured visual information of the people in the physical space.

Claims

1. A method for characterizing a physical retail space based on automatic measurement of a demographic composition of a plurality of shoppers in the physical retail space, comprising the following steps of:

a) dividing said physical retail space into a plurality of sub-regions, the sub-regions comprising at least one of aisles, entry points, exit points, or predefined areas of said physical retail space,

b) capturing a plurality of input video images of said plurality of shoppers in said physical retail space and said retail space sub-regions over a specified period of time by a plurality of cameras,

c) automatic measuring demographic information of each shopper in said plurality of shoppers in said plurality of input video images using a computer, said demographic information comprising age, gender and ethnicity information,

d) determining statistics of said demographic information and measuring said demographic composition of said plurality of shoppers in said physical retail space and said retail space sub-regions, said demographic composition comprising categories defining entity with a group of products, a group of product types, spaces, areas or display of a group of products, and

e) characterizing said physical retail space and said retail space sub-regions based on said demographic measuring and said demographic statistics, said physical retail space characterization comprising at least one of the following;

a characterization of said demographic composition of shoppers in said physical retail space and retail space sub-regions;

or

a characterization of said physical retail space or said retail space sub-regions by product match or media match based on said demographic composition;

or

a characterization of said physical retail space and retail space sub-regions according to a media effectiveness measurement based on said demographic composition measuring; and

f) outputting, by a computer, said physical retail space characterization of said physical retail space and retail space sub-regions.

2. The method according to claim 1, wherein the method further comprises a step of repeating the steps from a) to d) for a plurality of physical spaces in a network of spaces,

whereby said network of spaces comprises a large number of stores.

3. The method according to claim 1, wherein the method further comprises a step of taking samples from the universe of the plurality of shoppers for measuring the statistics of demographic information,

wherein the step of characterizing said physical space uses the sample statistics of demographic information.

4. The method according to claim 1, wherein the method further comprises a step of dividing said physical space into a plurality of sub-regions and characterizing said physical space per sub-region based on the demographic statistics and composition in each sub-region of said plurality of sub-regions in said physical space.

5. The method according to claim 1, wherein the method further comprises a step of re-zoning said physical space based on the demographic characterization in said physical space.

6. The method according to claim 1, wherein the method further comprises a step of measuring a pattern of changes in demographics due to changes in the product match or media match as a part of the physical space characterization.

7. The method according to claim 1, wherein the method further comprises a step of measuring media effectiveness based on the demographic characterization of said physical space,

whereby the measurement of the media effectiveness is achieved by analyzing what drives the plurality of shoppers in a certain demographic composition to buy a certain category of products.

8. The method according to claim 1, wherein the method further comprises a step of enabling actual verification of demographic composition for the physical space characterization, by measuring the demographic composition for those who actually visited the physical space.

9. The method according to claim 1, wherein the method further comprises a step of applying the physical space characterization to different servicing strategies, so that the application attracts a specific demographic group, based on the actually measured demographic data, by matching needs for the specific demographic group.

10. The method according to claim 1, wherein the method further comprises a step of providing web-based reporting of an aggregated demographic characterization data,

whereby the aggregated demographic characterization data comprises visualization of the aggregated demographic characterization data.

11. The method according to claim 1, wherein the method further comprises a step of positioning said plurality of cameras near an entrance and exit or a plurality of entrances and exits in said physical space,

wherein an aggregated measurement of the demographic composition at all of the entrances and exits of the physical space provides a solution for characterizing the entire physical space.

12. The method according to claim 1, wherein the method further comprises a step of positioning said plurality of cameras in the vicinity of a plurality of aisles in said physical space,

wherein the physical space characterization is performed to characterize the aisles, and wherein the demographic composition measurement at the vicinity of the plurality of aisles provides information for finer level of interest of the demographics for particular products at the specific aisle area.

13. The method according to claim 1, wherein the method further comprises a step of placing said plurality of cameras in a place within a predefined open space where a plurality of the fields of view from said plurality of cameras cover said predefined open space, wherein the physical space characterization is performed to characterize the predefined open space.

14. The method according to claim 1, wherein the method further comprises a step of automatically delimiting a predefined area in the views of said plurality of cameras, so that the method selects and processes only the faces of said plurality of shoppers dwelling in the delimited predefined area,

wherein placement of the cameras is dependent on the kind of physical space that needs to be characterized.

15. The method according to claim 1, wherein the method further comprises a step of positioning said plurality of cameras in the vicinity of a plurality of entry points in said physical space,

wherein the physical space characterization is performed to characterize the entry points, and wherein an entry point is defined as an area where people'"'"'s traffic exists although the area is not a hot spot in a physical space.

16. The method according to claim 1, wherein the method further comprises a step of segmenting body images of each shopper of said plurality of shoppers in the captured plurality of input images, so that the physical floor position of said each shopper is estimated,

whereby a lowest end of the segmented body image is marked as the footfall of said each shopper,

whereby when the footfall of said each shopper is within the region of interest in the views of said plurality of cameras, the shopper'"'"'s facial image and body image are sent to a demographics classification module, and

whereby when the footfall of said each shopper is not within the region of interest in the views of said plurality of cameras, the shopper'"'"'s facial image and body image are not sent to a demographics classification module.

17. The method according to claim 1, wherein the method further comprises a step of segmenting out the regions having human skin-like color pixel values in the captured plurality of input images.

18. The method according to claim 1, wherein the method further comprises a step of detecting faces of said plurality of shoppers in a crowded and complex indoor scene and a step of providing the locations and sizes of detected faces.

19. The method according to claim 1, wherein the method further comprises a step of detecting locations of facial features and estimating the position, size, and orientation of a facial image detected from a face detection scheme.

20. The method according to claim 1, wherein the method further comprises a step of aligning a facial image detected from a face detection step using a facial geometry estimation.

21. The method according to claim 20, wherein the method further comprises a step of automatically classifying the gender of a shopper in said plurality of shoppers based on a gender score of an aligned face image.

22. The method according to claim 20, wherein the method further comprises a step of automatically classifying the ethnicity of a shopper in said plurality of shoppers among multiple designated classes of ethnicity based on an ethnicity score of an aligned facial image.

23. The method according to claim 20, wherein the method further comprises a step of automatically classifying the age of a shopper in said plurality of shoppers among multiple designated classes of age groups based on an age score of an aligned facial image.

24. An apparatus for characterizing a physical retail space based on automatic measurement of a demographic composition of a plurality of shoppers in the physical retail space, comprising:

a) means for dividing said physical retail space into a plurality of sub-regions, the sub-regions comprising at least one of aisles, entry points, exit points, or predefined areas of said physical retail space,

b) means for capturing a plurality of input video images of said plurality of shoppers in said physical retail space and said retail space sub-regions over a specified period of time by a plurality of means for capturing images,

c) means for automatic measuring demographic information of each shopper in said plurality of shoppers in said plurality of input video images using a means for control and processing, said demographic information comprising age, gender and ethnicity information,

d) means for determining statistics of said demographic information and measuring said demographic composition of said plurality of shoppers in said physical retail space and said retail space sub-regions, said demographic composition comprising categories defining entity with a group of products, a group of product types, spaces, areas or display of a group of products, and

e) means for characterizing said physical retail space and said retail space sub-regions based on said demographic measuring and said demographic statistics, said physical retail space characterization comprising at least one of the following;

a characterization of said demographic composition of shoppers in said physical retail space and retail space sub-regions;

or

a characterization of said physical retail space or said retail space sub-regions by product match or media match based on said demographic composition;

or

a characterization of said physical retail space and retail space sub-regions according to a media effectiveness measurement based on said demographic composition measuring; and

f) means for outputting said physical retail space characterization of said physical retail space and retail space sub-regions.

25. The apparatus according to claim 24, wherein the apparatus further comprises means for repeatedly using the means from a) to d) for a plurality of physical spaces in a network of spaces,

whereby said network of spaces comprises a large number of stores.

26. The apparatus according to claim 24, wherein the apparatus further comprises means for taking samples from the universe of the plurality of shoppers for measuring the statistics of demographic information,

wherein the means for characterizing said physical space uses the sample statistics of demographic information.

27. The apparatus according to claim 24, wherein the apparatus further comprises means for dividing said physical space into a plurality of sub-regions and characterizing said physical space per sub-region based on the demographic statistics and composition in each sub-region of said plurality of sub-regions in said physical space.

28. The apparatus according to claim 24, wherein the apparatus further comprises means for re-zoning said physical space based on the demographic characterization in said physical space.

29. The apparatus according to claim 24, wherein the apparatus further comprises means for measuring a pattern of changes in demographics due to changes in the product match or media match as a part of the physical space characterization.

30. The apparatus according to claim 24, wherein the apparatus further comprises means for measuring media effectiveness based on the demographic characterization of said physical space,

whereby the measurement of the media effectiveness is achieved by analyzing what drives the plurality of shoppers in a certain demographic composition to buy a certain category of products.

31. The apparatus according to claim 24, wherein the apparatus further comprises means for enabling actual verification of demographic composition for the physical space characterization, by measuring the demographic composition for those who actually visited the physical space.

32. The apparatus according to claim 24, wherein the apparatus further comprises means for applying the physical space characterization to different servicing strategies, so that the application attracts a specific demographic group, based on the actually measured demographic data, by matching needs for the specific demographic group.

33. The apparatus according to claim 24, wherein the apparatus further comprises means for providing web-based reporting of an aggregated demographic characterization data, whereby the aggregated demographic characterization data comprises visualization of the aggregated demographic characterization data.

34. The apparatus according to claim 24, wherein the apparatus further comprises means for positioning said plurality of means for capturing images near an entrance and exit or a plurality of entrances and exits in said physical space, wherein an aggregated measurement of the demographic composition at all of the entrances and exits of the physical space provides a solution for characterizing the entire physical space.

35. The apparatus according to claim 24, wherein the apparatus further comprises means for positioning said plurality of means for capturing images in the vicinity of a plurality of aisles in said physical space, wherein the physical space characterization is performed to characterize the aisles, and wherein the demographic composition measurement at the vicinity of the plurality of aisles provides information for finer level of interest of the demographics for particular products at the specific aisle area.

36. The apparatus according to claim 24, wherein the apparatus further comprises means for placing said plurality of means for capturing images in a place within a predefined open space where a plurality of the fields of view from said plurality of means for capturing images cover said predefined open space, wherein the physical space characterization is performed to characterize the predefined open space.

37. The apparatus according to claim 24, wherein the apparatus further comprises means for automatically delimiting a predefined area in the views of said plurality of means for capturing images, so that the apparatus selects and processes only the faces of said plurality of shoppers dwelling in the delimited predefined area, wherein placement of the means for capturing images is dependent on the kind of physical space that needs to be characterized.

38. The apparatus according to claim 24, wherein the apparatus further comprises means for positioning said plurality of means for capturing images in the vicinity of a plurality of entry points in said physical space,

wherein the physical space characterization is performed to characterize the entry points, and wherein an entry point is defined as an area where people'"'"'s traffic exists although the area is not a hot spot in a physical space.

39. The apparatus according to claim 24, wherein the apparatus further comprises means for segmenting body images of each shopper of said plurality of shoppers in the captured plurality of input images, so that the physical floor position of said each shopper is estimated, whereby a lowest end of the segmented body image is marked as the footfall of said each shopper, whereby when the footfall of said each shopper is within the region of interest in the views of said plurality of means for capturing images, the shopper'"'"'s facial image and body image are sent to a demographics classification module, and

whereby when the footfall of said each shopper is not within the region of interest in the views of said plurality of means for capturing images, the shopper'"'"'s facial image and body image are not sent to a demographics classification module.

40. The apparatus according to claim 24, wherein the apparatus further comprises means for segmenting out the regions having human skin-like color pixel values in the captured plurality of input images.

41. The apparatus according to claim 24, wherein the apparatus further comprises means for detecting faces of said plurality of shoppers in a crowded and complex indoor scene and means for providing the locations and sizes of detected faces.

42. The apparatus according to claim 24, wherein the apparatus further comprises means for detecting locations of facial features and estimating the position, size, and orientation of a facial image detected from a face detection scheme.

43. The apparatus according to claim 24, wherein the apparatus further comprises means for aligning a facial image detected from a face detection using a facial geometry estimation.

44. The apparatus according to claim 43, wherein the apparatus further comprises means for automatically classifying the gender of a shopper in said plurality of shoppers based on a gender score of an aligned face image.

45. The apparatus according to claim 43, wherein the apparatus further comprises means for automatically classifying the ethnicity of a shopper in said plurality of shoppers among multiple designated classes of ethnicity based on an ethnicity score of an aligned facial image.

46. The apparatus according to claim 43, wherein the apparatus further comprises means for automatically classifying the age of a shopper in said plurality of shoppers among multiple designated classes of age groups based on an age score of an aligned facial image.