Method and system for automatically analyzing categories in a physical space based on the visual characterization of people

• US 8,189,926 B2
• Filed: 12/06/2007
• Issued: 05/29/2012
• Est. Priority Date: 12/30/2006
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Sharma, Rajeev, Mummareddy, Satish, Baboo, Priya, Hershey, Jeff, Jung, Namsoon
• Primary Examiner(s): Le, Brian Q
• Assistant Examiner(s): Park, Edward
• Application Number:
  US11/999,656
• Publication Number:
  US 20080159634A1
• Time in Patent Office: 1,636 Days
• Field of Search: 382/100, 382/103, 382/224, 382/231, 382/106, 382/107, 382/115, 382/153, 382/161, 382/181, 382/223, 707/722, 707/735
• US Class Current: 382/224
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

The present invention is a method and system for automatically analyzing a category in a plurality of the categories in a physical space based on the visual characterization, such as behavior analysis or segmentation, of the persons with regard to the category. The present invention captures a plurality of input images of the persons in the category by a plurality of means for capturing images. The present invention processes the plurality of input images in order to understand the shopping behavior of the persons with the sub-categories of the category and analyzes the level of engagement and decision process at the sub-category level. The processes are based on a novel usage of a plurality of computer vision technologies to analyze the visual characterization of the persons from the plurality of input images. The physical space may be a retail space, and the persons may be customers in the retail space.

Claims

1. A method for automatically analyzing a category in a physical space based on video-based behavior analysis of a plurality of persons with regard to said category in the physical space, comprising the following steps of:

a) capturing a plurality of input images of said plurality of persons by a plurality of means for capturing images in the area with regard to said category,

wherein the means for capturing images includes at least a camera,

b) processing said plurality of input images in order to process a video-based behavior analysis of each person in said plurality of persons using at least a computer that executes computer vision algorithms for said plurality of input images,

wherein the computer vision algorithms include algorithms for person tracking, hand motion detection, and demographic classification,

c) analyzing the category based on the accumulation of said video-based behavior analysis for said plurality of persons, and

d) measuring the effect after making a change at the category level,

whereby the analysis of said category facilitates an analysis of level of engagement and decision process at the category level, and

whereby said change comprises change in the quantity, price, and position of certain products in said category.

2. The method according to claim 1, wherein the method further comprises a step of measuring domain specific information in said category,

whereby said domain specific information comprises sequence of products shopped in said category, ranking of products shopped in said category, product purchase pattern, and comparison of products interacted by said plurality of persons within said category.

3. The method according to claim 1, wherein the method further comprises a step of analyzing the shopping pattern of said plurality of persons when a shelf space in shelves that are located in regards to said category is empty.

4. The method according to claim 1, wherein the method further comprises a step of calculating the frequency of replenishing a certain product in shelves that are located in regards to said category,

whereby the calculation increases the efficiency in inventory management of the product.

5. The method according to claim 1, wherein the method further comprises a step of calculating optimal size and height of shelves that are located in regards to said category based on shopping interaction analysis in regards to said shelves.

6. The method according to claim 1, wherein the method further comprises a step of calculating optimal layout for shelves that are located in regards to said category,

whereby said optimal layout provides information for optimal position of a product, optimal number of products, efficient product assortment, and optimal type of products in a shelf of said shelves and in said shelves.

7. The method according to claim 1, wherein the method further comprises a step of extracting analytic and statistical data from the category analysis that construct a map,

wherein the map comprises:

a) a traffic distribution map that shows penetration of said plurality of persons throughout the category,

b) a shopping distribution map that shows the activity of said plurality of persons throughout the category,

c) a shopping conversion map that shows traffic of said plurality of persons converted to shopping activity,

d) a sub-category level traffic distribution map,

e) a sub-category level shopping distribution map, and

f) a sub-category level shopping conversion map,

whereby the map uses color-coded symbolic expressions to differentiate the information on the map among said plurality of persons at the category.

8. The method according to claim 1, wherein the method further comprises a step of calculating quantitative measurement per sub-category,

whereby the quantitative measurement comprises a ratio between shopping interaction levels per sub-category, based on actual measurement for the shopping interaction levels of said plurality of persons.

9. The method according to claim 1, wherein the method further comprises a step of measuring dominant path at a location in said category,

whereby the dominant path measurement shows a specific decision pattern and the tendency of decision for the next path by said plurality of persons, and

whereby next region for the path direction is defined either point-level or polygon-level depending on the pre-defined granularity of the next region in said category.

10. The method according to claim 1, wherein the method further comprises a step of extracting analytic and statistical data from the category analysis that construct a shopping correlation analysis that is defined as the optimal distance measure among a plurality of sub-categories shopped.

11. The method according to claim 1, wherein the method further comprises a step of extracting analytic and statistical data from the category analysis that construct a shopping sequence analysis that is the order of a plurality of sub-categories shopped,

whereby the sequence analysis shows the order of shopping engagement by said plurality of persons.

12. The method according to claim 1, wherein the method further comprises a step of applying a set of predefined rules in a rule-base to the category analysis, comprising steps of:

a) augmenting the information units from said video-based behavior analysis for said plurality of persons, and

b) utilizing a polymorphic operator for applying said set of predefined rules to the information units.

13. The method according to claim 1, wherein the method further comprises a step of

dividing the region of interest in said category into logical grid elements by taking the perspective distortion into account through a grid generation module,

whereby said region of interest in said category comprises sub-regions on a shelf that make up a product group.

14. The method according to claim 1, wherein the method further comprises a step of matching all the grid elements that match each product group by checking for intersection of the grid element with the regions through a grid assignment module.

15. The method according to claim 1, wherein the method further comprises a step of

detecting temporal motion at every pixel in the image,

wherein the grid elements that have a minimum threshold of pixel with motion are marked as active grid elements, and

whereby a look up logic module is used to determine which region is being interacted with to find the right product group.

16. The method according to claim 1, wherein the method further comprises a step of measuring dwell time of said plurality of persons in said category.

17. The method according to claim 1, wherein the method further comprises a step of utilizing second means for capturing images to track the hand of each person in said plurality of persons in said category,

whereby the hand tracking provides sub-category level information for shopping interaction of said plurality of persons in said category.

18. The method according to claim 1, wherein the method further comprises a step of analyzing the syndicated data per category level as opposed to product level.

19. The method according to claim 1, wherein the method further comprises a step of analyzing said category in regards to the segmentation of said plurality of persons,

whereby said segmentation comprises demographic classification or shopping pattern of said plurality of persons.

20. The method according to claim 1, wherein the method further comprises a step of defining and utilizing shopping interaction levels for analyzing the behavior category of said plurality of persons in said category.

21. The method according to claim 1, wherein the method further comprises a step of analyzing the stopping power based on dwell time of said plurality of persons in said category,

whereby the stopping power comprises any logical and physical cause that stops said plurality of persons nearby a physical location in a category.

22. The method according to claim 1, wherein the method further comprises a step of utilizing prior knowledge about the physical location of said category in said physical space as an attribute for the category analysis.

23. The method according to claim 1, wherein the method further comprises a step of forming a string of behavior measurement for said plurality of persons during a window of time,

whereby each element in the string of behavior measurement consists of sub-category number, interaction level, and information with regard to the sub-category.

24. The method according to claim 1, wherein the method further comprises a step of building a model for product group interaction detection.

25. An apparatus for automatically analyzing a category in a physical space based on video-based behavior analysis of a plurality of persons with regard to said category in the physical space, comprising:

a) a plurality of means for capturing images that capture a plurality of input images of said plurality of persons,

wherein the plurality of means for capturing images are located in the area with regard to said category, and

wherein the means for capturing images includes at least a camera,

b) at least a computer that executes computer vision algorithms for said plurality of input images,

wherein the computer vision algorithms include algorithms for person tracking, hand motion detection, and demographic classification, and

wherein the computer performs the following steps of;

processing said plurality of input images in order to process a video-based behavior analysis of each person in said plurality of persons,

analyzing the category based on the accumulation of said video-based behavior analysis for said plurality of persons, and

measuring the effect after making a change at the category level,

whereby the analysis of said category facilitates an analysis of level of engagement and decision process at the category level, and

whereby said change comprises change in the quantity, price, and position of certain products in said category.

26. The apparatus according to claim 25, wherein the apparatus further comprises a computer for measuring domain specific information in said category,

whereby said domain specific information comprises sequence of products shopped in said category, ranking of products shopped in said category, product purchase pattern, and comparison of products interacted by said plurality of persons within said category.

27. The apparatus according to claim 25, wherein the apparatus further comprises a computer for analyzing the shopping pattern of said plurality of persons when a shelf space in said shelves that are located in regards to said category is empty.

28. The apparatus according to claim 25, wherein the apparatus further comprises a computer for calculating the frequency of replenishing a certain product in shelves that are located in regards to said category,

whereby the calculation increases the efficiency in inventory management of the product.

29. The apparatus according to claim 25, wherein the apparatus further comprises a computer for calculating optimal size and height of shelves that are located in regards to said category based on shopping interaction analysis in regards to said shelves.

30. The apparatus according to claim 25, wherein the apparatus further comprises a computer for calculating optimal layout for shelves that are located in regards to said category,

whereby said optimal layout provides information for optimal position of a product, optimal number of products, efficient product assortment, and optimal type of products in a shelf of said shelves and in said shelves.

31. The apparatus according to claim 25, wherein the apparatus further comprises a computer for extracting analytic and statistical data from the category analysis that construct a map,

wherein the map comprises:

a) a traffic distribution map that shows penetration of said plurality of persons throughout the category,

b) a shopping distribution map that shows the activity of said plurality of persons throughout the category,

c) a shopping conversion map that shows traffic of said plurality of persons converted to shopping activity,

d) a sub-category level traffic distribution map,

e) a sub-category level shopping distribution map, and

f) a sub-category level shopping conversion map,

whereby the map uses color-coded symbolic expressions to differentiate the information on the map among said plurality of persons at the category.

32. The apparatus according to claim 25, wherein the apparatus further comprises a computer for calculating quantitative measurement per sub-category,

whereby the quantitative measurement comprises a ratio between shopping interaction levels per sub-category, based on actual measurement for the shopping interaction levels of said plurality of persons.

33. The apparatus according to claim 25, wherein the apparatus further comprises a computer for measuring dominant path at a location in said category,

whereby the dominant path measurement shows a specific decision pattern and the tendency of decision for the next path by said plurality of persons, and

whereby next region for the path direction is defined either point-level or polygon-level depending on the pre-defined granularity of the next region in said category.

34. The apparatus according to claim 25, wherein the apparatus further comprises a computer for extracting analytic and statistical data from the category analysis that construct a shopping correlation analysis that is defined as the optimal distance measure among a plurality of sub-categories shopped.

35. The apparatus according to claim 25, wherein the apparatus further comprises a computer for extracting analytic and statistical data from the category analysis that construct a shopping sequence analysis that is the order of a plurality of sub-categories shopped,

whereby the sequence analysis shows the order of shopping engagement by said plurality of persons.

36. The apparatus according to claim 25, wherein the apparatus further comprises a computer for applying a set of predefined rules in a rule-base to the category analysis, wherein the computer performs the following steps of:

a) augmenting the information units from said video-based behavior analysis for said plurality of persons, and

b) utilizing a polymorphic operator for applying said set of predefined rules to the information units.

37. The apparatus according to claim 25, wherein the apparatus further comprises a computer for dividing the region of interest in said category into logical grid elements by taking the perspective distortion into account through a grid generation module,

whereby said region of interest in said category comprises sub-regions on a shelf that make up a product group.

38. The apparatus according to claim 25, wherein the apparatus further comprises a computer for matching all the grid elements that match each product group by checking for intersection of the grid element with the regions through a grid assignment module.

39. The apparatus according to claim 25, wherein the apparatus further comprises a computer for detecting temporal motion at every pixel in the image, wherein the grid elements that have a minimum threshold of pixel with motion are marked as active grid elements, and

whereby a look up logic module is used to determine which region is being interacted with to find the right product group.

40. The apparatus according to claim 25, wherein the apparatus further comprises a computer for measuring dwell time of said plurality of persons in said category.

41. The apparatus according to claim 25, wherein the apparatus further comprises a computer for utilizing second means for capturing images to track the hand of each person in said plurality of persons in said category,

whereby the hand tracking provides sub-category level information for shopping interaction of said plurality of persons in said category.

42. The apparatus according to claim 25, wherein the apparatus further comprises a computer for analyzing the syndicated data per category level as opposed to product level.

43. The apparatus according to claim 25, wherein the apparatus further comprises a computer for analyzing said category in regards to the segmentation of said plurality of persons,

whereby said segmentation comprises demographic classification or shopping pattern of said plurality of persons.

44. The apparatus according to claim 25, wherein the apparatus further comprises a computer for defining and utilizing shopping interaction levels for analyzing the behavior category of said plurality of persons in said category.

45. The apparatus according to claim 25, wherein the apparatus further comprises a computer for analyzing the stopping power based on dwell time of said plurality of persons in said category,

whereby the stopping power comprises any logical and physical cause that stops said plurality of persons nearby a physical location in a category.

46. The apparatus according to claim 25, wherein the apparatus further comprises a computer for utilizing prior knowledge about the physical location of said category in said physical space as an attribute for the category analysis.

47. The apparatus according to claim 25, wherein the apparatus further comprises a computer for forming a string of behavior measurement for said plurality of persons during a window of time,

whereby each element in the string of behavior measurement consists of sub-category number, interaction level, and information with regard to the sub-category.

48. The apparatus according to claim 25, wherein the apparatus further comprises a computer for building a model for product group interaction detection.