Method and system for automatically analyzing categories in a physical space based on the visual characterization of people

• US 20080159634A1
• Filed: 12/06/2007
• Published: 07/03/2008
• Est. Priority Date: 12/30/2006
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Baboo, Priya, Jung, Namsoon, Sharma, Rajeev, Hershey, Jeff, Mummareddy, Satish
• Granted Patent: US 8,189,926 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

The present invention is a method and system for automatically analyzing a category in a plurality of the categories in a physical space based on the visual characterization, such as behavior analysis or segmentation, of the persons with regard to the category. The present invention captures a plurality of input images of the persons in the category by a plurality of means for capturing images. The present invention processes the plurality of input images in order to understand the shopping behavior of the persons with the sub-categories of the category and analyzes the level of engagement and decision process at the sub-category level. The processes are based on a novel usage of a plurality of computer vision technologies to analyze the visual characterization of the persons from the plurality of input images. The physical space may be a retail space, and the persons may be customers in the retail space.

Claims

1. A method for automatically analyzing a category in a physical space based on video-based behavior analysis of a plurality of persons with regard to said category in the physical space, comprising the following steps of:

a) capturing a plurality of input images of said plurality of persons by a plurality of means for capturing images in the area with regard to said category,

b) processing said plurality of input images in order to process a video-based behavior analysis of each person in said plurality of persons, and

c) analyzing the category based on the accumulation of said video-based behavior analysis for said plurality of persons,

whereby the analysis of said category facilitates an analysis of level of engagement and decision process at the category level.

2. The method according to claim 1, wherein the method further comprises a step of repeating the steps from a) to c) for a plurality of categories in said physical space, wherein each category analysis for each category in said plurality of categories provides unique domain specific information about said each category.

3. The method according to claim 1, wherein the method further comprises a step of measuring domain specific information in said category,

whereby said domain specific information comprises sequence of products shopped in said category, ranking of products shopped in said category, product purchase pattern, and comparison of products interacted by said plurality of persons within said category.

4. The method according to claim 1, wherein the method further comprises a step of measuring dwell time of said plurality of persons in said category.

5. The method according to claim 1, wherein the method further comprises a step of utilizing second means for capturing images to track the hand of each person in said plurality of persons in said category,

whereby the hand tracking provides sub-category level information for shopping interaction of said plurality of persons in said category.

6. The method according to claim 1, wherein the method further comprises a step of analyzing shopping interaction by said plurality of persons in relation to shelves that are located in regards to said category,

whereby said shelves provide another dimension of measurement for the shopping interaction analysis in said category, and

whereby each interaction of a plurality of interactions in said shelves may have unique meaning because the intent of one interaction can be different from the intent of another interaction in said plurality of interactions.

7. The method according to claim 6, wherein the method further comprises a step of analyzing the shopping pattern of said plurality of persons when a shelf space in said shelves is empty.

8. The method according to claim 6, wherein the method further comprises a step of calculating the frequency of replenishing a certain product in said shelves, whereby the calculation increases the efficiency in inventory management of the product.

9. The method according to claim 6, wherein the method further comprises a step of calculating optimal size and height of said shelves based on shopping interaction analysis in regards to said shelves.

10. The method according to claim 6, wherein the method further comprises a step of calculating optimal layout for said shelves,

whereby said optimal layout provides information for optimal position of a product,

optimal number of products, efficient product assortment, and optimal type of products in a shelf of said shelves and in said shelves.

11. The method according to claim 1, wherein the method further comprises a step of measuring the effect after making a change at the category level,

whereby said change comprises change in the quantity, price, and position of certain products in said category.

12. The method according to claim 1, wherein the method further comprises a step of analyzing the syndicated data per category level as opposed to product level.

13. The method according to claim 1, wherein the method further comprises a step of analyzing said category in regards to the segmentation of said plurality of persons, whereby said segmentation comprises demographic classification or shopping pattern of said plurality of persons.

14. The method according to claim 1, wherein the method further comprises a step of defining and utilizing shopping interaction levels for analyzing the behavior category of said plurality of persons in said category.

15. The method according to claim 1, wherein the method further comprises a step of extracting analytic and statistical data from the category analysis that construct a map, wherein the map comprises:

a) a traffic distribution map that shows penetration of said plurality of persons throughout the category,

b) a shopping distribution map that shows the activity of said plurality of persons throughout the category,

c) a shopping conversion map that shows traffic of said plurality of persons converted to shopping activity,

d) a sub-category level traffic distribution map,

e) a sub-category level shopping distribution map, and

f) a sub-category level shopping conversion map,

whereby the map uses color-coded symbolic expressions to differentiate the information on the map among said plurality of persons at the category.

16. The method according to claim 1, wherein the method further comprises a step of calculating quantitative measurement per sub-category,

whereby the quantitative measurement comprises a ratio between shopping interaction levels per sub-category, based on actual measurement for the shopping interaction levels of said plurality of persons.

17. The method according to claim 1, wherein the method further comprises a step of measuring dominant path at a location in said category,

whereby the dominant path measurement shows a specific decision pattern and the tendency of decision for the next path by said plurality of persons, and

whereby next region for the path direction can be defined either point-level or polygon-level depending on the pre-defined granularity of the next region in said category.

18. The method according to claim 1, wherein the method further comprises a step of extracting analytic and statistical data from the category analysis that construct a shopping correlation analysis that is defined as the optimal distance measure among a plurality of sub-categories shopped,

whereby the correlation can be represented in a multi-dimensional adjacency table or map.

19. The method according to claim 1, wherein the method further comprises a step of extracting analytic and statistical data from the category analysis that construct a shopping sequence analysis that is the order of a plurality of sub-categories shopped, whereby the sequence analysis shows the order of shopping engagement by said plurality of persons.

20. The method according to claim 1, wherein the method further comprises a step of analyzing the stopping power based on dwell time of said plurality of persons in said category,

whereby the stopping power comprises any logical and physical cause that stops said plurality of persons nearby a physical location in a category.

21. The method according to claim 1, wherein the method further comprises a step of utilizing prior knowledge about the physical location of said category in said physical space as an attribute for the category analysis.

22. The method according to claim 1, wherein the method further comprises a step of forming a string of behavior measurement for said plurality of persons during a window of time,

whereby each element in the string of behavior measurement for a person of said plurality of persons can be a 3-tuple, consisting of sub-category number, interaction level, and information with regard to the sub-category.

23. The method according to claim 1, wherein the method further comprises a step of applying a set of predefined rules in a rule-base to the category analysis, comprising steps of:

a) augmenting the information units from said video-based behavior analysis for said plurality of persons, and

b) utilizing a polymorphic operator for applying said set of predefined rules to the information units.

24. The method according to claim 1, wherein the method further comprises a step of building a model for product group interaction detection.

25. The method according to claim 1, wherein the method further comprises a step of dividing the region of interest in said category into logical grid elements by taking the perspective distortion into account through a grid generation module,

whereby said region of interest in said category comprises sub-regions on a shelf that make up a product group.

26. The method according to claim 1, wherein the method further comprises a step of matching all the grid elements that match each product group by checking for intersection of the grid element with the regions through a grid assignment module.

27. The method according to claim 1, wherein the method further comprises a step of detecting temporal motion at every pixel in the image,

wherein the grid elements that have a minimum threshold of pixel with motion are marked as active grid elements, and

whereby a look up logic module is used to determine which region is being interacted with to find the right product group.

28. An apparatus for automatically analyzing a category in a physical space based on video-based behavior analysis of a plurality of persons with regard to said category in the physical space, comprising:

a) means for capturing a plurality of input images of said plurality of persons by a plurality of means for capturing images in the area with regard to said category,

b) means for processing said plurality of input images in order to process a video-based behavior analysis of each person in said plurality of persons, and

c) means for analyzing the category based on the accumulation of said video-based behavior analysis for said plurality of persons,

whereby the analysis of said category facilitates an analysis of level of engagement and decision process at the category level.

29. The apparatus according to claim 28, wherein the apparatus further comprises means for repeatedly using the means from a) to c) for a plurality of categories in said physical space,

wherein each category analysis for each category in said plurality of categories provides unique domain specific information about said each category.

30. The apparatus according to claim 28, wherein the apparatus further comprises means for measuring domain specific information in said category,

whereby said domain specific information comprises sequence of products shopped in said category, ranking of products shopped in said category, product purchase pattern, and comparison of products interacted by said plurality of persons within said category.

31. The apparatus according to claim 28, wherein the apparatus further comprises means for measuring dwell time of said plurality of persons in said category.

32. The apparatus according to claim 28, wherein the apparatus further comprises means for utilizing second means for capturing images to track the hand of each person in said plurality of persons in said category,

whereby the hand tracking provides sub-category level information for shopping interaction of said plurality of persons in said category.

33. The apparatus according to claim 28, wherein the apparatus further comprises means for analyzing shopping interaction by said plurality of persons in relation to shelves that are located in regards to said category,

whereby said shelves provide another dimension of measurement for the shopping interaction analysis in said category, and

whereby each interaction of a plurality of interactions in said shelves may have unique meaning because the intent of one interaction can be different from the intent of another interaction in said plurality of interactions.

34. The apparatus according to claim 33, wherein the apparatus further comprises means for analyzing the shopping pattern of said plurality of persons when a shelf space in said shelves is empty.

35. The apparatus according to claim 33, wherein the apparatus further comprises means for calculating the frequency of replenishing a certain product in said shelves, whereby the calculation increases the efficiency in inventory management of the product.

36. The apparatus according to claim 33, wherein the apparatus further comprises means for calculating optimal size and height of said shelves based on shopping interaction analysis in regards to said shelves.

37. The apparatus according to claim 33, wherein the apparatus further comprises means for calculating optimal layout for said shelves,

whereby said optimal layout provides information for optimal position of a product, optimal number of products, efficient product assortment, and optimal type of products in a shelf of said shelves and in said shelves.

38. The apparatus according to claim 28, wherein the apparatus further comprises means for measuring the effect after making a change at the category level, whereby said change comprises change in the quantity, price, and position of certain products in said category.

39. The apparatus according to claim 28, wherein the apparatus further comprises means for analyzing the syndicated data per category level as opposed to product level.

40. The apparatus according to claim 28, wherein the apparatus further comprises means for analyzing said category in regards to the segmentation of said plurality of persons,

whereby said segmentation comprises demographic classification or shopping pattern of said plurality of persons.

41. The apparatus according to claim 28, wherein the apparatus further comprises means for defining and utilizing shopping interaction levels for analyzing the behavior category of said plurality of persons in said category.

42. The apparatus according to claim 28, wherein the apparatus further comprises means for extracting analytic and statistical data from the category analysis that construct a map,

wherein the map comprises:

a) a traffic distribution map that shows penetration of said plurality of persons throughout the category,

b) a shopping distribution map that shows the activity of said plurality of persons throughout the category,

c) a shopping conversion map that shows traffic of said plurality of persons converted to shopping activity,

d) a sub-category level traffic distribution map,

e) a sub-category level shopping distribution map, and

f) a sub-category level shopping conversion map,

whereby the map uses color-coded symbolic expressions to differentiate the information on the map among said plurality of persons at the category.

43. The apparatus according to claim 28, wherein the apparatus further comprises means for calculating quantitative measurement per sub-category,

whereby the quantitative measurement comprises a ratio between shopping interaction levels per sub-category, based on actual measurement for the shopping interaction levels of said plurality of persons.

44. The apparatus according to claim 28, wherein the apparatus further comprises means for measuring dominant path at a location in said category,

whereby the dominant path measurement shows a specific decision pattern and the tendency of decision for the next path by said plurality of persons, and

whereby next region for the path direction can be defined either point-level or polygon-level depending on the pre-defined granularity of the next region in said category.

45. The apparatus according to claim 28, wherein the apparatus further comprises means for extracting analytic and statistical data from the category analysis that construct a shopping correlation analysis that is defined as the optimal distance measure among a plurality of sub-categories shopped,

whereby the correlation can be represented in a multi-dimensional adjacency table or map.

46. The apparatus according to claim 28, wherein the apparatus further comprises means for extracting analytic and statistical data from the category analysis that construct a shopping sequence analysis that is the order of a plurality of sub-categories shopped,

whereby the sequence analysis shows the order of shopping engagement by said plurality of persons.

47. The apparatus according to claim 28, wherein the apparatus further comprises means for analyzing the stopping power based on dwell time of said plurality of persons in said category,

whereby the stopping power comprises any logical and physical cause that stops said plurality of persons nearby a physical location in a category.

48. The apparatus according to claim 28, wherein the apparatus further comprises means for utilizing prior knowledge about the physical location of said category in said physical space as an attribute for the category analysis.

49. The apparatus according to claim 28, wherein the apparatus further comprises means for forming a string of behavior measurement for said plurality of persons during a window of time,

whereby each element in the string of behavior measurement for a person of said plurality of persons can be a 3-tuple, consisting of sub-category number, interaction level, and information with regard to the sub-category.

50. The apparatus according to claim 28, wherein the apparatus further comprises means for applying a set of predefined rules in a rule-base to the category analysis, comprising:

a) means for augmenting the information units from said video-based behavior analysis for said plurality of persons, and

b) means for utilizing a polymorphic operator for applying said set of predefined rules to the information units.

51. The apparatus according to claim 28, wherein the apparatus further comprises means for building a model for product group interaction detection.

52. The apparatus according to claim 28, wherein the apparatus further comprises means for dividing the region of interest in said category into logical grid elements by taking the perspective distortion into account through a grid generation module,

whereby said region of interest in said category comprises sub-regions on a shelf that make up a product group.

53. The apparatus according to claim 28, wherein the apparatus further comprises means for matching all the grid elements that match each product group by checking for intersection of the grid element with the regions through a grid assignment module.

54. The apparatus according to claim 28, wherein the apparatus further comprises means for detecting temporal motion at every pixel in the image,

wherein the grid elements that have a minimum threshold of pixel with motion are marked as active grid elements, and

whereby a look up logic module is used to determine which region is being interacted with to find the right product group.