Method and system for segmenting people in a physical space based on automatic behavior analysis

• US 8,295,597 B1
• Filed: 03/07/2008
• Issued: 10/23/2012
• Est. Priority Date: 03/14/2007
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Sharma, Rajeev, Mummareddy, Satish, Hershey, Jeff, Jung, Namsoon
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US12/075,089
• Time in Patent Office: 1,691 Days
• Field of Search: 382/103, 382/164, 382/173, 382/209, 382218-220, 382/224, 382/294, 375/240.08, 375/240.11, 375/240.16, 348/143, 348/150, 348/155, 348/169, 340/572.4, 340/572.7, 340/745, 704/270, 705/10, 706/12, 706/25, 706/47, 706/59, 715/862, 715/863
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

The present invention is a method and system for segmenting a plurality of persons in a physical space based on automatic behavior analysis of the persons in a preferred embodiment. The behavior analysis can comprise a path analysis as one of the characterization methods. The present invention applies segmentation criteria to the output of the video-based behavior analysis and assigns segmentation label to each of the persons during a predefined window of time. In addition to the behavioral characteristics, the present invention can also utilize other types of visual characterization, such as demographic analysis, or additional input sources, such as sales data, to segment the plurality of persons in another exemplary embodiment. The present invention captures a plurality of input images of the persons in the physical space by a plurality of means for capturing images. The present invention processes the plurality of input images in order to understand the behavioral characteristics, such as shopping behavior, of the persons for the segmentation purpose. The processes are based on a novel usage of a plurality of computer vision technologies to analyze the visual characterization of the persons from the plurality of input images. The physical space may be a retail space, and the persons may be customers in the retail space.

Claims

1. A method for segmenting a plurality of persons in a physical space based on behavior analysis of the plurality of persons, comprising the following steps of:

a) processing path analysis of each person in the plurality of persons based on tracking of the person in a plurality of input images captured by a plurality of means for capturing images using at least a means for control and processing,

b) processing behavior analysis of each person in the plurality of persons,

c) processing a video-based demographic analysis of each person in the plurality of persons,

d) constructing segmentation criteria for the plurality of persons based on a set of predefined rules, and

e) segmenting the plurality of persons by applying the segmentation criteria to outputs of the behavior analysis and the demographic analysis for the plurality of persons,

wherein attributes of the path analysis comprise information for initial point and destination, coordinates of the person'"'"'s position, temporal attributes, including trip time and trip length, and average velocity, and

wherein the outputs of the behavior analysis comprise a sequence of visits or a combination of visits to a predefined category in the physical space by the plurality of persons.

2. The method according to claim 1, wherein the method further comprises a step of differentiating levels of segmentation in a network of the physical spaces,

wherein first segmentation criteria are applied throughout the network, and

wherein second segmentation criteria are applied to a predefined subset of the network of the physical spaces to serve specific needs of the predefined subset.

3. The method according to claim 1, wherein the method further comprises a step of sampling of a subset of a network of the physical spaces rather than segmenting all physical spaces in the network of the physical spaces.

4. The method according to claim 1, wherein the method further comprises a step of assigning a segmentation label to each person in the plurality of persons during a predefined window of time.

5. The method according to claim 1, wherein the method further comprises a step of constructing the segmentation criteria based on a combination of a set of predefined rules,

whereby population in the physical space is divided into subpopulations by the combination of rules.

6. The method according to claim 1, wherein the method further comprises a step of constructing the segmentation criteria based on application of a learning algorithm based behavior analysis to a training population,

wherein clusters of trajectories are separated by determining probability density function of the trajectories.

7. The method according to claim 1, wherein the method further comprises a step of constructing the segmentation criteria based on example-based behavior analysis algorithm,

wherein one of trajectory classes is made as an example behavior so that a neural network creates a cluster for trajectories similar to the example behavior.

8. The method according to claim 1, wherein the method further comprises a step of applying different segmentation criteria per each segmentation group.

9. The method according to claim 1, wherein the method further comprises a step of defining domain-specific criteria for the segmentation criteria, whereby the domain-specific criteria comprise criteria that are constructed based on retail-specific rules.

10. The method according to claim 1, wherein the method further comprises steps of

a) receiving additional input in regards to the plurality of persons, and

b) segmenting the plurality of persons by applying segmentation criteria to the additional input,

whereby the additional input comprises transaction data by the plurality of persons.

11. The method according to claim 1, wherein the method further comprises a step of utilizing a rule application logic module for applying the segmentation criteria to the behavior analysis data.

12. The method according to claim 1, wherein the method further comprises a step of representing the output of the segmentation using layers of information.

13. An apparatus for segmenting a plurality of persons in a physical space based on behavior analysis of the plurality of persons, comprising:

a) a plurality of means for capturing images that capture a plurality of input images,

b) at least a means for control and processing that is programmed to perform the following steps of;

processing path analysis of each person in the plurality of persons based on tracking of the person in the plurality of input images,

processing behavior analysis of each person in the plurality of persons,

processing a video-based demographic analysis of each person in the plurality of persons, constructing segmentation criteria for the plurality of persons based on a set of predefined rules, and

segmenting the plurality of persons by applying the segmentation criteria to outputs of the behavior analysis and the demographic analysis for the plurality of persons,

wherein attributes of the path analysis comprise information for initial point and destination, coordinates of the person'"'"'s position, temporal attributes, including trip time and trip length, and average velocity, and

wherein the outputs of the behavior analysis comprise a sequence of visits or a combination of visits to a predefined category in the physical space by the plurality of persons.

14. The apparatus according to claim 13, wherein the apparatus further comprises means for differentiating levels of segmentation in a network of the physical spaces,

wherein first segmentation criteria are applied throughout the network, and

wherein second segmentation criteria are applied to a predefined subset of the network of the physical spaces to serve specific needs of the predefined subset.

15. The apparatus according to claim 13, wherein the apparatus further comprises means for sampling of a subset of a network of the physical spaces rather than segmenting all physical spaces in the network of the physical spaces.

16. The apparatus according to claim 13, wherein the apparatus further comprises means for assigning a segmentation label to each person in the plurality of persons during a predefined window of time.

17. The apparatus according to claim 13, wherein the apparatus further comprises means for constructing the segmentation criteria based on a combination of a set of predefined rules,

whereby population in the physical space is divided into subpopulations by the combination of rules.

18. The apparatus according to claim 13, wherein the apparatus further comprises means for constructing the segmentation criteria based on application of a learning algorithm based behavior analysis to a training population,

wherein clusters of trajectories are separated by determining probability density function of the trajectories.

19. The apparatus according to claim 13, wherein the apparatus further comprises means for constructing the segmentation criteria based on example-based behavior analysis algorithm,

wherein one of trajectory classes is made as an example behavior so that a neural network creates a cluster for trajectories similar to the example behavior.

20. The apparatus according to claim 13, wherein the apparatus further comprises means for applying different segmentation criteria per each segmentation group.

21. The apparatus according to claim 13, wherein the apparatus further comprises means for defining domain-specific criteria for the segmentation criteria, whereby the domain-specific criteria comprise criteria that are constructed based on retail-specific rules.

22. The apparatus according to claim 13, wherein the apparatus further comprises

a) means for receiving additional input in regards to the plurality of persons, and

b) means for segmenting the plurality of persons by applying segmentation criteria to the additional input,

whereby the additional input comprises transaction data by the plurality of persons.

23. The apparatus according to claim 13, wherein the apparatus further comprises means for utilizing a rule application logic module for applying the segmentation criteria to the behavior analysis data.

24. The apparatus according to claim 13, wherein the apparatus further comprises means for representing the output of the segmentation using layers of information.