Method and system for optimizing the observation and annotation of complex human behavior from video sources

• US 8,665,333 B1
• Filed: 01/25/2008
• Issued: 03/04/2014
• Est. Priority Date: 01/30/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Videomining Corporation
• Inventors: Sharma, Rajeev, Mummareddy, Satish, Schapira, Emilio, Jung, Namsoon
• Primary Examiner(s): Lazaro, David
• Assistant Examiner(s): DRABIK, SARAH E
• Application Number:
  US12/011,385
• Time in Patent Office: 2,230 Days
• Field of Search: 348 47- 48, 348/135, 348/139, 348/143, 348/153, 348/155, 348/159, 348/169
• US Class Current: 348/159
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19643 wherein the cameras play different roles, e.g. different resolution, different camera type, master-slave camera

Abstract

The present invention is a method and system for optimizing the observation and annotation of complex human behavior from video sources by automatically detecting predefined events based on the behavior of people in a first video stream from a first means for capturing images in a physical space, accessing a synchronized second video stream from a second means for capturing images that are positioned to observe the people more closely using the timestamps associated with the detected events from the first video stream, and enabling an annotator to annotate each of the events with more labels using a tool. The present invention captures a plurality of input images of the persons by a plurality of means for capturing images and processes the plurality of input images in order to detect the predefined events based on the behavior in an exemplary embodiment. The processes are based on a novel usage of a plurality of computer vision technologies to analyze the human behavior from the plurality of input images. The physical space may be a retail space, and the people may be customers in the retail space.

Claims

1. A method for efficiently annotating behavior and characteristics of a person or a plurality of persons in a first video stream and a second video stream in a physical space, comprising the following steps of:

a) capturing the first video stream of the person or the plurality of persons by a first means for capturing images,

b) processing the first video stream in order to track and detect predefined behavior and demographics of the person or the plurality of persons in a field of view of the first means for capturing images automatically using at least a means for control and processing that executes computer vision algorithms on the first video stream,

c) providing demographic segmentation of the person or the plurality of persons to create a plurality of demographic groups,

d) analyzing the behavior based on spatio-temporal primitives and a model for interaction levels of the person or the plurality of persons, wherein the behavior of each demographic group is analyzed to obtain segment-specific insights,

e) generating time-stamped lists of events based on the automatically detected predefined behavior and demographics, using a time server,

f) using the time-stamped lists of events and timestamps of events in the time-stamped lists of events to access at least a corresponding sub-stream for the events in the second video stream from a second means for capturing images,

g) manually annotating each of the events with a plurality of labels for a synchronized annotation using a user interface, and

h) utilizing the annotation for quantitative behavior analysis about interaction of the person or the plurality of persons with a plurality of commercial products in the physical space,

wherein the first video stream is synchronized with the second video stream, and whereby the user interface allows users to mark time-based annotations describing complex behavioral issues, including expressions of the person or the plurality of persons.

2. The method according to claim 1, wherein the method further comprises a step of utilizing a plurality of first means for capturing images,

wherein the plurality of first means for capturing images are synchronized with the second means for capturing images by using the time-stamped lists of events and the timestamps of events, and

whereby the first means for capturing images comprises an overhead top-down camera.

3. The method according to claim 1, wherein the method further comprises a step of utilizing a plurality of second means for capturing images,

whereby the second means for capturing images comprises a camera that is positioned to observe the person or the plurality of persons more closely for analyzing a specific event.

4. The method according to claim 1, wherein the method further comprises a step of using a graphical user interface for the annotation, whereby the graphical user interface is used to browse the first video stream and the second video stream based on the timestamps of the events, comprising beginning time and end time of the events.

5. The method according to claim 1, wherein the method further comprises a step of utilizing the time-stamped lists of events and the synchronized annotation to reduce time required to analyze a large amount of the first video stream and the second video stream.

6. The method according to claim 1, wherein the method further comprises a step of utilizing dwell time of the person or the plurality of persons to detect the predefined behavior,

whereby the predefined behavior based on the dwell time comprises passerby behavior and engaged shopper behavior.

7. The method according to claim 1, wherein the method further comprises a step of generating statistical reports by aggregating the annotated events.

8. The method according to claim 1, wherein the method further comprises a step of utilizing at least a wireless sensor and a wireless sensor based tracking to track and detect predefined behavior,

whereby the wireless sensor comprises a RFID and means for using the RFID.

9. The method according to claim 1, wherein the method further comprises a step of utilizing door sensors to detect predefined behavior.

10. The method according to claim 1, wherein the method further comprises a step of utilizing a rule-based logic module to synchronize the first video stream with the second video stream,

wherein the synchronization is adjusted dynamically based on rules defined in the rule-based logic module.

11. A system for efficiently annotating behavior and characteristics of a person or a plurality of persons in a first video stream and a second video stream in a physical space, comprising:

a) at least a first means for capturing images that captures the first video stream of the person or the plurality of persons,

b) at least a first means for control and processing that executes computer vision algorithms on the first video stream, performing the following steps of;

processing the first video stream in order to track and detect predefined behavior and demographics of the person or the plurality of persons in a field of view of the first means for capturing images automatically,

providing demographic segmentation of the person or the plurality of persons to create a plurality of demographic groups,

analyzing the behavior based on spatio-temporal primitives and a model for interaction levels of the person or the plurality of persons, wherein the behavior of each demographic group is analyzed to obtain segment-specific insights, and

generating time-stamped lists of events based on the automatically detected predefined behavior and demographics, using a time server, and

c) an annotation tool for using the time-stamped lists of events and timestamps of events in the time-stamped lists of events to access at least a corresponding sub-stream for the events in the second video stream from a second means for capturing images, and

for annotating each of the events with a plurality of labels for a synchronized annotation including a user interface for the annotation,

wherein the first video stream is synchronized with the second video stream,

wherein the annotation is utilized for quantitative behavior analysis about interaction of the person or the plurality of persons with a plurality of commercial products in the physical space, and

whereby the user interface allows users to mark time-based annotations describing complex behavioral issues, including expressions of the person or the plurality of persons.

12. The system according to claim 11, wherein the system further comprises at least an overhead top-down camera that is connected to the first means for control and processing,

wherein the overhead top-down camera is synchronized with the second means for capturing images by using the time-stamped lists of events and the timestamps of events.

13. The system according to claim 11, wherein the system further comprises a plurality of second means for capturing images,

whereby the second means for capturing images comprises a camera that is positioned to observe the person or the plurality of persons more closely for analyzing a specific event.

14. The system according to claim 11, wherein the system further comprises a graphical user interface for the annotation,

whereby the graphical user interface is used to browse the first video stream and the second video stream based on the timestamps of the events, comprising beginning time and end time of the events.

15. The system according to claim 11, wherein the system further comprises a means for control and processing for utilizing the time-stamped lists of events and the synchronized annotation to reduce time required to analyze a large amount of the first video stream and the second video stream.

16. The system according to claim 11, wherein the system further comprises a means for control and processing for utilizing dwell time of the person or the plurality of persons to detect the predefined behavior,

whereby the predefined behavior based on the dwell time comprises passerby behavior and engaged shopper behavior.

17. The system according to claim 11, wherein the system further comprises a means for control and processing for generating statistical reports by aggregating the annotated events.

18. The system according to claim 11, wherein the system further comprises at least a wireless sensor and a means for control and processing that utilizes a wireless sensor based tracking to track and detect predefined behavior,

whereby the wireless sensor comprises a RFID and means for using the RFID.

19. The system according to claim 11, wherein the system further comprises a means for control and processing for utilizing door sensors to detect predefined behavior.

20. The system according to claim 11, wherein the system further comprises a means for control and processing for utilizing a rule-based logic module to synchronize the first video stream with the second video stream, wherein the synchronization is adjusted dynamically based on rules defined in the rule-based logic module.