Method and system for efficient sampling of videos using spatiotemporal constraints for statistical behavior analysis

• US 8,570,376 B1
• Filed: 11/19/2008
• Issued: 10/29/2013
• Est. Priority Date: 11/19/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Videomining Corporation
• Inventors: Jung, Namsoon, Sharma, Rajeev
• Primary Examiner(s): Feild, Lynn
• Assistant Examiner(s): Liu, Lin
• Application Number:
  US12/313,359
• Time in Patent Office: 1,805 Days
• Field of Search: 348/143, 348/159, 382/103
• US Class Current: 348/159
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The present invention is a method and system for selecting and storing videos by applying semantically-meaningful selection criteria to the track sequences of the trips made by people in an area covered by overlapping multiple cameras. The present invention captures video streams of the people in the area by multiple cameras and tracks the people in each of the video streams, producing track sequences in each video stream. The present invention determines a first set of video segments that contains the trip information of the people, and compacts each of the video streams by removing a second set of video segments that do not contain the trip information of the people from each of the video streams. The present invention selects video segments from the first set of video segments based on predefined selection criteria for the statistical behavior analysis. The stored video data is an efficient compact format of video segments that contain the track sequences of the people and selected according to semantically-meaningful and domain-specific selection criteria. The final storage format of the videos is a trip-centered format, which sequences videos from across multiple cameras, and it can be used to facilitate multiple applications dealing with behavior analysis in a specific domain.

Claims

1. A method for storing selected video segments for a person or a plurality of persons that visited an area covered by a plurality of means for capturing images using at least a computer that executes computer vision algorithms on a plurality of video streams, comprising the following steps of:

a) finding information for the trip of the person or the plurality of persons based on track sequences calculated from tracking the person or the plurality of persons in video streams,

b) determining a first set of video segments that contain the trip information of the person or the plurality of persons and a second set of video segments that do not contain the trip information of the person or the plurality of persons in each of the video streams,

c) removing the second set of video segments from each of the video streams,

d) selecting video segments from the first set of video segments based on predefined selection criteria, applying multi-layered spatiotemporal constraints to select the video segments, for the statistical behavior analysis

e) applying domain-specific sampling criteria to the selection of video segments, and

f) compacting each of the video streams using the first set of video segments, after removing the second set of video segments from each of the video streams, and

whereby the application of the multi-layered spatiotemporal constraints further reduces the size of the selected video segments.

2. The method according to claim 1, wherein the method further comprises a step of selecting video segments from the compact video streams based on selection criteria, further reducing the size of video data.

3. The method according to claim 1, wherein the method further comprises a step of sampling the video segments based on the spatiotemporal constraints.

4. The method according to claim 1, wherein the method further comprises a step of sampling the video segments based on rule-based sampling criteria,

wherein the rule-based sampling criteria comprise a rule application module, and

whereby the rule application module comprises information unit verification technologies.

5. The method according to claim 1, wherein the method further comprises a step of applying semantics-based sampling criteria to the selection of video segments,

wherein the semantics are formulated for behavior analysis in the video segments, and

wherein the application facilitates intelligent sequencing of the video segments.

6. The method according to claim 1, wherein the method further comprises a step of analyzing the selected video segments based on demographics.

7. The method according to claim 1, wherein the method further comprises a step of creating a trip-centered format that stores the compacted video streams with the first set of video segments based on the track sequences from the plurality of means for capturing images.

8. The method according to claim 1, wherein the method further comprises a step of utilizing a format that comprises a data structure for associating the first set of video segments with the person or the plurality of persons based on the information for the trip of the person or the plurality of persons, including a table of indices, and

whereby the format facilitates multiple applications dealing with behavior analysis and provides an efficient storage.

9. The method according to claim 1, wherein the method further comprises a step of resolving disambiguation among a plurality of video segments in the first set of video segments if the person or the plurality of persons are visible in more than one video segment in the first set of video segments during the same window of time in order to reduce duplicates of stored video segments.

10. The method according to claim 1, wherein the method further comprises a step of adjusting the selection criteria in association with a selection rate in order to maintain the size of the selected video segments stored in storage according to the selection rate.

11. An apparatus for storing selected video segments for a person or a plurality of persons that visited an area covered by a plurality of means for capturing images, comprising:

a) a plurality of means for capturing images for capturing a plurality of video streams of the person or the plurality of persons in the area,

b) at least a means for video interface that is connected to the plurality of means for capturing images,

c) at least a computer that executes computer vision algorithms on the plurality of video streams, performing the following steps of;

finding information for the trip of the person or the plurality of persons based on track sequences calculated from tracking the person or the plurality of persons in video streams,

determining a first set of video segments that contain the trip information of the person or the plurality of persons and a second set of video segments that do not contain the trip information of the person or the plurality of persons in each of the video streams,

removing the second set of video segments from each of the video streams,

selecting video segments from the first set of video segments based on predefined selection criteria, applying multi-layered spatiotemporal constraints to select the video segments, for the statistical behavior analysis,

applying domain-specific sampling criteria to the selection of video segments, and

compacting each of the video streams using the first set of video segments, after removing the second set of video segments from each of the video streams.,

d) at least a means for storing data that stores the selected video segments, and

whereby the application of the multi-layered spatiotemporal constraints further reduces the size of the selected video segments.

12. The apparatus according to claim 11, wherein the apparatus further comprises a computer for selecting video segments from the compact video streams based on selection criteria, further reducing the size of video data.

13. The apparatus according to claim 11, wherein the apparatus further comprises a computer for sampling the video segments based on the spatiotemporal constraints.

14. The apparatus according to claim 11, wherein the apparatus further comprises a computer for sampling the video segments based on rule-based sampling criteria,

wherein the rule-based sampling criteria comprise a rule application module, and

whereby the rule application module comprises information unit verification technologies.

15. The apparatus according to claim 11, wherein the apparatus further comprises a computer for applying semantics-based sampling criteria to the selection of video segments,

wherein the semantics are formulated for behavior analysis in the video segments, and

wherein the application facilitates intelligent sequencing of the video segments.

16. The apparatus according to claim 11, wherein the apparatus further comprises a computer for analyzing the selected video segments based on demographics.

17. The apparatus according to claim 11, wherein the apparatus further comprises a computer for creating a trip-centered format that stores the compacted video streams with the first set of video segments based on the track sequences from the plurality of means for capturing images.

18. The apparatus according to claim 11, wherein the apparatus further comprises a computer for utilizing a format that comprises a data structure for associating the first set of video segments with the person or the plurality of persons based on the information for the trip of the person or the plurality of persons, including a table of indices, and whereby the format facilitates multiple applications dealing with behavior analysis and provides an efficient storage.

19. The apparatus according to claim 11, wherein the apparatus further comprises a computer for resolving disambiguation among a plurality of video segments in the first set of video segments if the person or the plurality of persons are visible in more than one video segment in the first set of video segments during the same window of time in order to reduce duplicates of stored video segments.

20. The apparatus according to claim 11, wherein the apparatus further comprises a computer for adjusting the selection criteria in association with a selection rate in order to maintain the size of the selected video segments stored in storage according to the selection rate.