Videore: method and system for storing videos from multiple cameras for behavior re-mining

• US 8,457,466 B1
• Filed: 09/29/2008
• Issued: 06/04/2013
• Est. Priority Date: 09/29/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Videomining Corporation
• Inventors: Sharma, Rajeev, Jung, Namsoon
• Primary Examiner(s): Shaw, Peter
• Assistant Examiner(s): HO, DAO Q
• Application Number:
  US12/286,138
• Time in Patent Office: 1,709 Days
• Field of Search: 386/46, 386/224
• US Class Current: 386/224
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The present invention is a method and system for storing videos by track sequences and selection of video segments in a manner to support “re-mining” by indexing and playback of individual visitors'"'"' entire trip to an area covered by overlapping cameras, allowing analysis and recognition of detailed behavior. The present invention captures video streams of the people in the area by multiple cameras and tracks the people in each of the video streams, producing track sequences in each video stream. Using the track sequences, the present invention finds trip information of the people. The present invention determines a first set of video segments that contain the trip information of the people, and compacts each of the video streams by removing a second set of video segments that do not contain the trip information of the people from each of the video streams. The video segments in the first set of video segments are associated with the people by indexing the video segments per person based on the trip information. The final storage format of the videos is a trip-centered format which sequences videos from across multiple cameras in a manner to facilitate multiple applications dealing with behavior analysis, and it is an efficient compact format without losing any video segments that contain the track sequences of the people.

Claims

1. A method for storing videos based on track sequences in video segments for a person or a plurality of persons that visited an area covered by a plurality of cameras that are connected to at least a computer through at least a video interface using computer vision technology, comprising the following steps of:

a) tracking the person or the plurality of persons in each of a plurality of video streams, producing track sequences, by executing a computer vision based tracking algorithm on the plurality of video streams utilizing the computer,

b) computing information for the trip of the person or the plurality of persons based on the track sequences,

c) executing a process in the computer to determine a first set of video segments that contain the trip information of the person or the plurality of persons and a second set of video segments that do not contain the trip information of the person or the plurality of persons in each of the video streams,

d) executing a process in the computer to compact each of the video streams by removing the second set of video segments from each of the video streams,

e) executing a process in the computer to associate the first set of video segments with the person or the plurality of persons based on the information for the trip of the person or the plurality of persons, in a data structure, and

f) creating a trip-centered format that stores the compacted video streams with the first set of video segments based on the track sequences from the plurality of cameras, wherein the format comprises the data structure for associating the first set of video segments with the person or the plurality of persons based on the information for the trip of the person or the plurality of persons, including a table of indices, and

wherein the stored videos based on track sequences in video segments support re-mining of behavior of the person or the plurality of persons.

2. The method according to claim 1, wherein the method further comprises a step of organizing the associated first set of video segments logically based on at least a rule, wherein the rule comprises a trip-centered video indexing scheme.

3. The method according to claim 1, wherein the method further comprises a step of indexing the first set of video segments for the person or each of the plurality of persons based on the information for the trip of the person or each of the plurality of persons and producing a table of indices,

wherein the indices point to relevant video segments in the first set of video segments.

4. The method according to claim 1, wherein the method further comprises a step of accessing the compacted video streams and the first set of video segments without a loss of the video segments that contain the trip information.

5. The method according to claim 1, wherein the method further comprises a step of annotating the compacted video streams and the first set of video segments, wherein the annotation facilitates multiple applications dealing with behavior analysis, including areas of human behavior analysis, loss prevention, investigation, security, animal research, medical research, and sociology research.

6. The method according to claim 1, wherein the method further comprises a step of processing a playback of the entire trip of the person or the plurality of persons, using a data structure that contains collages of video segments information, wherein a collage of video segments information is created according to trip video segments information per person.

7. The method according to claim 1, wherein the method further comprises a step of merging the track sequences into one track,

wherein the trip information of the person or the plurality of persons is calculated based on the track sequences, and

wherein the trip information includes coordinates for the positions of the person or the plurality of persons, temporal attributes of the track sequences, video stream IDs, video segment IDs, trip IDs, and person IDs for each person.

8. The method according to claim 1, wherein the method further comprises a step of resolving disambiguation of multiple appearances of a person'"'"'s track sequences among a plurality of video segments in the first set of video segments if the person or the plurality of persons is visible in more than one video segment in the first set of video segments.

9. An apparatus for storing videos based on track sequences in video segments for a person or a plurality of persons that visited an area covered by a plurality of cameras using computer vision technology, comprising:

a) cameras for capturing a plurality of video streams of the person or the plurality of persons in the area,

b) at least a video interface,

c) at least a computer that performs the following steps of;

tracking the person or the plurality of persons in each of the video streams, producing track sequences, by executing a computer vision based a tracking algorithm on the plurality of video streams,

computing information for the trip of the person or the plurality of persons based on the track sequences,

determining a first set of video segments that contain the trip information of the person or the plurality of persons and a second set of video segments that do not contain the trip information of the person or the plurality of persons in each of the video streams,

compacting each of the video streams by removing the second set of video segments from each of the video streams,

associating the first set of video segments with the person or the plurality of persons based on the information for the trip of the person or the plurality of persons, in a data structure, and

creating a trip-centered format that stores the compacted video streams with the first set of video segments based on the track sequences from the plurality of cameras,

wherein the format comprises the data structure for associating the first set of video segments with the person or the plurality of persons based on the information for the trip of the person or the plurality of persons, including a table of indices, and

wherein the stored videos based on track sequences in video segments support re-mining of behavior of the person or the plurality of persons.

10. The apparatus according to claim 9, wherein the apparatus further comprises a computer for organizing the associated first set of video segments logically based on at least a rule,

wherein the rule comprises a trip-centered video indexing scheme.

11. The apparatus according to claim 9, wherein the apparatus further comprises a computer for indexing the first set of video segments for the person or each of the plurality of persons based on the information for the trip of the person or each of the plurality of persons and producing a table of indices,

wherein the indices point to relevant video segments in the first set of video segments.

12. The apparatus according to claim 9, wherein the apparatus further comprises a computer for accessing the compacted video streams and the first set of video segments without a loss of the video segments that contain the trip information.

13. The apparatus according to claim 9, wherein the apparatus further comprises a computer for annotating the compacted video streams and the first set of video segments,

wherein the annotation facilitates multiple applications dealing with behavior analysis, including areas of human behavior analysis, loss prevention, investigation, security, animal research, medical research, and sociology research.

14. The apparatus according to claim 9, wherein the apparatus further comprises a computer for processing a playback of the entire trip of the person or the plurality of persons, using a data structure that contains collages of video segments information,

wherein a collage of video segments information is created according to trip video segments information per person.

15. The apparatus according to claim 9, wherein the apparatus further comprises a computer for merging the track sequences into one track, wherein the trip information of the person or the plurality of persons is calculated based on the track sequences, and

wherein the trip information includes coordinates for the positions of the person or the plurality of persons, temporal attributes of the track sequences, video stream IDs, video segment IDs, trip IDs, and person IDs for each person.

16. The apparatus according to claim 9, wherein the apparatus further comprises a computer for resolving disambiguation of multiple appearances of a person'"'"'s track sequences among a plurality of video segments in the first set of video segments if the person or the plurality of persons is visible in more than one video segment in the first set of video segments.