Video data storage, search, and retrieval using meta-data and attribute data in a video surveillance system

• US 7,460,149 B1
• Filed: 05/28/2007
• Issued: 12/02/2008
• Est. Priority Date: 05/28/2007
• Status: Expired due to Fees

• Current Assignee: SecureNet Solutions Group, LLC
• Original Assignee: KD Secure LLC
• Inventors: Hussain, Danir, Donovan, John J.
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US11/754,335
• Publication Number:
  US 20080297599A1
• Time in Patent Office: 554 Days
• Field of Search: 348/143, 348/135, 348/142, 348/154, 348/171, 348/172, 348/169, 348/155, 348/152, 348/231.99, 348/700, 348/231.2, 348/231.3, 348/231.5, 382/103, 382/107, 382/194, 382/235, 382/236, 382/224, 382/305, 382/190, 382/195, 382/232, 382/217, 382/173, 382/225, 382/226, 382/227, 382/228, 382/229, 382/230, 382/231, 382/159, 382/118, 382/170, 382/218, 386/46, 386/125, 375/240
• US Class Current: 348/143
• CPC Class Codes: G06F 16/7343 Query language or query format
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06F 16/787 using geographical or spatial information, e.g. location
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/329 on a disc [VTOC]
• CPC Class Codes: H04N 2005/91364 the video signal being scrambled
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

One embodiment is a method of storing video data from a video surveillance system having one or more cameras. Video data is captured from one or more surveillance cameras. Meta-data is automatically generated by performing video analysis on the captured video data from the surveillance cameras. A human operator may manually enter additional meta-data. Attribute data and associated weights, representing information about the relevance of the meta-data, is received. The video data is stored in a hierarchical video storage area; the meta-data, indexed by date and time stamp to the video data, is stored in a meta-data storage area; and the attribute data is stored in an attribute storage area. One or more alerts may be issued based on the past and present meta-data. The video data is secured by encrypting and storing the video data remotely, and audit trails are generated about who and when viewed the video data.

Claims

1. A method of storing video data, associated meta-data, and associated attribute weights from a video surveillance system, the method comprising:

capturing video data from one or more surveillance cameras;

generating meta-data by performing video analysis on the video data from the surveillance cameras, the meta-data representing events detected in the video data;

determining attribute weights, representing information about the relevance of the meta-data;

generating intersections of two or more subsets of the meta-data to generate intersection meta-data;

determining attribute weights associated with the intersection meta-data by multiplying the attribute weights for each subset of meta-data;

generating unions of two or more subsets of the meta-data to generate union meta-data;

determining attribute weights associated with the union meta-data by adding the attribute weights for each subset of meta-data and subtracting a multiple of the attribute weights of each subset of meta-data;

changing the attribute weights based on external events by computing future attribute weights from past attribute weights by composing past attribute weights with external event weights;

storing the video data in a video storage area;

storing the meta-data, indexed by date and time stamp to the video data, in a meta-data storage area; and

storing the attribute weights in an attribute storage area,

wherein attribute weights for the intersection meta-data is calculated using the equation;

W(M₁∩

M₂)=W(M₁)·

W(M₂),

wherein attribute weights for the union meta-data is calculated using the equation;

W(M₁∪

M₂)=W(M₁)+W(M₂)−

W(M₁)·

W(M₂), and

wherein M₁ and M₂ are two subsets of meta-data, W(M₁) is an attribute weight associated with subset M₁, W(M₂) is an attribute weight associated with subset M₂, W(M₁∪

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂, and W(M₁∪

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂.

2. The method of claim 1, wherein the attribute weights are changed based on external events by computing future attribute weights from past attribute weights by composing past attribute weights with external event weights as shown in the equation:

$\left[\begin{array}{c}{w}_1\\ w_2\\ \mathrm{\dots <br><br>}\\ w_j\end{array}\right]=\left[e_1,e_2,\mathrm{\dots <br><br>}<br><br>,e_n\right]·<br><br>\left[\begin{array}{c}{w}_1\\ w_2\\ \mathrm{\dots <br><br>}\\ w_i\end{array}\right],$

where w_j are future attribute weights, w_i are past attribute weights, and e_n are external event weights.

3. The method of claim 1, further comprising:

receiving video tips from one or more anonymous sources, the video tips being short video clips captured by citizens;

generating tip meta-data based on the video tips, the tip meta-data representing events detected in the video tips; and

determining tip attribute weights for the tip meta-data, representing information about the relevance of the tip meta-data.

4. The method of claim 1, further comprising:

providing additional meta-data generated by a human operator; and

storing the additional human generated meta-data, indexed to the video data by date and time stamp, in the meta-data storage module.

5. The method of claim 1, further comprising:

retrieving historical meta-data from the meta-data storage module;

evaluating a set of rules based on the historical meta-data and the generated meta-data; and

performing one or more actions based on the evaluation of the set of rules.

6. The method of claim 1, wherein the video storage module is a hierarchical storage module that archives the video data based at least on meta-data and attribute weights associated with the video data.

7. The method of claim 1, further comprising:

storing access privileges for the video data, the meta-data, and the attribute weights.

8. The method of claim 1, further comprising:

encrypting the captured video data before storing the video data.

9. The method of claim 1, wherein the video data is stored off-site.

10. A video surveillance system, comprising:

one or more surveillance cameras for capturing video data;

one or more video storage areas for storing video data;

a meta-data storage area for storing meta-data;

an attribute storage area for storing attribute weights; and

a processor, the processor coupled to the video storage areas, the meta-data storage area, and the attribute storage area, the processor adapted to execute program code to;

capture video data from one or more surveillance cameras;

generate meta-data by performing video analysis on the video data from the surveillance cameras, the meta-data representing events detected in the video data;

determine attribute weights, representing information about the relevance of the meta-data;

generate intersections of two or more subsets of the meta-data to generate intersection meta-data;

determine attribute weights associated with the intersection meta-data by multiplying the attribute weights for each subset of meta-data;

generate unions of two or more subsets of the meta-data to generate union meta-data;

determine attribute weights associated with the union meta-data by adding the attribute weights for each subset of meta-data and subtracting a multiple of the attribute weights of each subset of meta-data;

change the attribute weights based on external events by computing future attribute weights from past attribute weights by composing past attribute weights with external event weights;

store the video data in a video storage area;

store the meta-data, indexed by date and time stamp to the video data, in a meta-data storage area; and

store the attribute weights in an attribute storage area,

wherein attribute weights for the intersection meta-data is calculated using the equation;

W(M₁∩

M₂)=W(M₁)·

W(M₂),

wherein attribute weights for the union meta-data is calculated using the equation;

W(M₁∪

M₂)=W(M₁)+W(M₂)−

W(M₁)·

W(M₂), and

wherein M₁ and M₂ are two subsets of meta-data, W(M₁) is an attribute weight associated with subset M₁, W(M₂) is an attribute weight associated with subset M₂, W(M₁∩

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂, and W(M₁∪

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂.

11. The apparatus of claim 10, wherein the attribute weights are changed based on external events by computing future attribute weights from past attribute weights by composing past attribute weights with external event weights as shown in the equation:

$\left[\begin{array}{c}{w}_1\\ w_2\\ \mathrm{\dots <br><br>}\\ w_j\end{array}\right]=\left[e_1,e_2,\mathrm{\dots <br><br>}<br><br>,e_n\right]·<br><br>\left[\begin{array}{c}{w}_1\\ w_2\\ \mathrm{\dots <br><br>}\\ w_i\end{array}\right],$

where w_j are future attribute weights, w_i are past attribute weights, and e_n are external event weights.

12. The apparatus of claim 10, wherein the processor further comprises program code to:

receive video tips from one or more sources, the video tips being short video clips captured by citizens;

generate tip meta-data based on the video tips;

determine tip attribute weights for the tip meta-data; and

store the video tips in the video storage areas.

13. The apparatus of claim 10, wherein the processor further comprises program code to:

provide additional meta-data generated by a human operator; and

store the additional human generated meta-data, indexed to the video data by date and time stamp, in the meta-data storage module.

14. The apparatus of claim 10, wherein the processor further comprises program code to:

retrieve historical meta-data from the meta-data storage module;

evaluate a set of rules based on the historical meta-data and the generated meta-data; and

perform one or more actions based on the evaluation of the set of rules.

15. The apparatus of claim 10, further comprising:

a hierarchical video storage module adapted to archive the video data based at least on meta-data and attribute weights associated with the video data.

16. The apparatus of claim 10, further comprising:

a fiber optic line to an off-site location for archiving the video data off-site.

17. A method of searching and retrieving video data from a video surveillance system, the method comprising:

entering a search criteria;

searching meta-data associated with the video data, the meta-data generated by one or more video detection components and indexed to the video data;

retrieving meta-data matching the search criteria from a meta-data module;

retrieving video data indexed by the meta-data from a video storage module; and

retrieving attribute weights associated with the meta-data, the attribute weights representing reliability of the meta-data,

wherein attribute weights for intersection meta-data of two sub-sets of meta-data is calculated using the equation;

W(M₁∩

M₂)=W(M₁)·

W(M₂),

wherein attribute weights for union meta-data of two sub-sets of meta-data is calculated using the equation W(M₁∪

M₂)=W(M₁)+W(M₂)−

W(M₁)·

W(M₂),

wherein M₁ and M₂ are two subsets of meta-data, W(M₁) is an attribute weight associated with subset M₁, W(M₂) is an attribute weight associated with subset M₂, W(M₁∩

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂, and W(M₁∪

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂.

18. The method of claim 17, wherein the attribute weights includes data about the source of the meta-data.

19. The method of claim 17, further comprising:

storing audit information about who and when retrieved the video data.

20. An apparatus for storing video data, associated meta-data, and associated attribute weights from a video surveillance system, the apparatus comprising:

means for capturing video data from one or more surveillance cameras;

means for generating meta-data by performing video analysis on the video data from the surveillance cameras, the meta-data representing events detected in the video data;

means for determining attribute weights, representing information about the relevance of the meta-data;

means for generating intersections of two or more subsets of the meta-data to generate intersection meta-data;

means for determining attribute weights associated with the intersection meta-data by multiplying the attribute weights for each subset of meta-data;

means for generating unions of two or more subsets of the meta-data to generate union meta-data;

means for determining attribute weights associated with the union meta-data by adding the attribute weights for each subset of meta-data and subtracting a multiple of the attribute weights of each subset of meta-data;

means for changing the attribute weights based on external events by computing future attribute weights from past attribute weights by composing past attribute weights with external event weights;

means for storing the video data in a video storage area;

means for storing the meta-data, indexed by date and time stamp to the video data, in a meta-data storage area; and

means for storing the attribute weights in an attribute storage area,

wherein attribute weights for the intersection meta-data is calculated using the equation;

W(M₁∩

M₂)=W(M₁)·

W(M₂),

wherein attribute weights for the union meta-data is calculated using the equation;

W(M₁∪

M₂)=W(M₁)+W(M₂)−

W(M₁)·

W(M₂), and

wherein M₁ and M₂ are two subsets of meta-data, W(M₁) is an attribute weight associated with subset M₁, W(M₂) is an attribute weight associated with subset M₂, W(M_i∩

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂, and W(M₁∪

M₂) is a calculated attribute weight associated with the intersection meta-data of subset M₁ and subset M₂.

21. The apparatus of claim 20, wherein the attribute weights are changed based on external events by computing future attribute weights from past attribute weights by composing past attribute weights with external event weights as shown in the equation:

$\left[\begin{array}{c}{w}_1\\ w_2\\ \mathrm{\dots <br><br>}\\ w_j\end{array}\right]=\left[e_1,e_2,\mathrm{\dots <br><br>}<br><br>,e_n\right]·<br><br>\left[\begin{array}{c}{w}_1\\ w_2\\ \mathrm{\dots <br><br>}\\ w_i\end{array}\right],$

where w_j are future attribute weights, w_i are past attribute weights, and e_n are external event weights.

22. The apparatus of claim 20, further comprising:

means for receiving video tips from one or more anonymous sources, the video tips being short video clips captured by citizens;

means for generating tip meta-data based on the video tips, the tip meta-data representing events detected in the video tips; and

means for determining tip attribute weights for the tip meta-data, representing information about the relevance of the tip meta-data.