Intelligent, dynamic, long-term digital surveilance media storage system

• US 7,751,632 B2
• Filed: 02/15/2005
• Issued: 07/06/2010
• Est. Priority Date: 02/15/2005
• Status: Expired due to Fees

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Liu, Lipin, OZDEMIR, Hasan Timucin, Lee, K. C.
• Primary Examiner(s): Hung, Yubin
• Application Number:
  US11/058,558
• Publication Number:
  US 20060182357A1
• Time in Patent Office: 1,967 Days
• Field of Search: 382/232, 382/239, 382/305, 345/555, 348/143, 358/403, 358/404, 358/426.02, 358/426.05, 358/426.06, 707/204
• US Class Current: 382/239
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19669 Event triggers storage or change of storage policy

Abstract

An Intelligent, dynamic, long-term digital Surveillance Media storage System (ISMS), proposes an intelligent, dynamic, digital media storage mechanism for digital surveillance applications that can analyze media streams in real-time and provide flexible, scalable, and self-adjustable encoding and indexing mechanisms based on actual needs for digital surveillance systems. The system includes several subsystems, such as: (1) a media stream retriever/receiver; (2) a content-based description generator; (3) an intelligent engine with metadata description storage and media file storage (4) an event manager; (5) a media stream writer; and (6) an encoding optimizer.

Claims

1. A digital surveillance media storage system, comprising:

a media analyzer receiving a media stream and decomposing the media stream into a plurality of media segments based on scenes recorded in the media stream, each media segment of the plurality of media segments having an indicia of a starting location for the media segment in the media stream and a duration time for the media segment;

an encoding format selector receiving content semantic information for each media segment of the plurality of media segments and selecting an encoding format for each media segment of the plurality of media segments from plural encoding formats based on the content semantic information for that media segment and scenes recorded in that media segment, wherein the encoding format selector selects;

a low-frame rate or skipping frame encoding when no change in motion is detected;

an object based encoding when low motion objects are detected; and

a frame-based encoding when high motion video is detected;

an encoder engine receiving the plurality of media segments from the media analyzer and encoding the media segments according to the encoding format selected by said encoding format selector, where different encoding formats are selected for different media segments; and

a file writer receiving the plurality of encoded media segments from the encoder engine and storing information describing the plurality of encoded media segments for the media stream in a single media file.

2. The system of claim 1, wherein the media file further includes metadata description information for the media stream in the media file.

3. The system of claim 1, further comprising an encoding optimizer periodically re-encoding encoded media streams already stored in the media file to reduce storage utilization while maintaining accuracy of data based on a predefined set of application criteria, wherein a sampling rate of the re-encoding is changed based on an age of the media stream to be re-encoded.

4. The system of claim 3, wherein said encoding optimizer processes the media file in multiple stages over time, each stage including dynamic selection of an encoding format based on an amount of available storage space, stream lifetime, stream priority, and application-defined accuracy requirements.

5. The system of claim 4, wherein the plural encoding formats include a skipping frames encoding format that can be dynamically selected in multiple stages of file encoding processes over a life time of the media file in the system.

6. The system of claim 5, wherein the media file propagates over time from relatively high accuracy and large size to relatively low accuracy and small size, the media file eventually being reduced to only a description of the media content based on MPEG7 before it is archived to secondary storage.

7. The system of claim 1, wherein said encoding format selector responds to semantic based content extraction results and event metadata to decide importance of an image before selection of an encoding method.

8. The system of claim 7, wherein the media file is re-processed after a predefined period of time to further reduce its size, and semantic based multi-stage compression is driven by user defined rules.

9. The system of claim 1, wherein the plural encoding formats include an unchanged frame-based encoding format, wherein an unchanged portion of an image is stored in high resolution as a background frame.

10. The system of claim 9, wherein a background frame is updated regularly to adapt to changing background conditions.

11. The system of claim 1, further comprising an event manager receiving and managing events of the media stream, wherein said encoding format selector further receives input from said event manager and selects the one of plural encoding formats based on the input.

12. The system of claim 11, further comprising a media stream receiver receiving a media stream and routing received events to said event manager while routing received media stream to said media analyzer.

13. The system of claim 1, further comprising a descriptor generator supplying descriptors for the media segments based on analysis of the media stream.

14. The system of claim 13, further comprising a file writer receiving encoded media segments and descriptors and storing the encoded media segments with the descriptors in the media file.

15. A digital surveillance media storage system of claim 1, wherein the single media file includes (a) media creation information;

(b) pointers to the plurality of encoded media segments;

(c) media segmentation information, of the plurality of encoded media segments, including media encoding, media type, time, and media storage location; and

(d) content description of the plurality of encoded media segments.

16. The system of claim 1, wherein the media analyzer selects a plurality of base frames in a media segment to determine an occurrence of incremental changes in frames that do not exceed the first predetermined threshold.

17. The system of claim 1, wherein the importance of the media contents become less important as time passes.

20. The system of claim 16 where the encoding optimizer re-encodes media segments in the subset of media segments in a manner that reduces the size of a given media segment according to age of the given media segment and content semantic information associated with the given media segment.

18. A digital surveillance media storage system, comprising:

a media analyzer receiving a media stream and decomposing the media stream into a plurality of media segments based on scenes recorded in the media stream, each media segment of the plurality of media segments having a duration time associated therewith;

an encoding format selector receiving content semantic information for each media segment of the plurality of media segments and selecting an encoding format for each media segment of the plurality of media segments based on the content semantic information for that media segment and scenes recorded in that media segment, wherein the encoding format selector selects

a low-frame rate or skipping frame encoding when no change in motion is detected;

an object based encoding when low motion objects are detected; and

a frame-based encoding when high motion video is detected;

an encoder engine receiving the plurality of media segments from the media analyzer and encoding the media segments according to the encoding format selected by said encoding format selector, where different encoding formats are selected for different media segments

a file writer receiving the plurality of encoded media segments from the encoder engine and storing information describing the plurality of encoded media segments for the media stream in a single media file; and

an encoding optimizer periodically retrieves the media file and selectively re-encodes a subset of the plurality of media segments which comprise the media stream.

19. The system of claim 18 where the encoding optimizer re-encodes media segments in the subset of media segments in a manner that reduces the size of a given media segment based on the content semantic information associated with the given media segment.

21. The system of claim 18 further comprises a description generator that supplies descriptors for each media segment, wherein the encoding optimizer selectively deletes media segments from the media file but retains descriptors for the media segments in the media file.

22. A digital surveillance system, comprising:

a plurality of surveillance cameras monitoring a surveillance domain and generating a media stream of the surveillance domain;

a domain knowledge profile defining settings of the digital surveillance system based on security requirements of the surveillance domain;

a media analyzer receiving the media stream and decomposing the media stream into a plurality of media segments based on scenes recorded in the media stream, each media segment of the plurality of media segments having an indicia of a starting location for the media segment in the media stream and a duration time for the media segment;

an encoding format selector receiving content semantic information for each media segment of the plurality of media segments and selecting an encoding format for each media segment of the plurality of media segments from plural encoding formats based on a) the content semantic information for that media segment, b) scenes in that media segment, and c) the domain knowledge profile, wherein the encoding format selector selects

a low-frame rate or skipping frame encoding when no change in motion is detected;

an object based encoding when low motion objects are detected; and

a frame-based encoding when high motion video is detected;

an encoder engine receiving the plurality of media segments from the media analyzer and encoding the media segments according to the encoding format selected by said encoding format selector, where different encoding formats are selected for different media segments; and

a file writer receiving the plurality of encoded media segments from the encoder engine and storing information describing the plurality of encoded media segments for the media stream in a single media file.

23. A digital surveillance system of claim 22, wherein the surveillance domain is a site to be monitored by the plurality of surveillance cameras;

wherein the dominant knowledge profile further contains user configurations, for the surveillance system, of media quality, selected encoding modes, and customized conditions.

24. A digital surveillance system of claim 23, wherein the domain knowledge profile further contains:

(a) rules for selecting an encoding mode, which is configured based on actual needs of surveillance applications of the surveillance system;

(b) selectable encoder modules which are extendable for importing a new encoder and registering the new encoder in the surveillance system;

(c) metadata description scheme;

(d) settings and rules for determining actions for different application domains and security situations of the site.