Modular immersive surveillance processing system and method

• US 8,063,936 B2
• Filed: 06/01/2005
• Issued: 11/22/2011
• Est. Priority Date: 06/01/2004
• Status: Expired due to Fees

• Current Assignee: L-3 Communications Corporation (L3Harris Technologies, Inc.)
• Original Assignee: L-3 Communications Corporation (L3Harris Technologies, Inc.)
• Inventors: Arpa, Aydin, Hanna, Keith, Miller, Philip, Samarasekera, Supun, Kumar, Rakesh, Paragano, Vincent, Sawhney, Harpreet, Aggarwal, Manoj
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US11/628,376
• Publication Number:
  US 20070247525A1
• Time in Patent Office: 2,365 Days
• Field of Search: 348/143, 348/157, 348/159
• US Class Current: 348/159
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19667 Details realated to data compression, encryption or encoding, e.g. resolution modes for reducing data volume to lower transmission bandwidth or memory requirements
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

According to an aspect of the invention, a system for providing immersive surveillance a site has a plurality of cameras each producing a respective raw video of a respective portion of the site. A processing component receives the raw video from the cameras and generates processed video from it. A visualization engine is coupled to the processing system, and receives the processed video therefrom. The visualization engine renders real-time images corresponding to a view of the site in which at least a portion of the processed video is overlaid onto a rendering of an image based on a computer model of the site. The visualization engine displays the images in real time to a viewer. The processing component comprises first and second filter modules. The second filter module processes video received as output from the first filter module. A controller component controls all transmission of data and video between the first and second filter modules.

Claims

1. A system for providing immersive surveillance of a site, said system comprising:

a plurality of cameras each producing a respective raw video of a respective portion of the site;

a processing component receiving said raw video from the cameras and generating therefrom processed video;

a visualization engine coupled to the processing component and receiving said processed video therefrom, said visualization engine rendering real-time images corresponding to a view of the site in which at least a portion of said processed video is overlaid onto a rendering of an image based on a computer model of the site, the visualization engine displaying said images in real time to a viewer; and

said processing component comprising first and second filter modules, the second filter module processing video received as output from the first filter module; and

a controller component controlling all transmission of video from the first filter module to said second filter module; and

wherein the computer model is a 3-D model of the site.

2. The system of claim 1 wherein the first and second filter modules are software controlled processes run on separate computers.

3. The system of claim 1 wherein the first filter module comprises a data reading and converting module that reads and converts the raw video from the plurality of cameras into converted video having a format suitable for further processing.

4. The system of claim 3 wherein the second filter module comprises a video processing module coupled to the data reading and converting module and receiving the converted video output therefrom, and further processing the converted video to data-fusion ready video for fusion with meta-data.

5. The system of claim 4 wherein the processing component comprises a third filter module that receives the data-fusion ready video from the second filter module, said controller component controlling transmission of all data and video between the second and third filter modules.

6. The system of claim 5, wherein said third filter module performs data fusion on said fusion ready video to yield said processed video.

7. The system of claim 6 wherein the second and third filter modules are each processes run on different computers.

8. The system of claim 6, and further comprising a rule engine coupled to the third filter module.

9. The immersive surveillance system of claim 1, and further comprising storing the processed video with a data storage module.

10. A method for processing video in an immersive surveillance system for a site, said method comprising:

receiving raw video from a plurality of video cameras;

processing said raw video so as to yield processed video;

transmitting said processed video to a visualization engine applying at least part of said processed video onto a rendering of an image based on a 3-D computer model of the site; and

displaying the rendered image with said video overlaid to a user;

said processing of said raw video to processed video being performed in at least two discrete filter steps by at least two filter modules, one filter module processing output of the other filter module; and

controlling with a master controller transmission of said output between the two filter modules.

11. The method of claim 10 wherein the first and second filter processes are performed by different computers.

12. The method of claim 10 wherein the processing of one of said filter modules includes data reading or converting the raw video.

13. The method of claim 10 wherein the processing of one of said filter modules includes preparing the video for data fusion with meta-data.

14. The method of claim 10 wherein the processing of one of said filter modules includes fusing meta data with the video.

15. The method of claim 10 wherein the processing of said raw video to processed video is performed in at least three discrete filter steps by the two filter modules and a third filter module, with all data transmission therebetween controlled by said controller.

16. The method of claim 15 wherein the filter steps are data reading/converting;

component processing; and

data fusion, respectively.

17. The method of claim 10 wherein, while the second step of processing is being performed by the second filter module, the first filter module is performing the first filter step on a subsequently received raw video.

18. The method of claim 10 wherein said processed video is transmitted to said visualization engine.

19. The method of claim 10 wherein the filter modules are instantiated based on XML files, and wherein the transmission of data between said filter modules is configured by said XML files.