IMAGING SYSTEMS AND METHODS FOR IMMERSIVE SURVEILLANCE

• US 20120169842A1
• Filed: 12/15/2011
• Published: 07/05/2012
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Candell, Lawrence M., Panasyuk, Svetlana V., Chuang, Daniel B., Ross, William D., Beattie, Mark E., Fang, Cindy Y., Ren, Bobby, Blanchard, Jonathan P., Long, Gary M. Jr., White, Lauren L., Bury, Mark
• Granted Patent: US 9,007,432 B2
• Time in Patent Office: Days
• US Class Current: 348/39
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19619 Details of casing
• CPC Class Codes: G08B 13/19628 of wide angled cameras and camera groups, e.g. omni-directional cameras, fish eye, single units having multiple cameras achieving a wide angle view
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Security guards at large facilities, such as airports, monitor multiple screens that display images from individual surveillance cameras dispersed throughout the facility. If a guard zooms with a particular camera, he may lose image resolution, along with perspective on the surrounding area. Embodiments of the inventive Imaging System for Immersive Surveillance (ISIS) solve these problems by combining multiple cameras in one device. When properly mounted, example ISIS systems offer up to 360-degree, 240-megapixel views on a single screen. (Other fields of view and resolutions are also possible.) Image-stitching software merges multiple video feeds into one scene. The system also allows operators to tag and follow targets, and can monitor restricted areas and sound an alert when intruders breach them.

Claims

1. A system for monitoring a wide-area scene, the system comprising:

an array of first cameras, each first camera having a first field of view and configured to provide respective first imagery of a corresponding portion of the wide-area scene; and

an array of second cameras, each second camera having a second field of view different than the first field of view and configured to provide respective second imagery of a corresponding portion of the wide-area scene.

2. The system of claim 1 wherein each first camera comprises a first lens having a first focal length and wherein each second camera comprises a second lens having a second focal length different than the first focal length.

3. The system of claim 1 wherein each first field of view is a first angular field of view and wherein each second field of view is a second angular field of view.

4. The system of claim 3 wherein each first angular field of view is about 10 degrees to about 30 degrees and each second angular field of view is about 30 degrees to about 50 degrees.

5. The system of claim 1 wherein each first field of view overlaps with one or more other first fields of view and wherein each second field of view overlaps with one or more other second fields of view.

6. The system of claim 1 wherein at least one first field of view overlaps with one or more second fields of view.

7. The system of claim 1 wherein at least one of the first imagery and the second imagery comprises an annular portion of the wide-area scene.

8. The system of claim 1 wherein the first imagery and the second imagery, taken together, provide an image of substantially all of the wide-area scene.

9. The system of claim 8 wherein the image of substantially all of the wide-area scene has a spatial resolution of about 0.5 cm to about 50 cm.

10. The system of claim 1 further comprising:

an array of third cameras, each third camera having a respective third field of view different than the second field of view.

11. The system of claim 1 further comprising:

a processor operably coupled to the array of first cameras and the array of second cameras and configured to form a spatially continuous image of substantially all of the wide-area scene from the first imagery and the second imagery.

12. The system of claim 11 wherein the processor is further configured to register the first imagery to the second imagery to form the spatially continuous image.

13. The system of claim 11 wherein the processor is further configured to register the first imagery and the second imagery to a reference image of the wide-area scene, and further comprising:

a panoramic camera operably coupled to the processor and configured to provide the reference image.

14. The system of claim 11 wherein the processor is further configured to correct and balance color of the first imagery and the second imagery to form the spatially continuous image.

15. The system of claim 1 further comprising:

an image processor operably coupled to the array of first cameras and the array of second cameras and configured to decompose the first imagery and the second imagery into image tiles and to compress the image tiles at each of a plurality of resolutions;

a memory operably coupled to the compression engine and configured to store the compressed image data; and

a server operably coupled to the memory and configured to serve one or more image tiles at one of the plurality of resolutions in response to a request for an image of a particular portion of the wide-area scene.

16. The system of claim 1 further comprising:

a housing to hold the array of first cameras and the array of second cameras;

a heat exchanger in thermal communication with at least a portion of the housing to remove heat from the housing;

two surfaces defining a channel to convey gas through the housing and across at least a portion of the heat exchanger so as to remove heat from the heat exchanger; and

a chamfer connecting the two surfaces defining the channel.

17. A method of acquiring an image of a wide-area scene, the method comprising:

(A) acquiring first imagery of a plurality of first portions of the wide-area scene at a first field of view;

(B) acquiring second imagery of a plurality of second portions of the wide-area scene at a second field of view different than the first field of view;

(C) forming the image of the wide-area scene from the first imagery and the second imagery.

18. The method of claim 17 wherein each first field of view is a first angular field of view and wherein each second field of view is a second angular field of view.

19. The method of claim 18 wherein each first field of view is about 10 degrees to about 30 degrees and each second field of view is about 30 degrees to about 50 degrees.

20. The method of claim 17 wherein each first field of view overlaps with one or more other first fields of view and wherein each second field of view overlaps with one or more other second fields of view.

21. The method of claim 17 wherein at least one first field of view overlaps with one or more second fields of view.

22. The method of claim 17 wherein at least one of the first imagery and the second imagery comprises an annular portion of the wide-area scene.

23. The method of claim 17 wherein the first imagery and the second imagery, taken together, provide an image of substantially all of the wide-area scene.

24. The method of claim 17 wherein the image of substantially all of the wide-area scene has a spatial resolution of about 0.5 cm to about 50 cm.

25. The method of claim 17 further comprising:

acquiring third imagery of a plurality of third portions of the wide-area scene at a third field of view different than the second field of view; and

wherein (C) further comprises forming the image of the wide-area scene from the first, second, and third imagery.

26. The method of claim 17 wherein (C) further comprises:

forming a spatially continuous image of substantially all of the wide-area scene from the first imagery and the second imagery.

27. The method of claim 17 wherein (C) further comprises:

registering the first imagery to the second imagery to form the spatially continuous image.

28. The method of claim 17 wherein (C) further comprises:

registering the first imagery and the second imagery to a reference image of the wide-area scene.

29. The method of claim 17 wherein (C) further comprises:

correcting and balancing color of the first imagery and the second imagery.

30. The method of claim 17 further comprising:

(D) decomposing the first imagery and the second imagery into image tiles; and

(E) compressing the image tiles at each of a plurality of resolutions.

31. The method of claim 30 further comprising:

(F) receiving a request for an image of a particular portion of the wide-area scene;

(G) determining a minimum resolution of the plurality of resolutions necessary to render the image; and

(H) serving one or more image tiles at the minimum resolution in response to the request for the image.

32. The method of claim 30 further comprising:

storing the compressed image tiles in an image format.

33. A surveillance system comprising:

an array of first cameras, each first camera having a first angular field of view and configured to provide respective first real-time imagery of a corresponding portion of the wide-area scene; and

an array of second cameras, each second camera having a second angular field of view different than the first angular field of view and configured to provide respective second real-time imagery of a corresponding portion of the wide-area scene.

a processor operably coupled to the array of first cameras and the array of second cameras and configured to decompose the first real-time imagery and the second real-time imagery into image tiles and to compress the image tiles at each of a plurality of resolutions;

a server operably coupled to the processor and configured to serve one or more image tiles at one of the plurality of resolutions in response to a request for an image of a particular portion of the wide-area scene; and

an interface communicatively coupled to the server and configured to render a real-time image of the wide-area scene represented by the one or more image tiles.

34. A method of determining a model representing views of a scene from cameras in an array of cameras, each camera in the array of cameras having a field of view that overlaps with the field of view of another camera in the array of cameras, the method comprising:

(A) for each pair of overlapping fields of view;

(A1) selecting image features in a region common to the overlapping fields of view;

(A2) matching points corresponding to a subset of the image features in one field of view in the pair of overlapping fields of view to points corresponding the subset of the image features in the other field of view in the pair of overlapping fields to form a set of matched points;

(B) merging at least a subset of each set of the matched points to form a set of merged points; and

(C) estimating parameters associated with each field of view based on the set of merged points to form the model representing the views of the scene.

35. The method of claim 34 wherein (A1) comprises identifying candidate image features using a random sample and consensus procedure.

36. The method of claim 34 wherein (A2) comprises calculating a homography that maps points from one field of view in the pair of overlapping fields of view to the other field of view in the pair of overlapping fields of view.

37. The method of claim 34 wherein (A2) comprises:

projecting points corresponding to the image features from one field of view in the pair of overlapping fields of view onto the other field of view in the pair of overlapping fields of view; and

for each projected point, determining whether the projected point lies within a predefined distance of a corresponding point in the other field of view.

38. The method of claim 37 further comprising:

selecting the subset of each set of matched points from among those projected points that lie within the predefined distance of the corresponding points.

39. The method of claim 34 further comprising:

(D) calculating a re-projection error associated with model representing the views of the scene; and

(E) weighting the parameters to reduce the re-projection error.

40. A method of compensating for imbalances in color and white levels in color images of respective portions of a wide-area scene, each color image comprising red, green, and blue color channels acquired by a respective camera in a camera array disposed to image the wide-area scene, the method comprising:

(A) normalizing values representing the red, green, and blue color channels to a reference value representing a response of the cameras in the camera array to white light;

(B) equalizing the values representing the red, green, and blue color channels to red, green, and blue equalization values, respectively;

(C) identifying high- and low-percentile values among each of the red, green, and blue color channels; and

(D) scaling each of the red, green, and blue color channels based on the high- and low-percentile values to provide compensated values representing the red, green, and blue color channels.

41. The method of claim 40 wherein (B) further comprises:

(B1) calculating red, green, and blue median values for the red, green, and blue color channels, respectively;

(B2) calculating red, green, and blue equalization coefficients by dividing the red, green, and blue median values, respectively, by the mean of the red, green, and blue median values; and

(B3) scaling the values representing the red, green, and blue color channels by the red, green, and blue equalization coefficients, respectively.

42. An interface for a surveillance system that monitors a scene, the interface comprising:

a full-scene view configured to render a real-time panoramic image of the entire scene monitored by the surveillance system; and

a zoom view configured to render a close-up of a region of the panoramic view.

43. The interface of claim 42 wherein the full-scene view is further configured to enable a user to select a region of the scene for display in the zoom view.

44. The interface of claim 42 wherein the interface is further configured to enable at least one user to set at least one zone in the panoramic image to be monitored for activity.

45. The interface of claim 44 wherein the interface is further configured to alert the at least one user upon detection of activity in the at least one zone.

46. The interface of claim 44 wherein the interface is further configured to populate an activity database with an indication of detected activity in the at least one zone.

47. The interface of claim 46 further comprising:

an activity view configured to display the indication of detected activity to the at least one user in a manner that indicates a time and a location of the detected activity.

48. The interface of claim 46 wherein the interface is further configured to display images of detected activity in at least one of the full-scene view and the zoom view.

49. The interface of claim 42 wherein the interface is further configured to track a target throughout the scene and to display an indication of the target'"'"'s location in at least one of the full-scene view and the zoom view.

50. The interface of claim 49 wherein the interface is further configured to enable a user to select the target.

51. The interface of claim 42 wherein the interface is further configured to display a pre-warped image in at least one of the full-scene view and the zoom view.