Method and apparatus for providing immersive surveillance

• US 7,522,186 B2
• Filed: 07/24/2002
• Issued: 04/21/2009
• Est. Priority Date: 03/07/2000
• Status: Expired due to Fees

• Current Assignee: L-3 Communications Corporation (L3Harris Technologies, Inc.)
• Original Assignee: L-3 Communications Corporation (L3Harris Technologies, Inc.)
• Inventors: Hsu, Stephen, Nister, David, Samarasekera, Supun, Arpa, Aydin, Hanna, Keith J., Kumar, Rakesh, Aggarwal, Manoj, Sawhney, Harpreet Singh
• Primary Examiner(s): WONG, ALLEN C
• Application Number:
  US10/202,546
• Publication Number:
  US 20030085992A1
• Time in Patent Office: 2,463 Days
• Field of Search: 348/153, 348/152, 348/143, 725/131, 345/419, 345/582, 382/154
• US Class Current: 348/153
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method and apparatus for providing immersive surveillance wherein a remote security guard may monitor a scene using a variety of imagery sources that are rendered upon a model to provide a three-dimensional conceptual view of the scene. Using a view selector, the security guard may dynamically select a camera view to be displayed on his conceptual model, perform a walk through of the scene, identify moving objects and select the best view of those moving objects and so on.

Claims

1. A surveillance system for monitoring a scene comprising:

a three-dimensional model of the scene defining surfaces corresponding to surfaces of real objects in the scene;

a plurality of cameras each producing a respective video capturing at least a respective portion of the scene;

an image processor, coupled to said plurality of cameras, constructing a texture map from image data of one or more of the videos and projecting said texture map onto one or more of the surfaces of the model of the scene and rendering a sequence of images representing a view of the scene with the texture map of the image data on said one or more surfaces therein;

a display, coupled to the image processor, displaying the image to a user; and

a view controller, coupled to the image processor, for selecting views of the scene to be displayed; and

wherein the constructing of the texture map includes identifying, for each of said one or more surfaces or portions thereof, a subset of the videos that contain said surface or said portion of said surface; and

wherein said subset of videos does not include any of the videos that has a view of said surface or portion thereof that is occluded by any of the other surfaces of the model.

2. The surveillance system of claim 1 wherein said image processor further comprises:

means for identifying moving objects in each of the videos;

means for producing an icon representing each of the identified moving objects; and

means for inserting the icon into the combination.

3. The surveillance system of claim 2 further comprising:

means for identifying a person as a moving object; and

means for annotating the display with a description of the person.

4. The system of claim 1, wherein, where at least two of the video images contain the portion, the texture map is derived from the video images based on the video image of said subset of video images that is selected as having a most frontal view of the surface or portion thereof.

5. A method of providing surveillance of a scene comprising:

producing a plurality of videos of the scene;

producing a three dimensional model of the scene, said model defining surfaces corresponding to surfaces of real objects in the scene;

selecting a viewpoint for the three dimensional model;

aligning the plurality of videos with the three dimensional model;

rendering said plurality of videos onto the three dimensional model to form a combination view comprising a sequence of output images in which the videos are draped onto the three dimensional model, wherein texture maps on at least some of the surfaces of the three dimensional model are generated from the videos;

each of said videos having a resolution relative to each of the surfaces that said video contains, and each of the texture maps for the surfaces being generated by selecting from a subset of the videos each containing the surface the video having the resolution for said surface that is the highest of said resolutions of the videos of the subset; and

displaying the combination view.

6. The method of claim 5 wherein the selected viewpoint is a bird'"'"'s eye view.

7. The method of claim 5 further comprising:

selecting a location within the combination view; and

producing a view of the scene from the selected location.

8. The method of claim 5 further comprising:

identifying moving objects in the videos;

producing an icon representing each moving object; and

inserting the icon into the combination view.

9. The method of claim 8 wherein said identifying step further comprises:

identifying the object as a person; and

annotating the combination view with text representing the identification of the person.

10. The method of claim 5 further comprising:

storing the combination view for replay.

11. The method of claim 5, wherein the texture maps on the surfaces in the three dimensional model are derived by separating the surface into one or more portions of a predetermined size, and for each of the portions detecting a subset of video images that contain the portion and discarding those video images that are occluded from the portion by another surface of the model.

12. A system providing surveillance for a site, said system comprising:

a computer accessible memory storing data defining a computer-supported three-dimensional model of the site defining one or more surfaces that can be used to produce at least one computer-generated view of the site;

a plurality of cameras each having a respective actual view of a respective portion of the site under surveillance, each camera forming a respective video of said actual view, said video comprising electronic signals corresponding to a series of frame images captured by the camera;

a computerized image processing system forming a synthetic image of the site as seen from a viewpoint; and

a display device providing a display of said synthetic image so as to be viewed by a user;

said display of the synthetic image comprising a contextual view of the site wherein at least a first portion of the contextual view is derived from the video from one of the cameras by aligning the video with the model of the site, and wherein texture maps on one or more surfaces of the model of the site are generated from said video, the texture maps on each surface that is contained in a subset of more than one of the videos of said subset of videos being derived from the video containing said surface that is selected as having the highest resolution of said surface, said surfaces corresponding to surfaces of real objects in the site.

13. The system of claim 12, wherein the video in said display of said synthetic image is aligned with the model and displayed in real time in the contextual view, so that the user can use the system for real-time surveillance of the site.

14. The system of claim 13, wherein, in another portion of the contextual view that contains no video from the cameras, the image processor renders the view of the model from the viewpoint in said first portion by mapping a texture mopped onto the model.

15. The system of claim 13, wherein the video in the first portion is merged into said contextual view by texturing said surface or surfaces with said video and rendering the view of the textured model.

16. The system of claim 15, wherein aligned video from one or more of the cameras is textured onto said surfaces and viewed in the synthetic image.

17. The system of claim 13, wherein the system further comprises a viewpoint selector input device providing the user with a capability of changing the viewpoint for which the synthetic image is generated.

18. The system of claim 12, wherein the texture maps on the surfaces in the three dimensional model are derived by separating the surface into one or more portions of a predetermined size, and for each of the portions detecting a subset of video images that contain the portion and discarding those video images that are occluded from the portion by another surface of the model.

19. A method of providing surveillance of a real-world site, said method comprising:

providing a plurality of cameras each transmitting a respective video signal corresponding to a respective real-time view of a respective portion of the site, each video signal comprising signals defining a series of frames of said real-time view, each frame corresponding to said view during a respective time period;

receiving said video signals at a computerized image processing system having access to a computer accessible model of the real-world site under surveillance;

said computer accessible model being a three-dimensional model of the site and including data defining a plurality of surfaces of the model corresponding to surfaces of real objects of the real-world site;

rendering with the image processing system a real time surveillance display comprising a series of images each comprising a synthetic view of the model of the site from a viewpoint;

said synthetic view being derived from the model and one or more frames of the video signals selected such that the surveillance display output is a real-time display of the video signals used in the synthetic views;

said frames of the video signals being used to generate texture maps of one or more of the surfaces of the three-dimensional model in said rendering, the texture map for each of said surfaces in the model being derived by selecting the video image that, of a subset of the video images each containing the surface, has the highest resolution of the surface; and

displaying the surveillance display on a display device so as to be viewable by a user.

20. The method of claim 19, and further comprising:

receiving a viewpoint selection input from the user indicative of a desired change in the viewpoint; and

said image processor, responsive to said input, generating said surveillance display images as synthetic views of the model from a second viewpoint.

21. The method of claim 19, wherein the texture maps on the surfaces in the three dimensional model are derived by separating the surface into one or more portions of a predetermined size, and for each of the portions detecting a subset of video images that contain the portion and discarding those video images that are occluded from the portion by another surface of the model.

22. A method of providing surveillance of a scene, said method comprising:

creating a three-dimensional model of static parts of the scene, said model including a plurality of surfaces corresponding to surfaces of said static parts of the scene;

providing video cameras each transmitting respective video data comprising a series of video images each capturing appearance of a portion of the scene at respective time instants;

constructing one or more texture maps for the surfaces of the model so as to create a textured model, said texture maps being created by identifying, for one or more portions of the surfaces, a subset of the video images that each contain said portion and that are not occluded by another surface of the model, and projecting one or more color values for the portion of the surface from a pixel or pixels of one of the video images of said subset of video images, wherein said one of said video images has the highest resolution of the surface of said subset of videos; and

creating an output image sequence based on the textured model by rendering the textured model so as to form an image thereof as seen from a user-selected virtual viewpoint;

wherein the texture maps of the textured model are updated continuously using the incoming video data from the cameras so that the model contains the most currently available video data.

23. The method of claim 22, wherein the model is a polyhedral three-dimensional model and the surfaces are polygonal surfaces.

24. The method of claim 23, wherein said polygonal surfaces are separated into one or more approximately equally sized portions of a predetermined size.

25. The method of claim 22, wherein said rendering is with a graphics pipeline processor or processors.

26. The method of claim 22, wherein said video images of the subset are all from the same time instant.

27. The method of claim 22, wherein the selected virtual viewpoint is controlled manually by a user.

28. The method of claim 22, wherein the selected virtual viewpoint follows a predetermined pattern.

29. The method of claim 22, wherein Z buffering is used to detect the video images that contain said portion and that are not occluded by another surface of the model.

30. The method of claim 22, wherein the sequence of output images is displayed on a viewing screen.

31. The method of claim 22, wherein the video images are aligned in near real-time to the three-dimensional model.

32. A surveillance system for monitoring a scene comprising:

a plurality of cameras producing respective videos each capturing a respective portion of the scene being monitored;

a computer system supporting a polyhedral three-dimensional model of the scene having polygonal surfaces defining static objects of the scene corresponding to surfaces of real objects in the scene being monitored;

said computer system creating texture maps from incoming video data of the videos and projecting the texture maps onto some of the polygonal surfaces so that the model is a textured model, said texture maps being created by identifying, for one or more portions of a predetermined size of the polygonal surfaces, a subset of the video images that contain said portion and that are not occluded by any other surface of the polyhedral model, and determining one or more color values for the texture map on said portion from a pixel or pixels of one of the video images of said subset, wherein said one of the video images has the highest resolution or most frontal view of said portion of said subset of video images;

said computer system rendering the textured model so as to create an output image sequence of output images of the textured model of the scene as seen from a user-selected virtual viewpoint; and

an image display displaying the output images to a user;

the texture maps of the textured model being updated continuously using the incoming video data from the cameras so that the model contains the most currently available video data.

33. The system of claim 32, wherein the graphics hardware includes graphics pipeline processors.

34. The system of claim 32, wherein the computer system derives the texture map by selecting a polygonal surface of the model, dividing said selected surface into portions of said predetermined size, selecting one of the portions, detecting all incoming video images containing said portion and discarding any occluded images so as to obtain said subset of the video images, selecting the video image of said subset that has the highest resolution or most frontal view of the portion of the surface, and projecting a color value of the portion from the pixels in said highest resolution or most frontal view video image that corresponds to the portion.

35. The system of claim 32, wherein the output images are stored in digital memory so as to be rewound and replayed.

36. The system of claim 32, wherein at least one of said plurality of cameras is a PTZ camera that has pan, tilt and zoom (PTZ) capability, and wherein creating texture maps generated from said PTZ camera further includes capturing the orientation of the PTZ camera, determining the pose of the camera within the scene, and using said pose to align said video produced from said PTZ camera with the model.

37. The system of claim 32, wherein said computer system identifies moving objects in the videos, produces an icon representing each moving object, and inserts the icon into the output images.

38. The system of claim 32, wherein said computer system identifies moving objects in the videos, produces a two-dimensional object silhouette representing each moving object, and inserts the icon into the output images.

39. The system of claim 32, wherein said computer system further automatically selects a subset of the video images from a time instant containing a view of the scene that best displays a region of the scene for viewing a moving object.

40. A surveillance system for monitoring a scene comprising:

a plurality of cameras producing respective videos each capturing a respective portion the scene;

a computer system supporting a polyhedral three-dimensional model of the scene having polygonal surfaces defining static objects of the scene;

said computer system creating texture maps from incoming video data of the videos and projecting the texture maps onto some of the polygonal surfaces so that the model is a textured model, said texture maps being created by identifying, for one or more portions of a predetermined size of the polygonal surfaces, a subset of the video images from one of the time instants that contain said portion and that are not occluded by any other surface of the polyhedral model and determining one or more color values for the texture map on said portion from a pixel or pixels of one of the video images of said subset, wherein said one of the video images has the highest resolution view of said portion of said subset of video images;

said computer system having graphics hardware and rendering the textured model so as to create an output images sequence of output images of the textured model of the scene as seen from a user-selected virtual viewpoint; and

a viewing screen displaying the output images to a user;

the texture maps of the textured model being updated continuously using the incoming video data from the cameras so that the model contains the most currently available video data; and

wherein the computer system derives the texture map by selecting a polygonal surface of the model, dividing said selected surface into portions of said predetermined size, selecting one of the portions, detecting by Z-buffering all incoming video images containing said portion and discarding any occluded images so as to obtain said subset of the video images, selecting the video image of said subset that has the highest resolution view of the portion of the surface, and projecting a color value of the portion from the pixels in said highest resolution video image that correspond to the portion.