Intelligent high resolution video system

• US 9,584,710 B2
• Filed: 02/28/2008
• Issued: 02/28/2017
• Est. Priority Date: 02/28/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Analytics Corporation (Motorola Solutions, Inc.)
• Inventors: Marman, Douglas H., Saptharishi, Mahesh
• Primary Examiner(s): Vu, Viet
• Assistant Examiner(s): BELCHER, HERMAN A
• Application Number:
  US12/039,085
• Publication Number:
  US 20090219387A1
• Time in Patent Office: 3,288 Days
• Field of Search: 348/143
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19652 Systems using zones in a single scene defined for different treatment, e.g. outer zone gives pre-alarm, inner zone gives alarm
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/575 with a response composed of multiple slopes
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An automated electronic video surveillance system enables a high-resolution mega-pixel camera to capture high quality, detailed, magnified images at multiple locations, simultaneously with an overview of the whole scene. A preferred embodiment requiring no moving parts provides full 360-degree continuous viewing with up to 5× all-digital zoom capability. The system performs continuous surveillance and active resolution allocation in the form of a feedback control subsystem that dynamically allocates resources so that important details within a scene receive appropriate scrutiny, while uninteresting areas are imaged at a lower resolution.

Claims

1. A method of dynamically controlling an imager of a video camera to provide, with selectable resolution, image information relating to one or more regions of interest in a scene processed by the video camera, the imager having a specified maximum number of image pixels that establish a characteristic maximum image resolution of the imager and having a number of image pixels from which are produced video data that represent the scene, comprising:

controlling the imager to acquire high resolution video data representing substantially the specified maximum number of image pixels that capture a full field of view of the scene

controlling the imager to acquire a first series of sub-sampled video data from the high resolution video data corresponding to the scene at a first image resolution, the first series of sub-sampled video data produced by a first subset of image pixels composed of fewer image pixels than the specified maximum number of image pixels of the imager;

analyzing the sub-sampled video data in the first series to detect a region of interest within the scene; and

controlling the imager to acquire a second series of sub-sampled video data from the high resolution video data corresponding to the detected region of interest within the scene at a second image resolution thereby non-uniformly sub-sampling at least one frame of the video data, the second image resolution being higher than the first image resolution, the sub-sampled video data in the second series representing a windowed view of the detected region of interest in the scene.

2. The method of claim 1, in which the imager using the specified maximum number of image pixels captures a full field of view of the scene at the characteristic maximum image resolution, and in which the first series of sub-sampled video data represents a portion of the full field of view of the scene.

3. The method of claim 1, in which the sub-sampled video data in the second series representing a windowed view of the detected region of interest includes a magnified view portion as compared to the scene to which the sub-sampled video data of the first series corresponds.

4. The method of claim 1, further comprising analyzing the subsampled video data in the second series of sub-sampled video data to recognize a characteristic of the region of interest.

5. The method of claim 4, in which:

the characteristic of the region of interest is a presence of an object in the region of interest;

the first and second image resolutions represent respective lower and higher image resolutions;

the analysis of the sub-sampled video data in the first series is performed at the lower image resolution to detect the region of interest; and

the analysis of the sub-sampled video data in the second series is performed at the higher image resolution to more accurately recognize the object present in the region of interest.

6. The method of claim 4, further comprising controlling the imager to acquire a third series of sub-sampled video data from the high resolution video data corresponding to the scene, the sub-sampled video data in the third series representing the recognized characteristic in a magnified localized portion of the windowed view of the detected region of interest.

7. The method of claim 4, in which:

the recognized characteristic includes an object located in the region of interest, the object having a size;

the video camera produces video image data in response to light propagating from a scene; and

the second image resolution of the second series of sub-sampled video data is determined by recognition of the size of the object over time for the video image data produced by the imager.

8. The method of claim 1, in which the video camera produces video image data in response to light propagating from a scene, and further comprising:

analyzing the sub-sampled video data in the first series of sub-sampled video data to recognize an object in the region of interest, the object having a size; and

determining from the size of the object the second image resolution of the second series of sub-sampled video data for the video image data produced.

9. The method of claim 1, in which the region of interest is manually selected.

10. The method of claim 1, further comprising:

analyzing the sub-sampled video data in the second series of sub-sampled video data to recognize a characteristic of the region of interest; and

upon verification of the characteristic of the region of interest after the analysis of the subsampled video data, controlling the imager to acquire a continual stream of the second series of sub-sampled video data representing successive observations of the windowed view of the detected region of interest to closely observe for a time the recognized characteristic.

11. The method of claim 1, in which the windowed view has position coordinates representing its location in the scene, and further comprising:

analyzing the sub-sampled video data in the first series of sub-sampled video data to identify an object in the region of interest; and

controlling the imager to change the position coordinates of the windowed view to track motion of the object through the scene.

12. The method of claim 11, further comprising changing to different values the second image resolution of the windowed view as the motion of the object is tracked through the scene.

13. The method of claim 1, in which the windowed view has position coordinates representing its location in the scene, and further comprising:

analyzing the sub-sampled video data in the second series of sub-sampled video data to recognize a number of objects in the region of interest; and

controlling the imager to change the position coordinates of the windowed view to track motion of each of the number of objects through the scene.

14. The method of claim 13, in which the number of objects is greater than one and the windowed view is of a size that covers all of the number of objects recognized.

15. The method of claim 1, further comprising:

delivering from the imager the sub-sampled video data in the first and second series as interleaved images of the scene and the windowed view of the detected region of interest, respectively.

16. The method of claim 1, further comprising:

transmitting through a communication medium for delivery to a destination the first and second series of sub-sampled video data in combination as a video stream representing the scene and the windowed view of the detected region of interest in the scene at the different image resolutions.

17. The method of claim 1, further comprising:

transmitting through a communication medium for delivery to a destination the first and second series of sub-sampled video data as, respectively, a first video stream representing a first windowed view of the scene and a second video stream representing a second windowed view as the detected region of interest in the scene.

18. The method of claim 1, further comprising:

analyzing the sub-sampled video data in the first series of sub-sampled video data to recognize a characteristic of the region of interest; and

controlling the imager to acquire at a frame rate one of the first and second series of sub-sampled video data corresponding to the scene, the frame rate being set in response to the characteristic recognized by the analysis of the sub-sampled video data in the first series.

19. The method of claim 18, in which the recognized characteristic includes an object located in an area of the region of interest, in which the object and the area are characterized by brightness levels, and in which the frame rate is set to provide a frame integration time that is adequate to provide a brightness level contrast between the object and the area to render them visually distinguishable from each other.

20. The method of claim 18, in which the frame rate constitutes a first frame rate and in which the controlling the imager acquires the first series of sub-sampled video data at the first frame rate, and further comprising controlling the imager to acquire at a second frame rate the second series of sub-sampled video data.

31. The method of claim 1, in which the scene processed by the video camera changes over time;

and further comprising controlling the imager to acquire multiple time-displaced sets of the first series of subsampled video data and the second series of subsampled video data, in which the acquiring of the second series of sub-sampled video data of one of the multiple sets is followed by the acquiring of the first series of sub-sampled video data of a subsequent one of the multiple sets, thereby to enable continual detection of the region of interest and the windowed view of the detected region of interest.

21. An automated active video surveillance system, comprising:

a high image data capacity video camera operable to provide image information relating to a scene processed by the video camera, the camera including a high resolution imager that has a number of image pixels from which are produced image data representing a scene processed by the video camera;

a video analytics engine implemented to analyze data included in the image data produced by the imager;

an imager controller operatively associated with the imager and the video analytics engine to acquire high resolution video data representing substantially all of the number of image pixels that capture a full field of view of the scene and to provide multiple portions of the image data for analysis by the video analytics engine to detect regions of interest at different locations in the scene, the multiple portions including a first series of sub-sampled image data from the high resolution video data and a second series of sub-sampled image data from the high resolution video data, the first series of sub-sampled image data corresponding to the scene at a first image resolution for analysis by the video analytics engine to detect a first region of interest within the scene, the second series of sub-sampled image data representing a windowed view of the detected first region of interest within the scene at a second resolution that is higher than the first image resolution; and

a communications medium receiving and transmitting to a destination separate from the camera analysis output data provided by the video analytics engine upon analyzing the first and second series of sub-sampled image data,

wherein the first and second series of sub-sampled image data are formed by non-uniformly sub-sampling at least one frame of the image data.

22. The system of claim 21, in which the imager controller in response to the video analytics engine allocates resolution so that at least some of the multiple portions are analyzed with different image resolutions, thereby enabling smaller objects within the scene to be imaged at higher resolution to undergo appropriate scrutiny and larger objects within the scene to be imaged at lower resolution.

23. The system of claim 21, in which the multiple portions of the image data are provided substantially simultaneously for analysis.

24. The system of claim 21, further comprising an image display operatively connected to the communications medium to display as visual images data corresponding to the analysis output data.

25. The system of claim 21, further comprising an image display operatively connected to the communications medium, and in which data corresponding to the analysis output data represent a closer range view of a feature of the scene and a farther range view of a region of the scene, and in which the image display is operable to display simultaneously images of the closer range view and the farther range view.

26. The system of claim 21, further comprising a storage medium operatively connected to the communications medium to store the data corresponding to the analysis output data.

27. The system of claim 21, in which the video camera is attached to a motorized pan mechanism, and in which the motorized pan mechanism is driven by stepper motors controlled by the video analytics engine to move the camera in response to analysis by the video analytics engine detecting activity in at least one of the regions of interest.

28. The system of claim 21, in which the video camera includes a high image resolution lens.

29. The system of claim 28, in which the analysis of data performed by the video analytics engine includes adjustment of light exposure levels associated with the high image resolution lens to optimize recognition of an object of interest within the scene, the adjustment of light exposure levels excluding from the analysis output data portions corresponding to high contrast areas of the scene.

30. The system of claim 28, in which the high image resolution lens is of fixed focal length and an f-stop of 1.0 or smaller.