High resolution network camera with automatic bandwidth control

• US 7,623,152 B1
• Filed: 07/14/2004
• Issued: 11/24/2009
• Est. Priority Date: 07/14/2003
• Status: Active Grant

• Current Assignee: Arecont Vision Costar, LLC (Costar Technologies, Inc.)
• Original Assignee: Arecont Vision LLC (Costar Technologies, Inc.)
• Inventors: Kaplinsky, Michael
• Primary Examiner(s): Tran; Sinh
• Assistant Examiner(s): BERARDESCA, PAUL M
• Application Number:
  US10/890,914
• Time in Patent Office: 1,959 Days
• Field of Search: 348/211.3, 348/208.4, 348/208.16, 348/208.13, 348/211.99, 348/208.14, 348/211.14, 348/305, 348/208.1, 348/208.3, 348/699, 348/382, 348/107, 348/402.1, 348/208.12, 348/208.6, 348/208.5, 348/208.15, 348/208.17, 348/208.99, 375/240.16, 375/240.17, 375/240.02
• US Class Current: 348/208.16
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20

Abstract

The subject of this invention is the network camera comprising an image sensor, image processor, and network interface, where image processor is capable of detecting the motion in the field of view of the camera, extracting/windowing the portion (sub-window) of the image that corresponds to that motion and submitting thus identified sub-window to the network interface hardware for transmission and where said image processor is also capable of submitting to the network interface the full field of view reduced-resolution (decimated) image either as the only image to be transmitted or as the image to be transmitted in a time-interleaved fashion with the said image window containing the motion.

Claims

1. A network video camera supporting multiple output modes including output of full resolution full field of view images, output of reduced resolution full field of view images, and output of window-of-interest partial images, said camera comprising:

an image sensor for sensing and outputting images;

an image buffer memory for storing said output images;

an image processor for decimating said output images to generate images with said reduced resolution and full field of view, and for generating said window-of-interest partial images;

a network interface for transmission of image data off camera; and

a motion detection module configured to detect presence of moving objects in the field of view of the camera and to select between said output modes of the camera based on the results of said motion detection, wherein the network interface is capable of transmitting a selected decimated output of said reduced resolution full field of view and a selected output of window-of-interest partial field of view in a time-interleaved fashion, wherein said motion detection module is configured to select said full resolution full field of view images when the presence of moving objects in the field of view of the camera has been detected and to select said reduced resolution full field of view images when the presence of moving objects in the field of view of the camera has not been detected.

2. Network video camera of claim 1, wherein said motion detection module is configured to determine the size of objects moving in the field of view of the camera, and to configure a size of said window-of-interest based on said determined size of said moving objects.

3. Network video camera of claim 1, wherein said motion detection module selects for the output off camera window-of-interest partial images if the presence of moving objects in the field of view of the camera has been detected and wherein said motion detection module configures a size of said window-of-interest to contain all or most of particular moving objects having a specified size, based on a desired sensitivity to motion.

4. Network video camera of claim 1, wherein said motion detection module is configured to reduce camera frame rate when the presence of moving objects in the field of view of the camera has not been detected.

5. Network video camera of claim 1, wherein said motion detection module selects for the output off camera both reduced resolution full field of view images and window-of-interest partial images if the presence of moving objects in the field of view of the camera has been detected and wherein said images are transmitted off camera in a time-interleaved manner wherein transmission of one or more images in one of said output modes is followed by the transmission of one or more images in another of said output modes.

6. Network video camera of claim 1, wherein said camera selects said output modes based on the external input delivered by said network interface.

7. Network video camera of claim 1, wherein said network interface is Ethernet network interface.

8. Network video camera of claim 1, further comprising an image processor equipped to perform image compression.

9. Network video camera of claim 1, wherein said motion detection module is further configured to determine a number of disjoint areas of the image where the motion has been detected, to select as many windows-of-interest as the number of disjoint areas of the image, and to submit said selected number of windows-of-interest to the network interface for transmission off camera.

10. Network video camera of claim 9, wherein an identifier of location of each of said selected number of windows-of-interest with respect to the entire source image is included in each respective window-of-interest before transmission off camera.

11. Network video camera of claim 8, wherein said image processor also comprises image pre-processor equipped to perform image sensor control and color processing.

12. Network video camera of claim 8, wherein said image processor comprises dedicated logic for implementing an image processing pipeline, wherein image pixels are operated on by sequential stages of said image processing pipeline with each pipeline stage operating concurrently with all or most of the other pipeline stages.

26. Camera system of claim 1, in which an output of window of interest partial images is a zoom window.

27. Camera system of claim 1, in which an output of window of interest partial view moves around responsive to the moving objects.

28. Camera system of claim 26, in which said motion detection module is configured to automatically select a zooming of the zoom window responsive to the moving objects.

29. Camera system of claim 26, in which said motion detection module is configured to automatically select a position of the zoom window responsive to the moving objects.

30. Network video camera of claim 1, wherein said motion detection module is configured to determine a speed of an object moving in the field of view of the camera, and to configure a size of said window-of-interest based on said determined speed of said moving object.

13. Method of automatic optimization of network bandwidth for image transmission from a network video camera having a field of view, an output camera mode, and equipped with a motion detection module, the method comprising:

determining presence of moving objects in the field of view of the camera;

selecting the output camera mode to be a reduced resolution full field of view image output if said moving objects are not found in the field of view of the camera, otherwise determining a minimum size rectangular area W of an image that contains more than Z percent of all moving objects, wherein Z is a parameter selectable by a user; and

selecting the output camera mode to be a full resolution full field of view image output if said image area W is more than F percent of an entire image area, and selecting the output camera mode to be a window-of-interest output including the image area W, if said image area W is less than or equal to F percent of the entire image area, wherein F is a parameter selectable by the user.

14. The method of claim 13, wherein said determining of the presence of moving objects in the field of view of the camera comprises:

subdividing the entire image area into a grid of macropixels of equal areas, wherein said macropixels are small rectangular image areas comprising one or more pixels;

determining luminance values for all said macropixels by calculating average luminance value of said pixels comprising the macropixel;

storing luminance values of macropixels;

comparing average luminance values of each macropixel in two consecutive frames and determining macropixels potentially containing motion, wherein a macropixel potentially contains motion if its luminance values in two consecutive frames are determined to be substantially different;

grouping said potentially containing motion macropixels into motion clusters, wherein a motion cluster comprises two or more potentially containing motion macropixels, and wherein the centers of said macropixels can be joined by a continuous curve completely encompassed within the macropixels of said motion cluster; and

determining the presence of moving objects in the field of view of the camera by determining a presence of said motion clusters containing N or more potentially containing motion macropixels, wherein N is a parameter selectable by the user.

15. The method of claim 14, wherein said comparing of luminance values of each macropixel in two consecutive frames comprises:

determining an absolute value of difference between luminance values of said macropixel in two consecutive frames;

determining normalized luminance difference of said macropixel by dividing said absolute value of difference by the average luminance value of said macropixel in two consecutive frames; and

comparing normalized luminance difference with a motion threshold, wherein luminance values of said macropixel in two consecutive frames are determined to be substantially different if said normalized luminance difference exceeds said motion threshold.

16. The method of claim 15, wherein said motion threshold is determined by multiplying TH_motion by a measure of root mean square temporal noise reflected in the luminance values of the pixels comprising said macropixels, wherein TH_motion, is a constant selectable by the user.

17. The method of claim 13, further comprising determining a speed of an object moving in the field of view of the camera, and configuring the size of said window-of-interest based on said determined speed of said moving object.

18. Camera system comprising:

one or more network cameras equipped to support multiple output modes comprising output of full resolution full field of view images, output of reduced resolution full field of view images, and output of window-of-interest partial images;

an image buffer memory;

a network interface for transmission of image data off cameras, wherein said cameras select one of said output modes based on an external command delivered to the cameras by said network interface; and

a computer system coupled to said cameras configured to acquire the images from said cameras and including software running on said computer system, wherein said software is configured to detect the presence of moving objects in the fields of view of said cameras and to place said cameras in one of said output modes by communicating a command for selecting an output mode from said multiple output modes, to the camera via said network interface, and to determine a size of said window-of-interest having a size of a minimum rectangular area that contains more than Z percent of all moving objects by gradually enlarging the rectangular area W of the image until it contains over Z percent of all moving objects, where, Z is a parameter selectable by a user.

19. Camera system of claim 18, wherein said software selects as an output mode of camera full resolution full field of view images if the presence of moving objects in the field of view of the camera has been detected and wherein said software selects as an output mode of camera reduced resolution full field of view images if the presence of moving objects in the field of view of the camera has not been detected.

20. Camera system of claim 18, wherein said software is configured to determine the size of objects moving in the field of view of the camera, and to configure a size of said window-of-interest based on said determined size of said moving objects.

21. Camera system of claim 18, wherein said software selects as an output mode of camera window-of-interest partial images followed by the full field of view reduced resolution images if the presence of moving objects in the field of view of the camera has been detected and wherein said software configures a size of said window of interest to contain all or most of particular moving objects having a specified size, based on a desired sensitivity to motion.

22. Camera system of claim 18, wherein said cameras and said software are configured to reduce camera frame rate if the presence of moving objects in the field of view of the camera has not been detected.

23. Camera system of claim 18, further comprising one or more monitors connected to the said computer system, wherein said software displays images acquired from said cameras on said monitors.

24. Camera system of claim 23, wherein said image display on said monitors includes a mode wherein window of interest image and full field of view image are displayed simultaneously in a picture-in-picture format.

25. Camera system of claim 23, wherein said image display on said monitors includes a mode wherein window of interest image is displayed on one of said monitors and full field of view image is displayed on another of said monitors.