High resolution network video camera with massively parallel implementation of image processing, compression and network server

• US 20050140787A1
• Filed: 11/22/2004
• Published: 06/30/2005
• Est. Priority Date: 11/21/2003
• Status: Active Grant

• Current Assignee: Arecont Vision Costar, LLC (Costar Technologies, Inc.)
• Original Assignee: Arecont Vision LLC (Costar Technologies, Inc.)
• Inventors: Kaplinsky, Michael
• Granted Patent: US 7,548,258 B2
• Time in Patent Office: Days
• US Class Current: 348/207.100
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19663 Surveillance related processing done local to the camera
• CPC Class Codes: G08B 13/19667 Details realated to data compression, encryption or encoding, e.g. resolution modes for reducing data volume to lower transmission bandwidth or memory requirements
• CPC Class Codes: G08B 13/19691 Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/7921 for more than one processing mode

Abstract

The subject of the present invention is the network video camera capable of delivering high-resolution digital images, comprising over a million of pixels per image, at full motion video frame rate. The camera disclosed in the present invention comprises high resolution image sensor capable of operating at video frame rate, computer network interface for transmission off camera of image data streams originated by said image sensor, and also comprising multi stage pipelined digital image processor operating under control of low cost microprocessor capable of processing and compression of image data at the output rate of said image sensor, wherein said image processor comprises image processing pipeline formed by multiple distinct stages, wherein most of said stages perform distinctly different image processing operations and wherein each said stage has an output latch or buffer and accepts its input data from the output latch or buffer of one or more of other said stages, wherein image data propagates from stage to stage and wherein each of said stages performs its operations on image data concurrently and synchronously with most of other said stages performing their respective operations.

Claims

1. Network video camera comprising high resolution image sensor, computer network interface for transmission off camera of image data streams originated by said image sensor and also comprising multi stage pipelined digital image processor capable of processing and compression of image data at the output rate of said image sensor, wherein said image processor comprises image processing pipeline formed by multiple distinct stages, wherein most of said stages perform distinctly different image processing operations and wherein each said stage has an output latch or buffer and accepts its input data from the output latch or buffer of one or more of other said stages, wherein image data propagates from stage to stage and wherein each of said stages performs its operations on image data concurrently and synchronously with most of other said stages performing their respective operations.

2. Network camera of claim 1, also comprising microprocessor or CPU.

3. Network camera of claim 2, wherein microprocessor is configured to control the operation of the said image processing pipeline and wherein said control comprises initialization of the pipeline, communicating with pipeline measurements engine, setting the pipeline registers to control the dataflow through pipeline multiplexers and setting pipeline parameters utilized in automatic white balance and automatic exposure and gain control operations.

4. Network camera of claim 3, wherein said microprocessor also configured to perform relatively slow computations comprising processing of the statistics accumulated by said measurement engines of the image processing pipeline and also comprising calculations of exposure time and gain settings (AE) as well as the calculation of auto white balance (AWB) parameters.

5. Network camera of claim 2, wherein said microprocessor is configured to support network communication with external to the camera clients and wherein said microprocessor is configured to communicate with camera MAC hardware, to interpret incoming network packets and to form network packet headers wherein said microprocessor relies on the output logic of the image processing pipeline for the computation of partial packet checksums and for loading of the image data in the data fields of the outgoing network packets.

6. Network camera of claim 1, wherein said image processing pipeline comprises stages configured to perform color interpolation, image sharpening, color separation, gamma correction and color space conversion, wherein said color interpolation is configured to convert one-color-per-pixel image into 3 colors-per-pixel image, and wherein said color interpolation is configured to perform multiplication of red, green and blue (RGB) pixel components by a 3×

3 color correction matrix, and wherein said image sharpening is configured to apply 2-dimensional high-pass filter to the image, and wherein said color space conversion is configured to convert color components of the image to YUV, YCrCb or other color-difference representation suitable as an input to the image compression stages of said image processing pipeline.

7. Network camera of claim 2, wherein said image processing pipeline comprises measurement engines configured to accumulate image brightness and color statistics and wherein said measurement engines are equipped with the means of communicating said statistics to said microprocessor.

8. Network camera of claim 6, wherein said image processing pipeline also comprises stages configured to perform 2-dimensional pixel defect correction, compensation for image distortions introduced by the optics, color saturation control and histogram equalization of the image.

9. Network camera of claim 8, wherein said 2-dimensional pixel defect correction performed by substitution of the defective pixels with one of the non-defective pixels from the pixels immediately surrounding said defective pixel.

10. Network camera of claim 9, wherein, for all image pixels, the luminance value of the pixel currently being processed (current pixel) is compared to the luminance values of 8 immediately surrounding it pixels of the same color (surrounding pixels), and wherein said substitution is performed if the current pixel has a luminance value either above or below the luminance values of all said surrounding pixels and wherein said substitution is performed by the replacement of all color components of said current pixel with the corresponding color components of the replacement pixel, wherein said replacement pixel is one of the surrounding pixels and wherein said replacement pixel has a luminance value least different in magnitude from luminance value of said current pixel.

11. Network camera of claim 1, wherein said image processing pipeline comprises stages configured to perform image compression.

12. Network camera of claim 11, wherein said image compression is baseline JPEG compression and wherein said image processing pipeline comprises stages configured to perform formation of the minimally coded units (MCU), forward discrete cosine transform (FDCT), zigzag scan, quantization and entropy coding wherein said operations and terms are described in Ref. [1].

13. Network camera of claim 11, wherein said image compression is JPEG2000 or MPEG-2 compression.

14. Network video camera comprising high resolution image sensor, computer network interface for transmission off camera of image data streams originated by said image sensor and also comprising multi stage pipelined digital image processor capable of processing and compression of image data at the output rate of said image sensor, wherein said image processor comprises image processing pipeline formed by multiple distinct stages, wherein most of said stages perform distinctly different image processing operations and wherein each said stage has an output latch or buffer and accepts its input data from the output latch or buffer of one or more of other said stages, wherein image data propagates from stage to stage and wherein each of said stages performs its operations on image data concurrently and synchronously with most of other said stages performing their respective operations, and

wherein said network video camera is capable of concurrent output of image sequences of different sizes and resolutions and also comprises one or more image memory buffers and image memory buffer access controller, wherein said access controller is equipped to store images in the memory buffer and to read stored images from the memory buffer, wherein said memory access controller is further equipped to reduce resolution and size of images by performing image decimation and image windowing.

15. Network camera of claim 14, wherein camera comprises at least two image memory buffers wherein memory buffer access controller is configured to store incoming image frames in one of said memory buffers and to read frames or partial frames from other of said memory buffers and wherein images are stored to said image buffer at the rate not less than the pixel output rate of said image sensor and wherein the images are read from said other image buffer at the rate determined by the output bandwidth of said network interface and wherein memory access controller is equipped to access said image buffers so that once the image readout from one image buffer has been completed it is then used for storage of incoming image frame and while the image buffer that was previously used for storing the image frame is then used for readout of the image frame to be output from camera.

16. Network camera of claim 14, wherein camera comprises at least three image memory buffers wherein one of said image buffers is available for storage of the incoming image frame, another of said image buffers is available for the readout of the image frame to be output from camera and third image buffer holds next image frame to be output from camera, wherein once image buffer has been readout it is made available for storage of the new incoming image frame.

17. Network camera of claim 14, wherein said image memory buffer access controller is equipped to support concurrent output of multiple image formats, wherein image access controller forms output image frames by reading the same image stored in the memory buffer two or more times, wherein said memory access controller creates different output frame formats by reading different segments of the stored image and wherein image frames of different formats are output from camera one after another.

18. Network video camera of claim 17, wherein memory access controller first reads and decimate and an entire image frame stored in said image buffer and wherein said access controller subsequently reads a specified rectangular region of the same image, wherein causing the camera output format to continuously switch on a frame-by-frame basis between full field of view reduced resolution image and a full resolution rectangular image window of specified by the user dimensions.

19. Network camera of claim 14, also comprising microprocessor or CPU.

20. Network camera of claim 19, wherein microprocessor is configured to control the operation of the said image processing pipeline and wherein said control comprises initialization of the pipeline, communicating with pipeline measurements engine, setting the pipeline registers to control the dataflow through pipeline multiplexers and setting pipeline parameters utilized in automatic white balance and automatic exposure and gain control operations.

21. Network camera of claim 20, wherein said microprocessor also configured to perform relatively slow computations comprising processing of the statistics accumulated by said measurement engines of the image processing pipeline and also comprising calculations of exposure time and gain settings (AE) as well as the calculation of auto white balance (AWB) parameters.

22. Network camera of claim 19, wherein said microprocessor is configured to support network communication with external to the camera clients and wherein said microprocessor is configured to communicate with camera MAC hardware, to interpret incoming network packets and to form network packet headers wherein said microprocessor relies on the output logic of the image processing pipeline for the computation of partial packet checksums and for loading of the image data in the data fields of the outgoing network packets.

23. Network camera of claim 14, wherein said image processing pipeline comprises stages configured to perform color interpolation, image sharpening, color separation, gamma correction and color space conversion, wherein said color interpolation is configured to convert one-color-per-pixel image into 3 colors-per-pixel image, and wherein said color interpolation is configured to perform multiplication of red, green and blue (RGB) pixel components by a 3×

3 color correction matrix, and wherein said image sharpening is configured to apply 2-dimensional high-pass filter to the image, and wherein said color space conversion is configured to convert color components of the image to YUV, YCrCb or other color-difference representation suitable as an input to the image compression stages of said image processing pipeline.

24. Network camera of claim 19, wherein said image processing pipeline comprises measurement engines configured to accumulate image brightness and color statistics and wherein said measurement engines are equipped with the means of communicating said statistics to said microprocessor.

25. Network camera of claim 24, wherein said image processing pipeline also comprises stages configured to perform 2-dimensional pixel defect correction, compensation for image distortions introduced by the optics, color saturation control and histogram equalization of the image.

26. Network camera of claim 25, wherein said 2-dimensional pixel defect correction performed by substitution of the defective pixels with one of the non-defective pixels from the pixels immediately surrounding said defective pixel.

27. Network camera of claim 26, wherein, for all image pixels, the luminance value of the pixel currently being processed (current pixel) is compared to the luminance values of 8 immediately surrounding it pixels of the same color (surrounding pixels), and wherein said substitution is performed if the current pixel has a luminance value either above or below the luminance values of all said surrounding pixels and wherein said substitution is performed by the replacement of all color components of said current pixel with the corresponding color components of the replacement pixel, wherein said replacement pixel is one of the surrounding pixels and wherein said replacement pixel has a luminance value least different in magnitude from luminance value of said current pixel.

28. Network camera of claim 19, wherein said image processing pipeline comprises stages configured to perform image compression.

29. Network camera of claim 28, wherein said image compression is baseline JPEG compression and wherein said image processing pipeline comprises stages configured to perform formation of the minimally coded units (MCU), forward discrete cosine transform (FDCT), zigzag scan, quantization and entropy coding wherein said operations and terms are described in Ref. [1].

30. Network camera of claim 28, wherein said image compression is JPEG2000 or MPEG-2 compression.

31. Network video camera comprising high resolution image sensor, computer network interface for transmission off camera of image data streams originated by said image sensor and also comprising multi stage pipelined digital image processor capable of processing and compression of image data at the output rate of said image sensor, wherein said image processor comprises image processing pipeline formed by multiple distinct stages, wherein most of said stages perform distinctly different image processing operations and wherein each said stage has an output latch or buffer and accepts its input data from the output latch or buffer of one or more of other said stages, wherein image data propagates from stage to stage and wherein each of said stages performs its operations on image data concurrently and synchronously with most of other said stages performing their respective operations, and

wherein said network video camera also comprises, microprocessor, two or more network packet buffers, the means of data transfer between said microprocessor and said network packet buffers and the means of data transfer between output stage of said image processing pipeline and said network packet buffers.

32. Network camera of claim 31, wherein said network interface comprises Ethernet media access control interface (MAC) and wherein said camera comprises the means of communication between said microprocessor and said media access control interface.

33. Network camera of claim 31, wherein said microprocessor is a low cost microprocessor with one or more orders of magnitude smaller data processing capability than said image processing pipeline.

34. Network camera of claim 32, wherein said microprocessor is configured to receive and interpret incoming network data packets and to assemble outgoing packets in response to certain incoming network packets, wherein said incoming data packets may comprise image transmission requests, requests for access to camera registers, address resolution protocol (ARP) requests, internet control message protocol (ICMP) ping requests and other requests defined by relevant network standards.

35. Network camera of claim 32, wherein said microprocessor is equipped to respond to external requests for image transmission by forming packet headers of outgoing network packets, wherein said headers are formed in accordance with the requirements of computer network protocol used for image transmission, loading said headers in said network packet buffers and directing said image processing pipeline to load image data in the data field of said outgoing network packets.

36. Network camera of claim 32, wherein camera supports a minimal set of computer network protocols comprising UDP, TFTP, ARP, IP and ICMP, wherein said protocols are defined in their respective standards.

37. Network camera of claim 36, wherein camera also supports TCP and DHCP computer network protocols, wherein said protocols are defined in their respective standards.

38. Network camera of claim 32, wherein said camera comprises three or more outgoing network packet buffers, wherein one buffer is available for storage of data arriving from said image processing pipeline, another said buffer contains packet available for transmission or being transmitted to said MAC hardware and wherein one or more buffers hold packets that sequentially follow the packet currently available for transmission.

39. Network camera of claim 31, wherein said microprocessor is configured to disable image processing pipeline if all available network packet buffers are full and wherein said microprocessor is configured to enable said image processing pipeline until either just enough image data is processed to fill available network packet buffer or an end-of-frame (EOF) condition is detected.

40. Network camera of claim 31, wherein said camera is configured to transmit images by means of modified Trivial File Transfer Protocol (TFTP), wherein in contrast to standard TFTP implementation (RFC783), two or more initial image data packets are transmitted immediately one after another without waiting the corresponding acknowledgements being received by said camera from the remote client and wherein the arriving acknowledgements are inspected by said microprocessor only after said initial packets have been transmitted and wherein said microprocessor is configured to trigger initiate the transmission of additional data packets upon receiving by the camera of an acknowledgement for one of already transmitted packets.

41. Network camera of claim 40, wherein said microprocessor is configured to wait for the arrival of said packet acknowledgements for the duration of certain timeout period and wherein said microprocessor is configured to initiate the retransmission of the data stored in one of previously transmitted network packet buffers if said timeout period has expired prior to arrival of said acknowledgement.

42. Network camera of claim 40, wherein all outgoing packets have are numbered in sequential order of initial transmission and wherein the receiving network client is required to sort arriving packets in ascending order of said packets numbers.

43. Network camera of claim 31, wherein resolution, size, compression quality and other parameters of incoming image requests as well as the numerical values of the camera register access requests are specified in the “

filename”

field of TFTP headers.

44. Camera system comprising:

one or more network video cameras, wherein said video cameras comprise high resolution image sensor, computer network interface for transmission off camera of image data streams originated by said image sensor and also comprising multi stage pipelined digital image processor capable of processing and compression of image data at the output rate of said image sensor, wherein said image processor comprises image processing pipeline formed by multiple distinct stages, wherein most of said stages perform distinctly different image processing operations and wherein each said stage has an output latch or buffer and accepts its input data from the output latch or buffer of one or more of other said stages, wherein image data propagates from stage to stage and wherein each of said stages performs its operations on image data concurrently and synchronously with most of other said stages performing their respective operations; and

computer system receiving the images from said cameras and connected to the said cameras by means of computer network.

45. Camera system of claim 44, wherein said computer also equipped with the hard drive and monitor and wherein said computer is configured to acquire the images from one or more of said cameras, decompress and display said images on said monitor, store some or all of said images on said hard drive into digital archives and to respond to the requests for image transmission from the remote users by sending to said remote users complete or partial images acquired from said cameras.

46. Camera system of claim 44, wherein said computer is configured to acquire from all or some of said cameras image streams comprising images in different formats, wherein one type of image streams comprise full field of view decimated images interleaved on a frame-by-frame basis with full resolution partial images or image windows.

47. Camera system of claim 46, wherein said computer is configured to simultaneously output on said monitor images in said different formats.

48. Camera system of claim 47, wherein said computer outputs on said monitor full field of view image concurrently with the output of zoomed or partial window of said image image.