INTEGRATED INTELLIGENT SERVER BASED SYSTEM AND METHOD/SYSTEMS ADAPTED TO FACILITATE FAIL-SAFE INTEGRATION AND/OR OPTIMIZED UTILIZATION OF VARIOUS SENSORY INPUTS

• US 20120179742A1
• Filed: 01/11/2012
• Published: 07/12/2012
• Est. Priority Date: 01/11/2011
• Status: Active Grant

• Current Assignee: Videonetics Technology Private Limited
• Original Assignee: Videonetics Technology Private Limited
• Inventors: ACHARYA, TINKU, DALAL, TUTAI KUMAR, MAITY, SOUMYADIP, BHATTACHARYYA, DIPAK, DAS, SAWAN, DHAR, SOUMYADEEP, BOSE, TUHIN
• Granted Patent: US 9,704,393 B2
• Time in Patent Office: Days
• US Class Current: 709/202
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23116 involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Integrated intelligent system adapted for any operating system and/or multi-OS computing environment seamlessly having sensory input/data acquisition cum recording server group and/or analytics server group enabling fail-safe integration and/or optimized utilization of various sensory inputs for various utility applications. Also disclosed as added advancements include intelligent method/system for cost-effective and efficient band adaptive transferring/recording sensory data from single or multiple data sources to network accessible storage devices, fail safe and self sufficient server group based method for sensory input recording and live streaming in a multi-server environment, intelligent and unified method of colour coherent object analysis, face detection in video images and the like, resource allocation for analytical processing involving multi channel environment, multi channel join-split mechanism adapted for low and/or variable bandwidth network link, enhanced multi-colour and/or mono-colour object tracking and also an intelligent automated traffic enforcement system.

Claims

1. An integrated intelligent server based system having sensory input/data acquisition cum recording server group and/or analytics server group adapted to facilitate fail-safe integration and/or optimized utilization of various sensory inputs for various utility applications comprising at least one autonomous system having:

I) A) said sensory input acquisition cum recording server group comprising plurality of acquisition cum recording servers which are operatively linked to assess respective server capacity and operate as a group to enable fail-safe support when any of the servers in the group fail to operate the remaining operative servers in the group are adapted to distribute and take over the sensory input load of the non-operative server/s to render the system fail safe and self sufficient; and

/or B) said analytics server group comprising plurality of analytics server for intelligent analysis including resource dependent analytical accuracy control including means adapted for computing complexity of scenes and dynamically reconfigure the analytical processing steps for optimal analysis and/or availability of computational and other resources for on-line and real-time and/or on demand for efficient and user friendly streaming/analysis/detection/alert generation of events and/or follow up actions; and

II) an intelligent interface for operative connection to said sensory input acquisition cum recording server group; and

/or said analytics server group.

2. An integrated intelligent server based system as claimed in claim 1 wherein each said acquisition cum recording servers are adapted for bandwidth optimized fail-safe recording and/or join-split mechanism for multi channel sensory data/video streaming.

3. An integrated intelligent server based system as claimed in claim 1 wherein each said analytics server is adapted for anyone or more of (a) intelligent colour object analysis framework and colour coherent background estimation, (b) identifying moving, static, quasi-static objects, (c) enhanced object tracking, (d) content aware resource scheduling, (e) join split mechanism for multi channel video streaming, and (f) resource dependent accuracy control.

4. An integrated intelligent server based system as claimed in claim 1 wherein said intelligent interface is operatively connected to anyone or more (a) user management and client access controller (b) event controller and handler and (c) event and/or selected segments of sensory data distributor.

5. An integrated intelligent server based system as claimed in claim 1 comprising operative client modules comprises selectively standalone surveillance client, internet browser, web client, any hand held devices including mobile device client, and remote event and/or notification receiver.

6. An integrated intelligent server based system as claimed in claim 1 wherein said acquisition cum recording server is adapted to (i) collect inputs from various sensory sources, archiving, tagging, and indexing to seamlessly map in a database or data warehousing system involving any one or more of optimal usage of computing, communication and storage resources, facilitate efficient search, transcoding, retransmission, authentication of data, rendering and viewing of archived data at any point of time and (ii) Streaming input sensory data real time or on Demand including streaming video and other sensory content in multiple formats to multiple devices for purposes including live view in different matrix layout, relay of the content, local archiving, rendering of the sensory data in multiple forms and formats, by a fail-safe mechanism without affecting speed and performance of on-going operations and services.

7. An integrated intelligent server based system as claimed in claim 4 comprising means for auto registration of servers involving unique identification number, configuration data of the relevant server, means for recording sensory inputs in local storage and streaming the data to client modules and means for bandwidth adaptive uploading to central storage systems.

8. An integrated intelligent server based system as claimed in claim 1 wherein said analytics server comprises:

(a) sensory input analytics engine; and

(b) analytics engine controller.

9. An integrated intelligent server based system as claimed in claim 1 wherein said intelligent interface is adapted for anyone or more of (i) filtering and need based transmission of sensory inputs, (ii) directing distribution of alerts (iii) providing a common gateway for heterogeneous entities.

10. An integrated intelligent server based system as claimed in anyone of claim 5 wherein said client module comprises means enabling user to receive, view, analyze, search sensory inputs and include standalone surveillance clients, internet browsers, handheld devices, cell phones, PCs, Tablet PCs and the like.

11. An integrated intelligent server based system as claimed in claim 1 comprising remote event receiver adapted to receive and display messages and ALERTs from various components of the system which can further be multicast or broadcast.

12. An integrated intelligent server based system as claimed in claim 1 comprising central server adapted to serve as a gateway to plurality of said autonomous system and integrate the system into a single unified system.

13. An integrated intelligent server based system as claimed in claim 5 wherein each said acquisition cum recording server is adapted to accept requests through the intelligent interface and/or receive inputs from various other input sources, recording sensory inputs in local storage, intelligent uploading of the sensory input in a cluster of storage devices wherein said cluster comprises one or more network accessible storages in an efficient manner with fair share to individual sources utilizing optimal bandwidth in a cooperative manner, enabling searching of input and analytical sensory inputs and streaming of the sensory inputs in original or transcoded format to various other devices including surveillance clients.

14. An integrated intelligent server based system as claimed in claim 1 comprising means for recording sensory inputs in local storage and intelligent streaming of stored inputs continuously or on trigger from any external or internal services wherein the data stream is first segmented into small granular clips or segments of programmable and variable length sizes and said clips stored in the said local storage of the server, the clip metadata being stored in the local database.

15. An integrated intelligent server based system as claimed in claim 14 comprising bandwidth adaptive data uploading from channels to central storage system via said local storage comprises allocating a data source to a server group with multiple servers in the group, said servers comprising the server group adapted to exchange their respective capacity information such that in case of a breakdown of anyone or more of the servers in a group the remaining operative servers in the group share the load of the failed server/servers, each server also adapted to monitor the available bandwidth and also the data inflow rate for each channel into the server and accordingly adjust the upload rate for an input channel, means to segment the data stream into various sized clips and the rate of uploading the clips to the central storage adjusted depending upon the network bandwidth and data inflow rate for that particular channel.

16. An integrated intelligent server based system as claimed in claim 8 wherein said sensory input analytic engine comprises of (a) scene analyzer, (b) rule engine and (c) event decider.

17. An integrated intelligent server based system as claimed in claim 16 wherein said scene analyzer comprises means for intelligent scene adaptive colour coherent object analysis framework and method adaptive to the availability of computational bandwidth and memory enabling processing steps to be dynamically reconfigured.

18. An integrated intelligent server based system as claimed in claim 16 wherein said scene analyzer comprises means to generate meta-data against each frame for analysis and computing the complexity of the scene such as to dynamically reconfigure the processing steps based thereon for optimal analysis results depending upon the availability of the computational and other resources for on-line and real-time detection of events and follow up actions and further feeding the metadata along with the scene complexity measure to a controller adapted to decide the rate at which the frames of said channel should be decoded and sent to the analytic engine for processing;

said rule engine adapted to maintain history of the metadata and correlate the data across multiple frames to thereby decide the behavioural patterns of the objects in the scene for further determinations; and

said event decider is adapted to receive the behavioural patterns as detected by the rule engine and analyze the same to thereby detect various events in parallel and also to control user defined application of any external device for better decision making/study of the event identified.

19. An integrated intelligent server based system as claimed in claim 8 wherein said analytical engine controller comprises:

A) means to receive multiple sensory channel inputs and feed decoded frames of the multiple channels to the analytical engine wherein the said decoding and feeding of the decoded frames to the analytical engine is optimally controlled such that the number of frames sent per second for each channel is individually and automatically controlled depending on the requirement of the analytics engine and also on the computational bandwidth available in the system at any point of time; and

means adapted to stream sensory data along with analytical results either as individual streams for each channel or as joined single stream data for all or user requested channels involving joining the channels and transmitting resulting combined single channel over IP network adapted to varying and low bandwidth network connectivity, Or

B) means adapted to directly generate events without feeding any decoded frames to the analytical engine.

20. An integrated intelligent server based system as claimed in claim 1 wherein said intelligent interface is adapted to (i) auto register itself to the system, (ii) accept request from surveillance clients and relay the same to corresponding recording server and analytic server, (iii) receive configuration data from the surveillance clients and feed to the intended components of the system, (iv) receive event information from analytic server on-line and transmit to various recipients including remote event receiver, fetch outstanding event clips, if any, (v) periodically receive heartbeat signals along with status information from all active devices and relay that to other devices in same or other networks, (vi) stream live video, recorded video or event alerts at appropriate time, (vii) join multiple channel sensory inputs into a single combined stream to adapt to variable and low bandwidth network, (viii) enable search based on various criteria including data, time, event types, channels, signal features, and other system input and (ix) enable user to perform an user-interactive smart search to filter out desired segment of the sensory input from the database.

21. An integrated intelligent server based system as claimed in claim 1 wherein said acquisition cum recording server group comprise plurality of sensory data recording server adapted to:

record inputs from single/multiple data sources in at least one local storage space database with the URL of the files stored;

transfer the thus stored files from said local storage to a network based central storage provided for accessing the files for end use/applications,

said transfer of sensory data from source to the central storage via said local storage being carried out taking into consideration the data download speed (inflow rate) from data source to server along with the availability of network bandwidth at any given point of time for efficient network bandwidth sharing amongst multiple data sources to said storage device in the network.

22. An integrated intelligent server based system as claimed in claim 21 wherein said sensory data recording server is adapted to monitor available total network bandwidth and per channel inflow rate and based thereon decide rate of per channel video transfer from the server local storage to said central storage.

23. An integrated intelligent server based system as claimed in claim 22 wherein said sensory data from the source are recorded in the form of variable length clips wherein the clip duration is set by the user or set by the server itself.

24. An integrated intelligent server based system as claimed in claim 22 wherein said sensory data recording server is adapted for determining the optimal bit rate for uploading sensory inputs involving:

(a) average bit rate for each channel separately in periodic intervals wherein the sensory input streaming rate (D_i) of a particular source/camera (C_i) camera to the server is estimated and (b) identifying the available network bandwidth (B) at that instant from the system; and

finally (c) calculating the frequency of Clip upload for channel, based on;

U_i=[B×

k÷

Σ

D_i]×

D_i,

where 0<

k<

1, depending on how much of the remaining bandwidth is to be allocated for video uploading task.

25. An integrated intelligent server based system as claimed in claim 1 wherein the capacity of the respective servers in a server group is based on the memory, network bandwidth and current processor utilization within the server.

26. An integrated intelligent server based system as claimed in claim 25 wherein a server group is adapted to allocate any one of the operative servers in said group as the group master server and continuously monitor the servers in the group and their respective capacities and decide on the allocation and release of the input sensory source from any server within the Group.

27. An integrated intelligent server based system as claimed in claim 26 the said group master server is adapted to release or add a sensory input source based on required (a) addition of an input source, (b) deletion of an existing input source, (c) addition of a new recording server to the system or when a failed server again re-operates and (d) when a running server stops functioning.

28. An integrated intelligent server based system as claimed in claim 1 wherein each said analytical server is adapted for multiple component colour object analysis in a scene favouring scene analytic applications comprising:

multiple component colour coherent background estimation involving colour correlation of neighbouring pixels and inter-frame multiple component colour correlation using said multiple components as a composite data and using the relative values of these components to maintain accurate colour information and appearance of the true colour in the estimated background frame.

29. An integrated intelligent server based system as claimed in claim 28 wherein said analytical server is adapted for colour object analysis involving said unified colour coherent background estimation involving statistical pixel processing comprises using R,G,B components as a composite single structure in a unified manner to thereby preserve the mutual relationship of theses colour components in each individual pixel in order to maintain true colour appearances in the estimative colour background frame;

continuously readjusting modelled or predicted values for each colour pixel in a frame with all sequential forthcoming frames of the colour video;

correlate the spatial distribution of the colour values in a local region to model the pixel background colour value.

30. An integrated intelligent server based system as claimed in claim 28 wherein said analytical server is adapted for colour object analysis involving said colour analysis of each pixel comprising accumulating the colours in the above window in different colour clusters k consisting of a mean representative colour pixel value (μ

_R,μ

_G,μ

_B)_k with span of colour deviation (σ

_R,σ

_G,σ

_B)_k and a number of appearance (ν

_k) of a colour pixel in this cluster and based thereon

i) Matching the colour pixel (R,G,B) with colour cluster k to confirm if the same is within the span of colour deviation;

ii) If the colour of the pixel does not match with any cluster then create a new colour cluster with mean value (R, G, B) and default chosen allowed threshold for deviation (σ

_Th,σ

_Th,σ

_Th) and number of occurrence ν

=1

ii) Split the colour cluster (p) which have a large (σ

_R,σ

_G,σ

_B)_p value and merge all the colour cluster which have very close mean representative value, the probability of occurrence then adjusted in the same ratio of the estimated colour clusters for that population, to thereby achieve finer granular colour matching.

31. An integrated intelligent server based system as claimed in claim 1 wherein said analytical server is adapted for efficient face detection in video images and the like by limiting the search space involving motion detection technique and controlled computational requirements based on desired accuracy by carrying out prediction of number iterations and temporal parameter “

t”

.

32. An integrated intelligent server based system as claimed in claim 31 wherein said analytical server for said face detection is adapted for:

i) involving the grey image of cropped motion rectangular area from current frame to calculate said temporal parameter “

t” and

updating “

t”

with history and calculating possible number of iterations “

nIterations”

ii) calculating scale factor, no. of iterations and other parameter from look up table;

iii) using convolution on different scaled images to get probable face rectangles;

iv) grouping the probable faces with spatial information; and

v) obtaining therefrom the confirmed faces.

33. An integrated intelligent server based system as claimed in claim 1 comprising resource allocation for analytical servers involving:

estimating scene complexity relevant for frequency of frame processing;

spawning of processor threads based on physical CPU cores involving a controller;

allocation of threads to video channels for analytical processing based on requirements; and

feeding the frames for processing to a video analytics engine at an fps F, where F is calculated dynamically by the analytics engine itself depending upon its processing requirements based on scene complexity to thereby favour optimal sharing of resources eliminating unnecessary computing.

34. An integrated intelligent server based system as claimed in claim 33 wherein said scene complexity is determined based on (a) inter class difference of foreground and background (b) number of objects present and (c) extent of processing based on the particular processing task.

35. An integrated intelligent server based system as claimed in claim 33 comprising a Controller module for spawning a number of processing threads depending on the number of CPU cores present as available from the system hardware information and a task scheduler module for generating the sequence indicating the order in which the individual channels are to be served for analytics tasks.

36. An integrated intelligent server based system as claimed in claim 1 comprising multi channel join-split mechanism adapted for low and/or variable bandwidth network link comprising:

a sender unit adapted to receive multi channel inputs from site to join and compress into a single channel and a receiver unit at the client site to receive the inputs and extract the individual channels for the purposes of end use

said sender unit adapted to combine while transmitting multi channel inputs into a single channel, frame by frame, and controlling the transmission bit rate to avoid jittery out puts and/or any interference between individual channels and/or starvation for any single channel.

37. An integrated intelligent server based system as claimed in claim 35 comprising means for encoding the stream with variable bit rate depending upon the available bandwidth from server to the client, a frame header is transmitted with each frame of the combined stream, said frame header containing meta data about the constituent streams, said receiver unit adapted to split the combined stream into constituent streams based on said frame header.

38. An integrated intelligent server based system as claimed in claim 35 wherein the sender unit is adapted to receive raw inputs or decode the inputs to raw input and store in a memory allocated for inputs from a defined channel and generate an initial fps on request from a client, on request of a subset of channel from the client, a sample module is adapted to take the current frame from the channel specific memory area at a fixed rate for those channels and combines to a single frame along with generation of a look-up table to store the channel ID and its boundary within the combined frame and finally compressed and checked to identify all motion vectors which cross the allocated inter-frame boundary and forcibly set all such motion vectors to null to ensure that the video content of one constituent frame within the combined frame does not interfere with the content of another constituent frame, a frame header composed with meta data information about the position of the individual channels frames within the combined frame, the resolution of the individual frames and the time stamp;

said receiver unit is adapted to open a TCP connection with the sender and request for all or selected channels including selectively specifying the format for compression, additional commands to get the existing channel information, resolution of the channels, the fps of the individual channels at the senders end and other inputs directed to specifying the channels of interest and specifying other parameters as the transmitting fps (f), initial bit rate, etc.

39. An integrated intelligent server based system as claimed in claim 16 wherein said event decider means comprises an enhanced object tracking system comprising:

object tracking means in conjunction and one or more PTZ cameras wherein when an object is first detected in a fixed camera view of the said object tracking means the same is adapted to track the object and also generate and transmit the positional values along with a velocity prediction data to the PTZ camera controller;

said PTZ camera controller adapted to receive the positional information of the object in the PTZ camera view involving scene registration and coordinate transformation technique.

40. An integrated intelligent server based system as claimed in claim 38 adapted to carry out said coordinate transformation following:

a. identifying a set of points in the static camera as A, B, . . . and also corresponding points A′

, B′

, . . . in the PTZ camera by the user;

b. mapping any arbitrary point C in the static camera to the corresponding point C′

in the PTZ camera view dynamically wherein;

a_x, b_x, c_x are x-coordinates of points A, B and C respectively in the static Camera view and similarly a′

_x, b′

_x and c′

_x are for the corresponding points in PTZ view where point C is interpolated with the help of points A and B, with a confidence factor W_AB, where W_AB=(A_x−

B_x)÷

[Minimum of (C_x−

B_x, C_x−

A_x)] is determined to be

C′

_xAB=B_x′

+[(A_x′

−

B_x′

)×

(C_x−

B_x)÷

(A_x−

B_x)]

and wherein similarly, an estimate of x-coordinate of the same point C is generated for all pair of points (A, B) in the Static camera view based on;

C′

_x=Σ

[C′

_xAB×

W_AB]÷

Σ

W_AB

and similarly generating also the y-coordinate C′

_y for the point C.

41. An integrated intelligent server based system as claimed in claim 1 wherein said acquisition cum recording servers and said analytical server are adapted to carry out intelligent automated traffic enforcement involving a video surveillance system with video analytic servers adapted for carrying out sequential analytical process (a) configuration means (b) incident detection means (c) incident audit means (d) reporting generation means (e) synchronization means and (f) user management means.

42. An integrated intelligent server based system as claimed in claim 1 comprising a site map server installed within each autonomous system and also within the centralized server gateway to the entire system which is adapted to receive request from any authorised components of the system and respond with positional data corresponding to any component linked, said site layer preferably multilayered and components linked to any spatial position of the map in any layer.

43. A computer readable medium adapted for enabling and operating an integrated intelligent sensory input/data acquisition cum recording server group and/or analytics server group adapted to facilitate fail-safe integration and/or optimized utilization of various sensory inputs for various utility applications comprising at least one autonomous system having:

I) A) said sensory input acquisition cum recording server group comprising plurality of acquisition cum recording servers which are operatively linked to assess respective server capacity and operate as a group to enable fail-safe support when any of the servers in the group fail to operate the remaining operative servers in the group are adapted to distribute and take over the sensory input load of the non-operative server/s to render the system fail safe and self sufficient; and

/or B) said analytics server group comprising plurality of analytics server for intelligent analysis including resource dependent analytical accuracy control including means adapted for computing complexity of scenes and dynamically reconfigure the analytical processing steps for optimal analysis and/or availability of computational and other resources for on-line and real-time and/or on demand for efficient and user friendly streaming/analysis/detection/alert generation of events and/or follow up actions; and

II) an intelligent interface for operative connection to said sensory input acquisition cum recording server group; and

/or said analytics server group.