Integrated intelligent server based system for unified multiple sensory data mapped imagery analysis

• US 10,212,462 B2
• Filed: 07/10/2017
• Issued: 02/19/2019
• Est. Priority Date: 01/11/2012
• Status: Active Grant

• Current Assignee: Videonetics Technology Private Limited
• Original Assignee: Videonetics Technology Private Limited
• Inventors: Acharya, Tinku, Bose, Tuhin
• Primary Examiner(s): Mejia, Anthony
• Application Number:
  US15/644,925
• Publication Number:
  US 20170311013A1
• Time in Patent Office: 589 Days
• Field of Search: 709202
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23116 involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2404 Monitoring of server processing errors or hardware failure error or fault detection G06F11/07; monitoring in general G06F11/30
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An integrated intelligent server based system includes at least one autonomous system containing one or more analytical server for unified multiple sensory data mapped multi-modal or multi-sensory imagery analysis, generating analysis result and streaming of the sensory data with the generated analysis result to the receiver module. A fail safe and fault tolerant technique to generate, store and stream the multi-sensory images to the receiver module is also proposed. Once received, the receiver module extracts the sensory data and the results of the sensory data analysis from the image and video using a suitable decoder. The integrated intelligent server based system would enable embedding the results of the sensory data analysis within the image itself and streaming the multi-sensory image embedded with the results of said sensory data analysis to a receiver module. This results in adopting this new concept in Internet of Things scheme of activities as well.

Claims

1. An integrated intelligent server based system for unified multiple sensory data mapped imagery analysis of sensory data received from channels and streaming of the sensory data along with analysis result to a receiver module comprising

at least one autonomous system containing one or more analytical server;

a seamless and intelligent interconnection of said autonomous systems receiving said sensory data from said channels for unified multiple sensory data mapped imagery analysis including a cooperative communication channel between all of said autonomous systems enabling desired scalability of number of the autonomous systems spread across wide geographical regions and also allocating the sensory data from the channels to the analytical servers contained in said autonomous systems through said cooperative communication channel, wherein each of said analytical server is enabled for storing the sensory data received from the channels connected to that analytical server, analyzing the stored sensory data to generate the analysis result and streaming of the sensory data with the generated analysis result to the receiver module either as individual stream for each of the connected channel, or as a joined single stream of sensory data for all or user requested channels among the connected channels; and

an intelligent interface communication channel operatively linked to said cooperative communication channel, for carrying the frames with the analysis result of each said analytical server to the receiver module.

3. The integrated intelligent server based system as claimed in claim 1, wherein each of the channels corresponds to a sensor deployed over any region or a cluster of identical sensors deployed over multiple regions of any zone for sensing a parameter of that zone and generating sensory data.

19. The integrated intelligent server based system as claimed in claim 1 wherein the receiver module comprises standalone surveillance client, internet browser, web client, any hand-held devices including mobile device client, and remote event and/or notification receiver having operative communication with the controller over ip based network;

and

said receiver module is further configured for establishing operative communication with the network accessible central storage over ip based network for accessing the stored sensory data.

22. The integrated intelligent server based system as claimed in claim 1, wherein the intelligent interface communication channel (i) auto registers itself to the system, (ii) accepts request from surveillance clients or the receiver module and relays the same to corresponding recording server and analytical server, (iii) receives configuration data from the surveillance clients or the receiver module and feeds to the intended components of the system, (iv) receives the analysis result from the analytical server and transmits to various recipients including remote event receiver, fetches outstanding event frames, if any, (v) periodically receives heartbeat signals along with status information from all active servers and relays that to other devices in same or other networks, (vi) streams live frames, recorded frames or event alerts at appropriate time, (vii) joins multiple channel sensory inputs into a single combined stream to adapt to variable and low bandwidth network, (viii) enables search based on various criteria including data, time, event types, channels, signal features, and other system input and (ix) enables user to perform an user-interactive smart search to filter out desired segment of the sensory input.

23. The integrated intelligent server based system as claimed claim 1, comprises central sensory data management server operatively connected with all the autonomous systems for registering all the recording servers and the analytical servers into the autonomous systems generating unique Identification number;

and

said central sensory data management server comprises a central database to store all configuration data related to the server including identification of sensory data sources or channels it caters to, the central storage it uses, the corresponding receiver module against the generated unique Identification number.

24. The integrated intelligent server based system as claimed claim 1, comprises remote event receiver includes object tracking system comprising:

object tracking means in conjunction and one or more PTZ cameras wherein when an object is first detected in a fixed camera view of the object tracking means the same is adapted to track the object and also generate and transmit the positional values along with a velocity prediction data to the PTZ camera controller; and

said PTZ camera controller adapted to receive the positional information of the object in the PTZ camera view involving scene registration and coordinate transformation technique.

25. The integrated intelligent server based system as claimed in claim 24, wherein the PTZ camera controller carry out coordinate transformation following:

a. identifying a set of points in the static camera as A, B, and also corresponding points A′

, B′

, . . . in the PTZ camera by the user;

b. mapping any arbitrary point C in the static camera to the corresponding point C′

in the PTZ camera view dynamically wherein;

a_x, b_x, c_x are x-coordinates of points A, B and C respectively in the static Camera view and similarly a′

_x, b′

_x and c′

_x are for the corresponding points in PTZ view where point C is interpolated with the help of points A and B, with a confidence factor W_AB, where W_AB=(A_x−

B_x)÷

[Minimum of (C_x−

B_x, C_x−

A_x)] is determined to be

C′

_xAB=B_x′

+[(A_x′

−

B_x′

)×

(C_x−

B_x)÷

(A_x−

B_x)]

and wherein similarly, an estimate of x-coordinate of the same point C is generated for all pair of points (A, B) in the Static camera view based on;

C′

_x=Σ

[C′

_xAB×

W_AB]÷

Σ

W_AB

and similarly generating also the y-coordinate C′

_y for the point C.

2. The integrated intelligent server based system as claimed in claimed 1 for unified multiple sensory data mapped imagery analysis of sensory data received from channels and centralized storing and streaming of the sensory data along with the analysis result to a receiver module comprising

at least one autonomous system containing one or more said analytical server and one or recording server;

a seamless and intelligent interconnection of said autonomous system receiving said sensory data from said channels for unified multiple sensory data mapped imagery analysis and centralized storing including a cooperative communication channel between all of said autonomous systems enabling desired scalability of number of the autonomous systems spread across wide geographical regions and also allocating the sensory data from the channels to said analytical servers and said recording servers of said autonomous systems through said cooperative communication channel whereby said cooperative communication channel enabling assesing respective server capacity, configuration of each of the recording servers to operate them as a group of the recording servers and to enable fail-safe support when any of the recording servers in the group fail to operate, remaining operative recording servers in the group distribute and take over the sensory data load of said recording servers in the group which fail to operate to thus render the system fail safe and self-sufficient;

wherein, each of said recording server in the group comprises;

local storage for storing the sensory data in segmented clips of various size and monitor sensory data inflow rate for each of the channels into the recording server and available network bandwidth to a network accessible central storage; and

said intelligent interface communication channel operatively linked to said cooperative communication channel, for carrying the frames with the analysis result of each said analytical server to the receiver module and carrying said sensory data of each said recording server to the network accessible central storage involving intelligent network bandwidth sharing amongst said channels for transferring said sensory data received from the channels to the network accessible central storage based on real time available network bandwidth as well as each of said sensory data inflow rate, local storage space of the respective recording servers wherein each of said recording server is self enabled for adjusting of rate of uploading of said segmented clips received by it to said network accessible central storage from an individual channel based on the available network bandwidth for uploading to the central storage and the sensory data inflow rate for said individual channel to its connected recording server and free local storage space of said recording server connected to said individual channel for transferring the sensory data received from said individual channel to the network accessible central storage via the local storage of said recording server in a failsafe bandwidth optimized manner utilizing optimal bandwidth and share the available network bandwidth amongst the channels and their respective recording servers for uploading the sensory data to the network accessible central storage in fail safe manner.

17. The integrated intelligent server based system as claimed in claim 2, wherein all the recording servers and the analytical servers in the autonomous system auto register themselves by requesting and thereby getting a unique identification number.

18. The integrated intelligent server based system as claimed in claim 2 wherein the network accessible central storage comprises a cluster of one or more network accessible storage devices.

20. The integrated intelligent server based system as claimed in claim 2, wherein the segmented clips of the sensory data include small granular clips or segments of programmable and variable length sizes and said clips stored in the local storages of the recording servers, the clip metadata being stored in a local database.

21. The integrated intelligent server based system as claimed in claim 2, wherein the recording server determine the rate of uploading of the segmented clips of the sensory data received from the channels to the network accessible central storage by:

calculating average of the rate of uploading for each channel separately in periodic interval including;

calculating the rate of clip upload (U_i) for a particular channel by (a) estimating the sensory data inflow rate (D_i) of said particular channel to the recording server;

(b) identifying the available network bandwidth (B) at that instant from the system;

(c) calculating the rate of clip upload for the particular channel, based on;

U_i=[B×

k÷

Σ

D_i]×

D_i,

where 0<

k<

1, depending on how much of the remaining bandwidth is to be allocated for uploading task and amount of local storage space available in the recording server.

4. The integrated intelligent server based system as claimed in claimed 1, wherein each of said analytical server comprises:

local storage for storing the sensory data and forming sensory image for each of the channels connected to the analytical server;

image analyzer for receiving said sensory images corresponding to the connected channels in temporal sequence forming frames representing temporal variation of the sensory data and therefrom analyzing temporal and spatial association of objects to generate the analysis result; and

controller for adjusting number of the frames per second feeding into said image analyzer depending on available computational space of the image analyzer at any point of time, including streaming of the sensory data with the generated analysis result embedded within the images itself to the receiver module either as individual stream for each of the channel, or as a joined single stream of sensory data for all or user requested channels of the connected channels.

7. The integrated intelligent server based system as claimed in claim 4, wherein each of the frames corresponds to a group of multiple sixel based sensory image planes corresponding to different channels having identical timestamp.

8. The integrated intelligent server based system as claimed in claim 4, wherein the controller stores the sensory data in the local storage of the analytical server overwriting the existing stored sensory data which are fed into the processor and streamed into the receiver module.

9. The integrated intelligent server based system as claimed in claim 4, wherein the image analyzer comprises:

data input port to receive the frames via the controller;

processing memory to temporarily store the frames till interpretation of the composite values of the one or more sixels of the frames and infer dynamics of the objects in the stored frames; and

computing module operatively connected with said processor memory to access the stored frames and analyze them by including;

estimating a background reference corresponding to the frames stored in the processor memory, comprising;

computing an adaptive local window centring around each sixel of the frames;

accumulating all sixel values in the local window in different sensory clusters depending on their distinguishability criteria of appearance enabling each sensory cluster consist of mean representative sixel values corresponding to each of the frames under analysis with span of sensory value deviation and a number of appearance of a sixel in that cluster; and

splitting all the sensory cluster having large sensory value deviation and merging all the sense cluster which having close mean representative sixel value and constructing the background reference corresponding to the frames under analysis from sensory values of the generated merged cluster;

extracting foreground objects in the frames under analysis by including;

constructing a reference frame for each input frame stored in the processor memory from the estimated background reference corresponding to previous one or more of stored frames of that input frame;

comparing the input frame with its reference frame and constructing a difference image to extract the foreground objects; and

segmenting the difference image according to the extracted foreground objects;

analysing each segment individually for classification of the objects by including;

comparing each captured foreground object with typical object shape, silhouette, spectral feature in the computing module for categorizing the objects; and

analyzing the objects including associating with the previous frames to infer dynamics of the object using pre-determined application dependent rules to identify occurrences of any rule violation or nature of anomaly in the region;

said computing module is operatively connected with the controller to forward the generated analysis result and streaming to the receiver module.

10. The integrated intelligent server based system as claimed in claim 9, wherein the computing module computes the adaptive size local window by:

constructing an initial window of size (h, w) centring around the sixel;

computing normalized average intensity (k) of all the sixels in said initial window of size (h, w) by following

11. The integrated intelligent server based system as claimed in claim 9, wherein each of the sensory clusters corresponds to group of all neighbouring sixels that are different from other neighbouring sixels whereby the differences lie in the values of the sixels at any particular instant of time relative to their previous weighted-average values and the neighbouring sixels that have this relative values above a certain threshold value form a cluster, the threshold values are set based on type of particular sensory data as well as on the area of application.

12. The integrated intelligent server based system as claimed in claim 9, wherein the computing module discard the sixels of current frame of the frames belongs to the object sixel locations in previous frame of the frames from the estimation of the background reference corresponding to the frames.

13. The integrated intelligent server based system as claimed in claim 9, wherein the computing module includes computer system comprising of a first computer embodying executable first set of computer program instructions for estimating the background reference, a second computer embodying executable second set of computer program instructions for the extracting foreground objects and a third computer embodying executable third set of computer program instructions for analysing the segments.

14. The integrated intelligent server based system as claimed in claim 9, wherein the controller monitors availability of the processor memory for storing the frames and computational complexity of the computing module for analyzing different frames to dynamically adjust the number of the frames per second feeding into the processor.

15. The integrated intelligent server based system as claimed in claim 9, wherein the controller includes encoder for encoding and streaming the multi-sensory data in form of a single multi-sensory image sequences to receiver module with variable rate depending upon available bandwidth from the analytical server to the receiver and transmitting a frame header embedded with each frame containing metadata about position, identity of the channel frame within the combined frames, resolution of the individual frames, a timestamp of the constituent streams and the analysis result corresponding to the multi-sensory images of the frames to enable the receiver module to split and interpret the combined stream of frames based on said frame header.

16. The integrated intelligent server based system as claimed in claim 15, wherein the receiver module includes decoder which is a natural extension of standard visual image decoders, configured to extract one or more number of image planes as the multi sensory images received from the analytical server corresponding to different sensory values including usual spectral values of optical sensors/cameras.

5. The integrated intelligent server based system as claimed in claimed 4, wherein the local storage of the analytical server stores the sensory data with timestamp and generate sixel based sensory image for each of the channels connected to the analytical server.

6. The integrated intelligent server based system as claimed in claim 5, wherein the local storage of the analytical server includes memory locations having two dimensional address space to store the sensory data of each of the channels in two dimensional grid structure to form sixel based sensory image for each of the channels whereby the sixel corresponds to value of the sensory data in each grid point of the grid structure and each grid point corresponds to the region where the sensor has been deployed.