Multi-channel sensing system with embedded processing

• US 10,834,363 B1
• Filed: 03/26/2020
• Issued: 11/10/2020
• Est. Priority Date: 06/22/2017
• Status: Active Grant

• Current Assignee: INSIGHT Inc.
• Original Assignee: INSIGHT Inc.
• Inventors: Zhang, Yi, Xu, Li, Shu, Hua, Yang, Jason T.
• Primary Examiner(s): Chang, Daniel
• Application Number:
  US16/831,600
• Time in Patent Office: 229 Days
• Field of Search: 4554566, 4554042, 340 31, 34028602, 34053917, 340988, 348 1401, 348 1402, 382115
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 21/0423 detecting deviation from an expected pattern of behaviour or schedule
• CPC Class Codes: G08B 21/043 detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0492 Sensor dual technology, i.e. two or more technologies collaborate to extract unsafe condition, e.g. video tracking and RFID tracking
• CPC Class Codes: G08B 29/186 Fuzzy logic; neural networks
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

System and method for monitoring objects of interest including persons in an indoor or outdoor scene with high accuracy, strong privacy protection, high security, low volume data transmission and storage. The system includes: an embedded sensing device with multiple sensors and microprocessor(s) with built-in artificial intelligence as an end unit with parametric outputs, a controller that aggregate parametric outputs from multiple end units with local intelligence and configurable control outputs, and an optional gateway that aggregates multiple controllers output and connect with the network, cloud and/or human interface. The embedded sensing devices can capture data of the scene through multiple sensors with different functions and technologies; detect and track the objects of interest including persons; analyze features of the objects of interest; detect event and behavior associated with the detected objects or persons of interest based on the features and/or events; and only provide these parametric outputs to controller.

Claims

1. A parametric sensor network system comprising:

a plurality of embedded sensing devices, each including;

a plurality of sensors, including at least one imaging sensor, at least one depth sensor or distance sensor, and at least one audio sensor, the plurality of sensors configured to generate at least audio, image and video data; and

a data processing unit connected to and receiving the audio, image and video data from the plurality of sensors, the data processing unit including one or more processors, the data processing unit implementing an object detection algorithm to detect presence and locations of objects and humans, the data processing unit further implementing a human identification algorithm for identifying at least one specific human based on the audio and video data received from the sensors,

wherein the data processing unit generates parametric data based on the data received from the sensors using the object detection algorithm, wherein the parametric data include the identification and location of the specific human; and

a controller connected to and receiving data from each of the plurality of embedded sensing devices; and

a plurality of actuators coupled to the controller;

wherein each embedded sensing device outputs only the parametric data, which include the identification and location of the specific human, to the controller without automatically outputting any video, image or audio data generated by the sensors to the controller; and

wherein the parametric data generated by the data processing unit and outputted by the embedded sensing devices further include parameters describing a room type of a room, furniture and space utilization of the room, locations of indoor objects in the room, ambient light, occupancy, numbers of humans, a length of stay by the specific human, postures and gesture of the specific human, motions of the specific human, sound and voice of the specific human, a location of the specific human relative to the indoor objects, activities of the specific human in relation to the indoor objects, temporal length and frequency of the activities of the specific human, a journal of daily activities of the specific human, behaviors of the specific human, and a flag which indicates whether the behaviors of the specific human is normal or abnormal,

wherein the controller generates commands for controlling the actuators based on the parametric data received from the plurality of embedded sensing devices.

2. The system of claim 1, wherein the data processing unit includes a plurality of processing elements organized as a pipeline, wherein each processing element is configured to perform a defined processing task and to produce an output, and wherein an output of at least one of the plurality of processing elements is inputted to another one of the plurality of processing elements as input.

3. The system of claim 1, wherein each of the plurality of embedded sensing device further includes a user interface device configured to receive commands from a human or object, the user interface device including one or more of:

a remote control, voice recognition algorithm implemented in the data processing unit and configured to recognize a command, and a gesture and posture recognition algorithm implemented in the data processing unit and configured to recognize a gesture and posture.

4. The system of claim 1, wherein the plurality of actuators includes a lighting control actuator, a temperature control actuator, and an alarm actuator.

5. The system of claim 1, wherein the data processing unit further includes data storage devices configured to store data generated by the plurality of sensors.

6. The system of claim 1, wherein the controller is further connected to an external network, a telephone line, or an alarm device.

7. The system of claim 1, wherein the embedded sensing devices and the controller are configured to define a privacy level of the embedded sensing devices and the controller, and to define a monitoring area for each embedded sensing device.

8. The system of claim 7,

wherein either the privacy level is a high privacy level, wherein each embedded sensing device is configured to store no video or image data captured by the sensors;

or

the privacy level is an intermediate privacy level, wherein each embedded sensing device further includes a data storage device and the embedded sensing device is configured to store video or image data captured by the sensors in the data storage device of the embedded sensing device;

or

the privacy level is a low privacy level, wherein the controller further includes a data storage sub-system, and the controller is configured to pull video or image data from the embedded sensing devices and store the video or image data in the data storage sub-system.

9. The system of claim 1, wherein the controller further includes a data storage sub-system, and wherein the controller is further configured to, in response to detecting a predefined trigger event, pull video or image data from one or more of the embedded sensing devices and store the video or image data in the data storage sub-system;

and

wherein in response to detecting the predefined trigger event, the controller is further configured to generate predefined commands to the embedded sensing devices.

10. The system of claim 9, wherein the controller is further configured to generate a journal file of the controller containing spatial and temporal information based on the parametric data and the video or image data received from the embedded sensing devices.

11. The system of claim 10, wherein the data processing unit of each embedded sensing device is configured to generate multi-dimension time series data and to store the multi-dimension time series data in a journal file of the embedded sensing device;

and

wherein the controller is configured to extract information from the video or image data received from the embedded sensing devices, to generate aggregated multi-dimension time series data based on the parametric data received from the embedded sensing devices and the extract information, and to store the aggregated multi-dimension time series data in the journal file of the controller.

12. The system of claim 11, wherein the controller is further configured to perform event detection based on correlations in the aggregated multi-dimension time series data.

13. The system of claim 11, wherein the multi-dimension time series data includes rapid-changing time series data and slow-changing time series data;

wherein the controller is further configured to processes the slow-changing time series data.

14. The system of claim 1, wherein the object detection algorithm implemented in the data processing unit of each embedded sensing device is a machine leaning algorithm.

15. The system of claim 14, wherein machine learning algorithm is implemented by one or multiple deep neural network models trained in advance.