Holographic technology implemented security solution

• US 10,650,593 B2
• Filed: 12/15/2016
• Issued: 05/12/2020
• Est. Priority Date: 07/12/2016
• Status: Active Grant

• Current Assignee: Tyco Fire & Security GmbH (Johnson Controls International plc)
• Original Assignee: Tyco Fire & Security GmbH (Johnson Controls International plc)
• Inventors: Locke, Robert B., Rasband, Paul B., Cui, Rain, Schattmaier, Steve, Campero, Richard
• Primary Examiner(s): Hasan, Mainul
• Application Number:
  US15/379,647
• Publication Number:
  US 20180018867A1
• Time in Patent Office: 1,244 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/043 Distributed expert systems; Blackboards
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0281 Customer communication at a business location, e.g. providing product or service information, consulting
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/004 : Annotating, labelling
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G07C 9/257 : electronically G07C9/26 takes precedence
• CPC Class Codes: G08B 13/19608 : Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19656 : Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 : Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

A server system can receive an assertion of an alarm condition from a security system that processes sensor signals from sensors and that triggers the alarm condition. The server system is can send messages to determined nearby sensors to start sending data back to the server system according to the alarm condition. The server system can analyze sensor data received from the sensors. The analysis includes a verification of the alarm condition, a determination of how often queried data is requested, and a determination of which of selected data received from selected sensors to forward to one or more mixed reality devices. The server system can forward data to the one or more mixed reality.

Claims

1. A system, comprising:

one or more server systems configured to;

receive an assertion of an alarm condition from a security system that processes sensor signals from sensors and that triggers the alarm condition;

send messages to determined nearby sensors to start sending data back to the one or more server systems according to the alarm condition;

analyze sensor data received from the sensors, the analysis including a verification of the alarm condition, a determination of how often queried data is requested, a determination of which of selected data received from selected sensors to forward to one or more of a plurality of mixed reality devices, a determination of a specified function of one or more users associated with the plurality of mixed reality devices, and continually updated respective locations of the plurality of mixed reality devices; and

forward data to the one or more of the plurality of mixed reality devices, with the forwarded data being determined in part based on the analysis performed by the one or more server systems.

2. The system of claim 1 wherein the one or more server systems based on a nature of the condition, and network addresses of sensors determine the relevant nearby sensors;

and

generate messages that are sent to the security system to adjust operation of the relevant nearby sensors, based on the analysis of the sensor data.

3. The system of claim 1 wherein the one or more server systems receive video feeds from cameras in the facility having the security system;

and the system is further configured to;

send video feeds to the one or more of the plurality of mixed reality devices.

4. The system of claim 1 wherein the one or more server systems execute analytic processes to determine which sensors sent queries and construct the queries.

5. The system of claim 1 wherein the one or more server systems access a database for IP addresses of nearby sensors and form queries that are sent to those sensors through the network.

6. The system of claim 1, wherein the plurality of mixed reality devices each comprising:

a processor device;

a head mounted display device including a stereoscopic 3D display; and

a storage device that stores a program of computing instructions for execution by the processor, the program comprising instructions configured to cause the processor to;

receive data from the one or more server systems; and

send a request to the one or more server systems for additional data pertaining to the assertion of the alarm condition; and

render the information on the head mounted display device.

7. The system of claim 1 further comprising:

a session manager that controls interactions with the plurality of mixed reality devices over the Internet using a “

session portal”

that includes a web service API (application programming interface) or a dedicated socket with a transfer protocol.

8. A method, comprising:

receiving by a server system an assertion of an alarm condition from a security system at a facility, with the security system processing sensor signals from sensors and triggering the alarm condition;

sending by the server, system messages to determined nearby sensors to either turn on or start sending data back to the server system;

sending by the server system, queries to the sensors for sensor data according to the alarm condition;

receiving by the server system, video from cameras at the facility;

analyzing by the server system, sensor data received from the sensors, the analysis including a verification of the alarm condition, a determination of how often queried data is requested, a determination of which of selected data received from selected sensors to forward to one or more of a plurality of mixed reality devices, a determination of a specified function of one or more users associated with the plurality of mixed reality devices, and continually updated respective locations of the plurality of mixed reality devices; and

forwarding by the server system, data to the one or more of the plurality of mixed reality devices, with the forwarded data being determined in part based on the analysis performed by the server system with at least one of the one or more mixed reality devices including a processor device, memory in communication with the processor device and a head mounted display device including a stereoscopic 3D display.

9. The method of claim 8 further comprising:

receiving by the server system, a request for relevant information pertaining to the assertion of the alarm condition to be rendered on the head mounted display device.

10. The method of claim 8 wherein the server system, based on a nature of the condition, and network addresses of sensors determine the relevant nearby sensors;

and

generate messages that are sent to the security system to adjust operation of the relevant nearby sensors.

11. The method of claim 8 further comprising:

constructing by the server system the queries to send to specific cameras to transmit video feeds from the specific cameras in the facility to the server system; and

sending by the server system the video feeds from the specific cameras to the one or more of the plurality of mixed reality devices.

12. The method of claim 8, further comprising:

executing by the server system an analytic processes to determine which sensors are sent the queries; and

constructing by the server system the queries to send to the determined sensors.

13. The method of claim 12, further comprising:

accessing by the server system a database for IP addresses of nearby sensors to construct the queries.

14. A server system comprising one or more processing devices and a memory configured to:

receive an assertion of an alarm condition from a security system that processes sensor signals from sensors and that triggers the alarm condition;

send messages to determined nearby sensors to start sending data back to the server system according to the alarm condition;

analyze sensor data received from the sensors, the analysis including a verification of the alarm condition, a determination of how often queried data is requested, a determination of which of selected data received from selected sensors to forward to one or more of a plurality of mixed reality devices, a determination of a specified function of one or more users associated with the plurality of mixed reality devices, and continually updated respective locations of the plurality of mixed reality devices; and

forward data to the one or more of the plurality of mixed reality devices, with the forwarded data being determined in part based on the analysis performed by the server system.

15. The server system of claim 14 is further configured to:

based on a nature of the condition, and network addresses of sensors determine the relevant nearby sensors; and

generate messages that are sent to the security system to adjust operation of the relevant nearby sensors, based on the analysis of the sensor data.

16. The server system of claim 14 is further configured to:

receive video feeds from cameras in the facility having the security system; and

send video feeds to the one or more of the plurality of mixed reality devices.

17. The server system of claim 14 is further configured to:

execute analytic processes to determine which sensors sent queries and construct the queries.

18. The server system of claim 14 is further configured to:

access a database for IP addresses of nearby sensors and form queries that are sent to those sensors through the network.