Holographic technology implemented security solution

• US 10,769,854 B2
• Filed: 12/15/2016
• Issued: 09/08/2020
• Est. Priority Date: 07/12/2016
• Status: Active Grant

• Current Assignee: Tyco Fire & Security GmbH (Johnson Controls International plc)
• Original Assignee: Tyco Fire & Security GmbH (Johnson Controls International plc)
• Inventors: Locke, Robert B.
• Primary Examiner(s): Prince, Jessica M
• Application Number:
  US15/379,657
• Publication Number:
  US 20180018823A1
• Time in Patent Office: 1,363 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/043 Distributed expert systems; Blackboards
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0281 Customer communication at a business location, e.g. providing product or service information, consulting
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/004 : Annotating, labelling
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G07C 9/257 : electronically G07C9/26 takes precedence
• CPC Class Codes: G08B 13/19608 : Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19656 : Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 : Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

Disclosed are techniques that use mixed reality, e.g., augmented reality and virtual reality technologies to improve analysis of security situations as well as retail processes and activity in retail stores. For security these techniques merge the physical world embodied in security systems with the virtual world of policies and analytics. In the retail aspect, these techniques merge the physical world of retail items, displays, and spaces with the virtual world of policies and analytics.

Claims

1. A system, comprising:

server computer systems that receive video feeds from plural fixed video cameras in a facility, the server computer systems including a storage device that stores a program of computing instructions for execution by server computer system, the program comprising instructions configured to cause the server computer system to;

construct a virtual, visual 3D model of the facility, the virtual, visual 3D model comprising areas selectable by a user corresponding to areas of the facility; and

control a mixed reality system comprising a processor device and a memory in communication with the processor device, and a head mounted display device including a stereoscopic 3D display;

with the mixed reality system configured to;

render using the head mounted display device the virtual, visual 3D model of the facility;

receive, from a camera of the head mounted display device, images indicating hand gestures of a user of the head mounted display device;

determine a plurality of commands by translating the images to identify the gestures;

receive a first command of the plurality of commands to monitor video feeds associated with a specified area of the facility via selection of an area within the virtual, visual 3D model corresponding to the specified area;

send requests to the server computer system for the video feeds associated with the specified area of the facility;

receive the video feeds;

receive a second command of the plurality of commands to pin a specific one of the video feeds to a particular position as rendered on the stereoscopic 3D display; and

render using the head mounted display device, the specific one of the video feeds to occupy a substantial portion on the stereoscopic 3D display by clearing any of the video feeds but the specific video feed from the stereoscopic 3D display.

2. The system of claim 1 wherein included with the video feeds are data including location data that specifies a location and an identification data that identifies a surveilling camera.

3. The system of claim 2 wherein the server computer systems process the data using convention processing analytics for determining either potential of alarm conditions or verification of asserted alarm conditions.

4. The system of claim 1 wherein personnel access the video feeds in an organized manner by the mixed reality system being configured to:

execute a mode that arranges the video feeds and other sensor data on a virtual wall of video monitors that is rendered in the stereoscopic 3D display of the mixed reality system.

5. The system of claim 4 wherein the video feeds are rendered in the stereoscopic 3D display of the corresponding mixed reality system as being affixed to the virtual wall at a fixed location in the display generated, by the corresponding mixed reality system, in response to a user pin command received from the user.

6. The system of claim 4 further configured to:

receive from a user a selection of the virtual wall of one of the video monitors for closer examination.

7. The system of claim 6 further configured to:

cause the mixed reality system to enter a mode by which the camera of the head mounted display device captures user gestures that signal the mixed reality system to select the one of the virtual monitors.

8. The system of claim 7 further configured to:

clear the stereoscopic 3D display upon receiving a selection of the virtual monitor and place the selected virtual monitor in the stereoscopic 3D display.

9. The system of claim 1 further comprising:

a session manager that controls interactions with the mixed reality system over the Internet using a “

session portal”

that includes a web service API or a dedicated socket with a transfer protocol.

10. A method performed by a mixed reality system comprising a head mounted display device including a stereoscopic 3D display, comprising:

rendering using the head mounted display device a virtual, visual 3D model of a facility, the virtual, visual 3D model comprising areas selectable by a user corresponding to areas of the facility;

receiving, from a camera of the head mounted display device, images indicating hand gestures of a user of the head mounted display device;

determining a plurality of commands by translating the images to identify the gestures;

receiving a first command of the plurality of commands to monitor video feeds associated with a specified area of the facility via selection of an area within the virtual, visual 3D model corresponding to the specified area;

sending requests to a server system for the video feeds associated with the specified area of the facility;

receiving the video feeds;

receiving a second command of the plurality of commands to pin a specific one of the video feeds to a particular position as rendered on the stereoscopic 3D display; and

using the head mounted display device to render the specific one of the video feeds to occupy a substantial portion on the stereoscopic 3D display by clearing any of the video feeds but the specific video feed from the stereoscopic 3D display.

11. The method of claim 10, wherein the video feeds are data including location data that specifies a location and an identification data that identifies a surveilling camera.

12. The method of claim 11, wherein the server computer system process the data using convention processing analytics for determining either potential of alarm conditions or verification of asserted alarm conditions.

13. The method of claim 10, further comprising:

executing a mode that arranges the video feeds and other sensor data on a virtual wall of video monitors that is rendered in the stereoscopic 3D display of the mixed reality system.

14. The method of claim 13, wherein the video feeds are rendered in the stereoscopic 3D display of the corresponding mixed reality system as being affixed to the virtual wall at a fixed location in the display generated, by the corresponding mixed reality system, in response to a user pin command received from the user.

15. The method of claim 13, further comprising:

receiving from a user a selection of the virtual wall of one of the video monitors for closer examination.

16. The method of claim 15, further comprising:

entering a mode by which the camera of the head mounted display device captures user gestures that signal the mixed reality system to select the one of the virtual monitors.

17. The method of claim 16, further comprising:

clearing the stereoscopic 3D display upon receiving a selection of the virtual monitor and placing the selected virtual monitor in the stereoscopic 3D display.

18. A mixed reality system, comprising:

an interface coupled to at least one server computer that receives video feeds from video cameras in a facility;

a storage device that stores a program of computing instructions for execution a processor, the program comprising instructions configured to a head mounted display device including a stereoscopic 3D display to;

render using the head mounted display device a virtual, visual 3D model of the facility, the virtual, visual 3D model comprising areas selectable by a user corresponding to areas of the facility;

receive, from a camera of the head mounted display device, images indicating hand gestures of a user of the head mounted display device;

determine a plurality of commands by translating the images to identify the gestures;

receive a first command of the plurality of commands to monitor video feeds associated with a specified area of the facility via selection of an area within the virtual, visual 3D model corresponding to the specified area;

send request to the server for the video feeds associated with the specified area of the facility;

receive the video feeds;

receive a second command of the plurality of commands to pin a specific one of the video feeds to a particular position as rendered on the stereoscopic 3D display; and

render using the head mounted display device, the specific one of the video feeds to occupy a substantial portion on the stereoscopic 3D display by clearing any of the video feeds but the specific video feed from the stereoscopic 3D display.

19. The system of claim 18, wherein the video feeds comprise data comprising location data that specifies a location and an identification data that identifies a surveilling camera.

20. The system of claim 18, wherein the mixed reality system is configured to:

execute a mode that arranges the video feeds and other sensor data on a virtual wall of video monitors that is rendered in the stereoscopic 3D display of the mixed reality system.