Systems and methods for mixed reality with cognitive agents

• US 10,521,968 B2
• Filed: 11/26/2018
• Issued: 12/31/2019
• Est. Priority Date: 07/12/2016
• Status: Active Grant

• Current Assignee: Sensormatic Electronics LLC (Johnson Controls International plc)
• Original Assignee: Tyco Fire & Security GmbH (Johnson Controls International plc)
• Inventors: Locke, Robert B., Rasband, Paul B.
• Primary Examiner(s): Craddock, Robert J
• Application Number:
  US16/200,341
• Publication Number:
  US 20190139315A1
• Time in Patent Office: 400 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/043 Distributed expert systems; Blackboards
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0281 Customer communication at a business location, e.g. providing product or service information, consulting
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/004 : Annotating, labelling
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G07C 9/257 : electronically G07C9/26 takes precedence
• CPC Class Codes: G08B 13/19608 : Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19656 : Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 : Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

Disclosed are techniques that use mixed reality, e.g., augmented reality and virtual reality technologies to improve analysis of retail processes and activity in retail establishments. A server system receives images from a mixed reality device, which images including representations of physical objects and receives voice data from the mixed reality device and executes a cognitive agent to process the voice data received from the mixed reality device that generates a set of virtual objects that include data describing merchandising process concepts regarding the set of physical objects, based on input queries received in the voice data and sending by a server system, the set of virtual objects to the mixed reality device.

Claims

1. A mixed reality system comprising:

one or more memory devices configured to store instructions that, when executed on one or more processors, cause the one or more processors to;

receive one or more images from a mixed reality device, the one or more images comprising representations of one or more physical objects;

generating one or more virtual objects based on the representations of the one or more physical objects;

receive voice data from the mixed reality device;

execute a cognitive agent to process the voice data by;

generating a first data structure representing text input of the voice data;

generating a second data structure based on the first data structure, the second data structure comprising response logic representing a response; and

generating one or more response sentences based on the response logic; and

communicate the one or more virtual objects and the one or more response sentences to the mixed reality device.

2. The mixed reality system of claim 1, wherein the voice data comprises one or more user queries;

wherein generating the one or more virtual objects is further based on the one or more user queries.

3. The mixed reality system of claim 1, wherein the one or more virtual objects comprise data describing merchandising process concepts associated with the one or more physical objects.

4. The mixed reality system of claim 1, wherein the instructions cause the one or more processors to:

determine location information of the mixed reality device; and

generate the one or more virtual objects based on the location information.

5. The mixed reality system of claim 1, wherein the response logic comprises response concepts and response concept qualifiers;

wherein the cognitive agent comprises a micro-planner;

wherein generating the one or more response sentences further comprises executing the micro-planner of the cognitive agent to generate the one or more response sentences based on the response concepts and the response concept qualifiers of the second data structure.

6. The mixed reality system of claim 1, wherein the cognitive agent comprises a response macro-planner;

wherein generating the second data structure based on the first data structure comprises executing the response macro-planner of the cognitive agent to;

determine information content of the response from the first data structure; and

generate the response logic of the second data structure based on the information content.

7. The mixed reality system of claim 1, wherein the cognitive agent comprises a natural language pre-processor;

wherein generating the first data structure representing the text input of the voice data comprises executing the natural language pre-processor of the cognitive agent to;

parse text input of the voice data into tagged words and phrases; and

generate the first data structure based on the tagged words and phrases, wherein the first data structure comprises the tagged words and phrases connected to each other and annotated with meaning indicators.

8. The mixed reality system of claim 1, wherein the instructions cause the one or more processors to execute the cognitive agent to resolve ambiguities of the voice data by accessing input data received from the mixed reality device, wherein the input data specifies the one or more physical objects with a highlight applied by a user to the one or more physical objects.

9. The mixed reality system of claim 1, wherein the voice data comprises a question regarding a product displayed by the mixed reality device, wherein the one or more response sentences form an answer to the question regarding the product;

wherein the instructions cause the one or more processors to;

generate the answer to the question by;

executing the cognitive agent to determine a meaning of the question based on the voice data;

generating one or more queries for an information algorithm; and

executing the information algorithm based on the one or more queries to generate the answer; and

communicate the answer to the mixed reality device.

10. The mixed reality system of claim 1, wherein the instructions cause the one or more processors to:

receive user input from the mixed reality device; and

execute the cognitive agent to identify at least one of the one or more physical objects indicated by the user input.

11. The mixed reality system of claim 10, wherein the user input comprises or more highlights applied to the at least one of the one or more physical objects.

12. A method for mixed reality, the method comprising:

receiving, by a processing circuit, one or more images from a mixed reality device, the one or more images comprising representations of one or more physical objects;

generating, by the processing circuit, one or more virtual objects based on the representations of the one or more physical objects;

receiving, by the processing circuit, voice data from the mixed reality device;

executing, by the processing circuit, a cognitive agent to process the voice data by;

generating a first data structure representing text input of the voice data;

generating a second data structure based on the first data structure, the second data structure comprising response logic representing a response; and

generating one or more response sentences based on the response logic; and

communicating, by the processing circuit, the one or more virtual objects and the one or more response sentences to the mixed reality device.

13. The method of claim 12, wherein the voice data comprises one or more user queries;

wherein generating, by the processing circuit, the one or more virtual objects is further based on the one or more user queries.

14. The method of claim 12, further comprising:

determining, by the processing circuit, location information of the mixed reality device; and

generating, by the processing circuit, the one or more virtual objects based on the location information.

15. The method of claim 12, wherein the response logic comprises response concepts and response concept qualifiers;

wherein generating, by the processing circuit, the one or more response sentences further comprises executing a micro-planner of the cognitive agent to generate the one or more response sentences based on the response concepts and the response concept qualifiers of the second data structure.

16. The method of claim 12, wherein generating, by the processing circuit, the second data structure representing the response based on the first data structure comprises executing a response macro-planner of the cognitive agent to:

determine information content of the response from the the first data structure; and

generate the response logic of the second data structure based on the information content.

17. The method of claim 12, wherein generating, by the processing circuit, the first data structure representing the text input of the voice data comprises executing a natural language pre-processor of the cognitive agent to:

parse text input of the voice data into tagged words and phrases; and

generate the first data structure based on the tagged words and phrases, wherein the first data structure comprises the tagged words and phrases connected to each other and annotated with meaning indicators.

18. The method of claim 12, further comprising executing, by the processing circuit, the cognitive agent to resolve ambiguities of the voice data by accessing input data received from the mixed reality device, wherein the input data specifies the one or more physical objects with a highlight applied by a user to the one or more physical objects.

19. The method of claim 12, wherein the voice data comprises a question regarding a product displayed by the mixed reality device, wherein the one or more response sentences form an answer to the question regarding the product;

wherein the method further comprises;

generating, by the processing circuit, the answer to the question by;

executing the cognitive agent to determine a meaning of the question based on the voice data;

generating one or more queries for an information algorithm; and

executing the information algorithm based on the one or more queries to generate the answer; and

communicating, by the processing circuit, the answer to the mixed reality device.

20. A building system of a building, the system comprising:

one or more memory devices configured to stores instructions; and

one or more processors communicably coupled to the one or more memory devices, the one or more processors configured to execute the instructions to;

receive one or more images from a mixed reality device, the one or more images comprising representations of one or more physical objects of the building;

generating one or more virtual objects based on the representations of the one or more physical objects;

receive voice data from the mixed reality device;

execute a cognitive agent to process the voice data by;

generating a first data structure representing text input of the voice data;

generating a second data structure based on the first data structure, the second data structure comprising response logic representing a response; and

generating one or more response sentences based on the response logic; and

communicate the one or more virtual objects and the one or more response sentences to the mixed reality device.