AUGMENTED REALITY VIRTUAL PERSONAL ASSISTANT FOR EXTERNAL REPRESENTATION

• US 20140310595A1
• Filed: 06/24/2014
• Published: 10/16/2014
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Kumar, Rakesh, Acharya, Girish, Samarasekara, Supun, Yarnall, Louise, Wolverton, Michael John, Zhu, Zhiwei, Villamil, Ryan, Braznoi, Vlad, Carpenter, James F., Murray, Glenn A.
• Granted Patent: US 10,824,310 B2
• Time in Patent Office: Days
• US Class Current: 715/706
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A computing system for virtual personal assistance includes technologies to, among other things, correlate an external representation of an object with a real world view of the object, display virtual elements on the external representation of the object and/or display virtual elements on the real world view of the object, to provide virtual personal assistance in a multi-step activity or another activity that involves the observation or handling of an object and a reference document.

Claims

1. A computing system for virtual personal assistance, the computing system comprising, embodied in one or more computing devices:

a scene understanding module to algorithmically analyze a video depicting a real world scene of a multi-step activity and detect, in the video, a physical object involved in the multi-step activity;

a correlation module to determine a correlation between a document comprising an external representation of the physical object with the video depiction of the object in the real world scene;

an input understanding module to, using the correlation, interpret user input relating to a step of the multi-step activity and relating to at least a portion of the external representation; and

a visual output generator to, in response to the interpreted user input, display a virtual element on one or more of a real world view of the physical object and a view of the document, the virtual element based on the correlation.

2. The computing system of claim 1, wherein the virtual element comprises an augmented reality overlay, and the visual output generator is to display the augmented reality overlay on the real world view of the object.

3. The computing system of claim 1, wherein the virtual element comprises an annotated portion of the external representation, and the visual output generator is to display the annotated portion of the external representation on the view of the document.

4. The computing system of claim 1, wherein the virtual element comprises data relating to the correlation between the external representation of the object and the real world view of the object.

5. The computing system of claim 1, wherein the computing system is to provide virtual personal assistance with an activity involving manipulation of a real-world physical object comprising multiple components, and:

the scene understanding module is to detect, in the video, one or more components of the physical object;

the correlation module is to determine a correlation between the external representation of the one or more components of the object with the video depiction of the real-world physical object; and

the input understanding module is to interpret user input based on the correlation, the user input relating to the one or more of the components of the physical object;

6. The computing system of claim 1, wherein the visual output generator is to, in response to the interpreted user input, display an annotated portion of the external representation, the annotated portion based on the correlation.

7. The computing system of claim 1, wherein the visual output generator is to overlay the virtual element on a display of the external representation, and wherein the virtual element is to indicate a correlation between the external representation and the object in the real world scene.

8. The computing system of claim 1, wherein the user input comprises natural language speech, the input understanding module is to generate a semantic interpretation of the natural language speech, the correlation module is to correlate the semantic interpretation of the natural language speech with the external representation, and the visual output generator is to overlay the virtual element on a real world view of the object in response to the natural language speech.

9. The computing system of claim 1, wherein the scene understanding module is to generate a semantic interpretation of the real world scene, the input understanding module is to generate a semantic interpretation of the user input, the computing system comprises a document understanding module to generate a semantic interpretation of the external representation, and the correlation module is to correlate the semantic interpretation of the real world scene and the semantic interpretation of the user input with the semantic interpretation of the external representation.

10. The computing system of claim 1, wherein the external representation comprises a plurality of schematic diagrams of the object, the scene understanding module is to detect, in the video, a step of the multi-step activity, the correlation module is to select a schematic diagram corresponding to the detected step of the multi-step activity, and the visual output generator is to overlay a virtual element representing a portion of the selected schematic diagram on a real world view of the object.

11. The computing system of claim 1, comprising an activity tracking module to track completed steps of the multi-step activity, wherein the visual output generator is to overlay a new virtual element corresponding to a portion of the object involved in a completed step of the multi-step activity on a real world view of the portion of the object involved in the completed step of the multi-step activity.

12. The computing system of claim 1, wherein the scene understanding module is to detect, in the video, a step of the multi-step activity, and the visual output generator is to overlay a virtual element comprising an x-ray view of a state of the object corresponding to the detected step of the multi-step activity on a real world view of the object.

13. The computing system of claim 1, wherein the scene understanding module is to detect, in the video, a plurality of different steps of the multi-step activity, the correlation module is correlate external representations of different portions of the detected object with different portions of the video depiction of the object in response to the different steps of the multi-step activity;

and the visual output generator is to overlay a different virtual element on a real world view of the object for each of the different portions of the detected object.

14. The computing system of claim 1, wherein the scene understanding module is to detect, in the video, a plurality of different steps of the multi-step activity, the correlation module is correlate external representations of the object in different steps of the multi-step activity with different portions of the video depiction of the object in response to the different steps of the multi-step activity;

and the visual output generator is to overlay a different virtual element on a real world view of the object for each of the different steps of the multi-step activity.

15. The computing system of claim 1, comprising a natural language generator to output a natural language explanation of the correlation between the external representation of the object and the video depiction of the object in the real world scene.

16. The computing system of claim 1, comprising a first computing device and a second computing device communicatively coupled to the first computing device, wherein the computing system is to display the external representation on the first computing device and display an augmented reality view of the real world scene comprising the virtual element on the second computing device.

17. The computing system of claim 1, wherein the computing system is to display the external representation and an augmented reality view of the real world scene comprising the virtual element on the same computing device.

18. An augmented reality virtual personal assistant embodied in one or more machine accessible storage media and comprising instructions executable by one or more computing devices to:

analyze a video depicting a real world scene of a multi-step activity;

algorithmically detect a physical object involved in the multi-step activity;

correlate a document comprising content relating to the object detected in the video with the video depiction of the object in the real world scene; and

in response to user input relating to the multi-step activity, present a virtual overlay on one or more of a view of the real world scene and a view of the document, the virtual overlay indicative of the correlation between the content of the document and the object in the real world scene.

19. The augmented reality virtual personal assistant of claim 18, wherein the instructions are executable by one or more computing devices to, in response to the user input, select a portion of the document, and overlay a virtual element indicative of content of the selected portion of the document on a real world view of a portion of the object that corresponds to the selected portion of the document.

20. The augmented reality virtual personal assistant of claim 18, wherein the instructions are executable by one or more computing devices to, in response to the user input, select a portion of the video depiction of the object, and overlay a virtual element on a display of a portion of the document corresponding to the selected portion of the object, wherein the virtual element is indicative of a correlation between the portion of the document and the selected portion of the object.

21. The augmented reality virtual personal assistant of claim 18, wherein the instructions are executable by one or more computing devices to detect human interaction with the document, interpret the human interaction, determine a suggestion relating to the multi-step activity in response to the human interaction, and output the suggestion on a view of the real world scene.

22. The augmented reality virtual personal assistant of claim 18, wherein the instructions are executable by one or more computing devices to detect human interaction with the object in the real world environment, interpret the human interaction, determine a suggestion relating to the multi-step activity in response to the human interaction, and output the suggestion on a display of the document.

23. A method for augmented reality virtual personal assistance comprising, with at least one computing device:

analyzing a video depicting a real world scene of a multi-step activity;

algorithmically detecting a physical object involved in the multi-step activity;

correlating an external representation of the object detected in the video with the video depiction of the object in the real world scene, the external representation depicting the object in a different context than the real world scene; and

overlaying a portion of a real world view of the object with a virtual element indicative of the correlation between the detected object and the external representation of the object.

24. The method of claim 23, comprising annotating a view of a portion of the external representation with a different virtual element indicative of the correlation between the detected object and the external representation of the object.

25. The method of claim 23, comprising algorithmically detecting a second physical object in the video, determining a relationship between the second object and the first object, and overlaying a portion of a real world view of the first and second objects with a virtual element indicative of the relationship between the second object and the first object.

26. The method of claim 23, wherein the object is an operable device, the external representation comprises an animation of the operation of the device, and the method comprises overlaying a portion of the real world view of the device with the animation of the operation of the device.

27. The method of claim 23, comprising selecting an external representation of the object that is indicative of a part-whole relationship between two or more components of the detected object;

and overlaying the real-world view of the object with the selected portion of the external representation.

28. The method of claim 23, comprising correlating a plurality of different external representations of the detected object with the video depiction of the object in the real world scene, determining a virtual element corresponding to each of the different external representations, and dynamically changing the overlay to include different virtual elements corresponding to the different external representations.