METHOD AND APPARATUS FOR MENTORING VIA AN AUGMENTED REALITY ASSISTANT

• US 20140176603A1
• Filed: 12/20/2012
• Published: 06/26/2014
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: KUMAR, RAKESH, SAMARASEKERA, SUPUN, ZHU, ZHIWEI, ACHARYA, GIRISH, WOLVERTON, MICHAEL JOHN, AYAN, NECIP FAZIL, VILLAMIL, RYAN
• Granted Patent: US 10,573,037 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

A method and apparatus for training and guiding users comprising generating a scene understanding based on video and audio input of a scene of a user performing a task in the scene, correlating the scene understanding with a knowledge base to produce a task understanding, comprising one or more goals, of a current activity of the user, reasoning, based on the task understanding and a user'"'"'s current state, a next step for advancing the user towards completing one of the one or more goals of the task understanding and overlaying the scene with an augmented reality view comprising one or more visual and audio representation of the next step to the user.

Claims

1. A computer-implemented method for utilizing augmented reality to assist users comprising:

generating a scene understanding based on video and audio input of a scene comprising a user performing a task in the scene;

correlating the scene understanding with a knowledge base to produce a task understanding, comprising one or more goals, of a current activity of the user;

reasoning, based on the task understanding and a user'"'"'s current state, a next step for advancing the user towards completing one of the one or more goals of the task understanding; and

overlaying the scene with an augmented reality view comprising one or more visual and audio representations of the next step of the task to the user.

2. The method of claim 1 further comprising:

guiding a user to perform the task understanding during operation of the task understanding.

3. The method of claim 1 further comprising:

training a user to perform the task understanding as a training exercise.

4. The method of claim 1 wherein the overlaying further comprises:

aligning visual representations of the next step with the scene based on a pose of the user.

5. The method of claim 1 further comprising:

analyzing performance of the user'"'"'s actions in response to the augmented reality view, progress towards the one or more goals of the task understanding and the user'"'"'s current state; and

modifying the reasoning and the next step based on the analyzed performance.

6. The method of claim 1 further comprising:

analyzing the audio input to generate a language understanding based on natural language input in the audio; and

identifying one or more speakers in the natural language input where a bound is created for the natural language input.

7. The method of claim 6 further comprising:

distinguishing acoustic realizations of sounds in the natural language input using a statistical model to discriminate between a set of words.

8. The method of claim 7 further comprising:

determining a goal of the user in a given utterance in the natural language input; and

extracting a set of arguments associated with the goal.

9. The method of claim 1 further comprising:

estimating a gaze of a user on an object;

interpreting a natural language input by the user; and

inferring a goal of the user based on an alignment of the gaze of the user and the interpreted input by the user.

10. The method of claim 1 further comprising:

producing the task understanding based on prior scene understanding information stored in the knowledge base and a semantic frame representing a user goal up to the current time.

11. An apparatus for utilizing augmented reality in assisting users comprising:

an understanding module for generating a scene understanding based on video and audio input of a scene comprising a user performing a task in the scene;

a correlation module for correlating the scene understanding with a knowledge base to produce a task understanding, comprising one or more goals, of a current activity of the user;

a reasoning module for reasoning, based on the task understanding and a user'"'"'s current state, a next step for advancing the user towards completing one of the one or more goals of the task understanding; and

an augmented reality generator for overlaying the scene with an augmented reality view comprising one or more visual and audio representation of the next step of the task to the user,

wherein the assisting either trains users in completing a task understanding or guides users during operation of the task understanding.

12. The apparatus of claim 11 wherein the generating a scene understanding comprises observing visual cues and scene characteristics to generate the scene understanding.

13. The apparatus of claim 11 wherein the augmented reality generator is further configured for aligning visual representations of the next step with the scene based on a pose of the user.

14. The apparatus of claim 11 further comprising:

a performance module for analyzing performance of the user'"'"'s actions in response to the augmented reality view, progress towards the one or more goals of the task understanding and the user'"'"'s current state and modifying the reasoning and the next step based on the analyzed performance.

15. The apparatus of claim 11 further comprising:

a language module for analyzing the audio input to generate a language understanding based on natural language input in the audio and identifying one or more speakers in the natural language input where a bound is created for the natural language input.

16. The apparatus of claim 15, wherein the language module is further configured for distinguishing acoustic realizations of sounds in the natural language input using a statistical model to discriminate between a set of words.

17. The apparatus of claim 16 wherein the language module is further configured for determining a goal of the user in a given utterance in the natural language input and extracting a set of arguments associated with the goal.

18. The apparatus of claim 11 further comprising:

a scene module for estimating a gaze of a user on an object; and

a language module for interpreting a natural language input by the user; and

inferring a goal of the user based on an alignment of the gaze of the user and the interpreted input by the user.

19. The apparatus of claim 11 further comprising:

a task understanding module for producing the task understanding based on prior scene understanding information stored in the knowledge base and a semantic frame representing a user goal up to the current time.

20. A computer-implemented method for training users or for guiding users during operation comprising:

accessing a task understanding stored in a knowledge base, comprising one or more goals relating to a current activity of a user;

deriving a current state of the user based on video and audio input of a scene comprising the user;

reasoning, based on the task understanding and the user'"'"'s current state, a next step for advancing the user towards completing one of the one or more goals of the task understanding; and

overlaying a view of the scene for the user with an augmented reality view comprising one or more visual and audio representations of the next step of the task.