Word flow annotation

• US 10,607,609 B2
• Filed: 08/10/2017
• Issued: 03/31/2020
• Est. Priority Date: 08/12/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Sommers, Jeffrey, Devine, Jennifer M. R., Seuck, Joseph Wayne, Kaehler, Adrian
• Primary Examiner(s): Shah, Paras D
• Application Number:
  US15/673,894
• Publication Number:
  US 20180047395A1
• Time in Patent Office: 964 Days
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 25/84 for discriminating voice from noise
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

An augmented reality (AR) device can be configured to monitor ambient audio data. The AR device can detect speech in the ambient audio data, convert the detected speech into text, or detect keywords such as rare words in the speech. When a rare word is detected, the AR device can retrieve auxiliary information (e.g., a definition) related to the rare word from a public or private source. The AR device can display the auxiliary information for a user to help the user better understand the speech. The AR device may perform translation of foreign speech, may display text (or the translation) of a speaker'"'"'s speech to the user, or display statistical or other information associated with the speech.

Claims

1. An augmented reality (AR) system comprising:

an AR display configured to present virtual content to a user of the AR system;

an audio sensor configured to capture ambient sounds; and

a hardware processor in communication with the AR display and the audio sensor, the hardware processor programmed to;

receive the ambient sounds captured by the audio sensor;

detect presence of a speech in the ambient sounds;

convert the detected speech to text;

parse the text to recognize one or more words;

determine a position of the one or more words on a word frequency list, the word frequency list comprising a list of words ranked based at least in part on their frequency of detection by the audio sensor;

determine a threshold position on the word frequency list based on a user input;

categorize the one or more words as a rare word based on whether the position of the one or more words passes the threshold position on the word frequency list;

determine a decay factor associated with the rare word, the decay factor comprising a time period in which to suppress display of auxiliary information associated with the rare word;

retrieve the auxiliary information associated with the rare word based on the decay factor; and

cause the AR display to render the retrieved auxiliary information.

2. The AR system of claim 1, wherein the rare word is a word excluded from a common word dictionary.

3. The AR system of claim 1, wherein the hardware processor is further programmed to:

detect a condition for dismissing display of the auxiliary information; and

cause the AR display to dismiss the display of the auxiliary information in response to the detected condition.

4. The AR system of claim 3, wherein to detect the condition for dismissing the display, the hardware processor is programmed to perform at least one of:

determining a threshold period of time has elapsed, detecting another rare word, detecting a pose of a user, or receiving an input from a user input device.

5. The AR system of claim 1, wherein the hardware processor is further programmed to:

determine a source of the detected speech; and

detect the rare word and retrieve and display the auxiliary information upon determining the source of the detected speech is associated with a speaker other than a user of the AR display.

6. The AR system of claim 5, wherein the source of the detected speech is determined based on at least one of audio information collected from the audio sensor comprising one or more directional microphones or images acquired by an outward-facing imaging system of the AR system.

7. The AR system of claim 5, wherein to determine the source of the detected speech, the hardware processor is programmed to perform voice recognition on the detected speech to determine an identity of the speaker.

8. The AR system of claim 5, wherein the source of the detected speech comprises at least one of:

another computing in the user'"'"'s environment, another AR device associated with the speaker, or a person in the user'"'"'s environment.

9. The AR system of claim 8, where the user is in a telepresence session with the speaker and the source of the detected speech is the other user device associated with the speaker, the hardware processor is further programmed to cause the AR display to render a virtual avatar of the speaker and to render the auxiliary information near the virtual avatar of the other user.

10. The AR system of claim 5, to cause the AR display to render the retrieved auxiliary information, the hardware processor is programmed to determine a location of the source;

and render the auxiliary information in a position in the 3D space that is close to the source without obscuring a view of the source through the AR display.

11. The AR system of claim 1, wherein auxiliary information comprises an explanation of the rare word.

12. The AR system of claim 1, wherein the hardware processor is further programmed to:

receive an indication from a user to annotate the auxiliary information;

initiate storage of an annotation of the auxiliary information; and

cause the AR display to render the annotation and the auxiliary information in response to detecting another presence of the rare word.

13. The AR system of claim 1, wherein the word frequency list comprises a list of words ordered by frequency that the user encountered in the user'"'"'s past experience.

14. A method comprising:

under control of an augmented reality (AR) device comprising a hardware processor and an AR display configured to present virtual content in an environment of a user;

monitoring the environment of the user of the AR device;

detecting presence of an object of interest in the environment based on contextual information associated with at least one of the user or the environment;

determining a position of the object of interest on a word frequency list, the word frequency list comprising a list of words ranked based at least in part on their frequency of detection by the AR device;

determining a threshold position on the word frequency list based on a user input;

categorizing the object of interest as a rare word based on whether the position of the one or more words passes the threshold position on the word frequency list;

determining a decay factor associated with the object of interest, the decay factor comprising a time period in which to suppress display of auxiliary information associated with the object of interest;

retrieving auxiliary information for the object of interest based on the decay factor;

determining a display position of the auxiliary information in a field of view of the user, wherein the field of view comprises a portion of the environment which a user can perceive at a given time; and

causing the AR display to render the auxiliary information at the display position.

15. The method of claim 14, where detecting the presence of the object of interest comprises:

detecting a speech from ambient sounds in the environment; and

parsing the speech to identify a rare word.

16. The method of claim 14, wherein monitoring the environment comprises at least one of:

capturing and analyzing ambient sounds of the environment;

or

acquiring images of the environment.

17. The method of claim 14, wherein the contextual information is associated with the user'"'"'s past experience and the word frequency list comprises a list of words ordered by frequency that the user encountered in the user'"'"'s past experience.

18. The method of claim 14, wherein the auxiliary information comprises an explanatory text of the object of interest.

19. The method of claim 14, further comprising:

receiving an indication from the user to annotate the auxiliary information;

initiating storage of an annotation of the auxiliary information; and

causing the AR display to render the annotation and the auxiliary information in response to detecting another presence of the object of interest.

20. The method of claim 14, wherein determining a display position of the auxiliary information comprises:

determining a location of the object of interest; and

identifying the display position as a position in the environment that is close to the object of interest without obscuring a view of the object of interest through the AR display.