NATURAL LANGUAGE TRANSLATION IN AR

• US 20200134026A1
• Filed: 10/25/2018
• Published: 04/30/2020
• Est. Priority Date: 10/25/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Lovitt, Andrew, Miller, Antonio John, Robinson, Philip, Selfon, Scott
• Granted Patent: US 11,068,668 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10K 11/17823 Reference signals, e.g. ambient acoustic environment
• CPC Class Codes: G10K 11/17827 Desired external signals, e.g. pass-through audio such as music or speech
• CPC Class Codes: G10K 11/17873 using a reference signal without an error signal, e.g. pure feedforward
• CPC Class Codes: G10K 2210/3044 Phase shift, e.g. complex envelope processing
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: H04R 1/1083 Reduction of ambient noise active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06
• CPC Class Codes: H04R 2430/23 Direction finding using a sum-delay beam-former
• CPC Class Codes: H04R 2460/01 Hearing devices using active noise cancellation
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
• CPC Class Codes: H04R 2499/15 Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/304 For headphones

Abstract

The disclosed computer-implemented method for performing natural language translation in AR may include accessing an audio input stream that includes words spoken by a speaking user in a first language. The method may next include performing active noise cancellation on the words in the audio input stream so that the spoken words are suppressed before reaching a listening user. Still further, the method may include processing the audio input stream to identify the words spoken by the speaking user, and translating the identified words spoken by the speaking user into a second, different language. The method may also include generating spoken words in the second, different language using the translated words, and replaying the generated spoken words in the second language to the listening user. Various other methods, systems, and computer-readable media are also disclosed.

Claims

1. A computer-implemented method comprising:

accessing an audio input stream that includes one or more words spoken by a speaking user in a first language;

performing active noise cancellation on the one or more words in the audio input stream such that the spoken words are suppressed before reaching a listening user;

processing the audio input stream to identify the one or more words spoken by the speaking user;

translating the identified words spoken by the speaking user into a second, different language;

generating spoken words in the second, different language using the translated words; and

replaying the generated spoken words in the second language to the listening user.

2. The computer-implemented method of claim 1, wherein the generated spoken words are personalized to the speaking user, such that the generated spoken words in the second language sound as if spoken by the speaking user.

3. The computer-implemented method of claim 2, wherein personalizing the generated spoken words further comprises:

processing the audio input stream to determine how the speaking user pronounces one or more words or syllables; and

applying the determined pronunciations to the generated spoken words.

4. The computer-implemented method of claim 3, wherein during replay of the generated spoken words, personalizations are applied dynamically to the replayed words as the computer determines how the speaking user pronounces the words or syllables.

5. The computer-implemented method of claim 3, wherein the speaking user provides one or more voice samples, using which the computer determines how the speaking user pronounces the one or more of the words or syllables prior to receiving the audio input stream.

6. The computer-implemented method of claim 1, wherein replaying the generated spoken words to the listening user further comprises:

determining which direction the speaking user is speaking from; and

spatializing playback of the generated spoken words to sound as if the spoken words are coming from the determined direction of the speaking user.

7. The computer-implemented method of claim 6, wherein determining which direction the speaking user is speaking from further comprises:

receiving location data for a device associated with the speaking user;

determining which direction the speaking user is speaking from based on the received location data; and

spatializing playback of the generated spoken words to sound as if the spoken words are coming from the determined direction of the speaking user.

8. The computer-implemented method of claim 6, wherein determining which direction the speaking user is speaking from further comprises:

calculating a direction of arrival for sound waves coming from the speaking user;

determining which direction the speaking user is speaking from based on the calculated direction of arrival; and

spatializing playback of the generated spoken words to sound as if the spoken words are coming from the determined direction of the speaking user.

9. The computer-implemented method of claim 6, wherein determining which direction the speaking user is speaking from further comprises:

tracking movement of the listening user'"'"'s eyes;

determining which direction the speaking user is speaking from based on the tracked movements of the listening user'"'"'s eyes; and

spatializing playback of the generated spoken words to sound as if the spoken words are coming from the determined direction of the speaking user.

10. The computer-implemented method of claim 1, wherein processing the audio input stream to identify the one or more words spoken by the speaking user comprises implementing a speech-to-text (STT) program to identify the words spoken by the speaking user, and a text-to-speech (TTS) program to generate the translated spoken words.

11. A system comprising:

at least one physical processor;

physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to;

access an audio input stream that includes one or more words spoken by a speaking user in a first language;

perform active noise cancellation on the one or more words in the audio input stream such that the spoken words are suppressed before reaching a listening user;

process the audio input stream to identify the one or more words spoken by the speaking user;

translate the identified words spoken by the speaking user into a second, different language;

generate spoken words in the second, different language using the translated words; and

replay the generated spoken words in the second language to the listening user.

12. The system of claim 11, further comprising:

downloading a voice profile associated with the speaking user; and

using the downloaded voice profile associated with the speaking user to personalize the generated spoken words, such that the replayed generated spoken words in the second language sound as if spoken by the speaking user.

13. The system of claim 11, further comprising:

accessing one or more portions of stored audio data associated with the speaking user; and

using the accessed stored audio data to personalize the generated spoken words, such that the generated spoken words replayed in the second language sound as if spoken by the speaking user.

14. The system of claim 11, further comprising:

parsing the words spoken by the speaking user;

determining that at least one of the words is spoken in a language understood by the listening user; and

pausing active noise cancellation for the words spoken in the language understood by the listening user.

15. The system of claim 11, further comprising:

determining that the audio input stream includes words spoken by at least two different speaking users;

differentiating between the two speaking users according to one or more voice patterns; and

generating spoken words for a first speaking user, while performing active noise cancellation on both speaking users.

16. The system of claim 15, further comprising:

storing generated spoken words for the second speaking user until the first user has stopped talking for a specified amount of time; and

replaying the generated spoken words for the second speaking user.

17. The system of claim 16, further comprising personalizing the generated spoken words for each of the two speaking users, such that the generated spoken words in the second language sound as if coming from each speaking user'"'"'s voice.

18. The system of claim 11, wherein at least a portion of the computer-executable instructions stored on the physical memory are processed by at least one remote physical processor that is separate from the system.

19. The system of claim 18, wherein one or more policies indicate when and which portions of the computer-executable instructions are to be processed on the at least one remote physical processor that is separate from the system.

20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

access an audio input stream that includes one or more words spoken by a speaking user in a first language;

perform active noise cancellation on the one or more words in the audio input stream such that the spoken words are suppressed before reaching a listening user;

process the audio input stream to identify the one or more words spoken by the speaking user;

translate the identified words spoken by the speaking user into a second, different language;

generate spoken words in the second, different language using the translated words; and

replay the generated spoken words in the second language to the listening user.