LEVERAGING HEAD MOUNTED DISPLAYS TO ENABLE PERSON-TO-PERSON INTERACTIONS

• US 20140081634A1
• Filed: 01/11/2013
• Published: 03/20/2014
• Est. Priority Date: 09/18/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Forutanpour, Babak, Baker, Daniel S.
• Granted Patent: US 9,966,075 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces

Abstract

Various arrangements for using an augmented reality device are presented. Speech spoken by a person in a real-world scene may be captured by an augmented reality (AR) device. It may be determined that a second AR device is to receive data on the speech. The second AR device may not have been present for the speech when initially spoken. Data corresponding to the speech may be transmitted to the second augmented reality device.

Claims

1. A method for using augmented reality devices, the method comprising:

capturing, by a first augmented reality device, speech spoken by a person in a real-world scene;

determining, by the first augmented reality device, a second augmented reality device to receive text corresponding to the speech; and

transmitting the text corresponding to the speech to the second augmented reality device.

2. The method for using augmented reality devices of claim 1, wherein the second augmented reality device was not present for the speech.

3. The method for using augmented reality devices of claim 1, further comprising:

displaying, by a head-mounted display of the second augmented reality device, the text corresponding to the speech by the person such that the text is graphically attributed to the person by superimposing the text on the real-world scene.

4. The method for using augmented reality devices of claim 3, wherein:

the text is graphically attributed to the person via a graphical speech bubble comprising the text that is superimposed on the real-world scene.

5. The method for using augmented reality devices of claim 3, further comprising:

determining, by the second augmented reality device, one or more words categorized as important.

6. The method for using augmented reality devices of claim 5, wherein displaying, by the head-mounted display of the second augmented reality device, text corresponding to the speech by the person comprises:

emphasizing one or more words of the text that were determined to be important.

7. The method for using augmented reality devices of claim 1, further comprising:

transcribing the speech into the text corresponding to the speech.

8. The method for using augmented reality devices of claim 1, further comprising:

transmitting, by the first augmented reality device to the second augmented reality device, an indication of an identity of the person that spoke the speech.

9. The method for using augmented reality devices of claim 1, further comprising:

receiving, by the second augmented reality device, input from a user of the second augmented reality device to access the text corresponding to the speech.

10. The method for using augmented reality devices of claim 1, further comprising:

transmitting, by the first augmented reality device, data corresponding to the speech spoken by the person in the real-world scene to a remote server, wherein;

a user of the first augmented reality device uses a first language; and

the speech is spoken in a second language;

receiving, by the first augmented reality device, the text of the speech, wherein the text is in the first language; and

displaying, by a head-mounted display of the first augmented reality device, the text corresponding to the speech by the person such that the text is graphically attributed to the person by superimposing the text on the real-world scene, wherein the text is in the first language.

11. The method for using augmented reality devices of claim 10, wherein the text corresponding to the speech by the person is displayed such that the text is superimposed over a face of the person.

12. The method for using augmented reality devices of claim 11, wherein the text corresponding to the speech is scrolled across eyes of the person.

13. An augmented reality system comprising:

a first augmented reality device, comprising;

a microphone;

a communication module;

a first processor; and

a memory communicatively coupled with and readable by the first processor and having stored therein a first set of processor-readable instructions which, when executed by the first processor, cause the first processor to;

capture, from the microphone, speech spoken by a person in a real-world scene;

determine a second augmented reality device to receive text corresponding to the speech; and

cause the text corresponding to the speech to be transmitted to the second augmented reality device via the communication module.

14. The augmented reality system of claim 13, wherein the second augmented reality device was not present for the speech.

15. The augmented reality system of claim 13, wherein

the second augmented reality device comprises:

a head-mounted display;

a second processor; and

a second memory communicatively coupled with and readable by the second processor and having stored therein a second set of processor-readable instructions which, when executed by the second processor, cause the second processor to;

cause the head-mounted display to display the text corresponding to the speech by the person such that the text is graphically attributed to the person by superimposing the text on the real-world scene.

16. The augmented reality system of claim 15, wherein:

the text is graphically attributed to the person via a graphical speech bubble comprising the text that is superimposed on the real-world scene.

17. The augmented reality system of claim 15, wherein the second set of processor-readable instructions, when executed by the second processor, are further configured to cause the second processor to:

determine one or more words categorized as important.

18. The augmented reality system of claim 17, wherein the processor-readable instructions of the second set of processor-readable instructions that cause the head-mounted display to display the text corresponding to the speech by the person such that the text is graphically attributed to the person by superimposing the text on the real-world scene comprises processor-readable instructions configured to cause the second processor to:

emphasize one or more words of the text that were determined to be important.

19. The augmented reality system of claim 13, wherein the first set of processor-readable instructions, when executed by the first processor, are further configured to cause the first processor to:

transcribe the speech into the text corresponding to the speech.

20. The augmented reality system of claim 13, wherein the first set of processor-readable instructions, when executed by the first processor, are further configured to cause the first processor to:

cause an indication of an identity of the person that spoke the speech to be transmitted to the second augmented reality device via the communication module.

21. The augmented reality system of claim 15, wherein the processor-readable instructions, when executed by the second processor, are further configured to cause the second processor to:

receive input from a user of the second augmented reality device to access the text corresponding to the speech.

22. The augmented reality system of claim 13, wherein the first set of processor-readable instructions, when executed by the first processor, are further configured to cause the first processor to:

cause data corresponding to the speech spoken by the person in the real-world scene to be transmitted via the communication module to a remote server, wherein;

a user of the first augmented reality device uses a first language; and

the speech is spoken in a second language;

receive, via the communication module, the text of the speech, wherein

the text is in the first language; and

cause, by a head-mounted display of the first augmented reality device, the text corresponding to the speech by the person to be displayed such that the text is graphically attributed to the person by superimposing the text on the real-world scene, wherein the text is in the first language.

23. The augmented reality system of claim 22, wherein the text corresponding to the speech by the person is displayed such that the text is superimposed over a face of the person.

24. The augmented reality system of claim 23, wherein the text corresponding to the speech is scrolled across eyes of the person.

25. An augmented reality apparatus comprising:

means for capturing speech spoken by a person in a real-world scene;

means for determining a second augmented reality device to receive text corresponding to the speech; and

means for transmitting the text corresponding to the speech to the second augmented reality device.

26. The augmented reality apparatus of claim 25, wherein the second augmented reality device was not present for the speech.

27. The augmented reality apparatus of claim 25, further comprising:

means for transcribing the speech into the text corresponding to the speech.

28. The augmented reality apparatus of claim 25, further comprising:

means for transmitting, to the second augmented reality device, an indication of an identity of the person that spoke the speech.

29. The augmented reality apparatus of claim 25, further comprising:

means for transmitting data corresponding to the speech spoken by the person in the real-world scene to a remote server, wherein;

a user of the first augmented reality device uses a first language; and

the speech is spoken in a second language;

means for receiving the text of the speech, wherein the text is in the first language; and

means for displaying the text corresponding to the speech by the person such that the text is graphically attributed to the person by superimposing the text on the real-world scene, wherein the text is in the first language.

30. The augmented reality apparatus of claim 29, wherein the text corresponding to the speech by the person is displayed such that the text is superimposed over a face of the person.

31. The augmented reality apparatus of claim 30, wherein the text corresponding to the speech is scrolled across eyes of the person.

32. A computer program product residing on a non-transitory processor-readable medium for augmenting reality, the computer program product comprising processor-readable instructions configured to cause a processor of a first augmented reality device to:

capture speech spoken by a person in a real-world scene;

determine a second augmented reality device to receive text corresponding to the speech; and

cause the text corresponding to the speech to be transmitted to the second augmented reality device.

33. The computer program product for augmenting reality of claim 32, wherein the second augmented reality device was not present for the speech.

34. The computer program product for augmenting reality of claim 32, wherein the processor-readable instructions further comprise processor-readable instructions configured to cause the processor to:

transcribe the speech into the text corresponding to the speech.

35. The computer program product for augmenting reality of claim 32, wherein the processor-readable instructions further comprise processor-readable instructions configured to cause the processor to:

cause an indication of an identity of the person that spoke the speech to be transmitted to the second augmented reality device.

36. The computer program product for augmenting reality of claim 32, wherein the processor-readable instructions further comprise processor-readable instructions configured to cause the processor to:

cause data corresponding to the speech spoken by the person in the real-world scene to be transmitted to a remote server, wherein;

a user of the first augmented reality device uses a first language; and

the speech is spoken in a second language;

receive the text of the speech, wherein the text is in the first language; and

cause, via a head-mounted display, the text corresponding to the speech by the person to be displayed such that the text is graphically attributed to the person by superimposing the text on the real-world scene, wherein the text is in the first language.

37. The computer program product for augmenting reality of claim 36, wherein the text corresponding to the speech by the person is displayed such that the text is superimposed over a face of the person.

38. The computer program product for augmenting reality of claim 37, wherein the text corresponding to the speech is scrolled across eyes of the person.