Augmented reality speech balloon system

• US 10,614,828 B1
• Filed: 06/21/2018
• Issued: 04/07/2020
• Est. Priority Date: 02/20/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Cowburn, Piers, Pan, Qi, Pilipski, Eitan
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US16/014,193
• Time in Patent Office: 656 Days
• Field of Search: 704 2, 704235, 704258, 704260, 704270, 704275, 7042701, 345633, 381118
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Disclosed is an augmented reality system to generate and cause display of an augmented reality interface at a client device. Various embodiments may detect speech, identify a source of the speech, transcribe the speech to a text string, generate a speech bubble based on properties of the speech and that includes a presentation of the text string, and cause display of the speech bubble at a location in the augmented reality interface based on the source of the speech.

Claims

1. A system comprising:

a memory; and

at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising;

causing display at a client device, of a presentation of a space, the presentation of the space including at least a first person;

detecting, by the client device, speech that comprises speech properties, the speech properties including a volume of the speech;

identifying the first person as a source of the speech;

identifying a first language in response to the detecting the speech;

translating the speech from the first language to a second language in response to the identifying the first language;

selecting a speech bubble from a speech bubble library based on the volume of the speech, the speech bubble library comprising a plurality of speech bubbles; and

causing display of the speech bubble at a position proximate to the first person within the presentation of the space, the speech bubble containing a display of the translated text string.

2. The system of claim 1, wherein the selecting the speech bubble from the speech bubble library based on the speech properties further comprises:

transcribing the speech to a text string;

parsing the text string to a set of words;

determining a definition of each word among the set of words;

performing a comparison of the definition of each word to an emotional effect library;

determining an emotional effect of the speech based on the comparison; and

selecting the speech bubble based on the emotional effect.

3. The system of claim 2, wherein the first person has a set of facial landmarks, and wherein the determining the emotional effect of the speech further comprises:

capturing an image of the first person in response to identifying the first person as the source of the speech;

extracting the set of facial landmarks from the image of the first person; and

determining the emotional effect based on the set of facial landmarks and the comparison.

4. The system of claim 1, wherein the speech properties include at least a volume, and wherein the selecting the speech bubble from the speech bubble library includes:

determining a property of the volume of the speech based on the speech properties; and

selecting the speech bubble based on the property of the volume.

5. The system of claim 1, wherein the instructions cause the system to perform operations further comprising:

detecting a non-verbal sound;

comparing the non-verbal sound to an onomatopoeia library in response to the detecting the non-verbal sound;

identifying an onomatopoeia from the onomatopoeia library based on the non-verbal sound; and

causing display of a graphical element based on the onomatopoeia within the presentation of the space.

6. The system of claim 1, wherein the instructions cause the system to perform operations further comprising:

detecting movement of a mouth of the first person; and

identifying the first person as the source of the speech based on the movement of the mouth.

7. The system of claim 1, wherein the translating the speech from the first language to the second language includes:

accessing a user profile associated with the client device in response to the identifying the first language based on the speech, wherein the user profile includes a language preference, the language preference specifying the second language.

8. A method comprising:

causing display at a client device, of a presentation of a space, the presentation of the space including at least a first person;

detecting, by the client device, speech that comprises speech properties, the speech properties including a volume of the speech;

identifying the first person as a source of the speech;

identifying a first language in response to the detecting the speech;

translating the speech from the first language to a second language in response to the identifying the first language;

selecting a speech bubble from a speech bubble library based on the volume of the speech, the speech bubble library comprising a plurality of speech bubbles; and

causing display of the speech bubble at a position proximate to the first person within the presentation of the space, the speech bubble containing a display of the translated text string.

9. The method of claim 8, wherein the selecting the speech bubble from the speech bubble library based on the speech properties further comprises:

transcribing the speech to a text string;

parsing the text string to a set of words;

determining a definition of each word among the set of words;

performing a comparison of the definition of each word to an emotional effect library;

determining an emotional effect of the speech based on the comparison; and

selecting the speech bubble based on the emotional effect.

10. The method of claim 9, wherein the first person has a set of facial landmarks, and wherein the determining the emotional effect of the speech further comprises:

capturing an image of the first person in response to identifying the first person as the source of the speech;

extracting the set of facial landmarks from the image of the first person; and

determining the emotional effect based on the set of facial landmarks and the comparison.

11. The method of claim 8, wherein the speech properties include at least a volume, and wherein the selecting the speech bubble from the speech bubble library includes:

determining a property of the volume of the speech based on the speech properties; and

selecting the speech bubble based on the property of the volume.

12. The method of claim 8, wherein the instructions cause the system to perform operations further comprising:

detecting a non-verbal sound;

comparing the non-verbal sound to an onomatopoeia library in response to the detecting the non-verbal sound;

identifying an onomatopoeia from the onomatopoeia library based on the non-verbal sound; and

causing display of a graphical element based on the onomatopoeia within the presentation of the space.

13. The method of claim 8, wherein the method further comprises:

detecting movement of a mouth of the first person; and

identifying the first person as the source of the speech based on the movement of the mouth.

14. The method of claim 8, wherein the translating the speech from the first language to the second language includes:

accessing a user profile associated with the client device in response to the identifying the first language based on the speech, wherein the user profile includes a language preference, the language preference specifying the second language.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including:

causing display at a client device, of a presentation of a space, the presentation of the space including at least a first person;

detecting, by the client device, speech that comprises speech properties, the speech properties including a volume of the speech;

identifying the first person as a source of the speech;

identifying a first language in response to the detecting the speech;

translating the speech from the first language to a second language in response to the identifying the first language,

selecting a speech bubble from a speech bubble library based on the volume of the speech, the speech bubble library comprising a plurality of speech bubbles; and

causing display of the speech bubble at a position proximate to the first person within the presentation of the space, the speech bubble containing a display of the translated text string.

16. The non-transitory machine-readable storage medium of claim 15, wherein the selecting the speech bubble from the speech bubble library based on the speech properties includes:

transcribing the speech to a text string;

parsing the text string to a set of words;

determining a definition of each word among the set of words;

performing a comparison of the definition of each word to an emotional effect library;

determining an emotional effect of the speech based on the comparison; and

selecting the speech bubble based on the emotional effect.

17. The non-transitory machine-readable storage medium of claim 15, wherein the first person has a set of facial landmarks, and wherein the determining the emotional effect of the speech includes:

capturing an image of the first person in response to identifying the first person as the source of the speech;

extracting the set of facial landmarks from the image of the first person; and

determining the emotional effect based on the set of facial landmarks and the comparison.

18. The non-transitory machine-readable storage medium of claim 15, wherein the speech properties include at least a volume, and wherein the selecting the speech bubble from the speech bubble library includes:

determining a property of the volume of the speech based on the speech properties; and

selecting the speech bubble based on the property of the volume.

19. The non-transitory machine-readable storage medium of claim 15, wherein the instructions cause the system to perform operations further comprising:

detecting a non-verbal sound;

comparing the non-verbal sound to an onomatopoeia library in response to the detecting the non-verbal sound;

identifying an onomatopoeia from the onomatopoeia library based on the non-verbal sound; and

causing display of a graphical element based on the onomatopoeia within the presentation of the space.

20. The non-transitory machine-readable storage medium of claim 15, wherein the instructions cause the machine to perform operations further comprising:

detecting movement of a mouth of the first person; and

identifying the first person as the source of the speech based on the movement of the mouth.