Augmented expression system

• US 10,719,968 B2
• Filed: 04/17/2019
• Issued: 07/21/2020
• Est. Priority Date: 04/18/2018
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Cao, Chen, Gao, Yang, Xue, Zehao
• Primary Examiner(s): Cobb, Michael J
• Application Number:
  US16/387,092
• Publication Number:
  US 20190325631A1
• Time in Patent Office: 461 Days
• Field of Search: None
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/021 Flattening
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Embodiments described herein relate to an augmented expression system to generate and cause display of a specially configured interface to present an augmented reality perspective. The augmented expression system receives image and video data of a user and tracks facial landmarks of the user based on the image and video data, in real-time to generate and present a 3-dimensional (3D) bitmoji of the user.

Claims

1. A method comprising:

receiving a user input that assigns a graphical element to an expression definition at a client device;

causing display of a presentation of image data at a client device, the image data depicting a set of facial landmarks of a user;

generating a 3D model based on the set of facial landmarks in response to the causing display of the presentation of the image data;

overlaying the 3D model at a position upon on the presentation of the image data at the client device, the position based on the set of facial landmarks;

receiving a facial tracking input that comprises a movement of a facial landmark from among the set of facial landmarks of the user;

detecting the expression based on the movement of the facial landmark from among the set of facial landmarks;

applying the graphical element to a position upon the 3D model at the client device in response to the detecting the expression based on the movement of the facial landmark;

generating a flattened presentation of the 3D model at the client device based on the animation of the 3D model; and

generating a message that comprises a data packet, the data packet including the flattened presentation of the 3D model.

2. The method of claim 1, wherein the method further comprises:

identifying a user profile of the user based on the set of facial landmarks and an identifier of the client device; and

generating the 3D model based on the user profile and the set of facial landmarks.

3. The method of claim 1, wherein the method further comprises:

identifying an expression based on the facial tracking input;

selecting a graphical element from among a set of graphical elements based on the expression; and

causing display of the graphical element within the presentation of the image data at the client device.

4. The method of claim 3, wherein the facial tracking input is a first facial tracking input, and the method further comprises:

receiving a second facial tracking input, the second facial tracking input comprising a second movement of a facial landmark from among the set of facial landmarks; and

applying a change to the graphical element based on the second facial tracking input.

5. The method of claim 1, wherein the method further comprises:

detecting a rotation of a head of the user based on at least the facial tracking input; and

rotating the 3D model based on the rotation of the head of the user.

6. The method of claim 1, wherein the client device is a first client device, and the method further comprises:

distributing the data packet that includes the flattened presentation of the 3D model to a second client device; and

causing display of the flattened presentation at the second client device.

7. The method f claim 6, wherein the message includes an ephemeral message.

8. The method of claim 1, wherein the 3D model comprises a set of regions, each region of the 3D model corresponding to a facial landmark from among the set of facial landmarks of the user, and the method further comprises:

identifying a region of the 3D model based on the facial tracking input; and

dynamically animating the region in response to the receiving the facial tracking input.

9. A system comprising:

a memory; and

at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising;

receiving a user input that assigns a graphical element to an expression definition at a client device;

causing display of a presentation of image data at a client device, the image data depicting a set of facial landmarks of a user;

generating a 3D model based on the set of facial landmarks in response to the causing display of the presentation of the image data;

overlaying the 3D model at a position upon on the presentation of the image data at the client device, the position based on the set of facial landmarks;

receiving a facial tracking input that comprises a movement of a facial landmark from among the set of facial landmarks of the user;

detecting the expression based on the movement of the facial landmark from among the set of facial landmarks;

applying the graphical element to a position upon animating the 3D model at the client device in response to the detecting the expression based on the movement of the facial landmark;

generating a flattened presentation of the 3D model at the client device based on the animation of the 3D model; and

generating a message that comprises a data packet, the data packet including the flattened presentation of the 3D model.

10. The system of claim 9, wherein the instructions cause the system to perform operations further comprising:

identifying a user profile of the user based on the set of facial landmarks and an identifier of the client device; and

generating the 3D model based on the user profile and the set of facial landmarks.

11. The system of claim 9, wherein the instructions cause the system to perform operations further comprising:

identifying an expression based on the facial tracking input;

selecting a graphical element from among a set of graphical elements based on the expression; and

causing display of the graphical element within the presentation of the image data at the client device.

12. The system of claim 11, wherein the facial tracking input is a first facial tracking input, and the instructions cause the system to perform operations further comprising:

receiving a second facial tracking input, the second facial tracking input comprising a second movement of a facial landmark from among the set of facial landmarks; and

applying a change to the graphical element based on the second facial tracking input.

13. The system of claim 9, wherein the instructions cause the system to perform operations further comprising:

detecting a rotation of a head of the user based on at least the facial tracking input; and

rotating the 3D model based on the rotation of the head of the user.

14. The system of claim 9, wherein the client device is a first client device, and the instructions cause the system to perform operations further comprising:

distributing the data packet that includes the flattened presentation of the 3D model to a second client device; and

causing display of the flattened presentation at the second client device.

15. The system of claim 14, wherein the message includes an ephemeral message.

16. The system of claim 9, wherein the 3D model comprises a set of regions, each region of the 3D model corresponding to a facial landmark from among the set of facial landmarks of the user, and the instructions cause the system to perform operations further comprising:

identifying a region of the 3D model based on the facial tracking input; and

dynamically animating the region in response to the receiving the facial tracking input.

17. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving a user input that assigns a graphical element to an expression definition at a client device;

causing display of a presentation of image data at a client device, the image data depicting a set of facial landmarks of a user;

generating a 3D model based on the set of facial landmarks in response to the causing display of the presentation of the image data;

overlaying the 3D model at a position upon on the presentation e image data at the client device, the position based on the set of facial landmarks;

receiving a facial tracking input that comprises a movement of a facial landmark from among the set of facial landmarks of the user;

detecting the expression based on the movement of the facial landmark from anion the set of facial landmarks;

applying the graphical element to a position upon the 3D model at the client device in response to the detecting the expression based on the movement of the facial landmark;

generating a flattened presentation of the 3D model at the client device based on the animation of the 3D model; and

generating a message that comprises a data packet, the data packet including the flattened presentation of the 3D model.

18. The non-transitory machine-readable storage medium of claim 17, further comprising instructions that cause the machine to perform operations further comprising:

identifying a user profile of the user based on the set of facial landmarks and an identifier of the client device; and

generating the 3D model based on the user profile and the set of facial landmarks.

19. The non-transitory machine-readable storage medium of claim 17, further comprising instructions that cause the machine to perform operations further comprising:

identifying an expression based on the facial tracking input;

selecting a graphical element from among a set of graphical elements based on the expression; and

causing display of the graphical element within the presentation of the image data at the client device.

20. The non-transitory machine-readable storage medium of claim 19, further comprising instructions that cause the machine to perform operations further comprising:

receiving a second facial tracking input, the second facial tracking input comprising a second movement of a facial landmark from among the set of facial landmarks; and

applying a change to the graphical element based on the second facial tracking input.