System, method, and computer program for transmitting face models based on face data points

• US 10,708,545 B2
• Filed: 11/30/2018
• Issued: 07/07/2020
• Est. Priority Date: 01/17/2018
• Status: Active Grant

• Current Assignee: Duelight LLC
• Original Assignee: Duelight LLC
• Inventors: Rivard, William Guie, Kindle, Brian J., Feder, Adam Barry
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US16/206,241
• Publication Number:
  US 20190222807A1
• Time in Patent Office: 585 Days
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A system, method, and computer program are provided for transmitting face models based on face data points. In use, a first image is received and at least one face associated with the first image is identified. Next, a face model is created of the at least one face by determining a structure of the at least one face, wherein the face model includes one or more face data points. The face model is transmitted. Additionally, a real-time stream is enabled of the at least one face, and a real-time face model is determined of the real-time stream using the face model. The real-time face model is then transmitted.

Claims

1. A device, comprising:

a non-transitory memory storing instructions; and

one or more processors in communication with the non-transitory memory, wherein the one or more processors execute the instructions to;

receive a first image;

identify at least one face associated with the first image;

create a face model of the at least one face by determining a structure of the at least one face, wherein the face model includes one or more face data points;

transmit the face model;

provide a real-time stream of the at least one face;

determine a real-time face model of the real-time stream using the face model;

transmit the real-time face model, wherein the real-time face model is automatically transmitted based on a network latency threshold condition or a dropped packet threshold condition.

2. The device of claim 1, wherein the structure is a geometric mesh or collection of geometric objects.

3. The device of claim 1, wherein the real-time stream is a video chat.

4. The device of claim 1, wherein the device is further operable to execute the instructions to modify the face model based on the real-time face model.

5. The device of claim 1, wherein the face model and real-time face model are transmitted to at least a second device.

6. The device of claim 1, wherein the face model includes at least two of facial contours, minute data points, and main data points.

7. The device of claim 1, wherein the face model includes a plurality of face parameters.

8. The device of claim 7, wherein the plurality of face parameters are animation parameters associated with a particular movement and feature of the at least one face.

9. The device of claim 8, wherein the animation parameters include at least two of a movement of eyes, a movement of a nose, a movement of a jaw, a movement of a face bone structure, a movement of eyebrows, and a movement of lips.

10. The device of claim 7, wherein the device is operable such that the plurality of face parameters is correlated and occurs synchronously.

11. The device of claim 1, wherein the device includes audio input.

12. The device of claim 11, wherein the device is operable such that the audio input is used to create an inference of how the face model is to be animated according to face parameters.

13. The device of claim 1, wherein the face model includes an image depth map, an audio map, and a correlation map.

14. The device of claim 13, wherein the correlation map matches up audio associated with phonetics, intonations, and/or emotions, with one or more face data points.

15. The device of claim 1, wherein the device is operable such that transmitting the real-time face model requires less data usage than transmitting real time video frames associated with the at least one face.

16. The device of claim 1, wherein the device is operable such that the face model is further refined using historical data or additional data associated with the at least one face.

17. The device of claim 1, wherein the device is further operable to receive a selection of a second face model associated with a third party individual or character, transmit the second face model, and use the real-time face model to modify the second face model.

18. A method, comprising:

receiving, using a processor, a first image;

identifying, using the processor, at least one face associated with the first image;

creating, using the processor, a face model of the at least one face by determining a structure of the at least one face, wherein the face model includes one or more face data points;

transmitting, using the processor, the face model;

providing, using the processor, a real-time stream of the at least one face;

determining, using the processor, a real-time face model of the real-time stream using the face model;

transmitting, using the processor, the real-time face model, wherein the real-time face model is automatically transmitted based on a network latency threshold condition or a dropped packet threshold condition.

19. A computer program product comprising computer executable instructions stored on a non-transitory computer readable medium that when executed by a processor instruct the processor to:

receive a first image;

identify at least one face associated with the first image;

create a face model of the at least one face by determining a structure of the at least one face, wherein the face model includes one or more face data points;

transmit the face model;

provide a real-time stream of the at least one face;

determine a real-time face model of the real-time stream using the face model;

transmit the real-time face model, wherein the real-time face model is automatically transmitted based on a network latency threshold condition or a dropped packet threshold condition.