Emoji recording and sending

• US 10,845,968 B2
• Filed: 03/29/2018
• Issued: 11/24/2020
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Scapel, Nicolas, Guzman, Aurelio, Bauer, Sebastian, Yerkes, Giancarlo, Barlier, Guillaume Pierre Andre
• Primary Examiner(s): Ell, Matthew
• Assistant Examiner(s): Huynh, Linda
• Application Number:
  US15/940,017
• Publication Number:
  US 20180335929A1
• Time in Patent Office: 971 Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/08 Annexed information, e.g. attachments
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 51/58 Message adaptation for wireless communication
• CPC Class Codes: H04M 1/72436 for text messaging, e.g. short messaging services [SMS] or e-mails
• CPC Class Codes: H04M 1/72439 for image or video messaging
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

The present disclosure generally relates to generating and modifying virtual avatars. An electronic device having a camera and a display apparatus displays a virtual avatar that changes appearance in response to changes in a face in a field of view of the camera. In response to detecting changes in one or more physical features of the face in the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.

Claims

1. An electronic device, comprising:

one or more cameras;

a display apparatus;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying, via the display apparatus, a graphical representation that changes appearance in response to changes in a face in a field of view of the one or more cameras, wherein the graphical representation includes;

a first portion, and

a second portion that is different from the first portion;

while displaying, via the display apparatus, the graphical representation, detecting a change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the change in pose of the face, changing an appearance of the graphical representation, including;

in accordance with a determination that the change in pose of the face includes a first type of change in pose of the face that includes a rotational movement of the face, wherein the first portion is reactive to the rotational movement of the first type of change in pose of the face and the second portion has reduced or no reactivity to the rotational movement of the first type of change in pose of the face, changing the appearance of the graphical representation includes rotating the first portion of the graphical representation relative to the second portion of the graphical representation, including;

in accordance with a determination that the rotational movement of the first type of change in pose of the face has a first magnitude of rotation, rotating the first portion of the graphical representation by a first amount of rotation relative to the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face, wherein the first amount of rotation of the first portion of the graphical representation is greater than an amount of rotation of the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face; and

in accordance with a determination that the rotational movement of the first type of change in pose of the face has a second magnitude of rotation that is greater than the first magnitude of rotation, rotating the first portion of the graphical representation by a second amount of rotation relative to the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face, wherein the second amount of rotation of the first portion of the graphical representation relative to the second portion of the graphical representation is greater than the first amount of rotation of the first portion of the graphical representation relative to the second portion of the graphical representation, and wherein the second amount of rotation of the first portion of the graphical representation is greater than the amount of rotation of the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face; and

in accordance with a determination that the change in pose of the face includes a second type of change in pose of the face, wherein the first portion is reactive to the second type of change in pose of the face and the second portion is reactive to the second type of change in pose of the face, changing the appearance of the graphical representation includes changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on a magnitude of the second type of change in pose of the face.

2. The electronic device of claim 1, wherein:

the second portion has reduced reactivity to the first type of change in pose of the face; and

the amount of movement of the second portion of the graphical representation in response to the first type of change in pose of the face is a movement with a magnitude greater than zero.

3. The electronic device of claim 1, wherein the first portion is an upper portion of the graphical representation and the second portion is a lower portion of the graphical representation.

4. The electronic device of claim 1, wherein changing the appearance of the graphical representation further includes:

in accordance with a determination that the change in pose of the face includes both the first type of change in pose of the face and the second type of change in pose of the face, changing the appearance of the graphical representation includes;

moving the first portion of the graphical representation relative to the second portion of the graphical representation; and

changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on the magnitude of the second type of change in pose of the face.

5. The electronic device of claim 1, wherein the change in pose of the face includes only the first type of change, and wherein rotating the first portion of the graphical representation relative to the second portion of the graphical representation includes rotating the first portion of the graphical representation without rotating the second portion of the graphical representation.

6. The electronic device of claim 1, wherein the change in pose of the face includes only the second type of change, and wherein changing the appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on the magnitude of the second type of change in pose of the face includes maintaining a relative position of the first portion of the graphical representation with respect to a position of the second portion of the graphical representation.

7. The electronic device of claim 1, wherein the graphical representation further includes a third portion that is different from the first portion and the second portion, further in accordance with a determination that the change in pose of the face includes the first type of change in pose of the face, moving the third portion of the graphical representation relative to the second portion of the graphical representation, wherein the movement of the third portion relative to the second portion is less than the rotation of the first portion relative to the second portion.

8. The electronic device of claim 7, wherein the third portion of the graphical representation is positioned between the first portion of the graphical representation and the second portion of the graphical representation.

9. The electronic device of claim 8, wherein moving the third portion of the graphical representation relative to the second portion of the graphical representation includes moving the third portion of the graphical representation about an axis extending between the first portion of the graphical representation and the second portion of the graphical representation.

10. The electronic device of claim 1, wherein, prior to detecting the change in pose of the face within the field of view of the one or more cameras, the first portion of the graphical representation is not visually delineated from the second portion of the graphical representation.

11. The electronic device of claim 1, wherein the first portion of the graphical representation is at least a portion of a head feature of the graphical representation and the second portion of the graphical representation is at least a portion of a neck feature of the graphical representation.

12. The electronic device of claim 1, wherein:

prior to detecting the change in pose of the face within the field of view of the one or more cameras, the face is oriented in a first orientation with respect to the field of view of the one or more cameras; and

prior to detecting the change in pose of the face within the field of view of the one or more cameras, the graphical representation is displayed in a second orientation different than the first orientation.

13. The electronic device of claim 1, wherein movement of the first portion of the graphical representation or movement of the second portion of the graphical representation occurs in accordance to one or more physics models.

14. The electronic device of claim 1, the one or more programs further including instructions for:

while displaying the graphical representation, detecting a second change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the second change in pose of the face, changing the appearance of the graphical representation, including;

in accordance with a determination that the second change in pose of the face includes a third type of change in pose of the face, changing the appearance of the graphical representation includes moving the first portion of the graphical representation relative to the second portion of the graphical representation in accordance with a magnitude of the third type of change in pose of the face; and

in accordance with a determination that the second change in pose of the face includes a fourth type of change in pose of the face, changing the appearance of the graphical representation includes changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on a magnitude of the fourth type of change in pose of the face.

15. The electronic device of claim 14, wherein the third type of change in pose of the face is the first type of change in pose of the face, and the fourth type of change in pose of the face is the second type of change in pose of the face.

16. The electronic device of claim 1, wherein the graphical representation is a virtual avatar.

17. The electronic device of claim 1, wherein the amount of movement of the second portion of the graphical representation in response to the first type of change in pose of the face is no movement.

18. The electronic device of claim 1, wherein changing the appearance of the graphical representation includes deforming at least the first portion of the graphical representation to include a graphical feature that was not displayed prior to detecting the change in pose of the face.

19. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with one or more cameras and a display apparatus, the one or more programs including instructions for:

displaying, via the display apparatus, a graphical representation that changes appearance in response to changes in a face in a field of view of the one or more cameras, wherein the graphical representation includes;

a first portion, and

a second portion that is different from the first portion;

while displaying, via the display apparatus, the graphical representation, detecting a change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the change in pose of the face, changing an appearance of the graphical representation, including;

in accordance with a determination that the change in pose of the face includes a first type of change in pose of the face that includes a rotational movement of the face, wherein the first portion is reactive to the rotational movement of the first type of change in pose of the face and the second portion has reduced or no reactivity to the rotational movement of the first type of change in pose of the face, changing the appearance of the graphical representation includes rotating the first portion of the graphical representation relative to the second portion of the graphical representation, including;

in accordance with a determination that the rotational movement of the first type of change in pose of the face has a first magnitude of rotation, rotating the first portion of the graphical representation by a first amount of rotation relative to the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face, wherein the first amount of rotation of the first portion of the graphical representation is greater than an amount of rotation of the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face; and

in accordance with a determination that the rotational movement of the first type of change in pose of the face has a second magnitude of rotation that is greater than the first magnitude of rotation, rotating the first portion of the graphical representation by a second amount of rotation relative to the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face, wherein the second amount of rotation of the first portion of the graphical representation relative to the second portion of the graphical representation is greater than the first amount of rotation of the first portion of the graphical representation relative to the second portion of the graphical representation, and wherein the second amount of rotation of the first portion of the graphical representation is greater than the amount of rotation of the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face; and

in accordance with a determination that the change in pose of the face includes a second type of change in pose of the face, wherein the first portion is reactive to the second type of change in pose of the face and the second portion is reactive to the second type of change in pose of the face, changing the appearance of the graphical representation includes changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on a magnitude of the second type of change in pose of the face.

20. The non-transitory computer-readable storage medium of claim 19, wherein:

the second portion has reduced reactivity to the first type of change in pose of the face; and

the amount of movement of the second portion of the graphical representation in response to the first type of change in pose of the face is a movement with a magnitude greater than zero.

21. The non-transitory computer-readable storage medium of claim 19, wherein the first portion is an upper portion of the graphical representation and the second portion is a lower portion of the graphical representation.

22. The non-transitory computer-readable storage medium of claim 19, wherein changing the appearance of the graphical representation further includes:

in accordance with a determination that the change in pose of the face includes both the first type of change in pose of the face and the second type of change in pose of the face, changing the appearance of the graphical representation includes;

moving the first portion of the graphical representation relative to the second portion of the graphical representation; and

changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on the magnitude of the second type of change in pose of the face.

23. The non-transitory computer-readable storage medium of claim 19, wherein the change in pose of the face includes only the first type of change, and wherein rotating the first portion of the graphical representation relative to the second portion of the graphical representation includes rotating the first portion of the graphical representation without rotating the second portion of the graphical representation.

24. The non-transitory computer-readable storage medium of claim 19, wherein the change in pose of the face includes only the second type of change, and wherein changing the appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the first portion of the graphical representation and the second portion of the graphical representation based on the magnitude of the second type of change in pose of the face includes maintaining a relative position of the first portion of the graphical representation with respect to a position of the second portion of the graphical representation.

25. The non-transitory computer-readable storage medium of claim 19, wherein the graphical representation further includes a third portion that is different from the first portion and the second portion, further in accordance with a determination that the change in pose of the face includes the first type of change in pose of the face, moving the third portion of the graphical representation relative to the second portion of the graphical representation, wherein the movement of the third portion relative to the second portion is less than the rotation of the first portion relative to the second portion.

26. The non-transitory computer-readable storage medium of claim 25, wherein the third portion of the graphical representation is positioned between the first portion of the graphical representation and the second portion of the graphical representation.

27. The non-transitory computer-readable storage medium of claim 26, wherein moving the third portion of the graphical representation relative to the second portion of the graphical representation includes moving the third portion of the graphical representation about an axis extending between the first portion of the graphical representation and the second portion of the graphical representation.

28. The non-transitory computer-readable storage medium of claim 19, wherein, prior to detecting the change in pose of the face within the field of view of the one or more cameras, the first portion of the graphical representation is not visually delineated from the second portion of the graphical representation.

29. The non-transitory computer-readable storage medium of claim 19, wherein the first portion of the graphical representation is at least a portion of a head feature of the graphical representation and the second portion of the graphical representation is at least a portion of a neck feature of the graphical representation.

30. The non-transitory computer-readable storage medium of claim 19, wherein:

prior to detecting the change in pose of the face within the field of view of the one or more cameras, the face is oriented in a first orientation with respect to the field of view of the one or more cameras; and

prior to detecting the change in pose of the face within the field of view of the one or more cameras, the graphical representation is displayed in a second orientation different than the first orientation.

31. The non-transitory computer-readable storage medium of claim 19, wherein movement of the first portion of the graphical representation or movement of the second portion of the graphical representation occurs in accordance to one or more physics models.

32. The non-transitory computer-readable storage medium of claim 19, the one or more programs further including instructions for:

while displaying the graphical representation, detecting a second change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the second change in pose of the face, changing the appearance of the graphical representation, including;

in accordance with a determination that the second change in pose of the face includes a third type of change in pose of the face, changing the appearance of the graphical representation includes moving the first portion of the graphical representation relative to the second portion of the graphical representation in accordance with a magnitude of the third type of change in pose of the face; and

in accordance with a determination that the second change in pose of the face includes a fourth type of change in pose of the face, changing the appearance of the graphical representation includes changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on a magnitude of the fourth type of change in pose of the face.

33. The non-transitory computer-readable storage medium of claim 32, wherein the third type of change in pose of the face is the first type of change in pose of the face, and the fourth type of change in pose of the face is the second type of change in pose of the face.

34. The non-transitory computer-readable storage medium of claim 19, wherein the graphical representation is a virtual avatar.

35. The non-transitory computer-readable storage medium of claim 19, wherein the amount of movement of the second portion of the graphical representation in response to the first type of change in pose of the face is no movement.

36. The non-transitory computer-readable storage medium of claim 19, wherein changing the appearance of the graphical representation includes deforming at least the first portion of the graphical representation to include a graphical feature that was not displayed prior to detecting the change in pose of the face.

44. The method of claim 26, wherein the third portion of the graphical representation is positioned between the first portion of the graphical representation and the second portion of the graphical representation.

45. The method of claim 27, wherein moving the third portion of the graphical representation relative to the second portion of the graphical representation includes moving the third portion of the graphical representation about an axis extending between the first portion of the graphical representation and the second portion of the graphical representation.

37. A method, comprising:

at an electronic device having one or more cameras and a display apparatus;

displaying, via the display apparatus, a graphical representation that changes appearance in response to changes in a face in a field of view of the one or more cameras, wherein the graphical representation includes;

a first portion, and

a second portion that is different from the first portion;

while displaying, via the display apparatus, the graphical representation, detecting a change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the change in pose of the face, changing an appearance of the graphical representation, including;

in accordance with a determination that the change in pose of the face includes a first type of change in pose of the face that includes a rotational movement of the face, wherein the first portion is reactive to the rotational movement of the first type of change in pose of the face and the second portion has reduced or no reactivity to the rotational movement of the first type of change in pose of the face, changing the appearance of the graphical representation includes rotating the first portion of the graphical representation relative to the second portion of the graphical representation, including;

in accordance with a determination that the rotational movement of the first type of change in pose of the face has a first magnitude of rotation, rotating the first portion of the graphical representation by a first amount of rotation relative to the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face, wherein the first amount of rotation of the first portion of the graphical representation is greater than an amount of rotation of the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face; and

in accordance with a determination that the rotational movement of the first type of change in pose of the face has a second magnitude of rotation that is greater than the first magnitude of rotation, rotating the first portion of the graphical representation by a second amount of rotation relative to the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face, wherein the second amount of rotation of the first portion of the graphical representation relative to the second portion of the graphical representation is greater than the first amount of rotation of the first portion of the graphical representation relative to the second portion of the graphical representation, and wherein the second amount of rotation of the first portion of the graphical representation is greater than the amount of rotation of the second portion of the graphical representation in response to the rotational movement of the first type of change in pose of the face; and

in accordance with a determination that the change in pose of the face includes a second type of change in pose of the face, wherein the first portion is reactive to the second type of change in pose of the face and the second portion is reactive to the second type of change in pose of the face, changing the appearance of the graphical representation includes changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on a magnitude of the second type of change in pose of the face.

38. The method of claim 37, wherein:

the second portion has reduced reactivity to the first type of change in pose of the face; and

the amount of movement of the second portion of the graphical representation in response to the first type of change in pose of the face is a movement with a magnitude greater than zero.

39. The method of claim 37, wherein the first portion is an upper portion of the graphical representation and the second portion is a lower portion of the graphical representation.

40. The method of claim 37, wherein changing the appearance of the graphical representation further includes:

in accordance with a determination that the change in pose of the face includes both the first type of change in pose of the face and the second type of change in pose of the face, changing the appearance of the graphical representation includes;

moving the first portion of the graphical representation relative to the second portion of the graphical representation; and

changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on the magnitude of the second type of change in pose of the face.

41. The method of claim 37, wherein the change in pose of the face includes only the first type of change, and wherein rotating the first portion of the graphical representation relative to the second portion of the graphical representation includes rotating the first portion of the graphical representation without rotating the second portion of the graphical representation.

42. The method of claim 37, wherein the change in pose of the face includes only the second type of change, and wherein changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on the magnitude of the second type of change in pose of the face includes maintaining a relative position of the first portion of the graphical representation with respect to a position of the second portion of the graphical representation.

43. The method of claim 37, wherein the graphical representation further includes a third portion that is different from the first portion and the second portion, further in accordance with a determination that the change in pose of the face includes the first type of change in pose of the face, moving the third portion of the graphical representation relative to the second portion of the graphical representation, wherein the movement of the third portion relative to the second portion is less than the rotation of the first portion relative to the second portion.

46. The method of claim 37, wherein, prior to detecting the change in pose of the face within the field of view of the one or more cameras, the first portion of the graphical representation is not visually delineated from the second portion of the graphical representation.

47. The method of claim 37, wherein the first portion of the graphical representation is at least a portion of a head feature of the graphical representation and the second portion of the graphical representation is at least a portion of a neck feature of the graphical representation.

48. The method of claim 37, wherein:

prior to detecting the change in pose of the face within the field of view of the one or more cameras, the face is oriented in a first orientation with respect to the field of view of the one or more cameras; and

prior to detecting the change in pose of the face within the field of view of the one or more cameras, the graphical representation is displayed in a second orientation different than the first orientation.

49. The method of claim 37, wherein movement of the first portion of the graphical representation or movement of the second portion of the graphical representation occurs in accordance to one or more physics models.

50. The method of claim 37, the one or more programs further including instructions for:

while displaying the graphical representation, detecting a second change in pose of the face within the field of view of the one or more cameras; and

in response to detecting the second change in pose of the face, changing the appearance of the graphical representation, including;

in accordance with a determination that the second change in pose of the face includes a third type of change in pose of the face, changing the appearance of the graphical representation includes moving the first portion of the graphical representation relative to the second portion of the graphical representation in accordance with a magnitude of the third type of change in pose of the face; and

in accordance with a determination that the second change in pose of the face includes a fourth type of change in pose of the face, changing the appearance of the graphical representation includes changing an appearance of the first portion of the graphical representation proportional to a corresponding change in appearance of the second portion of the graphical representation based on a magnitude of the fourth type of change in pose of the face.

51. The method of claim 50, wherein the third type of change in pose of the face is the first type of change in pose of the face, and the fourth type of change in pose of the face is the second type of change in pose of the face.

52. The method of claim 37, wherein the graphical representation is a virtual avatar.

53. The method of claim 37, wherein the amount of movement of the second portion of the graphical representation in response to the first type of change in pose of the face is no movement.

54. The method of claim 37, wherein changing the appearance of the graphical representation includes deforming at least the first portion of the graphical representation to include a graphical feature that was not displayed prior to detecting the change in pose of the face.