Multi-participant live communication user interface

• US 10,362,272 B1
• Filed: 08/22/2018
• Issued: 07/23/2019
• Est. Priority Date: 05/07/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Van Os, Marcel, Broughton, Lee S., Callaway, Peter, Caro, Pablo F., Chang, Jae Woo, Dryer, Allison, Federighi, Craig M., Garcia, Robert, Lemay, Stephen O., Louch, John, Pham, Hoan
• Primary Examiner(s): Zenati, Amal S
• Application Number:
  US16/109,552
• Time in Patent Office: 335 Days
• Field of Search: 348 1403
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/043 using or handling presence information
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/224 providing notification on incoming messages, e.g. pushed notifications of received messages
• CPC Class Codes: H04M 1/72439 for image or video messaging
• CPC Class Codes: H04N 2007/145 Handheld terminals
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42224 Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

An electronic device includes a display, one or more processors, and memory storing one or more programs configured to be executed by the one or more processors. The one or more programs may include instructions for displaying a live video communication user interface that includes concurrently displaying representations of a first, second, and third participant in the live communication session. While displaying the live video communication user interface, the one or more programs may also include instructions for include receiving data indicating that an activity level of the second participant has increased above the activity level of the first participant. Further, in response to receiving the data, the one or more programs may further include instructions for changing a size and/or position of the representations of the first, second, and third participants.

Claims

1. An electronic device, comprising:

a display;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying a live video communication user interface that includes concurrently displaying a representation of a first participant in a live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session;

while displaying the live video communication user interface, receiving data indicating that an activity level of the second participant in the live video communication session has increased above an activity level of the first participant in the live video communication session, wherein the activity level of at least the first participant is based on at least one of audio from an audio feed received from a device of the first participant or movement identified in a video feed received from the device of the first participant that meets a set of movement criteria; and

in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;

changing a size and position of the representation of the first participant in the live video communication user interface;

changing a size and position of the representation of the second participant in the live video communication user interface; and

changing a size and position of the representation of the third participant in the live video communication user interface, wherein;

after changing the size and position of the representation of the first participant in the live communication user interface, the changed representation of the first participant occupies at least a portion of a region that was occupied by the representation of the first participant in the live communication user interface before changing the size and the position of the representation of the first participant,

after changing the size and position of the representation of the second participant in the live communication user interface, the changed representation of the second participant occupies at least a portion of a region that was occupied by the representation of the second participant in the live communication user interface before changing the size and the position of the representation of the second participant, and

after changing the size and position of the representation of the third participant in the live communication user interface, the changed representation of the third participant occupies at least a portion of a region that was occupied by the representation of the third participant in the live communication user interface before changing the size and the position of the representation of the third participant.

2. The electronic device of claim 1, the one or more programs further including instructions for:

after changing the size and position of the representation of the first participant in the live video communication user interface, receiving second data indicating that an activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session; and

in response to receiving the second data indicating that an activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session;

changing the size and/or position of the representation of the first participant in the live video communication user interface;

changing the size and/or position of the representation of the second participant in the live video communication user interface; and

maintaining the size and position of the representation of the third participant in the live communication session.

3. The electronic device of claim 1, the one or more programs further including instructions for:

further in response to receiving the first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session;

emphasizing a visual indication of the representation of the second participant in the live communication user interface; and

deemphasizing a visual indication of the representation of the first participant in the live communication user interface.

4. The electronic device of claim 1, wherein:

prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;

the representation of the first participant is displayed at a first size, and

the representation of the second participant is displayed at a second size smaller than the first size; and

changing the size and position of the representation of the second participant includes increasing the size of the representation of the second participant from the second size to the first size.

5. The electronic device of claim 1, wherein changing the size and position of the representation of the first participant includes:

changing the size and/or position of the representation of the first participant based on facial recognition data obtained from a video feed of the second participant, wherein changing the size and/or position of the representation of the first participant does not result in obscuring the face of the second participant.

6. The electronic device of claim 1, wherein, prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, one or more of the representations of the first participant, the second participant, and the third participant overlaps with at least one other representation of a participant.

7. The electronic device of claim 6, wherein:

prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, the representations of the first, second, and third participants in the live communication user interface are displayed according to a z-order; and

changing the sizes and positions of the representations of the first, second, and third participants in the live communication user interface includes maintaining the z-order of the representations of the first, second, and third participants.

8. The electronic device of claim 1, wherein a portion of the representation of a participant having the highest activity level of the first, second, and third participants is overlapped by a portion of the representation of another participant of the first, second, and third participants in the live communication user interface.

9. The electronic device of claim 1, wherein a portion of the representation of a participant having the lowest activity level of the first, second, and third participants overlaps a portion of the representation of another participant of the first, second, and third participants in the live communication user interface.

10. The electronic device of claim 1, the one or more programs further including instructions for:

after receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, detecting an input corresponding to the representation of the first participant; and

in response to detecting the input corresponding to the representation of the first participant, increasing the activity level of the first participant.

11. The electronic device of claim 1, the one or more programs further including instructions for:

receiving data indicating that a fourth participant in the live communication session is more active than a participant having the lowest activity level of the first, second, and third participants; and

in response to receiving the data indicating that the fourth participant in the live communication session is more active than a participant having the lowest activity level of the first, second, and third participants, replacing the display of the representation of the participant having the lowest activity level of the first, second, and third participants with display of a representation of the fourth participant.

12. The electronic device of claim 11, wherein replacing the display of the representation of the participant having the lowest activity level of the first, second, and third participants with display of a representation of the fourth participant includes:

displaying the representation of the fourth participant at a size larger than the size of the representation of the participant having the lowest activity level of the first, second, and third participants.

13. The electronic device of claim 1, wherein changing the size and position of the representation of the first participant includes gradually increasing the size of the representation from a first size to a second size.

14. The electronic device of claim 1, wherein the live communication interface includes a canvas region, and wherein the representation of the first participant in the live communication session, the representation of the second participant in the live communication session, and the representation of the third participant in the live communication session are displayed in the canvas region.

15. The electronic device of claim 14, the one or more programs further including instructions for:

detecting a canvas size change event; and

in response to detecting the canvas size change event;

changing a size of the canvas region; and

modifying a layout of the representations of the first, second, and third participants.

16. The electronic device of claim 15, the one or more programs further including instructions for:

further in response to detecting the canvas size change event, changing the size of one or more of the representations of the first, second, and third participants.

17. The electronic device of claim 1, the one or more programs further including instructions for:

displaying a representation of a view of a camera of the electronic device in the live communication session user interface; and

further in response to receiving the first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session;

maintaining a size of the representation of a view of the camera of the electronic device in the live communication session user interface.

18. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs including instructions for:

displaying a live video communication user interface that includes concurrently displaying a representation of a first participant in a live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session;

while displaying the live video communication user interface, receiving data indicating that an activity level of the second participant in the live video communication session has increased above an activity level of the first participant in the live video communication session, wherein the activity level of at least the first participant is based on at least one of audio from an audio feed received from a device of the first participant or movement identified in a video feed received from the device of the first participant that meets a set of movement criteria; and

in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;

changing a size and position of the representation of the first participant in the live video communication user interface;

changing a size and position of the representation of the second participant in the live video communication user interface; and

changing a size and position of the representation of the third participant in the live video communication user interface, wherein;

after changing the size and position of the representation of the first participant in the live communication user interface, the changed representation of the first participant occupies at least a portion of a region that was occupied by the representation of the first participant in the live communication user interface before changing the size and the position of the representation of the first participant,

after changing the size and position of the representation of the second participant in the live communication user interface, the changed representation of the second participant occupies at least a portion of a region that was occupied by the representation of the second participant in the live communication user before changing the size and the position of the representation of the second participant, and

after changing the size and position of the representation of the third participant in the live communication user interface, the changed representation of the third participant occupies at least a portion of a region that was occupied by the representation of the third participant in the live communication user interface before changing the size and the position of the representation of the third participant.

19. The non-transitory computer-readable storage medium of claim 18, the one or more programs further including instructions for:

after changing the size and position of the representation of the first participant in the live video communication user interface, receiving second data indicating that an activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session; and

in response to receiving the second data indicating that an activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session;

changing the size and/or position of the representation of the first participant in the live video communication user interface;

changing the size and/or position of the representation of the second participant in the live video communication user interface; and

maintaining the size and position of the representation of the third participant in the live communication session.

20. The non-transitory computer-readable storage medium of claim 18, the one or more programs further including instructions for:

further in response to receiving the first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session;

emphasizing a visual indication of the representation of the second participant in the live communication user interface; and

deemphasizing a visual indication of the representation of the first participant in the live communication user interface.

21. The non-transitory computer-readable storage medium of claim 18, wherein:

prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;

the representation of the first participant is displayed at a first size, and

the representation of the second participant is displayed at a second size smaller than the first size; and

changing the size and position of the representation of the second participant includes increasing the size of the representation of the second participant from the second size to the first size.

22. The non-transitory computer-readable storage medium of claim 18, wherein changing the size and position of the representation of the first participant includes:

changing the size and/or position of the representation of the first participant based on facial recognition data obtained from a video feed of the second participant, wherein changing the size and/or position of the representation of the first participant does not result in obscuring the face of the second participant.

23. The non-transitory computer-readable storage medium of claim 18, wherein, prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, one or more of the representations of the first participant, the second participant, and the third participant overlaps with at least one other representation of a participant.

24. The non-transitory computer-readable storage medium of claim 23, wherein:

prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, the representations of the first, second, and third participants in the live communication user interface are displayed according to a z-order; and

changing the sizes and positions of the representations of the first, second, and third participants in the live communication user interface includes maintaining the z-order of the representations of the first, second, and third participants.

25. The non-transitory computer-readable storage medium of claim 18, wherein a portion of the representation of a participant having the highest activity level of the first, second, and third participants is overlapped by a portion of the representation of another participant of the first, second, and third participants in the live communication user interface.

26. The non-transitory computer-readable storage medium of claim 18, wherein a portion of the representation of a participant having the lowest activity level of the first, second, and third participants overlaps a portion of the representation of another participant of the first, second, and third participants in the live communication user interface.

27. The non-transitory computer-readable storage medium of claim 18, the one or more programs further including instructions for:

after receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, detecting an input corresponding to the representation of the first participant; and

in response to detecting the input corresponding to the representation of the first participant, increasing the activity level of the first participant.

28. The non-transitory computer-readable storage medium of claim 18, the one or more programs further including instructions for:

receiving data indicating that a fourth participant in the live communication session is more active than a participant having the lowest activity level of the first, second, and third participants; and

in response to receiving the data indicating that the fourth participant in the live communication session is more active than a participant having the lowest activity level of the first, second, and third participants, replacing the display of the representation of the participant having the lowest activity level of the first, second, and third participants with display of a representation of the fourth participant.

29. The non-transitory computer-readable storage medium of claim 28, wherein replacing the display of the representation of the participant having the lowest activity level of the first, second, and third participants with display of a representation of the fourth participant includes:

displaying the representation of the fourth participant at a size larger than the size of the representation of the participant having the lowest activity level of the first, second, and third participants.

30. The non-transitory computer-readable storage medium of claim 18, wherein changing the size and position of the representation of the first participant includes gradually increasing the size of the representation from a first size to a second size.

31. The non-transitory computer-readable storage medium of claim 18, wherein the live communication interface includes a canvas region, and wherein the representation of the first participant in the live communication session, the representation of the second participant in the live communication session, and the representation of the third participant in the live communication session are displayed in the canvas region.

32. The non-transitory computer-readable storage medium of claim 31, the one or more programs further including instructions for:

detecting a canvas size change event; and

in response to detecting the canvas size change event;

changing a size of the canvas region; and

modifying a layout of the representations of the first, second, and third participants.

33. The non-transitory computer-readable storage medium of claim 32, the one or more programs further including instructions for:

further in response to detecting the canvas size change event, changing the size of one or more of the representations of the first, second, and third participants.

34. The non-transitory computer-readable storage medium of claim 18, the one or more programs further including instructions for:

displaying a representation of a view of a camera of the electronic device in the live communication session user interface; and

further in response to receiving the first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session;

maintaining a size of the representation of a view of the camera of the electronic device in the live communication session user interface.

35. A method comprising:

at an electronic device with a display;

displaying a live video communication user interface that includes concurrently displaying a representation of a first participant in a live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session;

while displaying the live video communication user interface, receiving data indicating that an activity level of the second participant in the live video communication session has increased above an activity level of the first participant in the live video communication session, wherein the activity level of at least the first participant is based on at least one of audio from an audio feed received from a device of the first participant or movement identified in a video feed received from the device of the first participant that meets a set of movement criteria; and

in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;

changing a size and position of the representation of the first participant in the live video communication user interface;

changing a size and position of the representation of the second participant in the live video communication user interface; and

changing a size and position of the representation of the third participant in the live video communication user interface, wherein;

after changing the size and position of the representation of the first participant in the live communication user interface, the changed representation of the first participant occupies at least a portion of a region that was occupied by the representation of the first participant in the live communication user interface before changing the size and the position of the representation of the first participant,

after changing the size and position of the representation of the second participant in the live communication user interface, the changed representation of the second participant occupies at least a portion of a region that was occupied by the representation of the second participant in the live communication user interface before changing the size and the position of the representation of the second participant, and

after changing the size and position of the representation of the third participant in the live communication user interface, the changed representation of the third participant occupies at least a portion of a region that was occupied by the representation of the third participant in the live communication user interface before changing the size and the position of the representation of the third participant.

36. The method of claim 35, further comprising:

after changing the size and position of the representation of the first participant in the live video communication user interface, receiving second data indicating that an activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session; and

in response to receiving the second data indicating that an activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session;

changing the size and/or position of the representation of the first participant in the live video communication user interface;

changing the size and/or position of the representation of the second participant in the live video communication user interface; and

maintaining the size and position of the representation of the third participant in the live communication session.

37. The method of claim 35, further comprising:

further in response to receiving the first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session;

emphasizing a visual indication of the representation of the second participant in the live communication user interface; and

deemphasizing a visual indication of the representation of the first participant in the live communication user interface.

38. The method of claim 35, wherein:

prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;

the representation of the first participant is displayed at a first size, and

the representation of the second participant is displayed at a second size smaller than the first size; and

changing the size and position of the representation of the second participant includes increasing the size of the representation of the second participant from the second size to the first size.

39. The method of claim 35, wherein changing the size and position of the representation of the first participant includes:

changing the size and/or position of the representation of the first participant based on facial recognition data obtained from a video feed of the second participant, wherein changing the size and/or position of the representation of the first participant does not result in obscuring the face of the second participant.

40. The method of claim 35, wherein, prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, one or more of the representations of the first participant, the second participant, and the third participant overlaps with at least one other representation of a participant.

41. The method of claim 40, wherein:

prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, the representations of the first, second, and third participants in the live communication user interface are displayed according to a z-order; and

changing the sizes and positions of the representations of the first, second, and third participants in the live communication user interface includes maintaining the z-order of the representations of the first, second, and third participants.

42. The method of claim 35, wherein a portion of the representation of a participant having the highest activity level of the first, second, and third participants is overlapped by a portion of the representation of another participant of the first, second, and third participants in the live communication user interface.

43. The method of claim 35, wherein a portion of the representation of a participant having the lowest activity level of the first, second, and third participants overlaps a portion of the representation of another participant of the first, second, and third participants in the live communication user interface.

44. The method of claim 35, further comprising:

after receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, detecting an input corresponding to the representation of the first participant; and

in response to detecting the input corresponding to the representation of the first participant, increasing the activity level of the first participant.

45. The method of claim 35, further comprising:

receiving data indicating that a fourth participant in the live communication session is more active than a participant having the lowest activity level of the first, second, and third participants; and

in response to receiving the data indicating that the fourth participant in the live communication session is more active than a participant having the lowest activity level of the first, second, and third participants, replacing the display of the representation of the participant having the lowest activity level of the first, second, and third participants with display of a representation of the fourth participant.

46. The method of claim 45, wherein replacing the display of the representation of the participant having the lowest activity level of the first, second, and third participants with display of a representation of the fourth participant includes:

displaying the representation of the fourth participant at a size larger than the size of the representation of the participant having the lowest activity level of the first, second, and third participants.

47. The method of claim 35, wherein changing the size and position of the representation of the first participant includes gradually increasing the size of the representation from a first size to a second size.

48. The method of claim 35, wherein the live communication interface includes a canvas region, and wherein the representation of the first participant in the live communication session, the representation of the second participant in the live communication session, and the representation of the third participant in the live communication session are displayed in the canvas region.

49. The method of claim 48, further comprising:

detecting a canvas size change event; and

in response to detecting the canvas size change event;

changing a size of the canvas region; and

modifying a layout of the representations of the first, second, and third participants.

50. The method of claim 49, further comprising:

further in response to detecting the canvas size change event, changing the size of one or more of the representations of the first, second, and third participants.

51. The method of claim 35, further comprising:

displaying a representation of a view of a camera of the electronic device in the live communication session user interface; and

further in response to receiving the first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session;

maintaining a size of the representation of a view of the camera of the electronic device in the live communication session user interface.