Systems and methods for chat with audio and video elements

• US 10,530,731 B1
• Filed: 03/27/2017
• Issued: 01/07/2020
• Est. Priority Date: 03/28/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Wu, William, Boyd, Nathan Kenneth, Al Majid, Newar Husam, Wagner, Erik
• Primary Examiner(s): Nguyen, Tu T
• Application Number:
  US15/470,624
• Time in Patent Office: 1,016 Days
• Field of Search: 709204
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 51/56 Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM]
• CPC Class Codes: H04L 51/58 Message adaptation for wireless communication
• CPC Class Codes: H04L 65/1016 IP multimedia subsystem [IMS]
• CPC Class Codes: H04L 65/1063 Application servers providing network services systems providing special services to telephonic subscribers H04M3/42
• CPC Class Codes: H04L 65/1069 Session establishment or de-establishment
• CPC Class Codes: H04L 65/4015 where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/632 using a connection between clients on a wide area network, e.g. setting up a peer-to-peer communication via Internet for retrieving video segments from the hard-disk of other client devices broadcast-related systems characterised by transmission among terminal devices H04H60/80; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/52 Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434

Abstract

Methods, devices, systems, and computer-readable media with instructions for text communications with imbedded audio or audiovisual elements are described. One embodiment involves displaying a user interface for chat communication with a time-ordered display space for communications. Text communications and audiovisual communications are each presented in associated rows within the user interface. Various embodiments enable audio or video notes as well as audio or video streaming or multi-user calls within the user interface with individual communications or connections presented within the time-ordered display. In some embodiments, communications within the display are deleted based on ephemeral message triggers.

Claims

1. A method comprising:

displaying, at a first device, a user interface for chat communication with a second device, the user interface comprising a time-ordered display space for communications with the second device;

communicating a text message with a second device;

displaying the text message in first one or more rows within the time-ordered display space of the user interface on the first device;

communicating a content message comprising audio data with the second device;

displaying an icon associated with the content message in second one or more rows within the time-ordered display space of the user interface on the first device;

receiving, via an input of the first device corresponding to the second one or more rows within the time-ordered display space of the user interface, a command to output content from the content message;

presenting, via one or more outputs of the first device, the content from the content message;

receiving, at the first device prior to communication of the content message, an audio stream notifier associated with the second device; and

displaying, in the second one or more rows within the time-ordered display space of the user interface, one or more response inputs associated with the audio stream notifier,

wherein the content message comprises a stream of audio data from the second device; and

wherein the one or more response inputs comprise at least a listen-only input, a join input, and an ignore input.

2. The method of claim 1 further comprising:

receiving, via the input of the first device, a selection of the join input; and

in response to the selection of the join input and receipt of the audio stream notifier, establishing a two-way data stream between the first device and the second device.

3. The method of claim 2 further comprising:

sensing, by the first device, data from a microphone of the first device; and

automatically adjusting, in response to the data from the first device, an audio connection between an idle state, a talking state, and a listening state based on the data from the microphone and the stream of audio data from the second device.

4. The method of claim 1 further comprising:

displaying a plurality of identifiers comprising an identifier associated with the second device,

wherein the audio stream notifier is associated with audio data from a plurality of devices associated with the plurality of identifiers, and

wherein the content message is received at the first device via a server computer.

5. The method of claim 4 further comprising:

monitoring, by the first device, an ephemeral message deletion trigger associated with the content message;

deleting the content message from the first device when a threshold associated with the ephemeral message deletion trigger has been met; and

communicating, to the server computer, a deletion notification message.

6. The method of claim 1 further comprising:

receiving, at the first device prior to communication of the content message, a video stream notifier associated with the second device,

wherein the icon comprises a portion of at least a first video frame from a stream of video data,

wherein the content message comprises a second stream of video data from the second device, and

wherein presenting the content from the content message comprises displaying video data from the second stream of video data in the second one or more rows within the time-ordered display space of the user interface.

7. The method of claim 6 further comprising:

receiving, at the first device, an input to switch from a video communication to an audio-only communication;

transmitting, by the first device to a server system, a notification of the switch from the video communication to the audio-only communication;

in response to the notification, receiving, from the server computer, a stream of audio data associated with the stream of video data; and

adjusting the presenting of the content from the content message from the displaying of the video data from the stream of video data to an audio-only output associated with an audio icon in the second one or more rows within the time-ordered display space of the user interface.

8. The method of claim 1 wherein the content message comprises an audio file, and

wherein presenting, via the one or more outputs of the first device, the content from the content message comprises outputting audio from the audio file via a speaker of the first device.

9. The method of claim 1 wherein the content message comprises a video clip, and

wherein presenting, via the one or more outputs of the first device, the content from the content message comprises outputting audio from the video clip via a speaker of the first device and displaying video frames from the video clip within a first circular area of the second one or more rows within the time-ordered display space of the user interface.

10. The method of claim 9 further comprising:

monitoring, by the first device, an ephemeral message deletion trigger associated with the content message; and

deleting the video clip from the first device when a threshold associated with the ephemeral message deletion trigger has been met.

11. The method of claim 9 further comprising:

receiving, at a record input area of the user interface, an input to initiate recording of a second content message;

generating, by the first device in response to the input, second content;

displaying, within third one or more rows within the user interface, a message icon associated with the second content; and

transmitting the second content and the message icon to the second device for display of the message icon in a second user interface of the second device.

12. A first device for display of a chat interface with additional content elements, the first device comprising:

a memory configured to store content data received from a second device via a server system;

a display screen; and

one or more processors coupled to the display screen and the memory and configured to;

display a user interface for chat communication with the second device, the user interface comprising a time-ordered display space for communications with the second device via the server system;

communicate a text message with the second device;

display the text message in first one or more rows within the time-ordered display space of the user interface on the first device;

communicate a content message comprising audio data with the second device;

display an icon associated with the content message in second one or more rows within the time-ordered display space of the user interface on the first device;

receive, via an input of the first device corresponding to the second one or more rows within the time-ordered display space of the user interface, a command to output content from the content message; and

present, via one or more outputs of the first device, the content from the content message;

receiving, at the first device prior to communication of the content message, an audio stream notifier associated with the second device; and

display, in the second one or more rows within the time-ordered display space of the user interface, one or more response inputs associated with the audio stream notifier,

wherein the content message comprises a stream of audio data from the second device; and

wherein the one or more response inputs comprise at least a listen-only input, a join input, and an ignore input.

13. The first device of claim 12, wherein the one or more processors are further configure to:

display a plurality of identifiers associated with a plurality of devices, the plurality of identifiers comprising an identifier associated with the second device, and the plurality of devices comprising the second device;

receive a plurality of content messages from the plurality of devices via the server system;

display a corresponding icon for each content message of the plurality of content messages within corresponding one or more rows of the time-ordered display area of the user interface;

monitor, an ephemeral message deletion trigger associated with each content message of the plurality of content messages;

delete each corresponding content message from the first device when a corresponding threshold associated with the ephemeral message deletion trigger for corresponding content messages has been met; and

communicate to the server system, for each deletion of each content message of the plurality of content messages, a corresponding deletion notification message.

14. The first device of claim 12, wherein the one or more processors are configured to receive, prior to communication of the content message, a video stream notifier associated with the second device,

wherein the icon comprises a portion of at least a first video frame from a stream of video data,

wherein the content message comprises a second stream of video data from the second device,

wherein presenting the content from the content message comprises displaying video data from the second stream of video data in the second one or more rows within the time-ordered display space of the user interface, and

wherein the one or more processors are further configured to:

receive, at the first device, an input to switch from a video communication to an audio-only communication;

transmit, by the first device to the server system, a notification of the switch from the video communication to the audio-only communication;

in response to the notification, receive, from the server system, a stream of audio data associated with the stream of video data; and

adjust the presenting of the content from the content message from the displaying of the video data from the stream of video data to an audio-only output associated with an audio icon in the second one or more rows within the time-ordered display space of the user interface.

15. The method of claim 12, wherein the content message comprises a video clip,

wherein presenting, via the one or more outputs of the first device, the content from the content message comprises outputting audio from the video clip via a speaker of the first device and displaying video frames from the video clip within a first circular area of the second one or more rows within the time-ordered display space of the user interface, and

wherein the one or more processors are further configured to monitor an ephemeral message deletion trigger associated with the content message and delete the video clip from the first device when a threshold associated with the ephemeral message deletion trigger has been met.

16. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a first device, cause the first device to perform operations comprising:

displaying a user interface for chat communication with a second device, the user interface comprising a time-ordered display space for communications with the second device;

communicating a text message with the second device;

displaying the text message in first one or more rows within the time-ordered display space of the user interface on the first device;

communicating a content message comprising audio data with the second device;

displaying an icon associated with the content message in second one or more rows within the time-ordered display space of the user interface on the first device;

receiving, via an input of the first device corresponding to the second one or more rows within the time-ordered display space of the user interface, a command to output content from the content message;

presenting, via one or more outputs of the first device, the content from the content messages

receiving, at the first device prior to communication of the content message, an audio stream notifier associated with the second device; and

displaying, in the second one or more rows within the time-ordered display space of the user interface, one or more response inputs associated with the audio stream notifier,

wherein the content message comprises a stream of audio data from the second device; and

wherein the one or more response inputs comprise at least a listen-only input, a join input, and an ignore input.

17. The non-transitory computer-readable media of claim 16, wherein the instructions further cause the first device to perform operations comprising:

receiving, via the input of the first device, a selection of the join input;

in response to the selection of the join input and receipt of the audio stream notifier, establishing a two-way data stream between the first device and the second device;

sensing, by the first device, data from a microphone of the first device; and

automatically adjusting, in response to the data from the first device, an audio connection between an idle state, a talking state, and a listening state based on the data from the microphone and the stream of audio data from the second device.