System and method for a media intelligence platform

• US 9,477,975 B2
• Filed: 02/02/2016
• Issued: 10/25/2016
• Est. Priority Date: 02/03/2015
• Status: Active Grant

• Current Assignee: Twilio, Inc.
• Original Assignee: Twilio, Inc.
• Inventors: Lawson, Jeff
• Primary Examiner(s): Eskandarnia, Arvin
• Application Number:
  US15/013,292
• Publication Number:
  US 20160225044A1
• Time in Patent Office: 266 Days
• Field of Search: 709/224
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/04 Billing or invoicing
• CPC Class Codes: H04L 12/1403 Architecture for metering, charging or billing
• CPC Class Codes: H04L 41/147 for predicting network behaviour
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04M 15/41 Billing record details, i.e. parameters, identifiers, structure of call data record [CDR]
• CPC Class Codes: H04M 15/765 Linked or grouped accounts, e.g. of users or devices
• CPC Class Codes: H04M 15/7652 shared by users
• CPC Class Codes: H04M 15/7655 shared by technologies
• CPC Class Codes: H04M 15/8066 According to the number of recipients
• CPC Class Codes: H04M 15/8228 Session based
• CPC Class Codes: H04W 4/24 Accounting or billing

Abstract

A multi-tenant media processing platform system and method. At least a first media analysis service of a plurality of media analysis services is activated for at least a portion of an active communication session of an entity in the platform system. The first activated media analysis service performs a first media analysis on media of the active communication session that is collected by the platform system. The first activated media analysis service performs the first media analysis on the collected media while the communication session is active to generate a first media analysis result. During the active communication session, at least one media analysis result is applied.

Claims

1. A method comprising:

at a multi-tenant media processing platform system that includes a plurality of media analysis services;

activating a first media analysis service and a second media analysis service of the plurality of media analysis services for at least a portion of an active communication session of an entity in the platform system, the platform system activating each media analysis service responsive to at least one of;

a determination by the platform system that at least one property of media of the active communication session that is collected by the platform system maps to at least one setting of entity platform configuration that enables media analysis, the entity platform configuration being platform configuration of the entity in the platform system,

a communication application instruction,

a REST API call, and

a SIP message;

the first activated media analysis service performing a first media analysis on the collected media while the communication session is active to generate a first media analysis result;

the second activated media analysis service performing a second media analysis on the collected media while the communication session is active to generate a second media analysis result, the second media analysis being different from the first media analysis;

during the active communication session, applying the first media analysis result and the second media analysis result by at least one of;

changing the active communication session in response to at least one of the first media analysis result and the second media analysis result, and

transmitting at least one application layer protocol message to at least one callback URI of an external system of the entity, the at least one callback URI being specified by the entity platform configuration, the at least one application layer protocol message providing at least one of the first media analysis result and the second media analysis result,

accounting for the use of the first activated media analysis service on behalf of the entity;

accounting for the use of the second activated media analysis service on behalf of the entity, the platform system accounting for the use of the second activated media analysis service independently from the accounting for the use of the first activated media analysis service;

generating billing information for the entity based on the accounting for the use of the first activated media analysis service and the accounting for the use of the second activated media analysis service,

wherein the entity platform configuration is provided by at least one external system via an account management interface of the media processing platform system, the account management interface including a least one of an account portal user interface and an account management API, and

wherein the first media analysis service and the second media analysis service are both active during at least a portion of the active communication session.

2. The method of claim 1, wherein the active communication session is a communication session between at least a first communication endpoint device and a second communication endpoint device, the first communication endpoint device and the second communication endpoint device each being external to the platform system, and wherein each media analysis service is constructed to perform media analysis on media of the communication session between at least the first communication endpoint device and the second communication endpoint device.

3. The method of claim 2, wherein the plurality of media analysis services includes at least one of a sentiment and emotion detection service, a language detection service, an intent detection service, a speaker detection service, a context analysis service, and a computer vision service.

4. The method of claim 2, wherein the active communication session is at least one of:

a peer-to-peer communication session; and

a real-time communication session.

5. The method of claim 4, wherein the media is a real-time media stream.

6. A method comprising:

at a multi-tenant media processing platform system that includes a plurality of media analysis services;

activating at least a first media analysis service and a second media analysis service of the plurality of media analysis services for at least a portion of an active communication session of an entity in the platform system;

the first media analysis service collecting media of the active communication session;

the first media analysis service performing a first media analysis on the collected media while the communication session is active to generate a first media analysis result;

the first media analysis service transmitting an application layer protocol message to a Uniform Resource Identifier (URI) of the second media analysis service, the application layer protocol message providing data relating to the first media analysis result;

the second media analysis service performing a second media analysis on the first media analysis result while the communication session is active to generate a second media analysis result, the second media analysis being different from the first media analysis;

during the active communication session, applying the second media analysis result,

wherein a callback URI parameter of the first media analysis service is configured to specify the URI of the second media analysis service, and wherein the first media analysis service and the second media analysis service are both active during at least a portion of the active communication session.

7. The method of claim 6, wherein applying the second media analysis result comprises transmitting at least one application layer protocol message to at least one callback URI of an external system of the entity, the at least one callback URI being specified by a REST API call provided by an external system of the entity, the at least one application layer protocol message providing the second media analysis result.

8. A method comprising:

at a multi-tenant media processing platform system that includes a plurality of media analysis services;

activating a first media analysis service of the plurality of media analysis services for at least a portion of an active communication session of an entity in the platform system, the platform system activating the first media analysis service responsive to a REST API call provided by an external system of the entity;

the first activated media analysis service performing a first media analysis on media of the active communication session that is collected by the first activated media analysis service, while the communication session is active, to generate a first media analysis result;

during the active communication session, applying the first media analysis result by transmitting at least one application layer protocol message to at least one callback URI of an external system of the entity, the at least one callback URI being specified by the REST API call, the at least one application layer protocol message providing the first media analysis result,

accounting for the use of the first activated media analysis service on behalf of the entity; and

generating billing information for the entity based on the accounting for the use of the first activated media analysis service,

wherein each external system of the entity is a system of an account holder of a platform account of the multi-tenant media processing platform system, each external system being external to the platform system.

9. The method of claim 8,

wherein the active communication session is a communication session between at least a first communication endpoint device and a second communication endpoint device, the first communication endpoint device and the second communication endpoint device each being external to the platform system, and

wherein the first media analysis service is constructed to perform media analysis on media of the communication session between at least the first communication endpoint device and the second communication endpoint device,

wherein the active communication session is at least one of:

a peer-to-peer communication session; and

a real-time communication session, and

wherein the media is a real-time media stream.

10. The method of claim 9,

the method further comprising:

at the multi-tenant media processing platform system;

activating a second media analysis service of the plurality of media analysis services, the platform system activating the second media analysis service responsive to a REST API call provided by an external system of the entity; and

the second activated media analysis service performing a second media analysis to generate a second media analysis result.

11. The method of claim 10,

wherein the media is real-time media, and the first media analysis service performs real-time media analysis on the real-time media, and

wherein the second activated media analysis service performs real-time media analysis on at least one of the real-time media and the first media analysis result.

12. The method of claim 11,

the method further comprising:

at the multi-tenant media processing platform system;

during the active communication session, applying the second media analysis result by transmitting at least one application layer protocol message to a communication service of the platform system, the at least one application layer protocol message providing the second media analysis result.

13. The method of claim 12, wherein the second media analysis is different from the first media analysis.

14. The method of claim 10,

wherein the media is real-time media, and the first media analysis service performs real-time media analysis on the real-time media, and

wherein the second activated media analysis service performs asynchronous media analysis on stored media.

15. The method of claim 9, wherein activating the first media analysis service comprises:

the platform system selecting at least a sentiment and emotion detection service as the first media analysis service from the plurality of media analysis services of the platform system, the plurality of media analysis services including the sentiment and emotion detection service and at least one of a context analysis service, and a computer vision service; and

the platform system activating each selected media analysis service.

16. The method of claim 15, wherein the REST API specifies at least the communication session and at least the first activated media analysis service.

17. The method of claim 15, the method further comprising at least one of:

at the multi-tenant media processing platform system, de-activating the first media analysis service responsive to a de-activation REST API call provided by an external system of the entity, the de-activation REST API call specifying the communication session; and

at the multi-tenant media processing platform system, configuring the first media analysis service responsive to a configuration REST API call provided by an external system of the entity, the configuration REST API call specifying the communication session.

18. The method of claim 16,

the method further comprising:

at the multi-tenant media processing platform system;

training at least one media analysis service by using the first media analysis result and user feedback provided by an external system via a feedback API of the platform system,

wherein the platform system receives user feedback via the feedback API, the feedback indicating a quality of media analysis provided by the first media analysis service, and the platform system trains the first media analysis service based on the received feedback, wherein the platform system maintains history data that includes media analysis results and corresponding media and user feedback of the first media analysis service, and the platform system trains the first media analysis service by using the history data.

19. The method of claim 8, wherein the REST API specifies program instructions for a customized media analysis service, and responsive to the REST API request, the platform system performs at least one of:

generation of a customized media analysis service according to the received program instructions; and

control of a communication service of the platform system to provide a media analysis service according to the received program instructions.

20. The method of claim 8,

the method further comprising:

at the multi-tenant media processing platform system;

responsive to a request for communication records that is provided by the external system of the entity, providing the external system with communication records that match media analysis results properties specified in the request,

wherein the provided communication records include information for a plurality of communication sessions of the entity including the communication session of the entity for which the first media analysis is performed,

wherein the information for the communication session of the entity includes at least information of the first media analysis result,

wherein the platform system indexes media analysis results information of the communication records, and

wherein the matching communication records include indexed media analysis results information that matches media analysis results properties specified in the request.