Configurable metadata-based automation and content classification architecture for cloud-based collaboration platforms

• US 10,708,321 B2
• Filed: 08/17/2017
• Issued: 07/07/2020
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Box Incorporated
• Original Assignee: Box Incorporated
• Inventors: Pearl, Annie, Huamonte, James, Tran, Brian, Low, Tiffany, Masse, Pierre-Alexandre, Blosser, Ted
• Primary Examiner(s): Abedin, Normin
• Application Number:
  US15/679,532
• Publication Number:
  US 20170346861A1
• Time in Patent Office: 1,055 Days
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/60 Subscription-based services using application servers or record carriers, e.g. SIM application toolkits

Abstract

Scalable architectures, systems, and services are provided herein for generating jobs by applying user-specified metadata rules to metadata events. More specifically, the scalable architecture described herein uses metadata to drive automations and/or polices in a cloud-based environment. In one embodiment, the architecture integrates a metadata service with an event-based automation engine to automatically trigger polices and/or automations based on metadata and/or changes in metadata changes. The metadata service can include customizable and/or pre-build metadata templates which can be used to automatically apply a metadata framework (e.g., particular fields) to files based on, for example, the upload or placement of a particular file in a particular folder. The architecture also provides for advanced metadata searching and data classification.

Claims

1. A computer-implemented method of facilitating metadata-based automations in a collaborative cloud-based environment, the method comprising:

receiving, by a job manager system of the collaborative cloud-based environment, a job request from a rule manager system of the collaborative cloud-based environment, the job request based on a metadata event indicative of a change in a metadata key-value pair associated with a particular work item in the collaborative cloud-based environment, the change in the metadata key-value pair resulting from an action performed on the particular shared work item by a collaborator of the collaborative cloud-based environment;

generating, by the job manager system, a job based on the received job request, the generated job including a set of parameters describing work to be executed in the collaborative cloud-based environment with respect to the particular work item; and

distributing, by the job manager system, the generated job into a particular job queue to be processed by one or more worker machines in a distributed computing cluster associated with the collaborative cloud-based environment, the particular job queue one of a plurality of job queues.

2. The computer-implemented method of claim 1, further comprising:

storing, by the job manager system, the received job request in persistent storage before generating the job.

3. The computer-implemented method of claim 2, wherein the persistent storage includes one or more geographically remote data centers.

4. The computer implemented method of claim 2, further comprising:

acknowledging, by the job manager system, to the rule manager system, receipt of the job request in response to storing the job request in persistent storage.

5. The computer-implemented method of claim 2, further comprising:

guaranteeing, by the job manager system, to the rule manager system, that the job will be processed in response to storing the job request in persistent storage.

6. The computer-implemented method of claim 1, further comprising:

selecting, by the job manager system, the particular job queue based on a type of the generated job.

7. The computer-implemented method of claim 1, wherein the particular job queue is designated for jobs of a particular job type, and wherein the particular job queue is serviced by one or more worker machines configured to process the particular job type.

8. The computer-implemented method of claim 1, wherein the particular job queue is a general purpose job queue serviced by one or more general purpose worker machines, each of the one or more general purpose worker machines configured to process a plurality of different job types.

9. The computer-implemented method of claim 1, further comprising:

monitoring, by the job manager system, performance of the generated job by the one or more worker machines processing the job.

10. The computer-implemented method of claim 1, further comprising:

injecting, by the job manager system, callback information into the generated job prior to distributing the job into the particular job queue, the callback information configured to cause a worker machine to transmit notifications to the job manager system while processing the job.

11. The computer-implemented method of claim 1, wherein the job request is received from the rule manager system via a load balancer configured to distribute job requests based on loads at a plurality of instances of the job manager system.

12. The computer-implemented method of claim 1, further comprising:

injecting, by the job manager system, a token into the job before distributing the job to the particular job queue;

wherein the token is used by the one or more worker machines when processing the job and follows any additional job requests initiated by the one or more worker machines;

wherein additional job requests received by the job manager system are not generated into jobs or distributed to job queues if the token has been received by the job manager more than a threshold number of times.

13. The computer implemented method of claim 1, wherein the generated job further includes action log data associated with the metadata event that triggered the job request.

14. A computer implemented method for facilitating metadata-based automations in a collaborative cloud-based environment, the method comprising:

receiving, by an event-based automation system, a metadata event, wherein the metadata event identifies a change in a metadata key-value pair associated with a particular work item in the collaborative cloud-based environment, the change in the metadata key-value pair resulting from an action performed on the particular shared work item by a collaborator of the collaborative cloud-based environment; and

automatically translating, by event-based automation system, the metadata event into a job request based on a pre-defined metadata rule that matches a key of the metadata key-value pair associated with the particular work item;

generating, by the event-based automation system, a job based on the job request, the generated job including a set of parameters describing work to be executed in the collaborative cloud-based environment with respect to the particular work item; and

distributing, by the event-based automation system, the generated job into a particular job queue to be processed by one or more worker machines in a distributed computing cluster associated with the collaborative cloud-based environment, the particular job queue one of a plurality of job queues.

15. The computer-implemented method of claim 14, wherein translating the received metadata event into a job request includes:

processing the metadata event to capture the metadata key-value pair associated with the particular work item;

accessing pre-defined metadata rules from a memory in communication with the event-based automation system;

scanning the pre-defined metadata rules to select the pre-defined metadata rule that matches the key of the metadata key-value pair; and

generating the job request associated with the first pre-defined metadata rule.

16. The computer-implemented method of claim 14, further comprising:

monitoring, by the event-based automation system, changes in the metadata occurring with respect to work items within the collaborative cloud-based environment.

17. The computer-implemented method of claim 16, wherein the changes in metadata occur as a result of events or job requests that are performed in the collaborative cloud-based environment.

18. The computer-implemented method of claim 14, further comprising:

processing, by the event-based automation system, a value of the metadata key-value pair associated with the particular work item;

processing, by the event-based automation system, the pre-defined metadata rule that matches the key of the key-value pair to identify a threshold value associated with the key-value pair; and

comparing by the event-based automation system, the value of the metadata key-value pair with the threshold value;

wherein the job request is conditionally generated based on the comparison.

19. The computer-implemented method of claim 14, further comprising:

storing, by the event-based automation system, the received job request in persistent storage before generating the job.

20. The computer-implemented method of claim 14, further comprising:

selecting, by the event-based automation system, the particular job queue based on a type of the generated job.

21. The computer implemented method of claim 14, wherein the particular job queue is designated for jobs of a particular job type, and wherein the particular job queue is serviced by one or more worker machines configured to process the particular job type.

22. The computer implemented method of claim 14, wherein the particular job queue is a general purpose job queue serviced by one or more general purpose worker machines, each of the one or more general purpose worker machines configured to process a plurality of different job types.

23. The computer implemented method of claim 14, further comprising:

monitoring, by the event-based automation system, performance of the generated job by the one or more worker machines processing the job.

24. The computer implemented method of claim 14, further comprising:

injecting, by the event-based automation system, callback information into the generated job prior to distributing the job into the particular job queue, the callback information configured to cause a worker to transmit notifications to the job manager while processing the job.

25. An event-based automation system for facilitating metadata-based automations in a collaborative cloud-based environment, comprising a non-transitory machine readable storage medium having instructions embodied thereon, the instructions when executed cause a processor to perform processing, the event-based automation system comprising:

an event dispatcher configured to;

monitor for changes in metadata occurring with respect to work items within the collaborative cloud-based environment; and

generate an event in response to detecting a change in a metadata key-value pair associated with a particular work item in the collaborative cloud-based environment, the change in the metadata key-value pair resulting from an action performed on the particular shared work item by a collaborator of the collaborative cloud-based environment;

a rule manager system communicatively coupled to the event dispatcher, the rule manager system configured to;

automatically translate the metadata event generated by the event dispatcher into a job request based on a pre-defined metadata rule that matches a key of the metadata key-value pair associated with particular work item;

a job manager system communicatively coupled to the rule manager system, the job manager system configured to;

generate a job based on the job request from the rule manager system, the generated job including a set of parameters describing work to be executed in the collaborative cloud-based environment with respect to the particular work item; and

distribute the generated job into a particular job queue for processing, the particular job queue one of a plurality of job queues; and

one or more worker machines in a distributed computing cluster, the one or more worker machines configured to;

receive the job generated by the job manager out of the particular job queue; and

process the job.