Resource allocation in distributed processing systems

• US 10,148,589 B2
• Filed: 09/29/2015
• Issued: 12/04/2018
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: Pearson Education Incorporated (Pearson plc)
• Original Assignee: Pearson Education Incorporated (Pearson plc)
• Inventors: Dronen, Nicholas A., Foltz, Peter W., Garner, Holly, Loring, Miles T., Kapoor, Vishal
• Primary Examiner(s): Lai, Michael C
• Application Number:
  US14/869,748
• Publication Number:
  US 20160094476A1
• Time in Patent Office: 1,162 Days
• Field of Search: 709224, 709226
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: H04L 47/783 Distributed allocation of resources, e.g. bandwidth brokers

Abstract

A distributed processing system is disclosed herein. The distributed processing system includes a server, a database server, and an application server that are interconnected via a network, and connected via the network to a plurality of independent processing units. The independent processing units can include an analysis engine that is machine-learning-capable, and thus uniquely completes its processing tasks. The server can provide one or several pieces of data to one or several of the independent processing units, can receive analysis results from these one or several independent processing units, and can update the result based on a value characterizing the machine learning of the independent processing unit.

Claims

1. A distributed processing system, the system comprising:

a source device configured to provide pieces of data for evaluation, wherein each of the pieces of data is associated with one or several user authors, wherein the pieces of data together comprise a processing task;

a plurality of independent processing units configured to receive a portion of the processing task, wherein the portion of the processing task comprises one or several of the pieces of data, and wherein the independent processing units are configured to characterize one or several aspects of the one or several of the pieces of data; and

a server communicatively connected to the source device and the plurality of independent processing units via a network, wherein the server is configured to;

receive a signal encoding the processing task;

identify a subset comprising some of the pieces of data in the processing task;

identify a plurality of features in each of the pieces of data;

generate an attribute vector for each of the pieces of data in the processing task, and wherein each of the plurality of attribute vectors comprises a dimension relating to the plurality of features of the corresponding piece of data for which the corresponding attribute vector is generated;

select pairs of attribute vectors from the plurality of attribute vectors;

determine a distance between ends of each of the pairs of attribute vectors;

identify a pair of the pairs of attribute vectors having ends separated by a greatest distance;

add the pieces of data associated with each of the attribute vectors having ends separated by a greatest distance to the subset of pieces of data; and

provide the subset of pieces of data to the plurality of independent processing units.

2. The distributed processing system of claim 1, wherein the server is configured to increment a count when the pieces of data are added to the subset of pieces of data.

3. The distributed processing system of claim 2, wherein the count identifies a number of pieces of data included in the subset of pieces of data.

4. The distributed processing system of claim 3, wherein the server is configured to compare the count to a desired size of the subset of pieces of data.

5. The distributed processing system of claim 4, wherein the server is configured to provide the subset when the count meets the desired size of the subset of pieces of data.

6. The distributed processing system of claim 5, wherein the server is configured to identify additional pieces of data for inclusion in the subset when the count does not meet the desired size of the subset of pieces of data.

7. The distributed processing system of claim 6, wherein identifying additional pieces of data for inclusion in the subset comprises generating attribute vectors for pieces of data not included in the subset.

8. The distributed processing system of claim 7, wherein the server is configured to:

compare to the attribute vectors for pieces of data not included in the subset to attribute vectors for pieces of data included in the subset.

9. The distributed processing system of claim 1, wherein determining the distance between the ends of the attribute vectors comprises:

setting attribute vectors in a pair of attribute vectors to a common origin;

calculating a distance between a pair of attribute vectors; and

adding the pair of attribute vectors having the greatest calculated distance to the subset.

10. A method for automatically providing a final subset of pieces of data to an independent processor, the method comprising:

receiving a signal encoding a processing task comprising a plurality of pieces of dada;

identifying a subset comprising some of the pieces of data in the processing task;

identifying a plurality of features in each of the pieces of data;

generating an attribute vector for each of the subset of the pieces of data in the processing task, wherein each of the plurality of attribute vectors comprises a dimension relating to the plurality of features of the corresponding piece of data for which the corresponding attribute vector is generated;

selecting pairs of attribute vectors from the plurality of attribute vectors;

determining a distance between ends of the attribute vectors in each of the pairs of selected attribute vectors;

identifying at least one pair of the pairs of attribute vectors having ends separated by a greatest distance;

generating a subset of pieces of data including a pair of attribute vectors having ends separated by the greatest distance; and

providing the subset of pieces of data to a plurality of independent processing units.

11. The method of claim 10, further comprising incrementing a count when the pieces of data are added to the subset of pieces of data.

12. The method of claim 11, wherein the count identifies a number of pieces of data included in the subset of pieces of data.

13. The method of claim 12, further comprising comparing the count to a desired size of the subset of pieces of data.

14. The method of claim 13, further comprising providing the subset when the count meets the desired size of the subset of pieces of data.

15. The method of claim 14, further comprising identifying additional pieces of data for inclusion in the subset when the count does not meet the desired size of the subset of pieces of data.

16. The method of claim 15, wherein identifying additional pieces of data for inclusion in the subset comprises generating attribute vectors for pieces of data not included in the subset.

17. The method of claim 16, further comprising:

comparing to the attribute vectors for pieces of data not included in the subset to attribute vectors for pieces of data included in the subset.

18. The method of claim 10, wherein determining the distance between the ends of the attribute vectors comprises:

setting attribute vectors to a common origin;

calculating a distance between pairs of attribute vectors; and

adding the pair of attribute vectors having the greatest calculated distance to the subset.