SYSTEMS AND METHODS TO CONTROL WORK PROGRESS FOR CONTENT TRANSFORMATION BASED ON NATURAL LANGUAGE PROCESSING AND/OR MACHINE LEARNING

• US 20140108103A1
• Filed: 10/15/2013
• Published: 04/17/2014
• Est. Priority Date: 10/17/2012
• Status: Abandoned Application

• Current Assignee: Gengo, Inc.
• Original Assignee: Gengo, Inc.
• Inventors: Romaine, Matthew M.I., Skyrm, Matthew James
• Application Number:
  US14/054,292
• Publication Number:
  US 20140108103A1
• Time in Patent Office: Days
• US Class Current: 705/7.41
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function

Abstract

Systems and methods are provided to compute indicators of completeness of the work output of a transformation of text-based content, worker capacity in performing the transformation, and/or the degree of matching between a unit of work and a worker, based on information collected about complexity of works, times and throughput of workers, rating of work outputs and using natural language processing techniques and machine learning techniques, such as language detection, longest common substring, length ratio, document similarity, etc. The indicators are utilized to optimize job pickup and output submission for online crowdsourcing tasks related to transformation of text-based content, such as transcription, translation, proofreading, etc.

Claims

1. A computer-implemented method, comprising:

collecting, in a computing apparatus, data about works to transform text-based content, information about workers performing the works, and ratings of work outputs provided by the workers;

generating, by the computing apparatus using the data, the information and the ratings and using natural language processing techniques and machine learning techniques, indicators of

completeness of work outputs corresponding to works, capacity of workers, and

degrees of matching between works and workers;

assigning, by the computing apparatus, works to workers based on the indicators; and

automating, by the computing apparatus, quality assurance check and work time limit management using at least some of the indicators.

2. The method of claim 1, wherein the computing apparatus comprises a crowdsourcing platform configured to distribute works to workers over Internet without contract.

3. The method of claim 1, wherein an indicator of the completeness of work outputs is computed based on at least one of:

length ratio between a source text to be transformed and a target text resulting from transforming the source txt;

longest common substring the source text and the target text; and

document similarity (DS) evaluated based on a cosine difference of a document vector representing the source text and a document vector representing the target text.

4. The method of claim 1, wherein an indicator of the completeness of work outputs is computed based on a combination of:

length ratio between a source text to be transformed and a target text resulting from transforming the source txt;

longest common substring the source text and the target text; and

document similarity (DS) evaluated based on a cosine difference of a document vector representing the source text and a document vector representing the target text.

5. The method of claim 4, wherein an indicator of the completeness of work outputs is computed further based on a distance of the length ratio to sample data.

6. The method of claim 4, wherein an indicator of the completeness of work outputs is computed further based on a deviation of the longest common substring length from a statistical mean.

7. The method of claim 1, wherein an indicator of the completeness of work outputs is computed based on a hypothesis function of:

a first completeness score based on length ratio between a source text to be transformed and a target text resulting from transforming the source txt;

a second completeness score based on longest common substring the source text and the target text; and

a third completeness score based on document similarity (DS) evaluated based on a cosine difference of a document vector representing the source text and a document vector representing the target text.

8. The method of claim 7, wherein the hypothesis function applies different weights to the first, second, and third completeness scores.

9. The method of claim 8, wherein the coefficients are adjusted in a batch process.

10. The method of claim 8, wherein the coefficients are adjusted via a neural network.

11. The method of claim 1, wherein an indicator of the degrees of matching between works and workers is based on document similarity between works to be performed and works that have been performed by respective works.

12. The method of claim 11, wherein an indicator of the degrees of matching between works and workers is further based on a complexity score computed based on a unit count and a number of unique words.

13. The method of claim 12, wherein an indicator of the degrees of matching between works and workers is augmented with natural language processing analytics.

14. The method of claim 1, wherein the indicator of the capacity of workers is based on a profile of statistical work output in a hour on a day in a week.

15. The method of claim 14, wherein the statistical work output is biased towards recent work activity.

16. The method of claim 1, further comprising:

presenting at least one multiple-choice question to a customer submitting a work to transform a text;

receiving from the customer an answer to the at least one multiple-choice question;

determining an expected quality metrics for the work based on the answer; and

controlling quality of transforming the text on a crowdsourcing platform based on the expected quality metrics.

17. The method of claim 16, further comprising:

iteratively, on the crowdsourcing platform, between transforming the text and evaluating output of the transforming of the text until the expected quality metrics is satisfied, a cost limit is reached, or a turnaround time is reached.

18. The method of claim 17, further comprising:

presenting a task to transform the text or evaluate a result of transforming the text to works in an order according to a just-in-time and best-in-time scheme.

19. A computing apparatus, comprising:

at least one processor; and

a memory storing instructions configured to instruct the at least one processor to;

collect data about works to transform text-based content, information about workers performing the works, and ratings of work outputs provided by the workers;

generate, using the data, the information and the ratings and using natural language processing techniques and machine learning techniques, indicators of

completeness of work outputs corresponding to works, capacity of workers, and

degrees of matching between works and workers;

assign works to workers based on the indicators; and

automate quality assurance check and work time limit management using at least some of the indicators.

20. A non-transitory computer-storage medium storing instructions configured to instruct a computing apparatus to perform at least:

collecting, in the computing apparatus, data about works to transform text-based content, information about workers performing the works, and ratings of work outputs provided by the workers;

generating, by the computing apparatus using the data, the information and the ratings and using natural language processing techniques and machine learning techniques, indicators of

completeness of work outputs corresponding to works, capacity of workers, and

degrees of matching between works and workers;

assigning, by the computing apparatus, works to workers based on the indicators; and

automating, by the computing apparatus, quality assurance check and work time limit management using at least some of the indicators.