Universal task management system, method and product for automatically managing remote workers, including assessing the work product and workers

• US 6,859,523 B1
• Filed: 11/14/2001
• Issued: 02/22/2005
• Est. Priority Date: 11/14/2001
• Status: Active Grant

• Current Assignee: Elance Incorporation
• Original Assignee: QGENISYS, INC.
• Inventors: Jilk, David J., Kellner, Brian L., Ganora, Victor, Kotsines, Thomas A.
• Primary Examiner(s): TRAN, QUOC DUC
• Application Number:
  US10/001,498
• Time in Patent Office: 1,196 Days
• Field of Search: 379/32.01, 379/32.02, 379/9, 379/9.02, 379/9.01, 379/9.03, 379/265.01, 379/265.02, 379/265.03, 379/265.04, 379/265.05, 379/265.06, 379/265.09, 379/265.1, 379/265.12, 379/266.07, 379/266.09, 705/8, 705/9, 705/11, 709/100, 709/104, 709/202, 709/223, 709/224
• US Class Current: 379/32.01
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

A computer implemented method, a system, and a software product to automatically manage one or more human workers carrying out a process of manipulating source data provided to produce result data. The process including a set of one or more task steps each having an input and resulting in a task result. The computer implemented method includes receiving units of source data from a customer. For each unit of source data and each task step for the unit of source data, the method includes dispatching the task step and its corresponding input unit to a worker and, after the worker carries out the dispatched task step on the input unit, receiving the task result corresponding to the dispatched task step and input unit from the worker. Each worker is certified to have one or more task skills, wherein each task step requires a corresponding task skill, and wherein the dispatching of any task step occurs automatically substantially without human management to a worker who is certified to have the corresponding task skill of the task step. The method further includes assessing at least some of the task results of at least some of the task steps for at least some of the units of source data. A version of the method further includes generating the result data for the unit of source data from one or more of the task results corresponding to the task steps of the set and sending the result data for the unit of source data to the customer.

Claims

1. A method of automatically managing a plurality of remote workers over a telecommunication network for carrying out a variety of jobs for one or more customers, each job including a process of a set of one or more task steps and a set of units of source data, the method comprising:

storing in a database information on each remote worker including one or more task skills of the worker that define the types of task steps the worker is certified to carry out;

storing in the database information on the customers;

storing in the database information on each process, including the customer of the process, the order of carrying out the task steps of the process, how the input for each task step is obtained from the results of prior task steps in the process, and any pre-processing and post-processing required;

receiving the units of source data from the customers;

carrying out any defined pre-processing for the received source data;

storing in a task data structure information on tasks to be completed, each task defined by a task step and a unit of input for the task step;

storing in the database information on each remote worker including one or more tasks skills of the worker that define the types of task steps the worker is certified to carry out;

receiving requests from one or more of the remote workers for tasks;

upon receiving a task request from a remote worker, dispatching a task from the stored tasks to be completed to the remote worker according to one or more task dispatch rules;

receiving the task result from the remote workers for the task dispatched to the workers;

carrying out any defined post-processing of the task result corresponding to the tasks of a process for a unit of source data to produce result data for the unit of source data;

sending the result data to the customers;

managing the capacity of the system based on information about the stored tasks,

assessing the quality of at least some of the task results; and

evaluating the workers who carried out the tasks that produced the task results whose quality is assessed,

such that the quality assessing and the worker evaluating occur automatically substantially without human management.

2. A method as recited in claim 1, wherein the variety of jobs include a plurality of members of the set consisting of:

data entry, telesales, voice transcription, translation, image categorization, sales lead incubation, auditing, repair of documents after OCR, photo retouching, paralegal processes, call center quality assurance, and editorial work.

3. A method as recited in claim 1, wherein the assessing the quality of at least some of the task results includes determining which task results of which tasks should undergo quality assessment.

4. A method as recited in claim 3, wherein the information stored in the database for at least some task steps includes a default QA sampling rate for the task step, and wherein the determining which task results should undergo quality assessment depends on the default QA sampling rate for the task step that produced the task result.

5. A method as recited in claim 1, wherein the quality assessing of a particular task result includes creating a QA task having the particular task result as input, the QA task being a new task for the method to be dispatched to a QA worker, the QA worker being a worker having a task skill to carry out the task step of the QA task.

6. A system for automatically managing a plurality of remote worker over a telecommunication network for carrying out a variety of jobs for one or more customers, each job including a process of a set of one or more task steps and a set of associated source data units, the system connected to a network, each worker having one or more task skills and able to communicate with the system using a worker terminal connectable to network, the system comprising:

a storage subsystem containing a task data structure to store tasks to be completed, each task defined by a task step and a unit of input for the task step;

a pre-processor coupled to the storage subsystem to accept units of source data from the customers and to carry out any defined pre-processing for the accepted source data;

a task dispatcher coupled to the network and to the task data structure to accept requests from one or more of the remote workers for tasks and to dispatch a task from the task data structure to a remote worker requesting tasks, the dispatching according to one or more task dispatch rules;

a task submission unit coupled to network to receive the task result from the remote workers for the task dispatched to the workers;

a capacity manager coupled to the storage subsystem to manage the capacity of the system based on task load information on the tasks in the task data structure, on the available workers, and on the available worker task skills;

a quality unit coupled to the task submission unit and having the task results as input to assess the quality of at least some of the task results;

a certification unit coupled to the dispatcher to certify workers as having one or more task skills;

an evaluation unit coupled to the storage subsystem to evaluate the workers who carried out the tasks that produced the task results whose quality is assessed; and

a post-processor coupled to the network and to the quality unit to produce result data from the task results corresponding to the tasks of a process for a unit of source data, including any defined post-processing of the task results, and to send the result data to the customer of the process,

such that the quality unit assesses the quality of the task results and the evaluation unit evaluates workers automatically substantially without human management.

7. A system as recited in claim 6,

wherein the storage subsystem further includes

a database storing information on each remote worker including one or more task skills of the worker that define the types of task steps the worker is certified to carry out, information on one or more customers, and information on each process, the process information including the customer of the process, the order of carrying out the task steps of the process, how the input for each task step is obtained from the results of prior task steps in the process, and any preprocessing and post-processing required;

and

a data store for storing input and output information for the tasks, and

wherein the coupling between the certification unit and the task dispatcher, the capacity manager, and the evaluation unit is via the database.

8. A system as recited in claim 6, wherein the variety of jobs include a plurality of members of the set consisting of:

data entry, telesales, voice transcription, translation, image categorization, sales lead incubation, auditing, repair of documents after OCR, photo retouching, paralegal processes, call center quality assurance, and editorial work.

9. A system for automatically managing a plurality of remote workers over a telecommunication network for carrying out a variety of jobs for one or more customers, each job including a process of a set of one or more task steps and a set of associated source data units, the system connected to a network, each worker able to communicate with the system using a worker terminal connectable to network, the system comprising:

a storage subsystem containing

a database storing information on each remote worker including one or more tasks skills of the worker that define the types of task steps the worker is certified to carry out, information on one or more customers, and information on each process, the process information including the customer of the process, the order of carrying out the task steps of the process, how the input for each task step is obtained from the results of prior task steps in the process, and any pre-processing and post-processing required;

a task data structure to store tasks to be completed, each task defined by a task step and a unit of input for the task step; and

a data store for storing input and output information for the tasks;

a mechanism coupled to the storage subsystem to accept units of source data from the customers;

a pre-processor coupled to the storage subsystem to carry out any defined preprocessing for the accepted source data;

a mechanism coupled to the network to accept requests from one or more of the remote workers for tasks;

a task dispatcher coupled to the storage subsystem and to the network for dispatching a task from the task data structure to a remote worker requesting tasks, the dispatching according to one or more task dispatch rules;

a task submission unit coupled to the storage subsystem to receive the task result from the remote workers for the task dispatched to the workers;

a post-processor coupled to the storage subsystem to carry out any defined post-processing of the task results corresponding to the tasks of a process for a unit of source data to produce result data for the unit of source data;

a mechanism coupled to the storage subsystem to send the result data to the customers;

a capacity manager coupled to the storage subsystem to manage the capacity of the system based on task load information on the tasks in the task data structure, on the available workers, and on the available worker task skills;

a quality unit coupled to the storage subsystem to assess the quality of at least some of the task results; and

an evaluation unit coupled to the storage subsystem to evaluate the workers who carried out the tasks that produced the task results whose quality is assessed,

such that the quality unit assesses the quality of task results and the evaluation unit evaluates workers automatically substantially without human management.

10. A system as recited in claim 9, wherein the variety of jobs include a plurality of members of the set consisting of:

data entry, telesales, voice transcription, translation, image categorization, sales lead incubation, auditing, repair of documents after OCR, photo retouching, paralegal processes, call center quality assurance, and editorial work.

11. A method of automatically managing a plurality of workers over a telecommunication network for carrying out a variety of jobs for one or more customers, each job including a process of a set of one or more task steps and a set of units of source data, the method comprising:

storing in a database information on each worker including one or more task skills of the worker that define the types of task steps the worker is certified to carry out;

storing in the database information on each process;

receiving the units of source data;

storing in a task data structure information on tasks to be completed, each task defined by a task step and input for the task step;

storing in the database information on each remote worker including one or more task skills of the worker that define the types of task steps the worker is certified to carry out;

dispatching a task from the stored tasks to be completed to a worker;

receiving the task result from the worker for the task dispatched to the worker after the worker completes the task; and

assessing the quality of at least some of the task results, and

wherein the quality assessing occurs automatically substantially without human management.

12. A method as recited in claim 11, further comprising:

producing result data from the task results of the tasks of a process and sending the result data to the customer of the process.

13. A method as recited in claim 11, wherein dispatching is to a remote worker via the Internet and wherein the remote worker completes the task at remote location.

14. A method as recited in claim 13, wherein the storing of process information includes storing information on any required pre-processing of source data and on any required post-processing, and wherein the source data receiving includes carrying out any preprocessing required for the source data according to the stored process information, and wherein the producing result data further includes carrying out any post-processing required according to the stored process information.

15. A method as recited in claim 11, wherein the dispatching occurs upon receiving a task request from the worker.

16. A method as recited in claim 11, wherein the task request is received from the worker automatically when the worker logs on.

17. A method as recited in claim 11, wherein the variety of jobs include a plurality of members of the set consisting of:

data entry, telesales, voice transcription, translation, image categorization, sales lead incubation, auditing, repair of documents after OCR, photo retouching, paralegal processes, call center quality assurance, and editorial work.

18. A method as recited in claim 11, further comprising:

certifying workers as having one or more task skills.

19. A method as recited in claim 18, wherein the dispatching occurs according to a set of one or more dispatch rules.

20. A method as recited in claim 19, wherein the dispatch rules includes that the worker a task is assigned to must have the task skill for the task step.

21. A method as recited in claim 19, wherein the dispatching further occurs to satisfy one or more task dispatch objectives.

22. A method as recited in claim 19, wherein the task data structure is part of the database and wherein the dispatching includes forming a query on the database.

23. A method as recited in claim 22, wherein the database is a relational database including a set of tables.

24. A method as recited in claim 18, wherein the assessing the quality of at least some of the task results includes determining which task results of which tasks should undergo quality assessment.

25. A method as recited in claim 24, wherein the information stored in the database for the skill levels of each worker includes a skill quality level, and wherein the determining which task results should undergo quality assessment depends on the skill quality level of the remote worker that produced the task result.

26. A method as recited in claim 24, wherein the information stored in the database for each remote worker includes a worker status, including whether or not the remote worker is on probation, and wherein the determining which task results should undergo quality assessment depends on whether or not the remote worker that produced the task result is on probation.

27. A method as recited in claim 24, wherein the information stored in the database for at least some task steps includes a default QA sampling rate for the task step, and wherein the determining which task results should undergo quality assessment depends on the default QA sampling rate for the task step that produced the task result.

28. A method as recited in claim 18, wherein the quality assessing of a particular task result includes creating a QA task having the particular task result as input, the QA task being a new task for the method to be dispatched to a QA worker, the QA worker being a worker having a task skill to carry out the task step of the QA task.

29. A method as recited in claim 28, wherein QA tasks are created for at least some of the task results according to a sampling rate for the task step that produced the particular task result.

30. A method as recited in claim 18, further comprising

evaluating the workers who carried out the tasks that produced the task results whose quality is assessed.

31. A method as recited in claim 30, wherein the information stored in the database for the skill levels of each worker includes a skill quality level.

32. A method as recited in claim 30, wherein the information stored in the database for each worker includes a worker status, including whether or not the worker is on probation.

33. A method as recited in claim 32, wherein the evaluating of a worker depends on the status of the worker.

34. A method as recited in claim 33, wherein the information stored in the database for the skill levels of each worker includes a skill quality level, and wherein the evaluation of the worker may have an effect on one or both of the status of the worker and of the skill quality level of the worker.

35. A system for automatically managing a plurality of worker over a telecommunication network for carrying out a variety of jobs for one or more customers, each job including a process of a set of one or more task steps and a set of associated source data units, the system connected to a network, each workers having one or more task skills and able to communicate with the system using a worker terminal connectable to network, the system comprising:

a storage subsystem containing a task data structure to store tasks to be completed, each task defined by a task step and input for the task step from source data received from the customer;

a task dispatcher coupled to the network and to the task data structure to dispatch a task from the task data structure to an available worker;

a task submission unit coupled to network to receive the task result from the worker for the task dispatched to the worker;

a quality unit having the task results as input to assess the quality of at least some of the task results; and

such that the quality unit assesses the quality of at least some of the task results automatically substantially without human management.

36. A system as recited in claim 35, further comprising:

a certification unit coupled to the dispatcher to certify workers as having one or more task skills.

37. A system as recited in claim 36, wherein the recruitment unit further is to screen potential workers, each successfully screened potential worker becoming an applicant.

38. A system as recited in claim 37, wherein the training unit further is to train applicants, and wherein the certification unit further is to certify applicants as having one or more task skills.

39. A system as recited in claim 36,

wherein the storage subsystem further includes

a database storing information on each remote worker including one or more task skills of the worker that define the types of task steps the worker is certified to carry out, information on one or more customers, and information on each process, the process information including the customer of the process, the order of carrying out the task steps of the process, how the input for each task step is obtained from the results of prior task steps in the process, and any preprocessing and post-processing required;

and

a data store for storing input and output information for the tasks.

40. A system as recited in claim 35, wherein dispatching is to a remote worker via the Internet and wherein the remote worker completes the task at remote location.

41. A system as recited in claim 35, wherein the dispatching occurs upon receiving a task request from the worker.

42. A system as recited in claim 35, wherein the task request is received from the worker automatically when the worker logs on to the system.

43. A system as recited in claim 35, wherein the variety of jobs include a plurality of members of the set consisting of:

data entry, telesales, voice transcription, translation, image categorization, sales lead incubation, auditing, repair of documents after OCR, photo retouching, paralegal processes, call center quality assurance, and editorial work.

44. A system as recited in claim 39, wherein the task data structure is part of the database.

45. A system as recited in claim 39, wherein the data store is part of the database.

46. A system as recited in claim 39, wherein the database is a relational database including a set of tables.

47. A system as recited in claim 39, wherein the quality unit is further to determine which task results of which tasks should undergo quality assessment.

48. A system as recited in claim 47, wherein the quality unit is coupled to the database, wherein information stored in the database for the skill levels of each worker includes a skill quality level, and wherein the determining which task results should undergo quality assessment depends on the skill quality level of the remote worker that produced the task result.

49. A system as recited in claim 47, wherein the quality unit is coupled to the database, wherein the information stored in the database for each remote worker includes a worker status, including whether or not the remote worker is on probation, and wherein the determining which task results should undergo quality assessment depends on whether or not the remote worker that produced the task result is on probation.

50. A system as recited in claim 47, wherein the quality unit is coupled to the database, wherein the information stored in the database for at least some task steps includes a default QA sampling rate for the task step, and wherein the determining which task results should undergo quality assessment depends on the default QA sampling rate for the task step that produced the task result.

51. A system as recited in claim 39, wherein the quality unit is coupled to the task dispatcher, and wherein the quality assessing of a particular task result includes creating a QA task having the particular task result as input, the QA task being a new task for the task dispatcher to be dispatched to a QA worker, the QA worker being a worker having a task skill to carry out the task step of the QA task.

52. A system as recited in claim 51, wherein the quality unit is coupled to the database, and wherein QA tasks are created for at least some of the task results according to a sampling rate for the task step that produced the particular task result.

53. A system as recited in claim 39, further comprising

an evaluation unit coupled to the quality unit and to the database to evaluate the workers who carried out the tasks that produced the task results whose quality is assessed.

54. A system as recited in claim 53, wherein the information stored in the database for the skill levels of each worker includes a skill quality level.

55. A system as recited in claim 53, wherein the information stored in the database for each worker includes a worker status, including whether or not the worker is on probation.

56. A system as recited in claim 55, wherein the evaluating of a worker depends on the status of the worker.

57. A system as recited in claim 56, wherein the information stored in the database for the skill levels of each worker includes a skill quality level, and wherein the evaluation of the worker may have an effect on one or both of the status of the worker and of the skill quality level of the worker.

79. A system as recited in claim 35, further comprising:

a post-processor coupled to the network and to the quality unit to produce result data from the task results of the tasks a process and to send the result data to the customer of the process.

58. A system for automatically managing a plurality of workers over a telecommunication network for carrying out a variety of jobs for one or more customers, each job including a process of a set of one or more task steps and a set of units of source data, the method comprising:

a storage means containing;

a database for storing information on each process and information on each worker including one or more task skills of the worker that define the types of task steps the worker is certified to carry out, and

a task data structure for storing information on tasks to be completed, each task defined by a task step and input for the task step;

means for receiving the units of source data;

means for dispatching a task from the stored tasks to be completed to a worker;

means for receiving the task result from the worker for the task dispatched to the worker after the worker completes the task;

means for producing result data from the task results of the tasks of a process and sending the result data to the customer of the process;

means for assessing the quality of at least some of the task results; and

means for certifying workers as having one or more task skills.

59. A system as recited in claim 58, wherein the system is coupled to the Internet and wherein the dispatching means dispatches to a remote worker via the Internet and wherein the remote worker completes the task at remote location.

60. A system as recited in claim 58, wherein the variety of jobs include a plurality of members of the set consisting of:

data entry, telesales, voice transcription, translation, image categorization, sales lead incubation, auditing, repair of documents after OCR, photo retouching, paralegal processes, call center quality assurance, and editorial work.

61. A system as recited in claim 58, wherein the assessing the quality of at least some of the task results includes determining which task results of which tasks should undergo quality assessment.

62. A system as recited in claim 61, wherein the information stored in the database for at least some task steps includes a default QA sampling rate for the task step, and wherein the determining which task results should undergo quality assessment depends on the default QA sampling rate for the task step that produced the task result.

63. A system as recited in claim 58, wherein the quality assessing of a particular task result includes creating a QA task having the particular task result as input, the QA task being a new task for the method to be dispatched to a QA worker, the QA worker being a worker having a task skill to carry out the task step of the QA task.

64. A system as recited in claim 58, further comprising

means for evaluating the workers who carried out the tasks that produced the task results whose quality is assessed.

65. A system as recited in claim 64, wherein the information stored in the database for the skill levels of each worker includes a skill quality level and wherein the evaluation of the worker may have an effect on the skill quality level of the worker.

66. A system as recited in claim 64, wherein the information stored in the database for each worker includes a worker status, including whether or not the worker is on probation, and wherein the evaluating of a worker depends on the status of the worker.

67. A carrier medium carrying computer readable code segments to instruct one or more processors of a processing system to carry out a method of automatically managing a plurality of workers over a telecommunication network for carrying out a variety of jobs for one or more customers, each job including a process of a set of one or more task steps and a set of units of source data, the medium comprising:

one or more code segments to instruct the one or more processors to store in a database information on each remote worker and on each process, the worker information including one or more task skills of the worker that define the types of task steps the worker is certified to carry out;

one or more code segments to instruct the one or more processors to store in a task data structure information on tasks to be completed, each task defined by a task step and input for the task step corresponding to source data from the customer of the process of the task step;

one or more code segments to instruct the one or more processors to dispatch a task from the stored tasks to be completed to a worker;

one or more code segments to instruct the one or more processors to accept task result from the worker for the task dispatched to the worker;

one or more code segments to instruct the one or more processors to send result data to the customers;

one or more code segments to instruct the one or more processors to assess the quality of at least some of the task results; and

one or more code segments to instruct the one or more processors to certify applicants and workers as having one or more task skills.

68. A carrier medium as recited in claim 67, wherein dispatching is to a remote worker via the Internet and wherein the remote worker completes the task at remote location.

69. A carrier medium as recited in claim 67, wherein the variety of jobs include a plurality of members of the set consisting of:

data entry, telesales, voice transcription, translation, image categorization, sales lead incubation, auditing, repair of documents after OCR, photo retouching, paralegal processes, call center quality assurance, and editorial work.

70. A carrier medium as recited in claim 67, wherein the assessing the quality of at least some of the task results includes determining which task result of which tasks should undergo quality assessment.

71. A carrier medium as recited in claim 67, wherein the quality assessing of a particular task result includes creating a QA task having the particular task result as input, the QA task being a new task for the method to be dispatched to a QA worker, the QA worker being a worker having a task skill to carry out the task step of the QA task.

72. A carrier medium as recited in claim 67, further comprising

one or more code segments to instruct the one or more processors to evaluate the workers who carried out the tasks that produced the task results whose quality is assessed.

73. A carrier medium as recited in claim 72, wherein the information stored in the database for the skill levels of each worker includes a skill quality level and wherein the evaluation of the worker may have an effect on the skill quality level of the worker.

74. A carrier medium as recited in claim 72, wherein the information stored in the database for each worker includes a worker status, including whether or not the worker is on probation, and wherein the evaluating of a worker depends on the status of the worker.

75. A computer implemented method of automatically managing one or more human workers over a telecommunication network for carrying out a process of manipulating source data provided in electronic form to produce result data in electronic form, the process including a set of one or more task steps, each task step having an input corresponding to the source data and when completed on the input resulting in a corresponding task result, the method comprising:

receiving units of source data from a customer;

for each unit of source data

for each task step of the set for the unit of source data;

dispatching the task step and its corresponding input unit to a worker; and

receiving from the worker, after the worker carries out the dispatched task step on the input unit, the task result corresponding to the dispatched task step and input unit,

wherein each worker is certified to have one or more task skills, wherein each task step requires a corresponding task skill, and wherein the dispatching of any task step occurs automatically substantially without human intervention to a worker who is certified to have the corresponding task skill of the task step,

the method further comprising assessing at least some of the task results of at least some of task steps for at least some of the units of source data.

76. A method as recited in claim 75, wherein the task step dispatching is from a server computer system over a network to a remote worker, and wherein the worker carries out the task step at a location remote from the server computer system.

77. A method as recited in claim 75, wherein the source data and the result data is provided in electronic form.

78. A method as recited in claim 75, further comprising, for each unit of source data,

generating the result data for the unit of source data from one or more of the task results corresponding to the task steps of the set;

and

sending the result data for the unit of source data to the customer.