Method and system for asynchronous online distributed problem solving including problems in education, business, finance, and technology

• US 7,155,157 B2
• Filed: 09/20/2001
• Issued: 12/26/2006
• Est. Priority Date: 09/21/2000
• Status: Expired due to Term

• Current Assignee: IQ Consulting Company, Inc.
• Original Assignee: I.Q. Consulting, Inc.
• Inventors: Kaplan, Craig A.
• Primary Examiner(s): MOSSER, KATHLEEN MICHELE
• Application Number:
  US09/957,656
• Publication Number:
  US 20020076674A1
• Time in Patent Office: 1,923 Days
• Field of Search: 434/323, 434/322, 434/350, 434/307.R, 434/118, 434/362, 434/353, 434/365
• US Class Current: 434/350
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

Systems and methods facilitating authoring and problem solving by joint contributors working separately but against a common goal. On-line asynchronous distributed authoring and problem solving system, method, and computer program for focusing attention toward particular authoring and problem solving topics using a threaded discussion group and reward matrix. System, method, computer program and computer program product for coordinating the activities of a plurality of people, where the plurality may be any number from two to thousands or more people. Mechanism for directing the attention and focus of large numbers of people who are solving problems using a tree-based problem space, where the tree based problem space may be a virtual problem space. Algorithms and procedures for evaluating nodes in the virtual problem space and assigning values via a pay-off matrix that serves to focus the attention of large numbers of problem solvers. Combination of threaded discussion groups with the pay-off matrix and a variety of algorithms to create useful system for solving multi-level problems leveraging human expertise.

Claims

1. A method for solving a complex problem by a plurality of problem solving contributors acting asynchronously over an interactive communication medium, said method comprising:

formulating the complex problem to be solved as a problem tree having a plurality of problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem;

associating a reward matrix with said problem tree having reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with at least one of said sub-problems;

receiving a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor;

dynamically modifying the complex problem formulation during its solution in response to at least one received problem solution contribution, the dynamically modifying the complex problem formulation comprises a contributor formulating and presenting at least one new sub-problem to be solved in response to problem solution contributions submitted; and

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor.

2. The method in claim 1, wherein the reward matrix comprises a payoff matrix and the reward values comprise monetary reward values.

3. The method in claim 2, wherein the monetary reward values are reward values in a recognized international currency.

4. The method in claim 3, wherein the recognized international currency is selected from the set consisting of United States Dollars, Japanese Yen, French Francs, Canadian Dollars, European Currency (EU), and English Pounds.

5. The method in claim 1, wherein in response to the receipt of a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor, the method further includes dynamically modifying the reward matrix to alter the incentive to provide problem solving contributions in response to the receipt of problem contributions to one or more of the sub-problems.

6. A method according to claim 5, wherein the act of modifying the reward matrix is performed at least in part by at least one problem solving contributor of the plurality of contributors.

7. The method in claim 1, wherein the reward matrix and the reward values of the reward matrix are selected to preferentially encourage problem solving contributions to particular ones of the sub-problems having particular more valuable reward values.

8. The method in claim 1, wherein said contributors access the problem tree and the reward matrix over a network communication medium.

9. The method in claim 1, wherein said contributors access the problem tree and the reward matrix via a threaded discussion group over the internet.

10. The method in claim 1, wherein the plurality of contributors comprise at least ten contributors.

11. The method in claim 1, wherein the plurality of contributors comprise at least one hundred contributors.

12. The method in claim 1, wherein the method further comprises matching skills of possible contributors to an existing problem tree or sub-problem.

13. The method in claim 1, wherein the matching includes comparing a database of expert contributors to all problems trees, or sub-problems for which solutions are sought.

14. The method in claim 1, further comprising detecting the occurrence or nonoccurrence of a predetermined condition or event and modifying the manner in which problem solution contributions are processed in response to identified ones of said predetermined conditions or events.

15. The method in claim 1, further comprising evaluating submitted contributions after a predetermined number of contributions have been submitted before permitting additional contributions to be submitted.

16. The method in claim 1, further comprising rating the problem solution contributions.

17. The method in claim 16, wherein the rating is done by at least one peer using a rating selected from the set consisting of:

(i) by a peer of the expert contributor;

(ii) by a peer of the expert contributor using any tree-based structure;

(iii) by a peer of the expert contributor using a threaded discussion group based tree-based structure;

(iv) by peer members in terms of their credibility; and

(v) by peer members of a threaded discussion group in which the peers themselves are evaluated in terms of their credibility.

18. The method in claim 17, further comprising rating the problem solution contributions by peer members of a threaded discussion group in which the peers themselves are evaluated in terms of their credibility, and wherein the peer ratings are analyzed and allocated a weighing based in part on historical performance and the demonstrated problem solving effectiveness of each peer-expert contributor.

19. The method in claim 16, wherein said problem solution contributions are rated for quality and the reward matrix reward values are modified to reflect the quality of the contribution.

20. The method in claim 19, wherein a more valuable reward value is allocated to a more highly rated problem solution contribution.

21. The method in claim 1, wherein method further comprising generating a record of a problem solution effort toward a final solution and generating a record of the ideas produced and an associated evaluation of each idea produced.

22. The method in claim 1, wherein method further comprises generating a record of a problem solution effort and assigning credit or blame for portions of the problem solution based on the record.

23. The method in claim 1, wherein the problem tree may be modified by a customer to control the problem solving effort.

24. The method in claim 1, wherein the reward matrix may be modified by a customer to control the problem solving effort.

25. The method in claim 1, wherein the problem tree and the reward matrix may be modified by a customer to control the problem solving effort.

26. The method in claim 1, wherein contributions to problem solution are directed in particular directions toward contributions of higher importance problem components in a particular order by incentivizing possible contributors to work on and submit contributions toward particular problem components by adjusting the reward matrix values so that high importance problem components are set with high reward matrix awards values and off-topic response contributions are discouraged by setting them to have a small reward matrix value or no reward matrix value.

27. The method in claim 1, wherein differential rewards are offered for problem solution contributions and dynamically adjusted during the complex problem solution in order to direct the attention of the potential problem solution contributors through the problem tree in a logical and efficient manner toward problem solution without any need to direct or assign problem or sub-problem solution responsibilities.

28. The method in claim 1, wherein contributions to a particular problem solution are received only during predetermined contribution acceptance time periods.

29. The method in claim 28, wherein the predetermined contribution acceptance time periods are time intervals selected from the set of time intervals consisting of one hour, 4 hours, 8 hours, one day, two days, five days, a week, a month, a number of days, a number of hours, a number of weeks, a number of months, and combinations thereof.

30. The method of claim 29, wherein the predetermined contribution time periods are date based time periods.

31. The method of claim 29, wherein the predetermined contribution time periods are variable time periods.

32. The method of claim 29, wherein the predetermined contribution time periods close a period of time for acceptance of contributions and initiate an evaluation of accepted contributions, before opening a new period for accepting contributions.

33. The method in claim 29, wherein the predetermined contribution acceptance time periods are selected to motivate contributions submittal and evaluation.

34. The method in claim 28, wherein the predetermined contribution acceptance time periods are controlled by a customer requesting solution to a problem.

35. The method in claim 1, wherein the problem tree is selectively pruned or filtered to display only node and branch portions or an equivalent list that have a higher than associated threshold reward value.

36. The method in claim 35, wherein the threshold reward value is established by a contributor desiring to make a contribution to solving the problem.

37. The method in claim 36, wherein the threshold reward value is established by a customer requesting solution to the problem.

38. The method in claim 1, wherein the reward values within the reward matrix are set as a maximum reward values and an actual reward remitted to a contributor is determined based on the extent to which the contribution actually contributed to the solution of the problem or to the sub-problem associated with the node.

39. The method in claim 38, wherein an actual reward remitted to a contributor includes a minimum reward for making a relevant contribution substantially independent of the quality of the contribution.

40. The method in claim 39 wherein the minimum reward is identified with the problem node.

41. The method in claim 1, further comprising controlling search capability using a tree structure.

42. The method in claim 41, further comprising controlling search characteristics using a tree structure and a search technique selected from the set of search techniques consisting of generate-and-test, hill climbing, breadth-first search, depth-first search, best-first search, problem reduction techniques, back-propagation techniques, constraint satisfaction techniques, means-ends analysis, branch-and-bound techniques, nearest neighbor algorithm, divide-and conquer techniques, back-chaining techniques, minimax searches with and without cutoffs, as well as the use of heuristic search functions, and combinations thereof.

43. A method according to claim 42, wherein the use of the search method comprises limiting a number of sub-problems that may be submitted in response to the particular one of the sub-problems.

44. The method in claim 1, further comprising controlling search capability using a threaded discussion group.

45. The method in claim 44, further comprising controlling search characteristics using a tree structure and a search technique selected from the set of search techniques consisting of generate-and-test, hill climbing, breadth-first search, depth-first search, best-first search, problem reduction techniques, back-propagation techniques, constraint satisfaction techniques, means-ends analysis, branch-and-bound techniques, nearest neighbor algorithm, divide-and conquer techniques, back-chaining techniques, minimax searches with and without cutoffs, as well as the use of heuristic search functions, and combinations thereof.

46. A method according to claim 45, wherein the use of the search method comprises limiting a number of sub-problems that may be submitted in response to the particular one of the sub-problems.

47. The method in claim 1, further comprising presenting a display of elements of the problem tree in a list on a computer display screen with an associated reward matrix reward value.

48. The method in claim 1, further comprising presenting a display of open problem elements of problem trees in a list with associated reward matrix reward values.

49. The method in claim 1, further comprising presenting a display of open problem elements of problem trees in a hierarchical list with each level shown as an indented sub-list with associated reward matrix reward values.

50. The method in claim 1, wherein the problem tree comprises a problem tree for rapid test item development.

51. The method in claim 1, wherein the problem tree comprises a problem tree for rapid scoring of constructed response and essay-types of test items.

52. The method in claim 1, wherein the problem tree comprises a problem tree for rapid association of educational content with test items or sections.

53. The method in claim 1, wherein the problem tree comprises a problem tree for rapid development of new educational content.

54. The method in claim 1, wherein the problem tree comprises a problem tree for identification, development, and sharing of educational best practices.

55. The method in claim 1, wherein the problem tree comprises a problem tree for financial analysis.

56. The method in claim 1, wherein the problem tree comprises a problem tree for technical analysis.

57. The method in claim 1, wherein the problem tree comprises a problem tree for scientific analysis.

58. A method according to claim 1, wherein said problem solution contribution comprises a contributed sub-problem.

59. A method according to claim 58, further comprising assigning at least a portion of the reward value associated with the particular one of the sub-problems to the contributed sub-problem.

60. A method according to claim 1, wherein:

the complex problem has no known predetermined algorithmic solution in advance of the receipt of the complex problem to be solved; and

the method further includes the step of receiving at least one input from an external source that is used to identify a set of procedures for contributing to the solution of the complex problem.

61. A method according to claim 60, wherein the at least one input from an external source comprises a plurality of inputs from a plurality of human contributors.

62. A method according to claim 1, wherein the dynamic modification to the complex problem formulation is performed between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified, and the modification including at least one of (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems.

63. A method according to claim 62, wherein the dynamically modifying the complex problem formulation in response to a received problem solution contribution comprises both (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems.

64. A computer program product for use with a computer stored on tangible media and including a program module containing instructions that direct the operation of the computer, the instructions including instructions for:

formulating the complex problem to be solved as a problem tree having a plurality of problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem;

associating a reward matrix with said problem tree having reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with an associated sub-problem;

receiving a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor;

dynamically modifying the complex problem formulation in response to the received problem solution contribution between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified, the dynamically modifying the complex problem formulation comprises a contributor formulating and presenting at least one new sub-problem to be solved in response to problem solution contributions submitted; and

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor.

65. A computer program product according to claim 64, wherein said problem solution contribution comprises a contributed sub-problem.

66. A computer program product according to claim 65, wherein the instructions further include instructions for assigning at least a portion of the reward value associated with the particular one of the sub-problems to the contributed sub-problem.

67. A computer program product according to claim 64, wherein the dynamic modification to the complex problem formulation including at least one of (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems.

68. A computer program product according to claim 67, wherein the new sub-problems to be solved are presented as a result of a contribution by a human problem solving contributor.

69. A computer program product according to claim 67, wherein the new sub-problems to be solved are presented as a contribution by a human problem solving contributor.

70. A method for solving an educational instructional materials preparation and evaluation complex problem by a plurality of problem solving contributors acting asynchronously over an internet based network said method comprising:

formulating the educational instructional materials preparation and evaluation problem to be solved as a educational problem tree having a plurality of educational materials preparation and evaluation problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem;

associating a reward matrix with said educational problem tree having reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with an associated sub-problem;

receiving a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor;

dynamically modifying the complex problem formulation in response to the received problem solution contribution between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified, the dynamically modifying the complex problem formulation comprises a contributor formulating and presenting new sub-problems to be solved in response to problem solution contributions submitted; and

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor.

71. A method for solving a complex problem by a plurality of problem solving contributors, the method comprising:

providing a virtual problem space having a tree-based structure comprising a plurality of levels and a plurality of branches, each level comprising one or more sub-problems whose solution contributes to a solution of the complex problem;

assigning a reward value to each of said plurality of branches;

receiving a response to at least one of said sub-problems;

dynamically modifying the virtual problem space including modifying at least one level of sub-problems in response to the received response to at least one of said sub-problems between the time the virtual problem space is initially presented for problem solution contributions and the time a final solution to the complex problem in the virtual problem space is identified, the dynamically modifying the problem space comprises a contributor formulating and presenting at least one new sub-problem to be solved in response to problem solution contributions submitted;

assigning a quality to said response; and

determining a reward for the response based on the quality of the response and the reward values.

72. A method according to claim 71, wherein the quality assigned to said response is determined by an automated system.

73. A method according to claim 71, further comprising:

determining that the reward for the response exceeds an amount of available funds; and

transferring funds from a second virtual problem space having no received responses.

74. A method according to claim 71, further comprising:

re-assigning a second reward value to each of said plurality of branches, based on said determined reward.

75. A method according to claim 71, wherein:

the complex problem has no known predetermined algorithmic solution in advance of the receipt of the complex problem to be solved; and

the method further includes the step of receiving at least one input from an external source that is used to identify a set of procedures for contributing to the solution of the complex problem.

76. A method according to claim 75, wherein the at least one input from an external source comprises a plurality of inputs from a plurality of human contributors.

77. A method according to claim 71, wherein the entire virtual problem space is accessible to each of the plurality of problem solving contributors.

78. A method according to claim 71 further comprising searching the virtual problem space for a solution to the complex problem.

79. A method according to claim 78, wherein the solution is an optimal solution chosen from a plurality of possible solutions.

80. A method according to claim 78, wherein the act of searching the virtual problem space comprises utilizing the quality assigned to the response to guide the search.

81. A method according to claim 71, wherein the quality assigned to said response is determined by at least one of the problem solving contributors.

82. A method for solving a complex problem by a plurality of human problem solving contributors acting asynchronously over an interactive communication medium, said method comprising:

formulating the complex problem to be solved as a problem tree having a plurality of problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem, the complex problem having no a priori known algorithmic solution in advance of the receipt of the complex problem by the human problem solving contributors and at least one human problem solving contribution identifying a set of procedures that contributes to the solution of at least one sub-problem;

associating a reward matrix with said problem tree having quantifiable incentive reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with at least one of said sub-problems, said reward values generating a motivation for said problem solving contributors to submit a problem solving contribution to any sub-problem;

matching to a set of human contributors and presenting the formulated complex problem for solution to the matched set of human contributors over a networked interactive communication medium, wherein the matching includes comparing a database of expert contributors to all problems trees, or sub problems for which solutions are sought;

receiving a particular problem solution contribution to a particular one of the sub-problems from a particular human problem solving contributor asynchronously of other problem solving contribution over said interactive communication medium, at least one of the received problem solving contributions including a contribution identifying a set of procedures for contributing to the solution of the complex problem;

dynamically modifying the complex problem formulation in response to the received problem solution contribution between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified, the dynamically modifying the complex problem formulation comprises a contributor formulating and presenting at least one new sub-problem to be solved in response to problem solution contributions submitted; and

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor.

83. A method for solving a complex problem by a plurality of problem solving contributors acting asynchronously over an interactive communication medium, said method comprising:

formulating the complex problem to be solved as a problem tree having a plurality of problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem;

associating a reward matrix with said problem tree having reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with at least one of said sub-problems;

receiving a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor;

dynamically modifying the complex problem formulation during its solution in response to at least one received problem solution contribution, the dynamic modification to the complex problem formulation is performed between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified, and the modification including at least one of (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems;

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor; and

the dynamic modification to the complex problem formulation comprises formulating and presenting new sub-problems to be solved.

84. A method for solving a complex problem by a plurality of problem solving contributors acting asynchronously over an interactive communication medium, said method comprising:

formulating the complex problem to be solved as a problem tree having a plurality of problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem;

associating a reward matrix with said problem tree having reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with at least one of said sub-problems;

receiving a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor;

dynamically modifying the complex problem formulation during its solution in response to at least one received problem solution contribution, the dynamic modification to the complex problem formulation is performed between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified, and the modification including at least one of (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems;

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor;

the dynamic modification to the complex problem formulation comprises formulating and presenting new sub-problems to be solved; and

the new sub-problems to be solved are presented as a result of a contribution by a human problem solving contributor.

85. A method for solving a complex problem by a plurality of problem solving contributors acting asynchronously over an interactive communication medium, said method comprising:

formulating the complex problem to be solved as a problem tree having a plurality of problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem;

associating a reward matrix with said problem tree having reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with at least one of said sub-problems;

receiving a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor;

dynamically modifying the complex problem formulation during its solution in response to at least one received problem solution contribution, the dynamic modification to the complex problem formulation is performed between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified, and the modification including at least one of (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems;

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor;

the dynamic modification to the complex problem formulation comprises formulating and presenting new sub-problems to be solved; and

the new sub-problems to be solved are presented as a contribution by a human problem solving contributor.

86. A computer program product for use with a computer stored on tangible media and including a program module containing instructions that direct the operation of the computer, the instructions including instructions for:

formulating the complex problem to be solved as a problem tree having a plurality of problem nodes and problem branches coupling said problem nodes, wherein each of said plurality of problem nodes comprises either a sub-problem or a solution or partial solution to a sub-problem, and wherein contributions to the sub-problem or solutions or partial solutions to the sub-problems contribute to solution of the complex problem;

associating a reward matrix with said problem tree having reward values identified with at least some of said sub-problems, each reward value comprising a reward that will be remitted to a problem solution contributor for making a problem solving contribution associated with an associated sub-problem;

receiving a problem solution contribution to a particular one of the sub-problems from a particular problem solving contributor;

dynamically modifying the complex problem formulation in response to the received problem solution contribution between the time the complex problem is initially presented for solution and the time a final solution to the complex problem is identified; and

remitting the reward or a portion of the reward associated with the particular one of the sub-problems to the particular problem solving contributor; and

wherein the dynamically modifying the complex problem formulation including at least one of (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems; and

wherein at least one of the following is true;

(a) the dynamic modification to the complex problem formulation comprises formulating and presenting new sub-problems to be solved;

(b) the new sub-problems to be solved are presented as a result of a contribution by a human problem solving contributor, and (c) the new sub-problems to be solved are presented as a contribution by a human problem solving contributor.

87. A computer program product according to claim 86, wherein the dynamically modifying the complex problem formulation in response to a received problem solution contribution comprises both (i) modification of the sub-problems presented for solution and (ii) modification of the reward matrix values assigned to the sub-problems.