Method and apparatus for testing automated valuation models

• US 20060122918A1
• Filed: 12/08/2004
• Published: 06/08/2006
• Est. Priority Date: 12/08/2004
• Status: Active Grant

• Current Assignee: Corelogic Solutions LLC (CoreLogic Incorporated)
• Original Assignee: Corelogic Solutions LLC (CoreLogic Incorporated)
• Inventors: Walker, Robert L., Graboske, Benjamin
• Granted Patent: US 8,001,024 B2
• Time in Patent Office: Days
• US Class Current: 705/30
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 30/0278 Product appraisal
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes
• CPC Class Codes: G06Q 40/12 Accounting
• CPC Class Codes: G06Q 50/16 Real estate

Abstract

A method and apparatus for real time testing of automated valuation models using various indicators of accuracy. These indicators are then weighted according to their value as indicators of accuracy using individualized weighting factors or an equation. A ranking is then computed based upon the factors and their weights. This method is preformed continuously, so as to achieve real-time or periodically updated automated valuation model accuracy rankings.

Claims

1. A computer-based method of ranking the accuracy of automated valuation models comprising the steps of:

receiving new valuation data for a subject property;

requesting an automated valuation of said subject property from a plurality of automated valuation models;

comparing said new valuation data for said subject property with said automated valuation of said subject property; and

computing a ranking of each of said plurality of automated valuation models according to predetermined accuracy criteria.

2. The method of claim 1, wherein said predetermined criteria include indicators of accuracy based upon said new valuation data and said automated valuation.

3. The method of claim 2, wherein said computing step further includes the weighting of said indicators of accuracy based upon said indicators value as indicators of accuracy.

4. The method of claim 1, further comprising the step of updating a database with said ranking of each of said plurality of automated valuation models.

5. The method of claim 1, further comprising the step of updating a database with said new valuation data.

6. The method of claim 1, further comprising the step of updating a database with said automated valuation.

7. The method of claim 1, wherein said automated valuation is requested and compared prior to each of said plurality of automated valuation models being updated with said new valuation data.

8. The method of claim 1, wherein said ranking of the accuracy of automated valuation models is performed in real time.

9. The method of claim 1, wherein said ranking of the accuracy of automated valuation models is performed periodically.

10. The method of claim 1, further comprising the step of updating a database with said automated valuation of said subject property.

11. The method of claim 1, wherein said new valuation data is property sale data.

12. The method of claim 1, wherein said new valuation data is an appraisal.

13. The method of claim 1, wherein said new valuation data is an automated valuation.

14. The method of claim 1, wherein said requesting step is performed in real time.

15. The method of claim 1, wherein said requesting step is performed at predetermined time intervals.

16. The method of claim 4, wherein said updating step is performed in real time.

17. The method of claim 4, wherein said updating step is performed at predetermined time intervals.

18. The method of claim 5, wherein said updating step is performed in real time.

19. The method of claim 5, wherein said updating step is performed at predetermined time intervals.

20. The method of claim 6, wherein said updating step is performed in real time.

21. The method of claim 6, wherein said updating step is performed at predetermined time intervals.

22. The method of claim 1 wherein said comparing step is performed using a percentage of properties for which said automated valuation is returned.

23. The method of claim 1 wherein said comparing step is performed using a percentage of properties for which said valuation data is within a predetermined percentage of said automated valuation.

24. The method of claim 1 wherein said comparing step is performed using a percentage of properties for which said valuation data is outside of a predetermined percentage of said automated valuation.

25. The method of claim 1 wherein said comparing step is performed using a median error percentage of said automated valuation with reference to said valuation data.

26. The method of claim 1 wherein said comparing step is performed using a mean squared error percentage of said automated valuation with reference to said valuation data.

27. The method of claim 1 wherein said comparing step is performed using an absolute mean error percentage of said automated valuation with reference to said valuation data.

28. The method of claim 22 wherein said comparing step further comprises the step of applying a weighting factor to said percentage of properties.

29. The method of claim 23 wherein said comparing step further comprises the step of applying a weighting factor to said percentage of properties.

30. The method of claim 24 wherein said comparing step further comprises the step of applying a weighting factor to said percentage of properties.

31. The method of claim 25 wherein said comparing step further comprises the step of applying a weighting factor to said median error percentage.

32. The method of claim 26 wherein said comparing step further comprises the step of applying a weighting factor to said mean squared error percentage.

33. The method of claim 27 wherein said comparing step further comprises the step of applying a weighting factor to said absolute mean error percentage.

34. The method of claim 22 wherein said comparing step further comprises the step of using an equation to weight the importance of said percentage of properties.

35. The method of claim 23 wherein said comparing step further comprises the step of using an equation to weight the importance of said percentage of properties.

36. The method of claim 24 wherein said comparing step further comprises the step of using an equation to weight the importance of said percentage of properties.

37. The method of claim 25 wherein said comparing step further comprises the step of using an equation to weight the importance of said median error percentage.

38. The method of claim 26 wherein said comparing step further comprises the step of using an equation to weight the importance of said mean squared error percentage.

39. The method of claim 27 wherein said comparing step further comprises the step of a using an equation to weight the importance of said absolute mean error percentage.

40. The method of claim 1 wherein said computing step is performed using the equation:

Score=f*((h+c)−

(|m|+a+s+o))

Where

f=multiplier factor

h=hit rate percentage

c=automated valuation model accuracy

m=median error to the reference value

a=absolute mean error

s=mean squared error

o=percentage incidence of outlier values.

41. The method of claim 1 further comprising the step of categorizing said subject property according to geographic area.

42. The method of claim 1 further comprising the step of categorizing said subject property according to economic tier.

43. The method of claim 1 further comprising the step of categorizing said subject property according to property type.

44. The method of claim 1 further comprising the steps of

updating a database with said ranking;

updating a database with said new valuation data; and

updating a database with said automated valuations.

45. A computer-based apparatus for ranking the accuracy of automated valuation models comprising:

input means for receiving new valuation data for a subject property;

connection means connected to said input means for requesting an automated valuation of said subject property from a plurality of automated valuation models;

comparison means connected to said connection means for comparing said new valuation data for said subject property with said automated valuation of said subject property; and

calculation means connected to said comparison means for calculating a ranking of each of said plurality of automated valuation models according to predetermined accuracy criteria.

46. The apparatus of claim 45, wherein said predetermined criteria include indicators of accuracy based upon said new valuation data and said automated valuation.

47. The apparatus of claim 45, wherein said calculation means further includes weighting means for weighting said indicators of accuracy based upon said indicators value as indicators of accuracy.

48. The apparatus of claim 45, further comprising a output means connected to said input means for updating a database with said ranking of each of said plurality of automated valuation models.

49. The apparatus of claim 45 further comprising an output means connected to said input means for updating a database with said new valuation data.

50. The apparatus of claim 45, further comprising an output means connected to said input means for updating a database with said automated valuation of said subject property.

51. The apparatus of claim 45, wherein said comparison means compares said new valuation data with said automated valuation prior to each of said plurality of automated valuation models being updated with said new valuation data.

52. The apparatus of claim 45, further comprising automation means connected to said calculation means for ranking of the accuracy of automated valuation models in real time.

53. The apparatus of claim 45, further comprising automation means connected to said calculation means for ranking of the accuracy of automated valuation models at predetermined time intervals.

54. The apparatus of claim 45, wherein said new valuation data is property sale data.

55. The apparatus of claim 45, wherein said new valuation data is an appraisal.

56. The apparatus of claim 45, wherein said new valuation data is an automated valuation.

57. The apparatus of claim 50, further comprising automation means connected to said output means for updating said database in real time.

58. The apparatus of claim 50, further comprising automation means connected to said output means for performing said updating step at predetermined time intervals.

59. The apparatus of claim 45 wherein said comparison means performs a comparison using a percentage of properties for which said automated valuation is returned.

60. The apparatus of claim 45 wherein said comparison means performs a comparison using a percentage of properties for which said new valuation data is within a predetermined percentage of said automated valuation.

61. The apparatus of claim 45 wherein said comparison means performs a comparison using a percentage of properties for which said new valuation data is outside of a predetermined percentage of said automated valuation.

62. The apparatus of claim 45 wherein said comparison means performs a comparison using a median error percentage of said automated valuation with reference to said new valuation data.

63. The apparatus of claim 45 wherein said comparison means performs a comparison using a mean squared error percentage of said automated valuation with reference to said valuation data.

64. The apparatus of claim 45 wherein said comparison means performs a comparison using an absolute mean error percentage of said automated valuation with reference to said new valuation data.

65. The apparatus of claim 59 wherein said calculation means further comprises an application means for applying a weighting factor to said percentage of properties.

66. The apparatus of claim 60 wherein said calculation means further comprises an application means for applying a weighting factor to said percentage of properties.

67. The apparatus of claim 61 wherein said calculation means further comprises an application means for applying a weighting factor to said percentage of properties.

68. The apparatus of claim 62 wherein said calculation means further comprises an application means for applying a weighting factor to said median error percentage.

69. The apparatus of claim 63 wherein said calculation means further comprises an application means for applying a weighting factor to said mean squared error percentage.

70. The apparatus of claim 64 wherein said calculation means further comprises an application means for applying a weighting factor to said absolute mean error percentage.

71. The apparatus of claim 59 wherein said calculation means further comprises an application means for using an equation to weight the importance of said percentage of properties

72. The apparatus of claim 60 wherein said calculation means further comprises an application means for using an equation to weight the importance of said percentage of properties.

73. The apparatus of claim 61 wherein said calculation means further comprises an application means for using an equation to weight the importance of said percentage of properties.

74. The apparatus of claim 62 wherein said calculation means further comprises an application means for using an equation to weight the importance of said median error percentage.

75. The apparatus of claim 63 wherein said calculation means further comprises an application means for using an equation to weight the importance of said mean squared error percentage.

76. The apparatus of claim 64 wherein said calculation means further comprises an application means for using an equation to weight the importance of said absolute mean error percentage.

77. The apparatus of claim 45 wherein said calculation means uses the equation:

Score=f*((h+c)−

(m+a+s+o))

Where

f=multiplier factor

h=hit rate percentage

c=automated valuation model accuracy

m=median error to the reference value

a=absolute mean error

s=mean squared error

o=percentage incidence of outlier values.

78. The apparatus of claim 45 further comprising a categorization means connected to said calculation means for categorizing said subject property according to geographic region.

79. The apparatus of claim 45 further comprising a categorization means connected to said calculation means for categorizing said subject property according to economic tier.

80. The apparatus of claim 45 further comprising a categorization means connected to said calculation means for categorizing said subject property according to property type.

81. The apparatus of claim 45 further comprising an output means for updating a database with said ranking, updating a database with said new valuation data and updating a database with said automated valuations.