Job categorization system and method

• US 7,702,674 B2
• Filed: 06/30/2005
• Issued: 04/20/2010
• Est. Priority Date: 03/11/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Khanna, Sandeep, Ting, Joseph, Hyder, Adam
• Primary Examiner(s): Cottingham; John R
• Assistant Examiner(s): Rayyan; Susan F
• Application Number:
  US11/173,470
• Publication Number:
  US 20060212466A1
• Time in Patent Office: 1,755 Days
• Field of Search: 707 1- 3, 707/10, 707/104.1, 707101-102, 705/11
• US Class Current: 707/710
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

A computer system and method for capture, managing and presenting data obtained from various often unrelated postings via the Internet for examination by a user. This system includes a scraping module having one or more scraping engines operable to scrape information data sets from listings on the corporate sites and web sites, direct feeds, and other sources, wherein the scraping module receives and stores the scraped listing information data sets in a database. The system also has a management platform coordinating all operation of and communication between the sources, system administrators and processing modules. The processing modules in the platform include scraping management module analyzing selected scraped data stored in the database, and a categorization module that examines and categorizes each data set stored in the database into one or more of a predetermined set of categories and returns categorized data sets to the database.

Claims

1. A computer system comprising:

a plurality of processors for managing capture and processing of job listing information data captured through a data network from a plurality of job related sources for compilation into a searchable data structure;

an administrative portal module implemented on one of the plurality of processors for providing system administration and operational control through a network interface;

a scraping management module implemented on one of the plurality of processors for coordinating operation of and communication between one or more job scraping engines to obtain scraped job information data sets from corporate career sites and job boards identified by a site management module in the administrative portal module and store the scraped data sets in a database, each scraped data set comprising data fields;

a quality management module implemented on one of the plurality of processors coupled to the scraping management module for comparing the data fields of each scraped job data set stored in the database with predetermined quality rules, wherein the quality rules include document rules, and wherein if the data set fails one or more of the document rules, the data set will not be indexed in the database and the data set will be flagged for quality review;

a job listing data categorization module implemented on one of the plurality of processors operable to examine and categorize each job data set stored in the database into one or more predetermined job categories based upon a volume of the scraped job data sets and return categorized job data sets to the database; and

a search bank synchronizer implemented on one of the plurality of processors for communicating with the database for compiling and transferring categorized job data sets from the database to a job search bank.

2. The system according to claim 1 wherein the data network includes the Internet.

3. The system according to claim 1 wherein the categorization module comprises:

a job categorization database; and

a categorization module determining a confidence value in each predetermined job category for each scraped job listing information data set by comparing text of each scraped job data set with previously categorized job data text in the job categorization database.

4. The system according to claim 3 wherein the administration portal comprises a categorization review module permitting a reviewer to verify categorizations determined by the document categorization platform service in the categorization module.

5. The system according to claim 3 wherein each job data set returned to the database includes an assigned job category determined by the categorization module and an assigned confidence value for that category.

6. The system according to claim 5 wherein each data set returned to the database further includes a confidence value for each predetermined job category.

7. The system according to claim 3 wherein each job data set returned to the database includes a manual review flag set if the assigned confidence value is below a predetermined threshold value.

8. The system according to claim 7 wherein the administration portal comprises a categorization review module permitting a reviewer to verify categorizations determined by the categorization module.

9. The system according to claim 1 wherein the administrative portal further comprises a quality review module communicating with the quality management module permitting a reviewer to manually examine job data sets that have been flagged.

10. A method of obtaining, handling and compiling job information data sets comprising:

scraping, by a processor, job information data sets from one or more job listings on one or more corporate career sites or job boards available through the Internet;

storing, by the processor, a job data set corresponding to each scraped job listing found in a database, each job data set comprising data fields;

comparing, by the processor, each data field of each scraped data set stored in the database with predetermined quality criteria, wherein the quality criteria include document rules, and wherein if the data set fails one or more of the document rules, the data set will not be indexed in the database and the data set will be flagged for quality review;

categorizing, by the processor, each data set stored in the database into one or more predetermined job categories based upon a volume of the scraped data sets and returning the categorized job information data set to the database; and

communicating, by the processor, with the database for compiling and transferring categorized job data sets from the database to a job search bank.

11. The method according to claim 10 further comprising obtaining job information data sets from one or more of customer sites through an XML feed.

12. The method according to claim 10 wherein the categorizing operation further comprises:

assigning a confidence value for each job information data set for each of the predetermined job categories.

13. The method according to claim 10 wherein the categorizing operation comprises:

comparing text of each scraped job information data set with text of previously categorized job information data sets in a job categorization database; and

determining a confidence value in each predetermined category for each scraped data set.

14. The method according to claim 13 further comprising:

flagging each categorized scraped data set that has a confidence value below a predetermined value for manual review; and

providing a manual review module permitting a reviewer to verify any flagged categorizations through an administration portal.

15. The method according to claim 10 further comprising assigning a confidence value for the job category assigned to each data set returned to the database.

16. The method according to claim 15 further comprising flagging any data set returned to the database having an assigned confidence level below a predetermined threshold.

17. The method according to claim 10 further comprising:

transferring selected categorized job information data sets from the job search bank through a web client server cluster to a job seeker in response to a query by the job seeker.

18. The method according to claim 10 wherein the scraping operation further comprises:

accessing one of the job boards or corporate career sites through the Internet;

flagging any scraped job information data set comprising data fields not meeting the predetermined quality criteria;

permitting a manual review of flagged job information data sets returned to the database, and the categorizing operation further comprises;

comparing data in each scraped job information data set with previously categorized job data set data in a categorization database; and

determining a confidence value in each predetermined job category for each scraped job information data set.

19. The method according to claim 18 further comprising:

flagging each categorized scraped data set that has a confidence value below a predetermined value for manual review; and

providing a manual review module in an administration portal permitting a reviewer to verify any flagged categorizations.

20. The method according to claim 18 further comprising transferring selected categorized data sets from the search bank through a web server to a user in response to a query by the user.

23. The method of claim 10 wherein the categorizing each data set stored in the database into one or more predetermined job categories further comprises choosing the value c′

for category c that maximizes p(c|x), expressed as c′

=arg max_c p(c|x) where x is a feature vector of the data set and p(c|x) is a conditional probability.

24. The method of claim 23 wherein the categorizing each data set stored in the database into one or more predetermined job categories further comprises computing a discriminant function d(x,c).

21. A computer readable medium tangibly encoding a computer program of instructions for executing a computer process for scraping job description data from corporate career sites and job boards, the computer process comprising:

scraping listing information data from one or more listings on sites available through the Internet;

storing a scraped data set corresponding to each scraped listing information in a database, each scraped data set comprising data fields;

comparing data fields of each scraped data set stored in the database with predetermined quality criteria, wherein the quality criteria include document rules, and wherein if the data set fails one or more of the document rules, the data set will not be indexed in the database and the data set will be flagged for quality review;

categorizing each data set stored in the database into one or more predetermined categories based upon a volume of the scraped data sets and returning the categorized data set to the database; and

communicating with the database for compiling and transferring categorized job data sets from the database to a job search bank.

22. The computer readable medium of claim 21 wherein the process further comprises:

flagging any scraped data set comprising data fields not meeting the predetermined quality criteria;

permitting a manual review of flagged data sets returned to the database, and wherein the categorizing operation further comprises;

comparing text in each scraped data set with previously categorized data set text in a categorization database; and

determining a confidence value in each predetermined category for each scraped data set.