Automated intelligent data scraping and verification

• US 10,489,377 B2
• Filed: 02/11/2015
• Issued: 11/26/2019
• Est. Priority Date: 02/11/2015
• Status: Active Grant

• Current Assignee: Best Collect SA de CV
• Original Assignee: Best Collect SA de CV
• Inventors: Abraham, Gabriel Diaz Garza, Lopez, Ricardo Suarez
• Primary Examiner(s): Beausoliel, Jr., Robert W
• Assistant Examiner(s): Rayyan, Susan F
• Application Number:
  US14/619,751
• Publication Number:
  US 20160232194A1
• Time in Patent Office: 1,749 Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

A server computer system for parsing non-uniformly presented data from a variety of unique non-uniform third-party web portals can comprise a scripting processor configured to automatically execute a web-portal specific script for each of the one or more third-party web portals accessed by a network communication device. Each of the web-portal specific scripts can be configured to imitate inputs from a user input device and to automatically adapt interactions with each of the one or more third-party web portals to access and parse data elements from one or more non-uniformly available data fields. Further, the server computer system can comprise a database processor configured to compare a first set of data received from the one or more non-uniformly available data fields with a second set of data, which is stored within the local database device.

Claims

1. At a server computer system that intelligently loads and verifies data in non-uniform formats against a second set of data, a computerized method for accessing and comparing non-uniformly stored data sets, the method comprising:

identifying, within one or more data structures, specific data entries of interest, wherein the one or more data structures are stored within a first locally accessible database;

accessing a particular third-party web portal, wherein the particular third-party web portal comprises information associated with the one or more data structures;

executing a particular script that is unique to the particular third-party web portal, wherein the particular script performs the method of;

interacting with the particular third-party web portal by imitating inputs from a user input device; and

automatically adapting interactions with the particular third-party web portal to access and parse data elements from one or more non-uniformly available data fields;

comparing a first set of data received from the one or more non-uniformly available data fields with the second set of data, which is stored within a second locally accessible database; and

generating a report based upon the compared data, wherein the report comprises notifications of discrepancies between the first set of data and the second set of data.

2. The method as recited in claim 1, wherein the particular third-party web portal is selected from a set of third-party web portals based upon information associated with the one or more data structures.

3. The method as recited in claim 2, further comprising:

accessing the second set of data with a first locally stored data file, wherein the second set of data comprises one or more invoices;

identifying a data entry and associated data elements within the first set of data;

determining that one or more of the data elements differ from the data elements of a corresponding data entry within the second set of data; and

generating a report identifying the data elements that differ.

4. The method as recited in claim 1, further comprising:

receiving an indication to compare records for a particular client;

accessing within a data store the specific data entries of interest, wherein the specific data entries of interest comprise one or more third-party web portals that are associated with the particular client;

automatically accessing each of the one or more third-party web portals; and

automatically executing on each of the one or more third-party web portals a script that is unique to each respective third-party web portal.

5. The method as recited in claim 1, wherein creating the script comprises:

activating a record feature within the system, wherein the record feature captures one or more input and output signals between a user and the system;

capturing and recording input signals received from a user mouse and a user keyboard, wherein the input signals comprise one or more third-party web portal addresses;

capturing and recording output signals received from one or more third-party web portals associated with the one or more third-party web portal addresses, wherein the output signals comprise one or more webpages that are displayed in response to the input signals;

identifying one or more output signals that are associated with respective input signals; and

identifying the one or more non-uniformly available data fields.

6. The method as recited in claim 5, further comprising:

receiving, from a user, an indication regarding a particular element within a webpage, wherein the element is associated with a non-uniformly available data field; and

mapping the particular element to a particular data field within the second set of data.

7. The method as recited in claim 6, wherein executing the script comprises:

automatically generating commands to move the user mouse in accordance with the captured and recorded input signals; and

after causing one or more commands to be issued, waiting until a corresponding output signal is received, wherein the corresponding output signal corresponds with a previously captured and recorded output signal.

8. The method as recited in claim 7, further comprising:

identifying the particular element within the webpage;

parsing one or more data elements from a data field positionally associated with the particular element in the webpage;

accessing the particular data field within the second set of data; and

identifying discrepancies between the parsed one or more data elements and data elements from the particular data field.

9. The method as recited in claim 8, wherein parsing one or more data elements comprises parsing one or more data elements that are not visible on the webpage but are present within local memory.

10. The method as recited in claim 8, further comprising:

identifying one or more elements from the data field positionally associated with the particular element in the webpage that are not visible on the webpage and are not present within local memory;

automatically adapting interactions with the particular third-party web portal by identifying a data field advancement element, wherein the data field advancement element comprises a command to display additional elements within the data field;

activating the data field advancement element by imitating one or more user inputs; and

parsing one or more data elements that were not previously visible on the webpage but are now visible due to the advancement of the data field.

11. A server computer system for parsing non-uniformly presented data from a variety of unique non-uniform third-party web portals, the system comprising:

a local database device comprising one or more data structures that comprise addresses to one or more third-party web portals;

a network communication device that comprises a web browser, wherein the web browser is configured to automatically access the one or more third-party web portals;

a scripting processor configured to automatically execute a web-portal specific script for each of the one or more third-party web portals accessed by the network communication device, wherein each of the web-portal specific scripts is configured to;

interact with each of the one or more third-party web portals by imitating inputs from a user input device; and

automatically adapt interactions with each of the one or more third-party web portals to access and parse data elements from one or more non-uniformly available data fields;

a database processor configured to compare a first set of data received from the one or more non-uniformly available data fields with a second set of data that is stored within the local database device, the first set of data comprising a plurality of records according to a first format, and the second set of data comprising records formatted according to a different format, wherein at least a subset of records from the first set of data corresponds to a subset of records from the second set of data; and

a user output device configured to display a report based upon the compared data, wherein the report comprises notifications of discrepancies between the first set of data and the second set of data.

12. The server computer system as recited in claim 11, further comprising a script creation module, wherein the script creation module is configured to:

capture and record input signals received from a user mouse and a user keyboard, wherein the input signals comprise one or more third-party web portal addresses;

capture and record output signals received from one or more third-party web portals associated with the one or more third-party web portal addresses, wherein the output signals comprise one or more webpage elements that are displayed in response to the input signals;

identify one or more output signals that are associated with respective input signals; and

identify the one or more non-uniformly available data fields.

13. The server computer system as recited in claim 12, wherein the script creation module is further configured to:

receive, from a user, an indication regarding a particular element within a webpage, wherein the element is associated with a non-uniformly available data field; and

map the particular element to a particular data field within the second set of data.

14. The server computer system as recited in claim 13, wherein the scripting processor is further configured to:

automatically generate commands to move a mouse in accordance with the captured and recorded input signals; and

after causing one or more commands to be issued, wait until a corresponding output signal is received, wherein the corresponding output signal corresponds with a previously captured and recorded output signal.

15. The server computer system as recited in claim 14, wherein the scripting processor is further configured to:

identify the particular element within the webpage;

parse one or more data elements from a data field positionally associated with the particular element in the webpage;

access the particular data field within the second set of data; and

identify discrepancies between the parsed one or more data elements and data elements from the particular data field.

16. The server computer system as recited in claim 15, wherein the scripting processor is further configured to:

identify one or more elements from the data field positionally associated with the particular element in the webpage that are not visible on the webpage and are not present within local memory;

automatically adapt interactions with the particular third-party web portal by identifying a data field advancement element, wherein the data field advancement element comprises a command to display additional elements within the data field;

activate the data field advancement element by imitating one or more user inputs; and

parse one or more data elements that were not previously visible on the webpage but are now visible due to the advancement of the data field.

17. The server computer system as recited in claim 11, further comprising one or more hand-coded web-portal specific scripts that are executed by the scripting processor once an associated webpage is accessed.

18. The server computer system as recited in claim 11, further comprising:

an optical character recognition module that is configured to recognize text within one or more images on a webpage, wherein the scripting processor can incorporate information from the recognized text into one or more scripts.

19. The server computer system as recited in claim 18, wherein the recognized text comprises information relating to the one or more non-uniformly available data fields.

20. A computer program product for use at a computer system, the computer program product comprising one or more non-transitory computer storage media having stored thereon computer-executable instructions that, when executed at a processor, cause the computer system to perform a method of accessing and comparing non-uniformly stored data sets, the computer program product, comprising:

identifying, within one or more data structures, specific data entries of interest, wherein the one or more data structures are stored within a first locally accessible database;

accessing a particular third-party web portal, wherein the particular third-party web portal comprises information associated with the one or more data structures;

executing a particular script that is unique to the particular third-party web portal, wherein the particular script performs the method of;

interacting with the particular third-party web portal by imitating inputs from a user input device; and

automatically adapting interactions with the particular third-party web portal to access and parse data elements from one or more non-uniformly available data fields;

comparing a first set of data received from the one or more non-uniformly available data fields with the second set of data, which is stored within a second locally accessible database, wherein the first set of data comprises invoicing records for the specific data entries and the second set of data comprises fulfillment records for the specific data entries; and

generating a report based upon the compared data, wherein the report comprises notifications of discrepancies between the first set of data comprising invoicing information and the second set of data comprising fulfillment information.