Transmitting card objects based on categorized and scraped data

• US 10,037,375 B2
• Filed: 08/28/2015
• Issued: 07/31/2018
• Est. Priority Date: 08/28/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ben-Tzur, Jonathan
• Primary Examiner(s): Chau, Dung K
• Application Number:
  US14/838,877
• Publication Number:
  US 20170060981A1
• Time in Patent Office: 1,068 Days
• Field of Search: 707737, 707709, 707767, 707723, 707999003
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06F 40/194 Calculation of difference between files

Abstract

A server configured to receive data corresponding with a web page and/or a native application, the data comprising a uniform resource locator (URL); compare the URL included in the data with URL patterns stored in a category data store; determine a data category of the data based on the comparison; determine a data scraper based on the data category, the data scraper comprising a scraper configuration that enables the data scraper to extract information from the web page and/or the native application; scrape the web page and/or the native application by extracting information from the web page and/or the native application using the data scraper; store the extracted information in one or more data stores; used the stored information to provide search results and/or card objects.

Claims

1. A server comprising:

a network communication device;

a storage device comprising;

a category data store that stores category records corresponding with data categories, each of the stored category records comprising a category identifier (ID) that identifies the category record and a uniform resource locator (URL) pattern associated with the data category,

an entity data store that stores entity records, and

a card data store that stores card records; and

a processing device that executes computer-readable instructions that, if executed by the processing device, cause the processing device to;

receive data corresponding with a web page and/or a native application, the data comprising a uniform resource locator (URL),

compare the URL included in the data with the URL patterns stored in the category data store, one of the URL patterns indicating a portion of the URL,

determine a data category of the data based on the comparison of the URL with the URL pattern,

determine a data scraper based on the determined data category of the data, the determined data scraper comprising a scraper configuration that enables the determined data scraper to extract information from the web page and/or the native application,

scrape the web page and/or the native application by extracting information from the web page and/or the native application using the determined data scraper,

populate one or more entity records in the entity data store and/or one or more card records in the card data store based on the scraped data,

receive a card request, the card request comprising one or more search terms and/or contextual data,

determine a card record from the card data store based on the received card request,

generate a card object based on the determined card record, and

transmit the generated card object via the network communication device.

2. The server of claim 1, wherein the comparing of the URL with the URL pattern comprises using an inverted index that indexes the URL patterns to the category records.

3. The server of claim 1, wherein the comparing of the URL with the URL patterns comprises, for each URL pattern, determining a Levenshtein distance between the URL and the URL pattern.

4. The server of claim 3, wherein the determining of the data category comprises identifying the URL pattern with the lowest Levenshtein distance and selecting the category ID associated therewith.

5. The server of claim 1, wherein the determining of the data category comprises identifying the URL pattern that matches the URL and selecting the category ID associated therewith.

6. The server of claim 1,

wherein the storage device further comprises a scraper data store that stores scraper configuration files comprising:

a file ID that identifies the scraper configuration file,

a category ID that identifies the data category associated with the scraper configuration file, and

scraping parameters, and

wherein the determining of the data scraper comprises retrieving the scraper configuration file associated with the data category of the data.

7. The server of claim 1, wherein the determining of the card record comprises performing a search of the card data store and/or the entity data store based on the search terms and/or the contextual data in the received card request.

8. The server of claim 1, wherein the generating of the card object comprises:

creating a data container, and

writing information from the determined card record to the data container.

9. A server comprising:

a network communication device;

a storage device comprising;

a category data store that stores category records corresponding with data categories, each of the category records comprising a category identifier (ID) that identifies the category record and a structural pattern associated with the data category, the structural pattern indicating the structure of a web page and/or a native application,

an entity data store that stores entity records, and

a card data store that stores card records; and

a processing device that executes computer-readable instructions that, if executed by the processing device, cause the processing device to;

receive data corresponding with a web page and/or a native application, the data comprising metadata that indicates a structure of the web page and/or the native application,

compare the structure indicated by the metadata with the structural patterns stored in the category data store,

determine a data category of the data based on the comparison of the metadata with the structural patterns,

determine a data scraper based on the determined data category of the data, the determined data scraper comprising a scraper configuration that enables the determined data scraper to extract information from the web page and/or the native application,

scrape the web page and/or the native application for scraped data by extracting information from the web page and/or the native application using the determined data scraper,

populate one or more entity records in the entity data store and/or one or more card records in the card data store based on the scraped data,

receive a card request, the card request comprising one or more search terms and/or contextual data,

determine a card record from the card data store based on the received card request,

generate a card object based on the determined card record, and

transmit the generated card object via the network communication device.

10. The server of claim 9, wherein the metadata indicating the structure of the web page and/or the native application comprises metatags that refer to graphical user interface (GUI) elements displayed by the web page and/or the native application, the GUI elements comprising data fields, images, labels, buttons and/or drop-down menus.

11. The server of claim 9, wherein the comparing of the structure with the structural patterns comprises using an inverted index that indexes the structural patterns to the category records.

12. The server of claim 9, wherein the comparing of the structure with the structural patterns comprises, for each structural pattern, determining a Levenshtein distance between the structure and the structural pattern.

13. The server of claim 12, wherein the determining of the data category comprises identifying the structural pattern with the smallest Levenshtein distance and selecting the category ID associated therewith.

14. The server of claim 9, wherein the determining of the data category comprises identifying the structural pattern that matches the structure and selecting the category ID associated therewith.

15. A server comprising:

a network communication device;

a storage device comprising;

a category data store that stores category records corresponding with data categories, each of the category records comprising a category identifier (ID) that identifies the category record and metatag patterns associated with the data category, the metatag patterns indicating content displayed by a web page and/or a native application, and

an entity data store that stores entity records; and

a processing device that executes computer-readable instructions that, if executed by the processing device, cause the processing device to;

receive data corresponding with a web page and/or a native application, the data comprising metadata that includes metatag pairs, each metatag pair comprising a tag label and a value interposed between the metatag pair,

compare the tag labels with the metatag patterns stored in the category data store,

determine a data category of the data based on the comparison of the metadata with the metatag patterns,

determine a data scraper based on the determined data category of the data, the determined data scraper comprising a scraper configuration that enables the determined data scraper to extract information from the web page and/or the native application,

scrape the web page and/or the native application by extracting information from the web page and/or the native application using the determined data scraper,

populate one or more entity records in the entity data store based on the scraped data,

receive a search request, the search request comprising one or more search terms and/or contextual data,

determine one or more entity records from the entity data store based on the search request,

generate a result object based on the determined entity records, and

transmit the result object via the network communication device.

16. The server of claim 15, wherein the comparing of the tag labels with the metatag patterns comprises using an inverted index that indexes the metatag patterns to the category records.

17. The server of claim 15, wherein the determining of the data category comprises identifying the metatag pattern that matches the tag labels and selecting the category ID associated therewith.

18. The server of claim 15, wherein the metatag patterns include vocabulary from a schema that is used to markup data.

19. The server of claim 15, wherein the metatag patterns include uniform resource locators (URLs) that identify entity types, relationships between entities and/or actions.

20. A server comprising:

a network communication device;

a storage device comprising an entity data store that stores entity records; and

a processing device that executes computer-readable instructions that, if executed by the processing device, cause the processing device to;

receive data corresponding with n states of a web page and/or a native application, n being greater than one, the data comprising n uniform resource locators (URLs) associated with respective states,

compare the data of the n states with each other to determine similarity between the n states,

cluster the n states into k clusters based on the comparison, k being less than n and greater than one,

for each cluster, determine a data scraper based on a structure of the data in the cluster, scrape the states of the web page and/or the native application indicated by the URLs in the cluster, and populate one or more entity records in the entity data store using the scraped data,

receive a search request, the search request comprising one or more search terms and/or contextual data,

determine one or more entity records from the entity data store based on the search request,

generate a result object based on the determined entity records, and

transmit the generated result object via the network communication device.

21. The server of claim 20, wherein the comparing of the data of the n states comprises comparing the n URLs with each other to determine the similarity between the n URLs.

22. The server of claim 21, wherein the clustering of the n states into the k clusters comprises clustering the URLs that are similar to each other.

23. The server of claim 22,

wherein the comparing of the URLs comprises determining a Levenshtein distance between the URLs, and

wherein the clustering of the URLs comprises clustering URLs in response to the Levenshtein distance between the URLs being less than a threshold.

24. The server of claim 20,

wherein the data comprises metadata for each of the n states, the metadata indicating graphical user interface (GUI) elements of the web page and/or the native applications, and

wherein the comparing of the data comprises comparing the GUI elements.

25. The server of claim 20, wherein the clustering of the n states into k clusters comprises using k-means clustering.

26. A server comprising:

a network communication device;

a storage device comprising an entity data store that stores entity records and a card data store that stores card records; and

a processing device that executes computer-readable instructions that, if executed by the processing device, cause the processing device to;

receive data corresponding with a web page and/or a native application, the data comprising a uniform resource locator (URL), metadata indicating a structure of the web page and/or the native application, and/or content displayed by the web page and/or the native application,

determine a category of the data based on the URL of the web page and/or the native application, the structure of the web page and/or the native application indicated by the metadata, and/or the content displayed by the web page and/or the native application,

determine a scraper based on the category of the data, the scraper comprising a configuration that enables the scraper to extract information from the web page and/or the native application,

scrape the web page and/or the native application for scraped data by extracting information from the web page and/or the native application using the scraper, and

populate one or more entity records in the entity data store and/or one or more card records in the card data store based on the scraped data.

27. The server of claim 26, wherein the computer-readable instructions further cause the processing device to:

receive a card request comprising one or more search terms and/or contextual data,

determine a card record from the card data store based on the received card request,

construct a card object that includes information from the determined card record, and

transmit the constructed card object.

28. The server of claim 26,

wherein the storage device stores URL patterns associated with known categories, and

wherein the determining of the category of the data comprises:

comparing the URL of the web page and/or the native application with the URL patterns of known categories stored in the storage device;

determining that the URL of the web page and/or the native application matches one of the URL patterns stored in the storage device; and

selecting the category associated with the matching URL pattern as the category of the data.

29. The server of claim 26,

wherein the storage device stores structural patterns associated with known categories, and

wherein the determining of the category of the data comprises:

comparing the metadata indicating the structure of the web page and/or the native application with the structural patterns of known categories stored in the storage device;

determining that the structure of the web page and/or the native application matches one of the structural patterns stored in the storage device; and

selecting the category associated with the matching structural pattern as the category of the data.

30. The server of claim 26,

wherein the storage device stores categorized content associated with known categories, and

wherein the determining of the category of the data comprises:

comparing the content displayed by the web page and/or the native application with the categorized content stored in the storage device;

determining that the content displayed by the web page and/or the native application matches the categorized content associated with one of the known categories; and

selecting the category associated with the matching categorized content as the category of the data.