Adaptive gathering of structured and unstructured data system and method

• US 9,466,066 B2
• Filed: 06/01/2015
• Issued: 10/11/2016
• Est. Priority Date: 07/25/2012
• Status: Active Grant

• Current Assignee: Avalara Incorporated
• Original Assignee: Indix Corporation (Avalara Incorporated)
• Inventors: Kalikivayi, Satyanarayana Rao, Muppalla, Rajesh, Parthasarathy, Sanjay
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US14/726,707
• Publication Number:
  US 20150339682A1
• Time in Patent Office: 498 Days
• Field of Search: 707/609, 707/687, 707/690, 707/697, 707/790, 707/821
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data

Abstract

Content is obtained from a webpage accessed via a URI, which URI is obtained from a URI queue. The content is parsed for price and product information according to a parse map, with the resulting parse result being stored. The priority of URIs in the URI queue is adjusted based on analysis of the parse result for changes in price and product attributes and according to other criteria. The parse map may be one associated with the URI or a general purpose parse maps. The parse result may be validated by human- and machine-based systems, including by graphically labeling price and product information in the content for human confirmation or correction.

Claims

1. A computer implemented method of managing a prioritized Uniform Resource Identifier (“

URI”

) queue comprising;

by a first computer processor, utilizing a first URI to access a first content in a first communication session with a first webserver associated with a first merchant at a first URI access time and utilizing the first URI to access a second content in a second communication session with the first webserver at a second URI access time subsequent to the first URI access time;

by the first or a second computer processor, parsing the first content for first price and product attribute values, saving the result as a first parse result associated with a first product in a first computer memory, and associating the first parse result with a first URI-specific product identifier;

by the first or the second computer processor, parsing the second content for second price and product attribute values, saving the result as a second parse result associated with the first product in the first computer memory, and associating the second parse result with a first merchant-specific identifier;

by the first or the second computer processor, determining at least one difference between the first price and product attribute values in the first parse result and the second price and product attribute values in the second parse result; and

setting a time to next check of the first URI in the prioritized URI queue at least according to the determined difference between the first price and product attribute values in the first parse result and the second price and product attribute values in the second parse result.

7. The method according to claim 1, wherein determining at least one difference between the first price and product attribute values in the first parse result and the second price and product attribute values in the second parse result comprises determining a change interval.

8. The method according to claim 1, further comprising utilizing a second URI to access a third content in a third communication session with a second webserver associated with a second merchant, parsing the third content for third price and product attribute values, saving the result as a third parse result associated with a second product in the first computer memory, and associating the third parse result with a second merchant-specific product identifier.

9. The method according to claim 1, further comprising determining that a price attribute of the second product has changed over time and moving the time to check of the first URI up in position in the prioritized URI queue.

10. The method according to claim 1, further comprising determining that there is an interest in a product or a product category associated with the URI-specific product identifier and moving the time to check of the first URI up in position in the prioritized URI queue.

14. The method according to claim 7, wherein the change interval is calculated at least in part by subtracting from a first time-date of the first parse result a second time-date of the second parse result and dividing this by a number of changes between the first and second parse results.

15. The method according to claim 8, further comprising, for a set of parse results for a set of content obtained from a set of webservers, wherein the set of parse results comprises the first and third parse result, wherein the set of webservers comprises the first and second webservers, and wherein the set of parse results comprises a set of price and product attributes comprising a first and a second product attribute, identifying at least one product attribute cluster in the set of parse results, and assigning a URI-independent identifier to the product attribute cluster, wherein the product attribute cluster comprises parse results comprising the first product attribute value within a cluster range.

16. The method according to claim 10, wherein the interest comprises search queries submitted by client computer devices.

18. The method according to claim 15, further comprising weighting the first and second product attributes by different weighting factors and wherein identifying the product attribute cluster in the set of parse results based at least in part on the first and the second product attribute in the set of product attributes comprises identifying the product attribute cluster based at least in part on the weighted first and second product attributes.

19. The method according to claim 15, wherein the first and second products are in the same product attribute cluster and are associated with a same merchant-independent identifier.

20. The method according to claim 19, further comprising determining that a price associated with a parse result for a category has changed over time and, in response, moving the time to check of the first URI up in position in the prioritized URI queue.

2. A computer apparatus for managing a prioritized Uniform Resource Identifier (“

URI”

) queue comprising;

a first computer processor and memory comprising a crawl agent software routine and a second computer processor and memory comprising a URI queue manager software routine;

wherein the crawl agent software routine is to utilize a first URI to access a first content in a first communication session with a first webserver associated with a first merchant at a first URI access time and utilize the first URI to access a second content in a second communication session with the first webserver at a second URI access time subsequent to the first URI access time;

wherein the URI queue manager software routine is to;

parse the first content for first price and product attribute values, save as a first parse result associated with a first product in a first computer memory, and associate the first parse result with a first URI-specific product identifier;

parse the second content for second price and product attribute values, save as a second parse result associated with the first product in the first computer memory, and associate the second parse result with a first merchant-specific identifier;

determine at least one difference between the first price and product attribute values in the first parse result and the second price and product attribute values in the second parse result; and

set a time to next check of the first URI in the prioritized URI queue according to the determined difference between the first price and product attribute values in the first parse result and the second price and product attribute values in the second parse result.

3. One or more computer-readable media comprising instructions that cause a computer device, in response to execution of the instructions by one or more processors of the computer device to perform the following:

utilize a first URI to access a first content in a first communication session with a first webserver associated with a first merchant at a first URI access time and utilize the first URI to access a second content in a second communication session with the first webserver at a second URI access time subsequent to the first URI access time;

parse the first content for first price and product attribute values, save as a first parse result associated with a first product in a first computer memory, and associate the first parse result with a first URI-specific product identifier;

parse the second content for second price and product attribute values, save as a second parse result associated with the first product in the first computer memory, and associate the second parse result with a first merchant-specific identifier;

determine at least one difference between the first price and product attribute values in the first parse result and the second price and product attribute values in the second parse result; and

set a time to next check of the first URI in the prioritized URI queue at least according to the determined difference between the first price and product attribute values in the first parse result and the second price and product attribute values in the second parse result.

4. A computer-implemented method for determining if products on different websites can be assigned same or different product identifiers, comprising:

by a first computer processor, receiving a first content from a first webpage from a first webserver associated with a first merchant,

by the first computer processor, parsing the first content for a first set of product attribute values, saving as a first parse result, and associating the first parse result with a first URI-specific product identifier;

by the first computer processor, receiving a second content from a second webpage from a second webserver associated with a second merchant;

by the first computer processor, parsing the second content for a second set of product attribute values, saving as a second parse result, and associating the second parse result with a second URI-specific product identifier;

for a set of parse results comprising the first and second parse result, identifying at least one product attribute cluster in the set of parse results, and assigning a URI-independent product identifier to the product attribute cluster, wherein the product attribute cluster comprises parse results comprising a first product attribute value within a cluster range.

11. The method according to claim 4, wherein assigning the URI-independent identifier to the product attribute cluster comprises identifying, within all URI-specific product identifiers associated with the product attribute cluster, a URI-specific product identifier associated with a largest number of product attribute values in the product attribute cluster, and assigning such URI-specific product identifier as the URI-independent product identifier.

12. The method according to claim 4, further comprising, prior to identifying at least one product attribute cluster in the set of parse results, weighting the product attribute values in the set of product attribute values.

13. The method according to claim 4, further comprising, for the first and second parse results, determining at least one category associated therewith and wherein identifying at least one product attribute cluster in the set of parse results comprises selecting a set of product attributes shared by products in the at least one category, organizing the set of parse results by a set of product attributes shared within the at least one category, and identifying at least one product attribute cluster in the set of parse results further according to the set of product attributes shared within the at least one category.

17. The method according to claim 13, wherein determining at least one category associated with the first and second parse results comprises, for each parse result, selecting a set of attribute extraction algorithms, executing the attribute extraction algorithms on the parse result, ranking the attribute extraction algorithms within the set based at least on data-type mismatches, and assigning the at least one category to the parse result based on a highest ranking attribute extraction algorithm.

5. A computer apparatus for determining if products on different websites can be assigned same or different product identifiers, comprising:

a first computer processor and memory comprising an identifier assignment routine wherein the identifier assignment routine is to;

receive a first content from a first webpage from a first webserver associated with a first merchant;

parse the first content for a first set of product attribute values, save as a first parse result, and associate the first parse result with a first URI-specific product identifier;

receive a second content from a second webpage from a second webserver associated with a second merchant;

parse the second content for a second set of product attribute values, save as a second parse result, and associate the second parse result with a second URI-specific product identifier;

for a set of parse results comprising the first and second parse result, identify at least one product attribute cluster in the set of parse results, and assign a URI-independent product identifier to the product attribute cluster, wherein the product attribute cluster comprises parse results comprising a first product attribute value within a cluster range.

6. One or more computer-readable media comprising instructions that cause a computer device, in response to execution of the instructions by one or more processors of the computer device to perform the following:

receive a first content from a first webpage from a first webserver associated with a first merchant;

parse the first content for a first set of product attribute values, save as a first parse result, and associate the first parse result with a first URI-specific product identifier;

receive a second content from a second webpage from a second webserver associated with a second merchant;

parse the second content for a second set of product attribute values, save as a second parse result, and associate the second parse result with a second URI-specific product identifier;

for a set of parse results comprising the first and second parse result, identify at least one product attribute cluster in the set of parse results, and assign a URI-independent product identifier to the product attribute cluster, wherein the product attribute cluster comprises parse results comprising a first product attribute value within a cluster range.