State Deduplication for Automated and Semi-Automated Crawling Architecture

• US 20160335286A1
• Filed: 09/29/2015
• Published: 11/17/2016
• Est. Priority Date: 05/13/2015
• Status: Abandoned Application

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Quixey, Inc.
• Inventors: DESINENI, Kalyan, SANKARANARASIMHAN, Manikandan
• Application Number:
  US14/869,810
• Publication Number:
  US 20160335286A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06F 16/9558 Details of hyperlinks; Management of linked annotations
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

A system for automated acquisition of content from an application includes state storage for storing state records. Each record includes a representation of content of a corresponding state of the application. A link tracking module controls the application to navigate to a first state of the application. A duplicate content detector calculates a representation of content of the first state and generates a comparison signal indicating whether the calculated representation matches any of the stored representations of content in the state records. The link tracking module creates a new state record in the state storage only in response to the comparison signal indicating that no match is found. The calculated representation is stored in the new state record. A scraper module, for each of the state records in the state storage, extracts text and metadata. Information based on the extracted text and metadata is stored in a data store.

Claims

1. A system for automated acquisition of content from an application, the system comprising:

a state storage module configured to store application state records, wherein each record of the application state records includes a representation of content of a corresponding application state of the application;

a link tracking module configured to control an executing instance of the application to navigate to a first application state of the application;

a duplicate content detector configured to (i) calculate a representation of content of the first application state, (ii) compare the calculated representation to the stored representations of content in the application state records, and (iii) generate a comparison signal based on the comparison, wherein;

the comparison signal indicates whether a match is found between the calculated representation and any of the stored representations of content in the application state records,

the link tracking module is configured to create a new application state record in the state storage module only in response to the comparison signal indicating that no match is found, and

the calculated representation is stored in the new application state record; and

a scraper module configured to, for each of the application state records in the state storage module, extract text and metadata from the corresponding application state, wherein information based on the extracted text and metadata is stored in a data store.

2. The system of claim 1 wherein:

each record of the application state records includes a unique identifier that uniquely identifies the corresponding application state among the application state records of the state storage module; and

the duplicate content detector is configured to calculate the representation of content of the first application state only in response to a unique identifier of the first application state not matching any of the stored unique identifiers in the application state records of the state storage module.

3. The system of claim 2 wherein the link tracking module is configured to:

in response to the comparison signal indicating that the match is found between the calculated representation and a stored representation of a second record of the application state records, add the unique identifier of the first application state to the second record; and

in response to the comparison signal indicating that no match is found, store the unique identifier of the first application state in the new application state record.

4. The system of claim 2 wherein:

for each record of the application state records, the unique identifier identifies a path followed within the executing application instance from a default state of the application to the corresponding application state; and

the unique identifier of the first application state indicates a path followed within the executing application instance from the default state of the application to the first application state.

5. The system of claim 4 wherein, for each record of the application state records, the path includes user interface interactions performed within the executing application instance in order to navigate from the default state of the application to the corresponding application state.

6. The system of claim 1 wherein:

each record of the application state records includes a user interface (UI) fingerprint of the corresponding application state; and

the duplicate content detector is configured to, in response to the calculated representation matching the stored representation of content of a second application state record;

determine a UI fingerprint of the first application state;

compare the UI fingerprint of the first application state to the stored UI fingerprint of the second application state record; and

generate the comparison signal indicating that no match is found in response to the UI fingerprint of the first application state not matching the UI fingerprint of the second application state record.

7. The system of claim 6 wherein, for each record of the application state records, the UI fingerprint includes at least one of (i) a width of a UI widget tree extracted from the corresponding application state and (ii) a depth of the UI widget tree extracted from the corresponding application state.

8. The system of claim 6 wherein, for each record of the application state records, the UI fingerprint includes a list of counts for predetermined UI widget types, each of the counts representing a number of occurrences of the predetermined UI widget type in the corresponding application state.

9. The system of claim 8 wherein the duplicate content detector is configured to generate the comparison signal indicating that no match is found in response to a percentage difference between the UI fingerprint of the first application state and the UI fingerprint of the second application state record being greater than a predetermined threshold.

10. The system of claim 1 wherein the duplicate content detector is configured to calculate the representation of the content of the first application state by calculating a hash value of the content of the first application state.

11. The system of claim 10 wherein the content of the first application state includes text displayed within the first application state.

12. The system of claim 1 wherein the link tracking module is configured to execute the instance of the application within an emulator.

13. The system of claim 1 wherein the link tracking module is configured to:

identify a set of application states reachable from the first application state, wherein each of the set of application states is reachable via a respective user interface interaction with the first application state; and

control the executing instance of the application to navigate to each of the set of application states in turn.

14. The system of claim 1 wherein:

the link tracking module is configured to identify a request made by the application to a remote server in order for the application to render the first application state; and

the duplicate content detector is configured to generate the comparison signal indicating a match between the first application state and a second application state record in response to parameters of the identified request matching stored request parameters in the second application state record.

15. A search system comprising:

the system of claim 1;

the data store;

a set generation module configured to, in response to a query from a user device, select records from the data store to form a consideration set of records;

a set processing module configured to assign a score to each record of the consideration set of records; and

a results generation module configured to respond to the user device with a subset of the consideration set of records, wherein the subset is selected based on the assigned scores, and wherein the subset identifies application states of applications that are relevant to the query.

16. A method for automated acquisition of content from an application, the method comprising:

storing application state records, wherein each record of the application state records includes a representation of content of a corresponding application state of the application;

controlling an executing instance of the application to navigate to a first application state of the application;

calculating a representation of content of the first application state;

comparing the calculated representation to the stored representations of content in the application state records;

generating a comparison signal based on the comparison, wherein the comparison signal indicates whether a match is found between the calculated representation and any of the stored representations of content in the application state records;

only in response to the comparison signal indicating that no match is found, creating and storing a new application state record for the first application state, wherein the calculated representation is stored in the new application state record; and

for each of the application state records, extracting text and metadata from the corresponding application state, wherein information based on the extracted text and metadata is stored in a data store.

17. The method of claim 16 wherein:

each record of the application state records includes a unique identifier that uniquely identifies the corresponding application state among the application state records; and

the calculating the representation of content of the first application state is performed only in response to a unique identifier of the first application state not matching any of the stored unique identifiers in the application state records.

18. The method of claim 17 further comprising:

in response to the comparison signal indicating that the match is found between the calculated representation and a stored representation of a second record of the application state records, adding the unique identifier of the first application state to the second record; and

in response to the comparison signal indicating that no match is found, storing the unique identifier of the first application state in the new application state record.

19. The method of claim 17 wherein:

for each record of the application state records, the unique identifier identifies a path followed within the executing application instance from a default state of the application to the corresponding application state; and

the unique identifier of the first application state indicates a path followed within the executing application instance from the default state of the application to the first application state.

20. The method of claim 19 wherein, for each record of the application state records, the path includes user interface interactions performed within the executing application instance in order to navigate from the default state of the application to the corresponding application state.

21. The method of claim 16 wherein:

each record of the application state records includes a user interface (UI) fingerprint of the corresponding application state; and

the method further comprises, in response to the calculated representation matching the stored representation of content of a second application state record;

determining a UI fingerprint of the first application state;

comparing the UI fingerprint of the first application state to the stored UI fingerprint of the second application state record; and

generating the comparison signal indicating that no match is found in response to the UI fingerprint of the first application state not matching the UI fingerprint of the second application state record.

22. The method of claim 21 wherein, for each record of the application state records, the UI fingerprint includes at least one of (i) a width of a UI widget tree extracted from the corresponding application state and (ii) a depth of the UI widget tree extracted from the corresponding application state.

23. The method of claim 21 wherein, for each record of the application state records, the UI fingerprint includes a list of counts for predetermined UI widget types, each of the counts representing a number of occurrences of the predetermined UI widget type in the corresponding application state.

24. The method of claim 23 further comprising generating the comparison signal indicating that no match is found in response to a percentage difference between the UI fingerprint of the first application state and the UI fingerprint of the second application state record being greater than a predetermined threshold.

25. The method of claim 16 further comprising calculating the representation of the content of the first application state by calculating a hash value of the content of the first application state.

26. The method of claim 25 wherein the content of the first application state includes text displayed within the first application state.

27. The method of claim 16 further comprising:

identifying a set of application states reachable from the first application state, wherein each of the set of application states is reachable via a respective user interface interaction with the first application state; and

controlling the executing instance of the application to navigate to each of the set of application states in turn.

28. The method of claim 16 further comprising:

identifying a request made by the application to a remote server in order for the application to render the first application state; and

generating the comparison signal indicating a match between the first application state and a second application state record in response to parameters of the identified request matching stored request parameters in the second application state record.