Unguided application crawling architecture

• US 10,120,876 B2
• Filed: 09/09/2015
• Issued: 11/06/2018
• Est. Priority Date: 05/13/2015
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Desineni, Kalyan, Sankaranarasimhan, Manikandan, Singh, Brahm, Mohan, Sudhir
• Primary Examiner(s): Aspinwall, Evan
• Application Number:
  US14/849,540
• Publication Number:
  US 20160335333A1
• Time in Patent Office: 1,154 Days
• Field of Search: 707602
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06F 16/9558 Details of hyperlinks; Management of linked annotations
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

A system for automated acquisition of content from an application includes a link tracking module that controls an instance of the application executing within an emulator. For a selected state, the link tracking module controls the executing application instance to navigate to the selected state and identifies a first set of application states reachable by user interface interaction. A state storage module stores records based on the first set. A first state record includes content of a first state of the first set and a unique identifier that uniquely identifies the first state. The unique identifier indicates a path followed within the executing application instance from a default state to the first state, including corresponding user interface interaction. A scraper module, for each of the records in the state storage module, navigates to the state specified by the unique identifier using the indicated path and extracts text from the state.

Claims

1. An apparatus for automated acquisition of content from an application and improved searching of the content in response to a query of a user device, the apparatus comprising:

at least one processor, the at least one processor configured to track links, the tracking of links including;

controlling an executing instance of the application; and

for a selected state of the application;

controlling the executing application instance to navigate to the selected state, and

identifying a first set of application states reachable from the selected state, each of the first set of application states being reachable via a respective user interface interaction with the selected state,

wherein the at least one processor is further configured to store records in a state storage based on the first set of application states, a first state record including;

(i) a representation of content of a first state of the first set of application states, and

(ii) a unique identifier that uniquely identifies the first state within the records of the state storage, the unique identifier of the first state indicating a path followed within the executing application instance from a default state of the application to the first state, and the path including the user interface interaction corresponding to the first state,

wherein the at least one processor is further configured to scrape records, including, for each of the stored records, extract text and metadata from the state, information based on the extracted text and metadata being stored in a data store, and

wherein the at least one processor is further configured to provide at least one record in response to the query from the user device.

2. The apparatus of claim 1, wherein the link tracking by the at least one processor comprises executing the instance of the application within an emulator.

3. The apparatus of claim 1 wherein the unique identifier of the first state is a uniform resource identifier (URI).

4. The apparatus of claim 1 wherein the unique identifier of the first state is based on a concatenation, in order, of each user interface interaction triggered when navigating from the default state to the first state.

5. The apparatus of claim 1 wherein the default state is a home state of the application and wherein, upon execution, the application presents the home state to a user of the application.

6. The apparatus of claim 1 wherein the representation of the content of the first state requires less storage space than does the content of the first state.

7. The apparatus of claim 6 wherein the representation of the content of the first state is based on a calculated hash of the content of the first state.

8. The apparatus of claim 1 wherein:

the first state record includes a second unique identifier that uniquely identifies the first state within the records of the state storage; and

the unique identifier of the first state indicates a second path followed within the application from the default state to the first state.

9. The apparatus of claim 8 wherein the at least one processor is further configured to detect duplicate content, the detection comprising:

determining whether the path and the second path both arrive at the first state, and

in response to determining that the path and the second path both arrive at the first state, adding the second unique identifier to the first state record instead of creating a second state record.

10. The apparatus of claim 8, wherein the at least one processor is further configured to select a path, the selection of a path including selecting a single unique identifier for each of the records for use by the at least one processor when scraping records.

11. The apparatus of claim 10 wherein the at least one processor is further configured to, when scraping records, for each of the records in the state storage, (i) navigate to the state specified by the selected single unique identifier using the path indicated by the selected single unique identifier and (ii) extract text and metadata from the state.

12. The apparatus of claim 10 wherein the selection of a path comprises selecting, for each of the records, the single unique identifier that identifies a fastest path to the respective state.

13. The apparatus of claim 1, wherein the link tracking by the at least one processor comprises:

identifying when an application programming interface call is available for navigating to the first state; and

storing a second unique identifier in the first state record, wherein the second unique identifier indicates the application programming interface call.

14. The apparatus of claim 13 wherein the link tracking by the at least one processor comprises, for a second state reachable from the first state via a first user interface interaction:

storing a second state record in the state storage including a unique identifier based on the application programming interface call and the first user interface interaction; and

adding a second unique identifier to the second state record, wherein the second unique identifier indicates the first user interface interaction and user interface interactions followed from the default state to the first state.

15. The apparatus of claim 1, wherein the providing of at least one record comprises:

in response to the query from the user device, selecting records from the data store to generate a set of records by forming a consideration set of records;

processing the consideration set of records by assigning a score to each record of the consideration set of records; and

generating results by responding to the user device with a subset of the consideration set of records,

wherein the subset is selected based on the assigned scores, and

wherein the subset identifies application states of applications that are relevant to the query.

16. A method for automated acquisition of content from an application and provision of content for searching by a user device, the method comprising:

executing, using at least one processor of an apparatus for automated content acquisition and provision, an instance of the application;

for a selected state of the application;

(i) controlling, using the at least one processor, the executing application instance to navigate to the selected state, and

(ii) identifying, using the at least one processor, a first set of application states reachable from the selected state, each of the first set of application states being reachable via a respective user interface interaction with the selected state;

storing records in a state storage based on the first set of application states, wherein a first state record includes;

(i) a representation of content of a first state of the first set of application states, and

(ii) a unique identifier that uniquely identifies the first state within the stored records, the unique identifier of the first state indicating a path followed within the executing application instance from a default state of the application to the first state, and the path including the user interface interaction corresponding to the first state;

for each of the stored records, extracting, using the at least one processor, text and metadata from the state; and

storing information based on the extracted text and metadata in a data store.

17. The method of claim 16 wherein the instance of the application is executed within an emulator.

18. The method of claim 16 wherein the unique identifier of the first state is a uniform resource identifier (URI).

19. The method of claim 16 further comprising forming the unique identifier of the first state by concatenating, in order, each user interface interaction triggered when navigating from the default state to the first state.

20. The method of claim 16 wherein the default state is a home state of the application and wherein, upon execution, the application presents the home state to a user of the application.

21. The method of claim 16 wherein the representation of the content of the first state requires less storage space than does the content of the first state.

22. The method of claim 21 further comprising generating the representation of the content of the first state by calculating a hash of the content of the first state.

23. The method of claim 16 wherein:

the first state record includes a second unique identifier that uniquely identifies the first state within the stored records; and

the unique identifier of the first state indicates a second path followed within the application from the default state to the first state.

24. The method of claim 23 further comprising:

determining whether the path and the second path both arrive at the first state; and

in response to determining that the path and the second path both arrive at the first state, adding the second unique identifier to the first state record instead of creating a second state record.

25. The method of claim 23 further comprising selecting a single unique identifier for each of the records for use in the extracting.

26. The method of claim 25 further comprising, for each of the stored records, navigating to the state specified by the selected single unique identifier using the path indicated by the selected single unique identifier before extracting text and metadata from the state.

27. The method of claim 25 wherein the selecting includes, for each of the records, selecting the single unique identifier that identifies a fastest path to the respective state.

28. The method of claim 16 further comprising:

identifying when an application programming interface call is available for navigating to the first state; and

storing a second unique identifier in the first state record, wherein the second unique identifier indicates the application programming interface call.

29. The method of claim 28 further comprising, for a second state reachable from the first state via a first user interface interaction:

storing a second state record including a unique identifier based on the application programming interface call and the first user interface interaction; and

adding a second unique identifier to the second state record, wherein the second unique identifier indicates the first user interface interaction and user interface interactions followed from the default state to the first state.

30. The method of claim 16, further comprising:

in response to receiving a query from a user device, selecting records from the data store to form a consideration set of records;

assigning a score to each record of the consideration set of records; and

responding to the user device with a subset of the consideration set of records, wherein the subset is selected based on the assigned scores, and wherein the subset identifies application states of applications that are relevant to the query.

31. A non-transitory computer-readable medium storing processor-executable instructions configured to perform the method of claim 16.