Grammar-based task analysis of web logs

• US 20060085788A1
• Filed: 09/29/2004
• Published: 04/20/2006
• Est. Priority Date: 09/29/2004
• Status: Active Grant

• Current Assignee: X Corp. (f/k/a Twitter, Inc.) (X Holdings Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Deshpande, Prasad Manikarao, Zbarsky, Vladimir, Amir, Arnon, Srinivasan, Savitha
• Granted Patent: US 7,694,311 B2
• Time in Patent Office: Days
• US Class Current: 718/100
• CPC Class Codes: G06F 8/30 Creation or generation of source code

Abstract

A method of detecting tasks performed by users wherein a single task is a sequence of web URLs invocation. Task patterns are detected in web logs to identify tasks performed by users and analyze task trends over time, across corporate divisions and geographies. A grammar-based framework is used to model and detect tasks from web log patterns. The framework has two components: a declarative unit—to generate a task grammar, and a processing unit—to detect tasks from access logs by generating a state machine for applying the task grammar to the tokens associated with the access records. By analyzing user tasks, rather than just URLs, useful business information can be extracted.

Claims

1. A method of defining a task grammar, said method comprising:

defining at least one task to be analyzed;

identifying at least one type of access record; and

defining a task grammar relating said task with said access record.

2. A method of defining a task grammar, according to claim 1, wherein said task grammar is implemented via at least one of the following:

a finite state machine and a regular expression search.

3. A method of defining a task grammar, according to claim 1, wherein said task grammar is a regular grammar.

4. A method of defining a task grammar, according to claim 3, wherein said regular grammar is represented in Backus Naur form.

5. A method of defining a task grammar, according to claim 1, wherein said task comprises at least one of the following:

a transaction, a session, a job, an event, a project, a chore, an operation and a form filling application.

6. A method of defining a task grammar, according to claim 1, wherein said task comprises one or more sub-tasks.

7. A method of defining a task grammar, according to claim 6, wherein said task grammar is defined in a hierarchical manner to detect said sub-tasks.

8. A method of defining a task grammar, according to claim 1, wherein said task is performed by at least one of the following:

one or more users, a plurality of machines, a plurality of applications, and accessing a plurality of logs or log files.

9. A method of defining a task grammar, according to claim 1, wherein said access record comprises at least one of the following:

a web access record, a database access record, a network access record, a URL entry, an HTTP log entry, an FTP log entry, an RTSP log entry, a network communication log entry, a packets transfer log entry, a keystroke log entry, a database log entry, a value entry into field in a user interface, a mouse clicks log entry, a user interface input log entry, an application-specific log entry, and a server log entry.

10. A method of defining a task grammar, according to claim 1, wherein said grammar detects and reports at least one of the following:

unfinished tasks, broken tasks, unsuccessful tasks, failed tasks, paused tasks, stopped tasks, and cancelled tasks.

11. A method of detecting tasks, said method comprising:

receiving an access log;

extracting access records from said access log;

labeling each of said access record with a token; and

applying a task grammar to said tokens to detect tasks.

12. A method of detecting tasks, according to claim 11, wherein said tasks are performed by at least one of the following:

one or more users, a plurality of machines, a plurality of applications, and accessing a plurality of logs or log files.

13. A method of detecting tasks, according to claim 11, wherein said tasks comprise at least one of the following:

a task that comprises one or more sub-tasks, a task that is carried over several sessions, a partial task in one session, and a group of tasks in one session.

14. A method of detecting tasks, according to claim 11, wherein said tasks comprise at least one of the following:

a transaction, a session, a job, a project, an event, a chore, an operation, and a form filling application.

15. A method of detecting tasks, according to claim 11, wherein said access record comprises at least one of the following:

a web access record, a database access record, a network access record, a URL entry, an HTTP log entry, an FTP log entry, an RTSP log entry, a network communication log entry, a packets transfer log entry, a keystroke log entry, a database log entry, a value entry into field in a user interface, a mouse clicks log entry, a user interface input log entry, an application-specific log entry, and a server log entry.

16. A method of detecting tasks, according to claim 11, wherein said tasks are associated with at least one attribute associated with at least one of the following entities:

user, user'"'"'s computer, user'"'"'s account, subject matter of the task, and application.

17. A method of detecting tasks, according to claim 16, wherein information associated with a user is retrieved from a database.

18. A method of detecting tasks, according to claim 16, wherein said tasks are analyzed based on at least one of the following said attributes:

demography, location, department, job, and title.

19. A method of detecting tasks, according to claim 11, wherein the time taken to accomplish said tasks is computed by using time associated with said access record.

20. A method of detecting tasks, according to claim 11, wherein said tasks are logged in at least one of the following:

a log file, a database, an archive, and a report.

21. A method of detecting tasks, according to claim 11, wherein said tasks are classified based on the content of said access record.

22. A system for modeling and detecting tasks contained therein, said system comprising:

a declarative unit to define a task grammar by defining at least one task to be analyzed, identifying at least one type of access record, and defining a task grammar relating said task with said access record; and

a processing unit to detect tasks from access logs by generating a parser for applying said task grammar to said tokens.

23. A system for modeling and detecting tasks, according to claim 22, wherein said parser is implemented via at least one of the following:

a finite state machine, and a regular expression search.

24. A system for modeling and detecting tasks, according to claim 22, wherein said tasks are performed by at least one of the following:

one or more users, a plurality of machines, a plurality of applications, and accessing a plurality of logs or log files.

25. A system for modeling and detecting tasks, according to claim 22, wherein said tasks comprise at least one of the following:

a task that comprises one or more sub-tasks, a task that is carried over several sessions, a partial task in one session, and a group of tasks in one session.

26. A system for modeling and detecting tasks, according to claim 22, wherein said tasks comprise at least one of the following:

a transaction, a session, a job, a project, an event, a chore, an operation, and a form filling application.

27. A system for modeling and detecting tasks, according to claim 22, wherein said access record comprises at least one of the following:

a web access record, a database access record, a network access record, a URL entry, an HTTP log entry, an FTP log entry, an RTSP log entry, a network communication log entry, a packets transfer log entry, a keystroke log entry, a database log entry, a value entry into field in a user interface, a mouse clicks log entry, a user interface input log entry, an application-specific log entry, and a server log entry.

28. A system for modeling and detecting tasks, according to claim 22, wherein said tasks are associated with at least one attribute associated with any of the following entities:

user, user'"'"'s computer, user'"'"'s account, subject matter of the task, and application.

29. A system for modeling and detecting tasks, according to claim 28, wherein information associated with a user is retrieved from a database.

30. A system for modeling and detecting tasks, according to claim 28, wherein said task is analyzed based on at least one of the following said attributes:

demography, location, department, job, and title.

31. A system for modeling and detecting tasks, according to claim 22, wherein the time taken to accomplish said tasks is computed by using time associated with said access record.

32. A system for modeling and detecting tasks, according to claim 22, wherein said tasks are logged in at least one of the following:

a log file, a database, an archive and a report.

33. A system for modeling and detecting tasks, according to claim 22, wherein said tasks are classified based on the content of said access record.

34. An article of manufacture comprising a computer usable medium having computer readable code embodied therein which implements a grammar-based framework, said medium comprising:

computer readable program code for receiving an access log;

computer readable program code for extracting access records from said access log;

computer readable program code for labeling each said access record with a token; and

computer readable program code for applying a task grammar to said tokens to detect tasks,

wherein said task grammar is defined by;

defining at least one task to be analyzed,

identifying at least one type of access record, and

relating said task with said access record.