System and method for main page identification in web decoding

• US 10,547,691 B2
• Filed: 01/04/2016
• Issued: 01/28/2020
• Est. Priority Date: 07/31/2011
• Status: Active Grant

• Current Assignee: Cognyte Technologies Israel Ltd. (Cognyte Software Ltd.)
• Original Assignee: Verint Systems Limited (Verint Systems Incorporated)
• Inventors: Weissblum, Ofer
• Primary Examiner(s): Trost, IV, William G
• Assistant Examiner(s): Moorad, Imran
• Application Number:
  US14/986,950
• Publication Number:
  US 20160191634A1
• Time in Patent Office: 1,485 Days
• Field of Search: 709227
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 61/2596 Translation of addresses of the same type other than IP, e.g. translation from MAC to MAC addresses
• CPC Class Codes: H04L 63/306 intercepting packet switched data communications, e.g. Web, Internet or IMS communications
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066
• CPC Class Codes: H04L 67/146 Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding

Abstract

Web pages may be rendered from a main page data element and a plurality of embedded data elements, which are separately fetched by a browser. Herein is provided a web decoder which includes a learning engine adapted to receive human indications of data elements which are unimportant and accordingly to adjust the web decoder'"'"'s procedures for determining which data elements are displayed to the user. The learning engine may receive human indications of important data elements and uses both types of indications in its further determinations. Optionally, rule generalizations are performed in a manner which searches for parameters which differentiate between important and unimportant data elements. The rule generalizations optionally concentrate on groups of data elements having at least a predetermined number of parameters having the same values for both important and unimportant data elements, reducing the chances that a generalization rule will find important data elements as unimportant.

Claims

1. A method for communication analysis via a decoding processor, the method comprising the steps of:

by the decoding processor, intercepting data communication packets exchanged between a user computer and one or more servers over a network during a web browsing session associated with a target user;

by the decoding processor, processing the packets so as to identify data elements viewed by the target user during the web browsing session;

by the decoding processor, for each identified data element, calculating a confidence level that the identified data element is main page;

by the decoding processor, determining identified data elements whose calculated confidence levels are below a threshold confidence level;

by the decoding processor, determining if a percentage of identified data elements whose calculated confidence levels are below the threshold confidence level exceeds a level; and

when it is determined that the percentage exceeds the level;

by the decoding processor, displaying the determined identified data elements to an operator on a display of an operator terminal in communication with the decoding processor, each of the determined identified data elements displayed as a separate web page with a request for feedback as to whether the displayed web page is a main web page;

receiving feedback from the operator via an input device of the operator terminal as to whether displayed data elements are main web pages that are displayed on their own in different pages in a web decoder or are embedded data elements that are displayed within a main web page and which are separately fetched from the main web page in which they are embedded and are not displayed on their own in different pages in the web decoder; and

by the decoding processor, determining which ones of determined identified data elements processed from packets accepted after receiving the feedback are to be displayed in the web decoder to the operator, responsive to the received feedback.

2. The method of claim 1, wherein displaying the determined identified data elements comprises displaying at least some of the data elements without a request for feedback and displaying at least some of the data elements with a request for feedback.

3. The method of claim 2, wherein displaying determined identified data elements comprises displaying data elements identified as a main page without a request for feedback and displaying data elements identified as possibly being a main page with a request for feedback.

4. The method of claim 2, wherein displaying determined identified data elements comprises displaying some of the identified data elements and not displaying at least one of the data elements identified as not being a main page.

5. The method of claim 4, wherein data elements identified as not being a main page comprise data elements identified as being downloaded by the target user for display on its own, not embedded in a different page.

6. The method of claim 1, wherein determining which ones of the identified data elements are to be displayed to the operator comprises determining using an artificial intelligence engine.

7. The method of claim 1, wherein the received feedback used in the determination comprises indications of embedded data elements.

8. The method of claim 7, wherein determining which ones of the identified data elements are to be displayed to the operator comprises determining for a plurality of data elements having equivalent values of at least one parameter, for which at least some of the elements were indicated as being a main page and at least some were indicated as being an embedded data element, one or more parameters to be used in differentiating between main pages and embedded data elements.

9. The method according to of claim 8, wherein the plurality of data elements having equivalent values comprise data elements from the same web address.

10. The method according to of claim 1, wherein determining which ones of the identified data elements are to be displayed to the operator comprises determining for identified data elements belonging to the same session as at least one of the elements for which the received feedback was used in the determination.

11. The method of claim 1, further comprising:

extracting, by the decoding processor, parameters of the displayed data elements;

defining, by the decoding processor, templates of data elements not to be displayed as main pages based on the received feedback and the extracted parameters.

12. The method of claim 11, wherein determining which ones of the determined identified data elements are to be displayed to the operator comprises comparing parameters of the determined identified data elements to the templates and marking those elements that match at least one of the templates as data elements not to be displayed.

13. The method of claim 11, wherein defining the templates of data elements not to be displayed as main pages comprises searching, by a learning engine, for parameter values of the extracted parameters that can be used to differentiate between main web pages and embedded data elements based on the received feedback.

14. The method of claim 1, wherein the identified data elements comprise data elements with a parameter value that is indicative of an embedded element, wherein the parameter value is present in more than a threshold number of identified main pages.

15. The method of claim 1, further comprising:

when it is determined that the percentage does not exceed the level;

by the decoding processor, displaying the determined identified data elements to the operator on the display of an operator terminal in communication with the decoding processor, each of the determined identified data elements displayed as a separate web page without a request for feedback as to whether the displayed web page is a main web page.

16. A communication analyzer, comprising:

a network interface configured to intercept data packets exchanged between a user computer and one or more servers over a network during a web browsing session associated with a target user;

a display screen;

an input device;

a processor configured to identify data elements viewed by the target user during the web browsing session, to, for each identified data element, calculate a confidence level that the identified data element is main page, to determine identified data elements whose calculated confidence levels are below a threshold confidence level, to determine if a percentage of identified data elements whose calculated confidence levels are below the threshold confidence level exceeds a level, and when it is determined that the percentage exceeds the level;

to display each of the determined identified data elements on the screen to an operator as a separate web page with a request for feedback as to whether the displayed web page is a main web page, to receive feedback via the input device from the operator as to whether displayed data elements are main web pages that are displayed on their own in different pages in a web decoder or are embedded data elements that are displayed within a main web page and which are separately fetched from the main web page within which they are embedded and are not displayed on their own in different pages in the web decoder, and to adjust its configuration for identifying data elements responsive to the received feedback.

17. The analyzer of claim 16, wherein the processor is configured to request feedback for some of the displayed web pages and not request feedback for others of the displayed web pages.

18. The analyzer of claim 16, wherein the processor is configured to determine parameter values of web pages indicated in the feedback as being main web pages and parameter values of web pages indicated in the feedback as embedded data elements, and to adjust its configuration for identifying data elements to identify data elements having those parameter values.