Automated forensic artifact reconstruction and replay of captured and recorded display interface streams

• US 9,996,915 B2
• Filed: 08/04/2016
• Issued: 06/12/2018
• Est. Priority Date: 08/04/2016
• Status: Active Grant

• Current Assignee: Altia Acquisition Corporation
• Original Assignee: Altia, Inc.
• Inventors: Stroupe, William J.
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US15/228,737
• Publication Number:
  US 20180040114A1
• Time in Patent Office: 677 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0733 in a data processing system embedded in an image processing device, e.g. printer, facsimile, scanner
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G09G 2330/12 Test circuits or failure detection circuits included in a display system, as permanent part thereof
• CPC Class Codes: G09G 3/18 using liquid crystals
• CPC Class Codes: G09G 3/20 for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters

Abstract

Novel tools and techniques might provide for implementing user interface validation or diagnostics, and, in particular embodiments, for implementing automated forensic artifact reconstruction and replay of captured and recorded display interface streams. In some embodiments, a computing system might retrieve stream data from a database, the stream data having been captured and recorded by a data stream recording system as display data is being sent from an embedded processor of a graphical user interface (“GUI”) system to a display device, without disturbing the display data being sent. The computing system subsequently parses pixel data from the display data, formats the parsed pixel data, reconstructs one or more display screen images based at least in part on the formatted pixel data, then compares each pixel of each screen image or frame of the reconstructed display screen images with each pixel of a model screen image of each corresponding screen image.

Claims

1. A method for implementing user interface diagnostics, comprising:

sending, with an embedded processor of a graphical user interface (“

GUI”

) system, a first stream of data to a display device;

receiving, with the display device, the first stream of data, wherein the first stream of data comprises display data and formatting data, the formatting data comprising information indicating a number of pixels per row and a number of rows, and the display data comprising pixel data comprising color depth for each pixel;

formatting, with the display device, the first stream of data to produce one or more first display screen images, based at least in part on the formatting data and on the display data;

displaying, with the display device, the one or more first display screen images, one screen image at a time;

capturing, with a data stream recording system, the first stream of data as the first stream of data is being sent from the embedded processor to the display device, without disturbing the data being sent to the display device;

recording, with the data stream recording system, the captured first stream of data;

retrieving, with a computing system, the captured and recorded first stream of data;

determining, with the computing system, a display protocol associated with the captured and recorded first stream of data, based at least in part on the formatting data of the captured and recorded first stream of data;

parsing, with the computing system, the pixel data from the display data of the captured and recorded first stream of data;

formatting, with the computing system, the parsed pixel data to produce one or more second display screen images, based at least in part on the determined display protocol and on the parsed pixel data;

determining, with the computing system, whether one or more successive display screen images of the one or more second display screen images is pixel-to-pixel identical to a display screen image of the one or more second display screen images;

based on a determination that one or more successive display screen images of the one or more second display screen images is pixel-to-pixel identical to one display screen image of the one or more second display screen images, compressing, with the computing system, the formatted pixel data to produce one or more third display screen images, by deleting the one or more successive display screen images that is determined to be pixel-to-pixel identical to the one display screen image;

reconstructing, with the computing system, the one or more first display screen images, based on the one or more third display screen images;

storing, with the computing system and on a data store, the reconstructed one or more first display screen images;

comparing, with the computing system, each pixel of each screen image of the reconstructed one or more first display screen images with each pixel of a model screen image of each corresponding unique screen image of the one or more first display screen images that is displayed on the display device;

based on a determination that at least one screen image of the reconstructed one or more first display screen images is not pixel-to-pixel identical to the model screen image of a corresponding unique screen image of the one or more first display screen images that is displayed on the display device, logging, with the computing system and in an error log, the at least one screen image as being pixel imperfect and indicating in the error log which pixels of the at least one screen image do not match with corresponding pixels in the model screen image of a corresponding unique screen image of the one or more first display screen images; and

based on a determination that each of the screen images of the reconstructed one or more first display images matches pixel-to-pixel with the model screen image of each corresponding unique screen image of the one or more first display screen images that is displayed on the display device, sending, with the computing system, a notification to a user that the one or more first display screen images match corresponding model screen images in a pixel perfect manner.

2. The method of claim 1, further comprising:

based on a determination that at least one screen image of the reconstructed one or more first display screen images is not pixel-to-pixel identical to the model screen image of a corresponding unique screen image of the one or more first display screen images, sending, with the computing system, a notification to a user indicating that the at least one screen image is pixel imperfect and indicating which pixels of the at least one screen image do not match with corresponding pixels in the model screen image of a corresponding unique screen image of the one or more first display screen images.

3. The method of claim 1, wherein the display device is a liquid crystal display (“

LCD”

).

4. The method of claim 1, wherein the data stream recording system captures the first stream data as the display data is being sent from the embedded processor to the display device, using a pass-through breakout connector that captures stream data without disturbing the stream data being sent.

5. The method of claim 1, further comprising:

performing, with the computing system, a self-diagnostic, by;

converting an input video file into a series of still images;

decomposing the series of still images into a binary recording file, the binary recording file comprising display data, a selected protocol, and synchronization signals;

parsing pixel data from the display data in the binary recording file;

formatting the parsed pixel data to produce one or more display screen images, based at least in part on the selected protocol and on the parsed pixel data;

reconstructing the one or more still images, based on the one or more display screen images;

comparing each pixel of each screen image of the reconstructed one or more still images with each pixel of each corresponding still image of the series of still images;

based on a determination that at least one screen image of the reconstructed one or more still images is not pixel-to-pixel identical to the corresponding still image of the series of still images, notifying a user that the computing system has an error; and

based on a determination that each screen image of the reconstructed one or more still images is pixel-to-pixel identical to the corresponding still image of the series of still images, notifying a user that the computing system is ready to diagnose the GUI system of the display device.

6. An apparatus for implementing user interface diagnostics, comprising:

one or more processors; and

a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon computer software comprising a set of instructions that, when executed by the one or more processors, causes the apparatus to;

retrieve, from a first data store, a first stream of data, the first stream of data comprising display data for producing one or more first display screen images, the first stream data being previously captured and recorded on the first data store by a data stream recording system as the display data is being sent from an embedded processor of a graphical user interface (“

GUI”

) system to a display device, using a pass-through breakout connector that captures stream data without disturbing the pixel data being sent to the display device, wherein the first stream of data further comprises formatting data comprising information indicating a number of pixels per row and a number of rows, and the display data comprises pixel data comprising color depth for each pixel;

determine a display protocol associated with the captured and recorded first stream of data, based at least in part on the formatting data of the captured and recorded first stream of data;

parse pixel data from the display data of the captured and recorded first stream of data;

format the parsed pixel data to produce one or more second display screen images, based at least in part on the determined display protocol and on the parsed pixel data;

determine whether one or more successive display screen images of the one or more second display screen images is pixel-to-pixel identical to a display screen image of the one or more second display screen images;

based on a determination that one or more successive display screen images of the one or more second display screen images is pixel-to-pixel identical to one display screen image of the one or more second display screen images, compress the formatted pixel data to produce one or more third display screen images, by deleting the one or more successive display screen images that is determined to be pixel-to-pixel identical to the one display screen image;

reconstruct the one or more first display screen images, based on the one or more third display screen images;

store the reconstructed one or more first display screen images on a data store;

compare each pixel of each screen image of the reconstructed one or more first display screen images with each pixel of a model screen image of each corresponding screen image of the one or more first display screen images;

based on a determination that at least one screen image of the reconstructed one or more first display screen images is not pixel-to-pixel identical to the model screen image of a corresponding unique screen image of the one or more first display screen images that is displayed on the display device, log, in an error log, the at least one screen image as being pixel imperfect and indicating in the error log which pixels of the at least one screen image do not match with corresponding pixels in the model screen image of a corresponding unique screen image of the one or more first display screen images; and

based on a determination that each of the screen images of the reconstructed one or more first display images matches pixel-to-pixel with the model screen image of each corresponding unique screen image of the one or more first display screen images that is displayed on the display device, send a notification to a user that the one or more first display screen images match corresponding model screen images in a pixel perfect manner.

7. The apparatus of claim 6, wherein the set of instructions, when executed by the one or more processors, further causes the apparatus to:

based on a determination that at least one screen image of the reconstructed one or more first display screen images is not pixel-to-pixel identical to the model screen image of a corresponding unique screen image of the one or more first display screen images, send a notification to a user indicating that the at least one screen image is pixel imperfect and indicate which pixels of the at least one screen image do not match with corresponding pixels in the model screen image of a corresponding unique screen image of the one or more first display screen images.

8. The apparatus of claim 6, wherein the display device is a liquid crystal display (“

LCD”

).

9. A system for implementing user interface diagnostics, comprising:

a data stream recording system, comprising;

a pass-through breakout connector that captures stream data without disturbing the stream data being sent;

one or more first processors; and

a first non-transitory computer readable medium, the first non-transitory computer readable medium having stored thereon computer software comprising a first set of instructions that, when executed by the one or more first processors, causes the data stream recording system to;

capture, using the pass-through breakout connector, a first stream of data as the first stream of data is being sent from an embedded processor of a graphical user interface (“

GUI”

) system to a display device, without disturbing the data being sent to the display device, the first stream of data comprising display data for producing one or more first display screen images, wherein the first stream of data further comprises formatting data comprising information indicating a number of pixels per row and a number of rows, and the display data comprises pixel data comprising color depth for each pixel; and

record, with the data stream recording system, the captured first stream of data on a first data store;

a computing system, comprising;

one or more second processors; and

a second non-transitory computer readable medium, the second non-transitory computer readable medium having stored thereon computer software comprising a second set of instructions that, when executed by the one or more second processors, causes the computing system to;

retrieve, from the first data store, the first stream of data;

determine a display protocol associated with the captured and recorded first stream of data, based at least in part on the formatting data of the captured and recorded first stream of data;

parse pixel data from the display data of the captured and recorded first stream of data;

format the parsed pixel data to produce one or more second display screen images, based at least in part on the determined display protocol and on the parsed pixel data;

determine whether one or more successive display screen images of the one or more second display screen images is pixel-to-pixel identical to a display screen image of the one or more second display screen images;

based on a determination that one or more successive display screen images of the one or more second display screen images is pixel-to-pixel identical to one display screen image of the one or more second display screen images, compress the formatted pixel data to produce one or more third display screen images, by deleting the one or more successive display screen images that is determined to be pixel-to-pixel identical to the one display screen image;

reconstruct the one or more first display screen images, based on the one or more third display screen images;

store the reconstructed one or more first display screen images on a data store;

compare each pixel of each screen image of the reconstructed one or more first display screen images with each pixel of a model screen image of each corresponding screen image of the one or more first display screen images;

based on a determination that at least one screen image of the reconstructed one or more first display screen images is not pixel-to-pixel identical to the model screen image of a corresponding unique screen image of the one or more first display screen images that is displayed on the display device, log, in an error log, the at least one screen image as being pixel imperfect and indicating in the error log which pixels of the at least one screen image do not match with corresponding pixels in the model screen image of a corresponding unique screen image of the one or more first display screen images; and

based on a determination that each of the screen images of the reconstructed one or more first display images matches pixel-to-pixel with the model screen image of each corresponding unique screen image of the one or more first display screen images that is displayed on the display device, send a notification to a user that the one or more first display screen images match corresponding model screen images in a pixel perfect manner.

10. The system of claim 9, wherein the second set of instructions, when executed by the one or more second processors, further causes the computing system to:

based on a determination that at least one screen image of the reconstructed one or more first display screen images is not pixel-to-pixel identical to the model screen image of a corresponding unique screen image of the one or more first display screen images, send a notification to a user indicating that the at least one screen image is pixel imperfect and indicate which pixels of the at least one screen image do not match with corresponding pixels in the model screen image of a corresponding unique screen image of the one or more first display screen images.

11. The system of claim 9, wherein the display device is a liquid crystal display (“

LCD”

).