System and method for objectively measuring user experience of touch screen based devices

• US 10,261,611 B2
• Filed: 12/03/2012
• Issued: 04/16/2019
• Est. Priority Date: 12/03/2012
• Status: Active Grant

• Current Assignee: Apkudo, Inc.
• Original Assignee: Apkudo, LLC
• Inventors: Matthews, Joshua Scott, Teitelbaum, David Michael, Tseng, Benjamin Kou-An
• Primary Examiner(s): Pappas, Claire X
• Assistant Examiner(s): Casarez, Benjamin X
• Application Number:
  US13/692,154
• Publication Number:
  US 20140152584A1
• Time in Patent Office: 2,325 Days
• Field of Search: None
• CPC Class Codes: G06F 11/2221 to test input/output devices or peripheral units
• CPC Class Codes: G06F 11/3041 where the computing system component is an input/output interface interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units G06F13/00
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: Y10S 901/14 Arm movement, spatial

Abstract

One aspect of the invention relates to an exemplary method for measuring latency of display response of a touch screen on a touch screen based device, comprising recording, by a video camera, an interaction with the touch screen, correlating, by a processor in operative communication with the video camera, data retrieved from the recording to the interaction; and determining, by the processor, a latency of the response of the touch screen to the first contact. An interaction may comprise at least a first contact between a member and the touch screen, and a response of the touch screen to the first contact, including at least a first change of a frame of a display of the touch screen.

Claims

1. A method of measuring latency related to touch screen response, the method comprising:

generating, by a camera, a video of an interaction with a touch screen of a device, wherein the interaction comprises at least a first contact between a member and the touch screen and (ii) a response of the touch screen to the first contact;

monitoring, by a processor, frame change rates of the touch screen based on the interaction video;

determining, by the processor, a response end time of the touch screen response based on the monitored frame change rates of the touch screen;

determining, by the processor, a contact time of the first contact based on the interaction video; and

determining, by the processor, a latency of the touch screen response to the first contact based on the contact time and the response end time.

2. The method of claim 1, wherein the first contact comprises at least a first type of contact between the member and the touch screen, the first type of contact including:

an initial contact between the member and the touch screen; and

a completion of the first type of contact between the member and the touch screen.

3. The method of claim 2, wherein a type of contact may comprise one of a touch, a swipe or a button press.

4. The method of claim 1, wherein the interaction comprises a plurality of contacts between the member and the touch screen, including at least the first contact of a first type and a second contact of a second type different from the first type of contact, with one or more of the plurality of contacts including:

an initial contact between the member and the touch screen; and

a completion of the contact between the member and the touch screen.

5. The method of claim 4, wherein the plurality of contacts comprises at least a plurality of contacts of the first type of contact and a plurality of contacts of the second type of contact.

6. The method of claim 5, further comprising:

determining, by the processor, a first average latency of response of the touch screen to the first type of contact; and

determining, by the processor, a second average latency of response of the touch screen to the second type of contact.

7. The method of claim 1, comprising:

accessing, by the processor, storage for data corresponding to one or more interactions including a contact of a same type as the first contact;

computing, by the processor, an average latency of response of the touch screen to contacts of the same type as the first contact based on the determined latency and the accessed data.

8. The method of claim 1, wherein the touch screen response to the first contact comprises:

an initial response of the touch screen including at least a first change of frame of the touch screen that occurs at a beginning of the touch screen response; and

a completion of the touch screen response including at least a plurality of frame changes.

9. The method of claim 1, wherein the camera has a recording rate of at least a 60 frames-per-second.

10. The method of claim 1, wherein determining the response end time comprises determining, based on the monitored frame change rates, the response end time to be a time at which no frame change subsequently occurs for at least a predetermined amount of time.

11. The method of claim 1, further comprising:

obtaining, by the processor, information specifying latencies of responses of the touch screen to contacts, the touch screen responses comprising the touch screen response and the contacts comprising the first contact;

determining, by the processor, respective amounts of time between contact times at which at least some of the contacts occurred and end times of the touch screen responses to other ones of the contacts, wherein the other ones of the contacts immediately preceded respective ones of the at least some contacts; and

determining, by the processor, latency patterns of the touch screen based on the respective amounts of time and the latencies of the touch screen responses to the at least some contacts.

12. The method of claim 1, wherein the member comprises a robotic member, the method further comprising:

causing, by the processor, the robotic member to move to effectuate the interaction with the touch screen.

13. The method of claim 1, the method further comprising:

obtaining, by the processor, a program of a plurality of contacts, including the first contact, to be performed by the member on the touch screen; and

causing, by the processor, the member to perform the plurality of contacts based on the program.

14. The method of claim 13, wherein the plurality of contacts specified by the program is based on a user input of the plurality of contacts via a user interface.

15. The method of claim 13, wherein the program is predefined and is stored among a plurality of predefined programs.

16. The method of claim 1, the method further comprising:

receiving, by the processor, an indication that the interaction video has been stored in a recording storage; and

obtaining the interaction video from the recording storage based on the indication.

17. The method of claim 1, the method further comprising:

determining, by the processor, a change in a number of pixels of the touch screen;

obtaining, by the processor, one or more characteristics of the touch screen being tested; and

determining, by the processor, whether the change in the number of pixels is sufficient to determine that the touch screen has responded to the first contact based on the one or more characteristics of the touch screen.

18. The method of claim 1, wherein the touch screen device is of a first type of touch screen device, the method further comprising:

generating, by the processor, a report comprising the latency of the response of the touch screen device and a second latency of response of a second type of touch screen device different from the first type of touch screen device.

19. The method of claim 1, wherein the first contact is of a first type of contact, the method further comprising:

causing the device to display a first application to respond to the first contact;

causing the touch screen device to display a second application, different from the first application, to respond to a second contact that is of the first type of contact;

generating, by the processor, a report comprising a first latency of response of the first application to the first contact and a second latency of response of the second application to the second contact.

20. A system for measuring latency related to touch screen response, the system comprising:

a computer system comprising one or more processors programmed with computer program instructions that, when executed by the one or more processors, cause the computer system to;

generate a video of an interaction with a touch screen of a device, wherein the interaction comprises at least (i) a first contact between a member and the touch screen and (ii) a response of the touch screen to the first contact;

monitor frame change rates of the touch screen based on the interaction video;

determine a response end time of the touch screen response based on the monitored frame change rates of the touch screen;

determine a contact time of the first contact based on the interaction video; and

determine a latency of the touch screen response to the first contact based on the contact time and the response end time.

21. The system of claim 20, wherein the first contact comprises at least a first type of contact between the member and the touch screen, the first type of contact including:

an initial contact between the member and the touch screen; and

a completion of the first type of contact between the member and the touch screen.

22. The system of claim 21, wherein a type of contact may comprise one of a touch, a swipe or a button press.

23. The system of claim 20, wherein the interaction comprises a plurality of contacts between the member and the touch screen, including at least the first contact of a first type and a second contact of a second type different from the first type of contact, with one or more of the plurality of contacts including:

an initial contact between the member and the touch screen; and

a completion of the contact between the member and the touch screen.

24. The system of claim 23, wherein the plurality of contacts comprises at least a plurality of contacts of the first type of contact and a plurality of contacts of the second type of contact.

25. The system of claim 24, wherein the computer system is further caused to:

determine a first average latency of response of the touch screen to the first type of contact; and

determine a second average latency of response of the touch screen to the second type of contact.

26. The system of claim 20, wherein the computer system is further caused to:

access storage for data corresponding to one or more interactions including a contact of a same type as the first contact; and

compute an average latency of response of the touch screen to contacts of the same type as the first contact based on the determined latency and the accessed data.

27. The system of claim 20, wherein the touch screen response to the first contact comprises:

an initial response of the touch screen including at least a first change of frame of the touch screen that occurs at a beginning of the touch screen response; and

a completion of the touch screen response including at least a plurality of frame changes.

28. The system of claim 20, wherein the video is generated by a camera, and wherein the camera has a recording rate of at least a 60 frames-per-second.

29. The system of claim 20, wherein determining the response end time comprises determining, based on the monitored frame change rates, the response end time to be a time at which no frame change subsequently occurs for at least a predetermined amount of time.

30. The system of claim 20, wherein the computer system is further caused to:

obtain information specifying latencies of responses of the touch screen to contacts, the touch screen responses comprising the touch screen response and the contacts comprising the first contact;

determine respective amounts of time between contact times at which at least some of the contacts occurred and end times of the touch screen responses to other ones of the contacts, wherein the other ones of the contacts immediately preceded respective ones of the at least some contacts; and

determine latency patterns of the touch screen based on the respective amounts of time and the latencies of the touch screen responses to the at least some contacts.

31. The system of claim 20, wherein the member comprises a robotic member, the robotic member configured to move to effectuate the interaction with the touch screen.

32. A system for measuring latency related to touch screen response, comprising:

a housing;

a holder coupled to the housing, the holder configured to hold a device having a touch screen in a fixed position;

a member coupled to the housing, the member configured to effectuate an interaction with the touch screen; and

a video camera coupled to the housing, the video camera comprising;

a lens, the lens being positioned such that it views the holder for the touch screen based device;

a recorder configured to;

generate a video of the interaction with the touch screen, the interaction comprising;

at least a first contact between the member and the touch screen; and

a response of the touch screen to the first contact; and

a processor in operative communication with the video camera, the processor programmed with computer program instructions to;

monitor frame change rates of the touch screen based on the interaction video;

determine a response end time of the touch screen response based on the monitored frame change rates of the touch screen;

determine a contact time of the first contact based on the interaction video; and

determine a latency of the touch screen response to the first contact based on the contact time and the response end time.

33. The system of claim 32, wherein the first contact comprises at least a first type of contact between the member and the touch screen, the first type of contact including:

an initial contact between the member and the touch screen; and

a completion of the first type of contact between the member and the touch screen.

34. The system of claim 33, wherein a type of contact may comprise one of a touch, a swipe or a button press.

35. The system of claim 32, wherein the interaction comprises a plurality of contacts between the member and the touch screen, including at least the first contact of a first type and a second contact of a second type different from the first type of contact, with one or more of the plurality of contacts including:

an initial contact between the member and the touch screen; and

a completion of the contact between the member and the touch screen.

36. The system of claim 35, wherein the plurality of contacts comprises at least a plurality of contacts of the first type of contact and a plurality of contacts of the second type of contact.

37. The system of claim 36, wherein the processor is further programmed to:

determine a first average latency of response of the touch screen to the first type of contact; and

determine a second average latency of response of the touch screen to the second type of contact.

38. The system of claim 32, wherein the processor is further programmed to:

access storage for data corresponding to one or more interactions including a contact of a same type as the first contact; and

compute an average latency of response of the touch screen to contacts of the same type as the first contact based on the determined latency and the accessed data.

39. The system of claim 32, wherein the touch screen response to the first contact comprises:

an initial response of the touch screen including at least a first change of frame of the touch screen that occurs at a beginning of the touch screen response; and

a completion of the touch screen response including at least a plurality of frame changes.

40. The system of claim 32, wherein the camera has a recording rate of at least a 60 frames-per-second.

41. The system of claim 32, wherein determining the response end time comprises determining, based on the monitored frame change rates, the response end time to be a time at which no frame change subsequently occurs for at least a predetermined amount of time.

42. The system of claim 32, wherein the processor is further programmed to:

obtain information specifying latencies of responses of the touch screen to contacts, the touch screen responses comprising the touch screen response and the contacts comprising the first contact;

determine respective amounts of time between contact times at which at least some of the contacts occurred and end times of the touch screen responses to other ones of the contacts, wherein the other ones of the contacts immediately preceded respective ones of the at least some contacts; and

determine latency patterns of the touch screen based on the respective amounts of time and the latencies of the touch screen responses to the at least some contacts.

43. The system of claim 32, wherein the member comprises a robotic member, the system further comprising an actuator coupled to the robotic member, the actuator configured to effectuate movement of the robotic member.