Automatically fixing inaccessible widgets during mobile application execution

• US 10,394,579 B2
• Filed: 03/27/2014
• Issued: 08/27/2019
• Est. Priority Date: 03/27/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nandakumar, Vikrant, Sharma, Vivek, Ekambaram, Vijay, Mathur, Ashish K.
• Primary Examiner(s): Vu, Kieu D
• Assistant Examiner(s): Basom, Blaine T.
• Application Number:
  US14/227,252
• Publication Number:
  US 20150279070A1
• Time in Patent Office: 1,979 Days
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: H04M 1/72481 for visually impaired users

Abstract

Methods, systems, and articles of manufacture for automatically fixing inaccessible widgets during mobile application execution are provided herein. A method includes identifying an image element from a set of one or more user interface elements of an application that are inaccessible to a given set of users; generating a text description associated with the image element; extracting a portion of the text description related to an accessibility property of the image element; and adjusting the accessibility property of the image element to render the image element accessible to the given set of users.

Claims

1. A method comprising:

identifying, during execution of a mobile application, an image element from a set of one or more user interface elements of the mobile application that is inaccessible to a set of users, wherein said set of users comprises at least one of (i) one or more users with a hearing impairment and (ii) one or more users with a vision impairment, wherein said identifying comprises determining that the image element is one of;

an ImageButton type element and an ImageView type element based on corresponding user interface element type information derived from a user interface view hierarchy of a given user interface screen, wherein the user interface view hierarchy is an extensible markup language representation of the given user interface screen and comprises multiple items of information pertaining to each user interface element in the screen, wherein said multiple items of information comprise user interface element type, user interface element label, and bounding coordinates of the user interface element;

transmitting an image associated with the image element to an image-based content retrieval application programming interface;

generating, during execution of the mobile application and via the image-based content retrieval application programming interface, a text description associated with the image element;

extracting, during execution of the mobile application, at least a portion of the text description related to an accessibility property of the image element; and

adjusting, during execution of the mobile application, the accessibility property of the image element to render the image element accessible to the set of users;

wherein said identifying, said transmitting, said generating, said extracting, and said adjusting are carried out by at least one computing device.

2. The method of claim 1, comprising:

executing an instrumentation process on the mobile application to generate an instrumented version of the mobile application, wherein said instrumentation process comprises inserting one or more lines of code in the mobile application to facilitate accessibility testing.

3. The method of claim 1, comprising:

launching the mobile application to obtain an identification of each of multiple user interface elements.

4. The method of claim 3, comprising:

determining whether each of the multiple user interface elements is accessible to the set of users.

5. The method of claim 1, wherein the ImageButton type element and the ImageView type element are defined based on a mobile operating system of the at least one computing device.

6. A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:

identify, during execution of a mobile application, an image element from a set of one or more user interface elements of the mobile application that is inaccessible to a set of users, wherein said set of users comprises at least one of (i) one or more users with a hearing impairment and (ii) one or more users with a vision impairment, wherein identifying comprises determining that the image element is one of;

an ImageButton type element and an ImageView type element based on corresponding user interface element type information derived from a user interface view hierarchy of a given user interface screen, wherein the user interface view hierarchy is an extensible markup language representation of the given user interface screen and contains multiple items of information pertaining to each user interface element in the screen and, wherein said multiple items of information comprise user interface element type, user interface element label, and bounding coordinates of the user interface element;

transmit an image associated with the inaccessible image element to an image-based content retrieval application programming interface;

generate, during execution of the mobile application and via the image-based content retrieval application programming interface, a text description associated with the image element;

extract, during execution of the mobile application, a portion of the text description related to an accessibility property of the image element; and

adjust, during execution of the mobile application, the accessibility property of the image element to render the image element accessible to the set of users.

7. The computer program product of claim 6, wherein the ImageButton type element and the ImageView type element are defined based on a mobile operating system of the at least one computing device.

8. A method comprising steps of:

identifying, during execution of a mobile application, an image element and a text element, from a set of one or more user interface elements of the mobile application, that are inaccessible to a set of users, wherein said set of users comprises at least one of;

(i) one or more users with a hearing impairment and (ii) one or more users with a vision impairment, wherein said identifying comprises determining that the text element is a TextField type element and the image element is one of;

an ImageButton type element and an ImageView type element based on corresponding user interface element type information derived from a user interface view hierarchy of a given user interface screen, wherein the user interface view hierarchy is an extensible markup language representation of the given user interface screen and contains data pertaining to each user interface element in the screen;

transmitting the text element to an associated label mapping application programming interface and an image associated with the inaccessible image element to an image-based content retrieval application programming interface;

determining, during execution of the mobile application and via the associated label mapping application programming interface, an associated label within a vicinity of the identified text element;

extracting, during execution of the mobile application, multiple properties of the associated label, wherein said multiple properties are related to an accessibility property of the identified text element, and wherein said multiple properties comprise label type, label description, and bounding coordinates relative to the inaccessible text element;

generating, during execution of the mobile application and via the image-based content retrieval application programming interface, a text description associated with the inaccessible image element;

extracting, during execution of the mobile application, at least a portion of the text description related to an accessibility property of the inaccessible image element; and

adjusting, during execution of the mobile application, the accessibility property of each of the identified text element and the identified image element to render the identified text element and the identified image element accessible to the set of users;

wherein the steps are carried out by at least one computing device.

9. The method of claim 8, comprising:

executing an instrumentation process on the mobile application to generate an instrumented version of the mobile application, wherein said instrumentation process comprises inserting one or more lines of code in the mobile application to facilitate accessibility testing.

10. The method of claim 8, comprising:

launching the mobile application to obtain an identification of each of multiple user interface elements.

11. The method of claim 10, comprising:

determining whether each of the multiple user interface elements is accessible to the set of users.

12. The method of claim 8, wherein the ImageButton type element, the ImageView type element and the TextField type element are defined based on a mobile operating system of the at least one computing device.

13. A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:

identify, during execution of a mobile application, an image element and a text element, from a set of one or more user interface elements of the mobile application, that are inaccessible to a set of users, wherein said set of users comprises at least one of;

(i) one or more users with a hearing impairment and (ii) one or more users with a vision impairment, wherein said identifying comprises determining that the text element is a TextField type element and the image element is one of;

an ImageButton type element and an ImageView type element based on corresponding user interface element type information derived from a user interface view hierarchy of a given user interface screen, wherein the user interface view hierarchy is an extensible markup language representation of the given user interface screen and contains data pertaining to each user interface element in the screen;

transmit the text element to an associated label mapping application programming interface and an image associated with the inaccessible image element to an image-based content retrieval application programming interface;

determine, during execution of the mobile application and via the associated label mapping application programming interface, an associated label within a vicinity of the identified text element;

extract, during execution of the mobile application, multiple properties of the associated label, wherein said multiple properties are related to an accessibility property of the identified text element, and wherein said multiple properties comprise label type, label description, and bounding coordinates relative to the inaccessible text element

generate, during execution of the mobile application and via the image-based content retrieval application programming interface, a text description associated with the inaccessible image element;

extract, during execution of the mobile application, at least a portion of the text description related to an accessibility property of the inaccessible image element; and

adjust, during execution of the mobile application, the accessibility property of each of the identified text element and the identified image element to render the identified text element and the identified image element accessible to the set of users.

14. The computer program product of claim 13, wherein the ImageButton type element, the ImageView type element, and the TextField type element are defined based on a mobile operating system of the at least one computing device.