User interface conversion method and apparatus

• US 5,956,029 A
• Filed: 09/08/1997
• Issued: 09/21/1999
• Est. Priority Date: 09/09/1996
• Status: Expired due to Fees

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Yamanaka, Katsuhiro, Okada, Yoshihiko
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): Jackson, Chadwick A.
• Application Number:
  US08/925,950
• Time in Patent Office: 743 Days
• Field of Search: 345/326, 345/333, 345/334, 434/112, 434/113, 434/114
• US Class Current: 715/746
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

In a user interface conversion method of converting a picture interface provided by an application program running on an operating system having a graphical user interface to generate and provide a new picture interface, picture information of the application program is acquired in response to, as a trigger, a change in the picture provided by the application program. A target point in the acquired picture information is determined. Converted picture information is generated from the determined target point by referring to conversion template information. A converted picture is displayed in accordance with the generated converted picture information. A user interface conversion apparatus is also disclosed.

Claims

1. A user interface conversion method of converting a picture interface provided by an application program running on an operating system having a graphical user interface to generate and provide a new picture interface, comprising the steps of:

acquiring picture information of the application program in response to, as a trigger, a change in the picture provided by the application program;

determining a target point in the acquired picture information;

generating converted picture information from the determined target point by referring to conversion template information stored in a conversion template storage section;

generating specific converted picture information by referring to specific conversion information stored in a specific conversion information storage section;

displaying a converted picture in accordance with the generated converted picture information and the generated specific converted picture information.

2. A user interface conversion apparatus including input means, output means, an operating system having a graphical user interface, and an application program which runs on the operating system, and adapted to convert a picture interface provided by the application program so as to generate and provide a new picture interface, comprising:

event acquiring means for detecting an input event from said input means and an event of a picture change in a window, a menu, and a button on the picture interface;

picture information acquiring means for extracting picture information including at least logic structure information, layout information, and attribute information of interactive components including a window, a menu, and a button displayed on a picture, as elements, in response to a picture change event detected by said event acquiring means;

target point extracting means for setting a target point on one of a display window of the application program and currently focused interactive components by using target point information preset in the picture information extracted by said picture information acquiring means, and extracting target point picture information as picture information at the set target point;

converted interface generating means for generating a converted picture from the target point picture information extracted by said target point extracting means, and generating converted picture information including at least logic structure information, layout information, and attribute information of interactive components on the generated converted picture, link information with respect to interactive components on the picture interface provided by the original application program, and information of a corresponding event, as elements;

converted interface control means for presenting a converted picture to an output unit in accordance with the converted picture information generated by said converted interface generating means, and generating an event for driving the application program by referring to pre-stored converted picture information upon reception of an operation event for the converted picture from said event acquiring means; and

wherein said converted interface generating means comprises information converting means for performing picture conversion in accordance with the picture information extracted by said target point extracting means so as to allow various types of picture conversion, and wherein said information converting means is constituted by at least one of pairs including

a first pair of attribute conversion information storage means for storing attribute conversion information for converting attribute information including a size and position of an interactive component and a caption (item name) and attribute conversion executing means for executing attribute conversion on the basis of the attribute conversion information stored in said attribute conversion information storage means,

a second pair of component replacement information storage means for storing component replacement information for replacing an interactive component including replacement of a given button with another button, and component replacing means for executing component replacement on the basis of the component conversion information stored in said component replacement information storage means, and

a third pair of virtual component addition information storage means for storing virtual component addition information for adding a new interactive component which does not exist on extracted picture information and virtual component addition executing means for adding a virtual component on the basis of the virtual component addition information stored in said virtual component addition information storage means.

3. An apparatus according to claim 2, further comprising:

picture information storage means for storing picture information acquired by said picture information acquiring means and outputting the information to said target point extracting means;

converted picture information storage means for storing converted picture information generated by said converted interface generating means, and outputting the information to said converted interface control means; and

target point information storage means for storing target point information to be used by said target point extracting means.

4. An apparatus according to claim 2, wherein said converted interface generating means further comprises

converted interface generation control means for controlling a picture converting operation of said information converting means.

5. An apparatus according to claim 2, wherein said converted interface generating means further comprises:

conversion template storage means for storing template information for information conversion; and

specific information conversion means for converting specific information constituted by at least one of a specific picture and an interactive component by referring to the template information stored in said conversion template storage means so as to allow picture conversion in detail.

6. An apparatus according to claim 5, wherein said converted interface generating means further comprises:

converted picture editing means for allowing a picture producer to generate and edit a converted picture and a converted component by referring to a picture interface provided by an original application program;

generated component linking means for generating a link between the converted component generated by said converted picture editing means and an interactive component on the original application program; and

conversion template generating means for generating template information stored in said conversion template storage means from converted picture information of the converted picture edited by said converted picture editing means.

7. An apparatus according to claim 3, wherein said converted interface control means comprises:

converted interface control executing means for presenting the picture generated by said converted interface generating means to said output means, and generating an event for starting the application program by referring to the converted picture information stored in said converted picture information storage means upon reception of an operation event for a converted picture from said event acquiring means;

media addition information storage means for storing media addition information used for a visually handicapped user and a user of advanced age;

media addition means for performing information conversion for synthetic voice, braille, and tactile pin display by referring to the media addition information stored in said media addition information storage means upon occurrence of an event such as movement of a mouse pointer or a touch operation;

target point change information storage means for storing target point change information for changing a target point with respect to an interactive component regarded as a target component by a user; and

target point change control means for performing target point control in accordance with an operation performed by the user by referring to the target point change information stored in said target point change information storage means.

8. A user interface conversion method of converting a picture interface provided by an application program running on an operating system having a graphical user interface to generate and provide a new picture interface, comprising the steps of:

acquiring picture information of the application program in response to, as a trigger, a change in the picture provided by the application program;

determining a target point in the acquired picture information;

generating converted picture information from the determined target point by referring to conversion template information;

displaying a converted picture in accordance with the generated picture information;

acquiring an operation event from an input unit;

executing a media addition in accordance with the operation event and interactive component information corresponding to the operation event and stored in an interactive component information storage section.

9. A user interface conversion apparatus including input means, output means, an operating system having a graphical user interface, and an application program which runs on the operating system, and adapted to convert a picture interface provided by the application program so as to generate and provide a new picture interface, comprising:

event acquiring means for detecting an input event from said input means and an event of a picture change in a window, a menu, and a button on the picture interface;

picture information acquiring means for extracting picture information including at least logic structure information, layout information, and attribute information of interactive components including a window, a menu, and a button displayed on a picture, as elements, in response to a picture change event detected by said event acquiring means;

target point extracting means for setting a target point on one of a display window of the application program and currently focused interactive components by using target point information preset in the picture information extracted by said picture information acquiring means, and extracting target point picture information as picture information at the set target point;

converted interface generating means for generating a converted picture from the target point picture information extracted by said target point extracting means, and generating converted picture information including at least logic structure information, layout information, and attribute information of interactive components on the generated converted picture, link information with respect to interactive components on the picture interface provided by the original application program, and information of a corresponding event, as elements; and

converted interface control means for presenting a converted picture to an output unit in accordance with the converted picture information generated by said converted interface generating means, and generating an event for driving the application program by referring to pre-stored converted picture information upon reception of an operation event for the converted picture from said event acquiring means, and wherein said converted interface control means comprises media addition information storage means for storing media addition information used for a visually handicapped user and a user of advanced age, and media addition means for performing information conversion for synthetic voice, braille, and tactile pin display by referring to the media addition information stored in said media addition information storage means upon occurrence of an event such as movement of a mouse pointer or a touch operation.

10. A user interface conversion apparatus including input means, output means, an operating system having a graphical user interface, and an application program which runs on the operating system, and adapted to convert a picture interface provided by the application program so as to generate and provide a new picture interface, comprising:

event acquiring means for detecting an input event from said input means and an event of a picture change in a window, a menu, and a button on the picture interface;

picture information acquiring means for extracting picture information including at least logic structure information, layout information, and attribute information of interactive components including a window, a menu, and a button displayed on a picture, as elements, in response to a picture change event detected by said event acquiring means;

target point extracting means for setting a target point on one of a display window of the application program and currently focused interactive components by using target point information preset in the picture information extracted by said picture information acquiring means, and extracting target point picture information as picture information at the set target point;

converted interface generating means for generating a converted picture from the target point picture information extracted by said target point extracting means, and generating converted picture information including at least logic structure information, layout information, and attribute information of interactive components on the generated converted picture, link information with respect to interactive components on the picture interface provided by the original application program, and information of a corresponding event, as elements;

converted interface control means for presenting a converted picture to an output unit in accordance with the converted picture information generated by said converted interface generating means, and generating an event for driving the application program by referring to pre-stored converted picture information upon reception of an operation event for the converted picture from said event acquiring means; and

wherein said converted interface generating means comprises conversion template storage means for storing template information for information conversion, and specific information conversion means for converting specific information constituted by at least one of a specific picture and an interactive component by referring to the template information stored in said conversion template storage means so as to allow picture conversion in detail; and

wherein said specific information converting means is constituted by at least one of pairs including

a first pair of specific attribute conversion information storage means for storing specific attribute conversion information for converting specific attribute information including a size and position of a specific interactive component and a caption (item name) and specific attribute conversion executing means for executing specific attribute conversion on the basis of the specific attribute conversion information stored in said specific attribute conversion information storage means,

a second pair of specific component replacement information storage means for storing specific component replacement information for replacing a specific interactive component including replacement of a specific given button with another button, and specific component replacing means for executing specific component replacement on the basis of the specific component conversion information stored in said specific component replacement information storage means, and

a third pair of specific virtual component addition information storage means for storing specific virtual component addition information for adding a new specific interactive component which does not exist on extracted picture information and specific virtual component addition executing means for adding a specific virtual component on the basis of the specific virtual component addition information stored in said specific virtual component addition information storage means.