Method for adaptive interaction with a legacy software application

• US 8,928,582 B2
• Filed: 02/17/2012
• Issued: 01/06/2015
• Est. Priority Date: 02/17/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Senanayake, Rukman, Denker, Grit, Lincoln, Patrick D., Murray, John, Weiner, Steven S.
• Primary Examiner(s): MARINELLI, PATRICK
• Application Number:
  US13/399,183
• Publication Number:
  US 20130215005A1
• Time in Patent Office: 1,054 Days
• Field of Search: 345156-184, 178 1801- 2004, 340 1222- 1255, 715/740
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04809 Textured surface identifying touch areas, e.g. overlay structure for a virtual keyboard
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0362 with detection of 1D translations or rotations of an operating part of the device, e.g. scroll wheels, sliders, knobs, rollers or belts
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: H01H 2215/05 electromechanical
• CPC Class Codes: H01H 2217/022 Part of keyboard not operable
• CPC Class Codes: H01H 2217/038 Prompting
• CPC Class Codes: H01H 2239/07 UV or IR detection, e.g. of human body

Abstract

Methods are disclosed to support adaptive interaction with legacy software applications, without a need for rewriting those applications. The methods are for use with an interactive electronic system including a processor, a display, and an input device with user-manipulated controls. When the legacy application is executed, a supplemental software program, such as a plugin, is also executed and is utilized in order to identify currently relevant interactive features of the legacy application during execution. Functionality is dynamically assigned to the various user-manipulated controls based on the identified features. In one embodiment, detection of objects (particularly the user'"'"'s hands) proximate to the input controls is also employed in determining the assignment of functionality and/or in displaying a visual representation to the user of the available interactive choices. In another embodiment, the user-manipulated input controls are dynamically and physically reconfigured under control of the processor based on the identified features.

Claims

1. A method for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system comprising a computing system and an input device, the computing system comprises of the legacy software application, a display, a processor, and one or more sensors for detecting a present object, and the input device comprises of a plurality of user-manipulated input controls, the method comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify a plurality of interactive features for the legacy application;

dynamically assigning functionality to the plurality of user-manipulated input controls based on the plurality of identified interactive features;

displaying a visual depiction of the assigned functionality on the display of the computing system;

receiving an input from the user via one or more of the user-manipulated input controls;

processing the input at least partly by the supplemental software program; and

providing a result of said processing to the legacy application

wherein the functionality assigned to any given one of the plurality of user-manipulated input controls varies dynamically depending upon whether the one or more sensors detect an object proximate to the given one of the plurality of user-manipulated controls,

wherein the plurality of identified interactive features includes at least one feature that does not correspond to an interactive feature provided by the legacy application, and

wherein the supplemental software program is a plugin.

2. The method of claim 1, wherein the detected object is an extremity of the user.

3. The method of claim 2, further comprising tracking a current position concurrently for each one of a plurality of user extremities.

4. The method of claim 1, wherein the visual depiction of the assigned functionality reflects a spatial layout of corresponding ones of the plurality of user-manipulated input controls.

5. The method of claim 1 wherein the visual depiction further depicts a location of the object relative to the plurality of user-manipulated input controls.

6. The method of claim 1, wherein the dynamic assignment of functionality is based on a current position of the detected object relative to the plurality of user-manipulated input controls.

7. The method of claim 1, wherein assigning functionality to the plurality of user-manipulated input controls includes assigning a plurality of alternative functions to at least one of the plurality of user-manipulated input controls, and the method further comprises allowing the user to select among the plurality of alternative functions.

8. The method of claim 7, wherein allowing the user to select comprises displaying a visual identification for each of the alternative functions, when an object is detected proximate to said input control having the alternative functions.

9. The method of claim 8, wherein displaying a visual identification for each of the alternative functions comprises cycling through said alternative functions by displaying each corresponding visual identification one at a time.

10. A method for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system including a computing system and an input device, the computing system comprises of a display, a processor, and one or more sensors for detecting a present object, the input device comprises of a plurality of user-manipulated input controls, the method comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify a plurality of interactive features for the legacy application, wherein the one or more interactive features include at least one feature that does not correspond to an interactive feature provided by the legacy application; and

physically reconfiguring the plurality of user-manipulated input controls and assigning functionality to said plurality of user-manipulated input controls as reconfigured, by the processor and based on the plurality of identified interactive features;

receiving an input from the user via one or more of the user-manipulated input controls;

processing the input at least partly by the supplemental software program; and

providing a result of said processing to the legacy application,

wherein the supplemental software program is a plugin, and

wherein said physically reconfiguring and said assigning functionality are further based on a current position of the user relative to the user-manipulated input controls as detected by the sensors.

11. The method of claim 10, wherein said physically reconfiguring comprises modifying a size and number of the plurality of user-manipulated input controls that can be individually actuated.

12. The method of claim 10, further comprising displaying a visual depiction of the assigned functionality.

13. The method of claim 12, wherein the visual depiction reflects a spatial layout of corresponding reconfigured ones of the plurality of user-manipulated input controls.

14. A non-transitory computer readable storage device containing an executable program for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system comprising a computing system and an input device, the computing system comprises of the legacy software application, a display, a processor, and one or more sensors for detecting a present object, and the input device comprises of a plurality of user-manipulated input controls, where the program performs steps comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify a plurality of interactive features for the legacy application;

dynamically assigning functionality to the plurality of user-manipulated input controls based on the plurality of identified interactive features;

displaying a visual depiction of the assigned functionality on the display of the computing system;

receiving an input from the user via one or more of the user-manipulated input controls;

processing the input at least partly by the supplemental software program; and

providing a result of said processing to the legacy application

wherein the functionality assigned to any given one of the plurality of user-manipulated input controls varies dynamically depending upon whether the one or more sensors detect an object proximate to the given one of the plurality of user-manipulated controls,

wherein the plurality of identified interactive features includes at least one feature that does not correspond to an interactive feature provided by the legacy application, and

wherein the supplemental software program is a plugin.

15. A non-transitory computer readable storage device containing an executable program for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system including a computing system and an input device, the computing system comprises of a display, a processor, and one or more sensors for detecting a present object, the input device comprises of a plurality of user-manipulated input controls, where the program performs steps comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify a plurality of interactive features for the legacy application, wherein the one or more interactive features include at least one feature that does not correspond to an interactive feature provided by the legacy application; and

physically reconfiguring the plurality of user-manipulated input controls and assigning functionality to said plurality of user-manipulated input controls as reconfigured, by the processor and based on the plurality of identified interactive features;

receiving an input from the user via one or more of the user-manipulated input controls;

processing the input at least partly by the supplemental software program; and

providing a result of said processing to the legacy application,

wherein the supplemental software program is a plugin, and

wherein said physically reconfiguring and said assigning functionality are further based on a current position of the user relative to the user-manipulated input controls as detected by the sensors.