METHOD FOR ADAPTIVE INTERACTION WITH A LEGACY SOFTWARE APPLICATION

• US 20130215005A1
• Filed: 02/17/2012
• Published: 08/22/2013
• Est. Priority Date: 02/17/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: SENANAYAKE, RUKMAN, Denker, Grit, Lincoln, Patrick D., Weiner, Steven S., Murray, John
• Granted Patent: US 8,928,582 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04809 Textured surface identifying touch areas, e.g. overlay structure for a virtual keyboard
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0362 with detection of 1D translations or rotations of an operating part of the device, e.g. scroll wheels, sliders, knobs, rollers or belts
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: H01H 2215/05 electromechanical
• CPC Class Codes: H01H 2217/022 Part of keyboard not operable
• CPC Class Codes: H01H 2217/038 Prompting
• CPC Class Codes: H01H 2239/07 UV or IR detection, e.g. of human body

Abstract

Methods are disclosed to support adaptive interaction with legacy software applications, without a need for rewriting those applications. The methods are for use with an interactive electronic system including a processor, a display, and an input device with user-manipulated controls. When the legacy application is executed, a supplemental software program, such as a plugin, is also executed and is utilized in order to identify currently relevant interactive features of the legacy application during execution. Functionality is dynamically assigned to the various user-manipulated controls based on the identified features. In one embodiment, detection of objects (particularly the user'"'"'s hands) proximate to the input controls is also employed in determining the assignment of functionality and/or in displaying a visual representation to the user of the available interactive choices. In another embodiment, the user-manipulated input controls are dynamically and physically reconfigured under control of the processor based on the identified features.

Claims

1. A method for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system comprising a display, a processor, one or more sensors for detecting a present object, and an input device having a plurality of user-manipulated input controls, the method comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify one or more interactive features for the legacy application;

dynamically assigning functionality to the user-manipulated input controls based on the identified interactive features; and

displaying a visual depiction of the assigned functionality,

wherein at least one of the assigning of functionality and the visual depiction is based on the sensors detecting an object proximate to the input controls.

2. The method of claim 1, wherein the detected object is an extremity of the user.

3. The method of claim 2, further comprising tracking a current position concurrently for each one of a plurality of user extremities.

4. The method of claim 1, wherein the visual depiction of the assigned functionality reflects a spatial layout of the corresponding input controls.

5. The method of claim 1 wherein the visual depiction further depicts a location of the object relative to the input controls.

6. The method of claim 1, wherein the dynamic assignment of functionality is based on a current position of the detected object relative to the input controls.

7. The method of claim 1, further comprising receiving an input from the user via one or more of the user-manipulated input controls, processing the input at least partly by the supplemental software program, and providing a result of said processing to the legacy application.

8. The method of claim 1, wherein the additional software program is a plugin.

9. The method of claim 1, wherein assigning functionality to the user-manipulated input controls includes assigning a plurality of alternative functions to at least one of the user-manipulated input controls, and the method further comprises allowing the user to select among the plurality of alternative functions.

10. The method of claim 9, wherein allowing the user to select comprises displaying a visual identification for each of the alternative functions, when an object is detected proximate to said input control having the alternative functions.

11. The method of claim 10, wherein displaying a visual identification for each of the alternative functions comprises cycling through said alternative functions by displaying each corresponding visual identification one at a time.

12. A method for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system including a display, a processor, and an input device having a plurality of user-manipulated input controls, the method comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify one or more interactive features for the legacy application; and

physically reconfiguring the user-manipulated input controls and assigning functionality to said input controls as reconfigured, by the processor and based on the identified interactive features.

13. The method of claim 12, wherein said physically reconfiguring comprises modifying a size and number of the input controls that can be individually actuated.

14. The method of claim 12, wherein the system further comprises one or more sensors for detecting a present object, the sensors being communicatively coupled to the processor.

15. The method of claim 14, wherein said physically reconfiguring and said assigning functionality are further based on a current position of the user relative to the user-manipulated input controls as detected by the sensors.

16. The method of claim 12, further comprising displaying a visual depiction of the assigned functionality.

17. The method of claim 16, wherein the visual depiction reflects a spatial layout of the corresponding reconfigured user-manipulated input controls.

18. The method of claim 12, further comprising receiving an input from the user via one or more of the user-manipulated input controls, processing the input at least partly by the supplemental software program, and providing a result of said processing to the legacy application.

19. The method of claim 12, wherein the additional software program is a plug n.

20. The method of claim 12, wherein the identified interactive features include at least one feature that does not correspond to an interactive feature provided by the legacy application.

21. A computer readable storage device containing an executable program for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system comprising a display, a processor, one or more sensors for detecting a present object, and an input device having a plurality of user-manipulated input controls, where the program performs steps comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify one or more interactive features for the legacy application;

dynamically assigning functionality to the user-manipulated input controls based on the identified interactive features; and

displaying a visual depiction of the assigned functionality,

wherein at least one of the assigning of functionality and the visual depiction is based on the sensors detecting an object proximate to the input controls.

22. A computer readable storage device containing an executable program for facilitating an interaction between an electronic system and a user in accordance with a legacy software application, the electronic system including a display, a processor, and an input device having a plurality of user-manipulated input controls, where the program performs steps comprising:

executing the legacy application with the processor;

executing a supplemental software program with the processor, to identify one or more interactive features for the legacy application; and

physically reconfiguring the user-manipulated input controls and assigning functionality to said input controls as reconfigured, by the processor and based on the identified interactive features.