METHOD AND APPARATUS OF REMOTE MANAGEMENT OF COMPUTER SYSTEM USING VOICE AND GESTURE BASED INPUT

• US 20140282045A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: American Megatrends International, LLC
• Original Assignee: American Megatrends Incorporated
• Inventors: Ayanam, Varadachari Sudan, Christopher, Samvinesh
• Granted Patent: US 9,176,649 B2
• Time in Patent Office: Days
• US Class Current: 715/740
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Method, application, and medium of remote management of a computer system using voice and gesture based input are disclosed. An application module is executed on a processor of a computing device. The application module (a) displays a first user interface on the computing device, the interface allowing a user to enter a first input which includes at least one of a voice input and a gesture input, (b) receives input data which include at least one of audio input data and multitouch input data from the computing device representing the first input, (c) determines, based on the input data, a first command from a plurality of commands targeted to a baseboard management controller (BMC) managing a computer system, (d) constructs a first message including the determined command, and (e) sends the first message to the BMC over a network.

Claims

1. A management application, comprising:

an application module executed on a processor of a computing device and configured to

display, on the computing device, a first user interface allowing a user to enter a first input, the first input including at least one of a voice input and a gesture input;

receive input data from the computing device representing the first input, the input data including at least one of audio input data and multitouch input data;

determine, based on the input data, a first command from a plurality of commands targeted to a baseboard management controller (BMC) managing a computer system;

construct a first message including the determined command; and

send the first message to the BMC over a network.

2. The management application of claim 1, wherein the plurality of commands include commands for managing at least one of the following management functions of the BMC:

a KVM management function, a Virtual Media management function, a sensors management function, a Field Replaceable Unit (FRU) management function, an Event Log management function, and a configuration management function.

3. The management application of claim 1, wherein the plurality of commands include commands for managing at least one of the following devices of the computer system:

an IPM Device, a Sensor Device, a SDR Repository Device, a SEL Device, an FRU Inventory Device, an Event Receiver Device, an Event Generator Device, an Application Device, a PEF Device, an Alert Processing Device, and a Chassis Device.

4. The management application of claim 1, wherein the application module is configured to

store a plurality of record entries mapped to the plurality of commands, respectively;

generate an input entry based on the input data;

compare the input entry to the plurality of record entries and select a first record entry of the plurality of record entries based on the comparison; and

obtain the first command that is mapped to the first record entry.

5. The management application of claim 4, wherein the input data are audio input data, wherein the plurality of record entries and the input entry each include a text string, wherein the application module is configured to

recognize, through voice recognition, a first text string from the audio input data and include the first text string in the input entry;

and

match the first text string with the text strings included in the plurality of record entries.

6. The management application of claim 4, wherein the input data are audio input data, wherein the plurality of record entries and the input entry each define a set of audio characteristics, wherein the application module is configured to

determine a first set of audio characteristics from the input data and use the input entry to define the first set of audio characteristics;

and

compare the first set of audio characteristics with the sets of audio characteristics defined in the plurality of record entries.

7. The management application of claim 4, wherein the input data are multitouch input data, wherein the plurality of record entries and the input entry each define a gesture, wherein the application module is configured to

recognize, through gesture recognition, a first gesture from the multitouch input data and use the input entry to define the first gesture;

and

match the first gesture with the gestures defined in the plurality of record entries.

8. The management application of claim 4, wherein the input data are multitouch input data, wherein the plurality of record entries and the input entry each define a set of multitouch characteristics, wherein the application module is configured to

determine a first set of multitouch characteristics from the input data and use the input entry to define the first set of multitouch characteristics;

and

compare the first set of multitouch characteristics with the sets of multitouch characteristics defined in the plurality of record entries.

9. The management application of claim 1, wherein the application module is configured to

generate the first user interface, wherein the first user interface shows options that allow a user to select at least one of a voice input and a gesture input to manage the BMC.

10. The management application of claim 1, wherein the application module is configured to

encapsulate the first message in at least one first network packet;

determine a network address of the BMC; and

transmit the at least one first network packet to the network address through the network.

11. The management application of claim 10, wherein the first message is an IPMI message, wherein the application module is configured to

receive at least one second network packet encapsulating an IPMI response message;

extract the IPMI response message from the second network packets;

extract response data from the IPMI response message; and

display a second interface on the computing device, the second interface having the response data.

12. The management application of claim 1, comprising:

a management module configured to manage interactions between the management application and the BMC as well as between the management application and a user;

an input module configured to

receive the audio input data from an audio input device of the computing device; and

receive the multitouch input data from a multitouch device of the computing device, wherein the input module determines, based on the input data, the first command from the plurality of commands;

a user interface module configured to display, as instructed by the management module, user interfaces for interacting with the user, including the first user interface, on a display of the computing device;

a KVM module configured to receive, from the BMC, video output from the computer system redirected by the BMC;

a virtual media module configured to instruct the BMC to emulate a selected type of media to the computer system and to provide data to the computer system through the emulated media;

a message module configured to construct the first message, wherein the first message is an IPMI message; and

a communication module configured to establish a communication channel between the management application and the BMC and to send the first message to the BMC.

13. The management application of claim 12, wherein the communication module communicates with the BMC through a TCP/IP socket or a UDP/IP socket.

14. A method of remote management of a computer system using voice and gesture based input, comprising:

generating a first user interface of a management application, wherein the first user interface shows options, shows options that allow a user to select at least one of a voice input and a gesture input to manage a baseboard management controller (BMC);

displaying, on a computing device, the first user interface allowing a user to enter a first input, the first input including at least one of a voice input and a gesture input;

receiving input data from the computing device representing the first input, the input data including at least one of audio input data and multitouch input data;

determining, based on the input data, a first command from a plurality of commands targeted to a BMC managing a computer system;

constructing a first message including the determined command; and

sending the first message to the BMC over a network.

15. The method as claimed in claim 14, wherein the input data are audio input data, wherein the plurality of record entries and the input entry each include a text string, and the method comprising:

storing a plurality of text record entries mapped to the plurality of commands, respectively;

recognizing, through voice recognition, a first text string from the audio input data and include the first text string in the text input entry;

matching the first text string with the text strings included in the plurality of text record entries;

generating a text input entry based on the audio input data;

comparing the text input entry to the plurality of text record entries and select a first text record entry of the plurality of text record entries based on the comparison; and

obtaining the first command that is mapped to the first text record entry.

16. The method as claimed in claim 14, wherein the input data are multitouch input data, wherein the plurality of record entries and the input entry each define a gesture, wherein the method comprising:

determining a first set of multitouch characteristics from the multitouch input data and using a multitouch input entry to define the first set of multitouch characteristics;

comparing the first set of multitouch characteristics with the sets of multitouch characteristics defined in the plurality of multitouch record entries;

recognizing, through gesture recognition, a first gesture from the multitouch input data and using the multitouch input entry to define the first gesture; and

matching the first gesture with the gestures defined in the plurality of multitouch record entries.

17. The method as claimed in claim 14, comprising:

encapsulating the first message in at least one first network packet, wherein the first message is an IPMI message;

determining a network address of the BMC;

transmitting the at least one first network packet to the network address through the network;

receiving at least one second network packet encapsulating an IPMI response message;

extracting the IPMI response message from the second network packets;

extracting response data from the IPMI response message; and

displaying the response data on a second interface on the computing device.

18. The method as claimed in claim 14, comprising:

managing, at a management module, interactions between the management application and the BMC as well as between the management application and a user;

receiving, at an input module, the audio input data from an audio input device of the computing device and the multitouch input data from a multitouch device of the computing device, wherein the input module determines, based on the input data, the first command from the plurality of commands;

displaying at a user interface module and as instructed by the management module, user interfaces for interacting with the user, including the first user interface, on a display of the computing device;

receiving at a KVM module and from the BMC, video output from the computer system redirected by the BMC;

instructing, at a virtual media module, the BMC to emulate a selected type of media to the computer system and to provide data to the computer system through the emulated media;

constructing at a message module, a first IPMI message; and

establishing, at a communication module, a communication channel between the management application and the BMC and to send the first IPMI message to the BMC.

19. The method as claimed in claim 18, wherein the communication module communicates with the BMC through a TCP/IP socket or a UDP/IP socket.

20. A non-transitory computer storage medium having computer-executable instructions stored thereon which, when executed by a processor of a computing device, cause the processor to:

generate a first user interface showing options that allow a user to select at least one of a voice input and a gesture input to manage a baseboard management controller (BMC);

display the first user interface on the computing device for the user to enter a first input, the first input including at least one of a voice input and a gesture input;

receive input data from the computing device representing the first input, the input data including at least one of audio input data and multitouch input data;

determine, based on the input data, a first command from a plurality of commands targeted to a BMC managing a computer system;

construct a first message including the determined command; and

send the first message to the BMC over a network.

21. The non-transitory computer storage medium of claim 20, wherein the input data are audio input data, wherein the plurality of record entries and the input entry each include a text string, and the computer-executable instructions cause the processor to:

store a plurality of text record entries mapped to the plurality of commands, respectively;

recognize, through voice recognition, a first text string from the audio input data and include the first text string in the text input entry;

match the first text string with the text strings included in the plurality of text record entries;

generate a text input entry based on audio input data;

compare the text input entry to the plurality of text record entries and select a first text record entry of the plurality of text record entries based on the comparison; and

obtain the first command that is mapped to the first text record entry.

22. The non-transitory computer storage medium of claim 20, wherein the input data are multitouch input data, wherein the plurality of record entries and the input entry each define a gesture, and the computer-executable instructions cause the processor to:

determine a first set of multitouch characteristics from the multitouch input data and use a multitouch input entry to define the first set of multitouch characteristics; and

compare the first set of multitouch characteristics with the sets of multitouch characteristics defined in the plurality of multitouch record entries;

recognize, through gesture recognition, a first gesture from the multitouch input data and use the multitouch input entry to define the first gesture; and

match the first gesture with the gestures defined in the plurality of multitouch record entries.

23. The non-transitory computer storage medium of claim 20, wherein the computer-executable instructions cause the processor to:

encapsulate the first message in at least one first network packet, wherein the first message is an IPMI message;

determine a network address of the BMC;

transmit the at least one first network packet to the network address through the network;

receive at least one second network packet encapsulating an IPMI response message;

extract the IPMI response message from the second network packets;

extract response data from the IPMI response message; and

display a second interface on the computing device, the second interface having the response data.

24. The non-transitory computer storage medium of claim 20, wherein the computer-executable instructions cause the processor to:

at a management module, manage interactions between the management application and the BMC as well as between the management application and a user;

at an input module, receive the audio input data from an audio input device of the computing device and the multitouch input data from a multitouch device of the computing device, wherein the input module determines, based on the input data, the first command from the plurality of commands;

at a user interface module, display, as instructed by the management module, user interfaces for interacting with the user, including the first user interface, on a display of the computing device;

at a KVM module, receive, from the BMC, video output from the computer system redirected by the BMC;

at a virtual media module, instruct the BMC to emulate a selected type of media to the computer system and to provide data to the computer system through the emulated media;

at a message module, construct the first message, wherein the first message is an IPMI message; and

at a communication module, establish a communication channel between the management application and the BMC and to send the first message to the BMC.

25. The non-transitory computer storage medium of claim 24, wherein the communication module communicates with the BMC through a TCP/IP socket or a UDP/IP socket.