Method and apparatus of remote management of computer system using voice and gesture based input

• US 9,176,649 B2
• Filed: 03/15/2013
• Issued: 11/03/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: American Megatrends International, LLC
• Original Assignee: American Megatrends Incorporated
• Inventors: Ayanam, Varadachari Sudan, Christopher, Samvinesh
• Primary Examiner(s): Harrell, Robert B
• Application Number:
  US13/843,820
• Publication Number:
  US 20140282045A1
• Time in Patent Office: 963 Days
• Field of Search: 340/531, 709/208
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Method, application, and medium of remote management of a computer system using voice and gesture based input are disclosed. An application module is executed on a processor of a computing device. The application module (a) displays a first user interface on the computing device, the interface allowing a user to enter a first input which includes at least one of a voice input and a gesture input, (b) receives input data which include at least one of audio input data and multitouch input data from the computing device representing the first input, (c) determines, based on the input data, a first command from a plurality of commands targeted to a baseboard management controller (BMC) managing a computer system, (d) constructs a first message including the determined command, and (e) sends the first message to the BMC over a network.

Claims

1. A system, comprising:

a computer system;

a baseboard management controller (BMC) communicatively connected to the computer system, wherein the BMC is configured to manage the computer system; and

a computing device remotely communicatively connected to the BMC through a network, the computing device having;

a processor,

an audio input device configured to receive a voice input, and to convert the voice input to audio input data,

a multitouch device configured to receive a gesture input, to detect multiple touch points simultaneously based on the gesture input, and to convert the detected multiple touch points to multitouch input data, and

a non-volatile memory storing an application module of a management application,

wherein the application module, when executed on the processor, is configured to

display, on the computing device, a first user interface allowing a user to enter a first input, the first input including at least one of the voice input and the gesture input;

receive input data from the computing device representing the first input, the input data including at least one of the audio input data and the multitouch input data;

determine, based on the input data, a first command from a plurality of commands targeted to the BMC;

construct a first message including the determined first command; and

send the first message from the computing device to the BMC over the network,

wherein the BMC is configured to receive the first message from the computing device, and to perform the first command to manage the computer system based on the received first message.

2. The system of claim 1, wherein the application module is configured to

store a plurality of record entries mapped to the plurality of commands, respectively;

generate an input entry based on the input data;

compare the input entry to the plurality of record entries and select a first record entry of the plurality of record entries based on the comparison; and

obtain the first command that is mapped to the first record entry.

3. The system of claim 2, wherein the input data is the audio input data, wherein the plurality of record entries and the input entry each include a text string, wherein the application module is configured to

recognize, through voice recognition, a first text string from the audio input data and include the first text string in the input entry;

and

match the first text string with the text strings included in the plurality of record entries.

4. The system of claim 2, wherein the input data is the audio input data, wherein the plurality of record entries and the input entry each define a set of audio characteristics, wherein the application module is configured to

determine a first set of audio characteristics from the input data and use the input entry to define the first set of audio characteristics;

and

compare the first set of audio characteristics with the sets of audio characteristics defined in the plurality of record entries.

5. The system of claim 2, wherein the input data is the multitouch input data, wherein the plurality of record entries and the input entry each define a gesture, wherein the application module is configured to

recognize, through gesture recognition, a first gesture from the multitouch input data and use the input entry to define the first gesture;

and

match the first gesture with the gestures defined in the plurality of record entries.

6. The system of claim 2, wherein the input data is the multitouch input data, wherein the plurality of record entries and the input entry each define a set of multitouch characteristics, wherein the application module is configured to

determine a first set of multitouch characteristics from the input data and use the input entry to define the first set of multitouch characteristics;

and

compare the first set of multitouch characteristics with the sets of multitouch characteristics defined in the plurality of record entries.

7. The system of claim 1, wherein the application module is configured to

encapsulate the first message in at least one first network packet;

determine a network address of the BMC; and

transmit the at least one first network packet to the network address through the network.

8. The system of claim 7, wherein the first message is an Intelligent Platform Management Interface (“

IPMI”

) message, wherein the application module is configured to

receive, from the BMC, at least one second network packet encapsulating an IPMI response message;

extract the IPMI response message from the second network packets;

extract response data from the IPMI response message; and

display a second interface on the computing device, the second interface having the response data.

9. The system of claim 1, wherein the application module comprises:

a management module configured to manage interactions between the management application and the BMC as well as between the management application and the user;

an input module configured to

receive the audio input data from the audio input device of the computing device;

receive the multitouch input data from the multitouch device of the computing device; and

determine, based on the input data, the first command from the plurality of commands;

a user interface module configured to display, as instructed by the management module, user interfaces for interacting with the user, including the first user interface, on a display of the computing device;

a KVM module configured to receive, from the BMC, video output from the computer system redirected by the BMC;

a virtual media module configured to instruct the BMC to emulate a selected type of media to the computer system and to provide data to the computer system through the emulated media;

a message module configured to construct the first message, wherein the first message is an Intelligent Platform Management Interface (“

IPMI”

) message; and

a communication module configured to establish a communication channel between the management application and the BMC and to send the first message to the BMC.

10. The system of claim 9, wherein the communication module communicates with the BMC through a TCP/IP socket or a UDP/IP socket.

11. The system of claim 1, wherein the plurality of commands include commands for managing at least one of the following management functions of the BMC:

a KVM management function, a Virtual Media management function, a sensors management function, a Field Replaceable Unit (FRU) management function, an Event Log management function, and a configuration management function.

12. The system of claim 1, wherein the plurality of commands include commands for managing at least one of the following devices of the computer system:

an IPM Device, a Sensor Device, a SDR Repository Device, a SEL Device, an FRU Inventory Device, an Event Receiver Device, an Event Generator Device, an Application Device, a PEF Device, an Alert Processing Device, and a Chassis Device.

13. The system of claim 1, wherein the application module is configured to

generate the first user interface, wherein the first user interface shows options that allow the user to select at least one of the voice input and the gesture input to manage the BMC.

14. A method of remote management of a computer system using voice and gesture based input, comprising:

generating, by an application module of a management application executed at a computing device, a first user interface of the management application, wherein the first user interface shows options that allow a user to select at least one of a voice input and a gesture input for a baseboard management controller (BMC) communicatively connected to the computer system, wherein the computing device is remotely communicatively connected to the BMC through a network, and the BMC is configured to manage the computer system;

displaying, by the application module on the computing device, the first user interface allowing the user to enter a first input, the first input including at least one of the voice input and the gesture input, wherein the computing device has an audio input device configured to receive the voice input, and to convert the voice input to audio input data, and a multitouch device configured to receive the gesture input, to detect multiple touch points simultaneously based on the gesture input, and to convert the detected multiple touch points to multitouch input data;

receiving, by the application module, input data from the computing device representing the first input, the input data including at least one of the audio input data and the multitouch input data;

determining, by the application module based on the input data, a first command from a plurality of commands targeted to the BMC managing the computer system;

constructing, by the application module, a first message including the determined first command; and

sending, by the application module, the first message from the computing device to the BMC over the network,

wherein the BMC is configured to receive the first message from the computing device, and to perform the first command to manage the computer system based on the received first message.

15. The method of claim 14, wherein the application module comprises:

a management module configured to manage interactions between the management application and the BMC as well as between the management application and the user;

an input module configured to

receive the audio input data from the audio input device of the computing device;

receive the multitouch input data from the multitouch device of the computing device; and

determine, based on the input data, the first command from the plurality of commands;

a user interface module configured to display, as instructed by the management module, user interfaces for interacting with the user, including the first user interface, on a display of the computing device;

a KVM module configured to receive, from the BMC, video output from the computer system redirected by the BMC;

a virtual media module configured to instruct the BMC to emulate a selected type of media to the computer system and to provide data to the computer system through the emulated media;

a message module configured to construct the first message, wherein the first message is an Intelligent Platform Management Interface (“

IPMI”

) message; and

a communication module configured to establish a communication channel between the management application and the BMC and to send the first message to the BMC.

16. The method as claimed in claim 15, wherein the communication module communicates with the BMC through a TCP/IP socket or a UDP/IP socket.

17. The method as claimed in claim 14, wherein the input data is the audio input data, and wherein the method comprises:

storing a plurality of text record entries mapped to the plurality of commands, respectively;

generating a text input entry based on the audio input data, wherein the plurality of text record entries and the text input entry each include a text string;

recognizing, through voice recognition, a first text string from the audio input data and include the first text string in the text input entry;

matching the first text string with the text strings included in the plurality of text record entries;

comparing the text input entry to the plurality of text record entries and select a first text record entry of the plurality of text record entries based on the comparison; and

obtaining the first command that is mapped to the first text record entry.

18. The method as claimed in claim 14, wherein the input data is the multitouch input data, and wherein the method comprises-comprising:

storing a plurality of multitouch record entries mapped to the plurality of commands, respectively;

generating a multitouch input entry based on the multitouch input data, wherein the plurality of multitouch record entries and the multitouch input entry each define a set of audio characteristics and a gesture;

determining a first set of multitouch characteristics from the multitouch input data and using the multitouch input entry to define the first set of multitouch characteristics;

comparing the first set of multitouch characteristics with the sets of multitouch characteristics defined in the plurality of multitouch record entries;

recognizing, through gesture recognition, a first gesture from the multitouch input data and using the multitouch input entry to define the first gesture; and

matching the first gesture with the gestures defined in the plurality of multitouch record entries.

19. The method as claimed in claim 14, further comprising:

encapsulating, by the application module, the first message in at least one first network packet, wherein the first message is an Intelligent Platform Interface (“

IPMI”

) message;

determining, by the application module, a network address of the BMC;

transmitting, by the application module, the at least one first network packet to the network address through the network;

receiving, from the BMC, at least one second network packet encapsulating an IPMI response message;

extracting, by the application module, the IPMI response message from the second network packets;

extracting, by the application module, response data from the IPMI response message; and

displaying, by the application module, the response data on a second interface on the computing device.

20. A non-transitory computer storage medium having computer-executable instructions stored thereon which, when executed by a processor of a computing device, cause the processor to:

generate a first user interface showing options that allow a user to select at least one of a voice input and a gesture input for a baseboard management controller (BMC) communicatively connected to a computer system, wherein the computing device is remotely communicatively connected to the BMC through a network, and the BMC is configured to manage the computer system;

display the first user interface on the computing device for the user to enter a first input, the first input including at least one of the voice input and the gesture input, wherein the computing device has an audio input device configured to receive the voice input, and to convert the voice input to audio input data, and a multitouch device configured to receive the gesture input, to detect multiple touch points simultaneously based on the gesture input, and to convert the detected multiple touch points to multitouch input data;

receive input data from the computing device representing the first input, the input data including at least one of the audio input data and the multitouch input data;

determine, based on the input data, a first command from a plurality of commands targeted to the BMC managing the computer system;

construct a first message including the determined first command; and

send the first message from the computing device to the BMC over the network,

wherein the BMC is configured to receive the first message from the computing device, and to perform the first command to manage the computer system based on the received first message.

21. The non-transitory computer storage medium of claim 20, wherein the computer-executable instructions comprise:

a management module configured to manage interactions between the computing device and the BMC as well as between the computing device and the user;

an input module configured to

receive the audio input data from the audio input device of the computing device;

receive the multitouch input data from the multitouch device of the computing device; and

determine, based on the input data, the first command from the plurality of commands;

a user interface module configured to display, as instructed by the management module, user interfaces for interacting with the user, including the first user interface, on a display of the computing device;

a KVM module configured to receive, from the BMC, video output from the computer system redirected by the BMC;

a virtual media module configured to instruct the BMC to emulate a selected type of media to the computer system and to provide data to the computer system through the emulated media;

a message module configured to construct the first message, wherein the first message is an Intelligent Platform Management Interface (“

IPMI”

) message; and

a communication module configured to establish a communication channel between the computing device and the BMC and to send the first message to the BMC.

22. The non-transitory computer storage medium of claim 21, wherein the communication module communicates with the BMC through a TCP/IP socket or a UDP/IP socket.

23. The non-transitory computer storage medium of claim 20, wherein the input data is the audio input data, and the computer-executable instructions cause the processor to:

store a plurality of text record entries mapped to the plurality of commands, respectively;

generate a text input entry based on the audio input data, wherein the plurality of text record entries and the text input entry each include a text string;

recognize, through voice recognition, a first text string from the audio input data and include the first text string in the text input entry;

match the first text string with the text strings included in the plurality of text record entries;

compare the text input entry to the plurality of text record entries and select a first text record entry of the plurality of text record entries based on the comparison; and

obtain the first command that is mapped to the first text record entry.

24. The non-transitory computer storage medium of claim 20, wherein the input data is the multitouch input data, and the computer-executable instructions cause the processor to:

store a plurality of multitouch record entries mapped to the plurality of commands, respectively;

generate a multitouch input entry based on the multitouch input data, wherein the plurality of multitouch record entries and the multitouch input entry each define a set of audio characteristics and a gesture;

determine a first set of multitouch characteristics from the multitouch input data and use the multitouch input entry to define the first set of multitouch characteristics; and

compare the first set of multitouch characteristics with the sets of multitouch characteristics defined in the plurality of multitouch record entries;

recognize, through gesture recognition, a first gesture from the multitouch input data and use the multitouch input entry to define the first gesture; and

match the first gesture with the gestures defined in the plurality of multitouch record entries.

25. The non-transitory computer storage medium of claim 20, wherein the computer-executable instructions cause the processor to:

encapsulate the first message in at least one first network packet, wherein the first message is an Intelligent Platform Management Interface (“

IPMI”

) message;

determine a network address of the BMC;

transmit the at least one first network packet to the network address through the network;

receive, from the BMC, at least one second network packet encapsulating an IPMI response message;

extract the IPMI response message from the second network packets;

extract response data from the IPMI response message; and

display a second interface on the computing device, the second interface having the response data.