Voice control of interactive whiteboard appliances

• US 10,409,550 B2
• Filed: 03/04/2016
• Issued: 09/10/2019
• Est. Priority Date: 03/04/2016
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Malatesha, Rathnakara, Wong, Lana, Kitada, Hiroshi
• Primary Examiner(s): Le, Thuykhanh
• Application Number:
  US15/061,806
• Publication Number:
  US 20170255446A1
• Time in Patent Office: 1,285 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A method and apparatus for providing voice command functionality to an interactive whiteboard appliance is provided. An interactive whiteboard appliance comprises: one or more processors; a non-transitory computer-readable medium having instructions embodied thereon, the instructions when executed by the one or more processors cause performance of: detecting, during execution of an annotation window on the interactive whiteboard appliance, a voice input received from a user; storing, in an audio packet, a recording of the voice input; transmitting the audio packet to a speech-to-text service; receiving, from the speech-to-text service, a command string comprising a transcription of the recording of the voice input; using voice mode command processing in a command processor, identifying, from the command string, an executable command that is executable by the interactive whiteboard appliance; causing the application of the interactive whiteboard appliance to execute the executable command.

Claims

1. An interactive whiteboard appliance comprising:

one or more processors;

a non-transitory computer-readable medium having instructions embodied thereon, the instructions when executed by the one or more processors cause performance of;

detecting at the interactive whiteboard appliance, during execution of an annotation window on the interactive whiteboard appliance, a voice input received from a user, wherein the annotation window on the interactive whiteboard appliance allows a user to view and edit content;

storing at the interactive whiteboard appliance, in an audio packet, a recording of the voice input;

transmitting the audio packet to a speech-to-text service that is physically separate from the interactive whiteboard appliance;

receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, a command string comprising a transcription of the recording of the voice input;

modifying at the interactive whiteboard appliance, the transcription of the recording by adding or removing one or more characters from the transcription to generate a modified transcription;

using voice mode command processing in a command processor of the interactive whiteboard appliance, identifying, from the command string that comprises the modified transcription of the recording of the voice input, an executable command that is executable by the interactive whiteboard appliance by comparing the modified transcription of the recording of the voice input to one or more pre-determined executable commands to identify a match between the modified transcription of the recording of the voice input and the one or more pre-determined executable commands;

causing an application of the interactive whiteboard appliance to execute the executable command.

2. The interactive whiteboard appliance of claim 1, wherein the instructions, when executed, further cause performance of:

detecting, during execution of the annotation window on an interactive whiteboard appliance, a stylus input from a user in the annotation window;

using pen mode command processing in the command processor, identifying, from the stylus input, an additional command;

causing the application of the interactive whiteboard appliance to execute the additional command.

3. The interactive whiteboard appliance of claim 1, wherein detecting the voice input comprises:

operating a microphone in a continuous listening mode that captures a continuous audio feed;

capturing the voice input from the continuous audio feed.

4. The interactive whiteboard appliance of claim 1, wherein detecting the voice input comprises:

operating a microphone in a start command mode;

detecting a start command;

in response to detecting the start command, capturing the voice input from a subsequent audio feed.

5. The interactive whiteboard appliance of claim 1, wherein receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, the command string comprises:

using a client device associated with the interactive whiteboard appliance, detecting, during execution of an application on the client device associated with the interactive whiteboard appliance, a second voice input from a second user;

using the client device, storing, in a second audio packet, a recording of the second voice input;

using the client device, transmitting the second audio packet to the speech-to-text service that is physically separate from the interactive whiteboard appliance;

wherein the instructions, when executed, further cause performance of;

at the interactive whiteboard appliance, receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, a command string comprising a transcription of the recording of the second voice input;

at the interactive whiteboard appliance, using voice mode command processing in the command processor, identifying, from the transcription of the recording of the second voice input, a second executable command by comparing the transcription of the recording of the second voice input to the one or more pre-determined executable commands to identify a match between the transcription of the recording of the second voice input and the one or more pre-determined executable commands;

at the interactive whiteboard appliance, causing the application of the interactive whiteboard appliance to execute the second executable command.

6. The interactive whiteboard appliance of claim 5, wherein the client device is associated with the interactive whiteboard appliance using a selection of the interactive whiteboard appliance received from the user via the application on the mobile device.

7. The interactive whiteboard appliance of claim 5, wherein the client device comprises a mobile device.

8. The interactive whiteboard appliance of claim 5, wherein the client device comprises another interactive whiteboard appliance.

9. The interactive whiteboard appliance of claim 1, wherein the audio packet includes an identifier of the interactive whiteboard appliance that identifies the interactive whiteboard appliance from a plurality of interactive whiteboard appliances.

10. The interactive whiteboard appliance of claim 9, wherein transmitting the audio packet to the speech-to-text service that is physically separate from the interactive whiteboard appliance comprises:

transmitting the audio packet via an audio packet queue that stores a plurality of queued audio packets transmitted by the plurality of interactive whiteboard appliances.

11. The interactive whiteboard appliance of claim 9, wherein receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, the command string comprises:

receiving the command string via a command string queue stores a plurality of queued command strings to be sent to the plurality of interactive whiteboard appliances;

determining that the identifier of the interactive whiteboard appliance included in the audio packet matches the identifier of the interactive whiteboard appliance.

12. The interactive whiteboard appliance of claim 1, wherein

the match between the modified transcription and the one or more pre-determined executable commands is a best match.

13. The interactive whiteboard appliance of claim 1, wherein causing the application of the interactive whiteboard appliance to execute the executable command comprises:

communicating the executable command to an operating system of the interactive whiteboard appliance via an application programming interface (API).

14. A method comprising:

detecting at the interactive whiteboard appliance, during execution of an annotation window on an interactive whiteboard appliance, a voice input received from a user, wherein the annotation window on the interactive whiteboard appliance allows a user to view and edit content;

storing at the interactive whiteboard appliance, in an audio packet, a recording of the voice input;

transmitting the audio packet to a speech-to-text service that is physically separate from the interactive whiteboard appliance;

receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, a command string comprising a transcription of the recording of the voice input;

modifying at the interactive whiteboard appliance, the transcription of the recording by adding or removing one or more characters from the transcription to generate a modified transcription;

using voice mode command processing in a command processor of the interactive whiteboard appliance, identifying, from the command string that comprises the modified transcription of the recording of the voice input, an executable command that is executable by the interactive whiteboard appliance by comparing the modified transcription of the recording of the voice input to one or more pre-determined executable commands to identify a match between the modified transcription of the recording of the voice input and the one or more pre-determined executable commands;

causing an application of the interactive whiteboard appliance to execute the executable command.

15. The method of claim 14, wherein receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, the command string comprises:

using a client device associated with the interactive whiteboard appliance, detecting, during execution of an application on the client device associated with the interactive whiteboard appliance, a second voice input from a second user;

using the client device, storing, in a second audio packet, a recording of the second voice input;

using the client device, transmitting the second audio packet to the speech-to-text service that is physically separate from the interactive whiteboard appliance;

wherein the instructions, when executed, further cause performance of;

at the interactive whiteboard appliance, receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, a command string comprising a transcription of the recording of the second voice input;

at the interactive whiteboard appliance, using voice mode command processing in the command processor, identifying, from the transcription of the recording of the second voice input, a second executable command by comparing the transcription of the recording of the second voice input to the one or more pre-determined executable commands to identify a match between the transcription of the recording of the second voice input and the one or more pre-determined executable commands;

at the interactive whiteboard appliance, causing the application of the interactive whiteboard appliance to execute the second executable command.

16. The method of claim 15, wherein the client device is associated with the interactive whiteboard appliance using a selection of the interactive whiteboard appliance received from the user via the application on the mobile device.

17. The method of claim 15, wherein the client device comprises a mobile device.

18. The method of claim 15, wherein the client device comprises another interactive whiteboard appliance.

19. A non-transitory computer-readable medium having instructions embodied thereon, the instructions when executed by the one or more processors cause performance of:

detecting at the interactive whiteboard appliance, during execution of an annotation window on an interactive whiteboard appliance, a voice input received from a user, wherein the annotation window on the interactive whiteboard appliance allows a user to view and edit content;

storing at the interactive whiteboard appliance, in an audio packet, a recording of the voice input;

transmitting the audio packet to a speech-to-text service that is physically separate from the interactive whiteboard appliance;

receiving, from the speech-to-text service that is physically separate from the interactive whiteboard appliance, a command string comprising a transcription of the recording of the voice input;

modifying at the interactive whiteboard appliance, the transcription of the recording by adding or removing one or more characters from the transcription to generate a modified transcription;

using voice mode command processing in a command processor of the interactive whiteboard appliance, identifying, from the command string that comprises the modified transcription of the recording of the voice input, an executable command that is executable by the interactive whiteboard appliance by comparing the modified transcription of the recording of the voice input to one or more pre-determined executable commands to identify a match between the modified transcription of the recording of the voice input and the one or more pre-determined executable commands;

causing an application of the interactive whiteboard appliance to execute the executable command.

20. The non-transitory computer-readable medium of claim 19, wherein detecting the voice input comprises one or more of operating a microphone in a continuous listening mode that captures a continuous audio feed or operating a microphone in a start command mode and detecting a start command.