DYNAMIC MULTILINGUAL SPEECH RECOGNITION

• US 20200134267A1
• Filed: 12/30/2019
• Published: 04/30/2020
• Est. Priority Date: 09/28/2018
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Liu, Su, Hwang, Inseok, Rozner, Eric, Yoo, Chungkuk
• Granted Patent: US 11,526,681 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

A method, computer program product, and a system where a processor(s), monitors multilingual switches performed on a client on behalf of a given user. Based on the monitoring, the processor(s) identifies switch patterns of the given user to generate a service profile for the user of machine learned multilingual switch patterns for the given user. The processor(s) determines a priority order for languages comprising the voice input streams, for the given user. The processor(s) obtains a new translation request initiated by the client, on behalf of the given user and applies the priority order to identify one or more languages spoken in a voice input stream of the new translation request. The processor(s) transmits indicators of the identified one or more languages to the client, where upon receiving the indicators, the client translates the voice input stream from the identified one or more languages to one or more target languages.

Claims

1. A computer-implemented method, comprising:

monitoring, by one or more processors, multilingual switches performed on one or more clients on behalf of a given user, wherein service requests initiated by the one or more clients on behalf of the given user comprise a portion of the multilingual switches, wherein a portion of the service requests each comprise a voice input stream;

based on the monitoring, identifying, by the one or more processors, based on the multilingual switches performed on the one or more clients, switch patterns of the given user;

generating, by the one or more processors, a service profile for the user, wherein the service profile comprises machine learned multilingual switch patterns for the given user;

determining, by the one or more processors, based on the machine learned multilingual switch patterns for the given user, a priority order for languages comprising the voice input streams, for the given user;

obtaining, by the one or more processors, a new service request comprising a voice input stream, initiated by a given client of the one or more clients, on behalf of the given user;

applying, by the one or more processors, the priority order to identify one or more languages spoken in the voice input stream of the new service request; and

transmitting, by the one or more processors, indicators of the identified one or more languages to the given client, wherein upon receiving the indicators, the given client translates the voice input stream from the identified one or more languages to one or more target languages.

2. The computer-implemented method of claim 1, wherein the portion of the service requests further comprise contextual data related to the voice input stream.

3. The computer-implemented method of claim 1, wherein the given user is selected from the group consisting of:

an individual user and a group of users.

4. The computer-implemented method of claim 1, wherein identifying switch patterns of the given user to generate the service profile for the user further comprises:

identifying, by the one or more processors, for the monitored service requests, dependencies between factors in the contexts of the service requests and input languages of the voice input streams of the portion of the service requests; and

generating, by the one or more processors, a set of rules representing the identified dependencies, wherein the machine learned multilingual switch patterns comprise the set of rules.

5. The computer-implemented method of claim 1, wherein the monitoring further comprises:

generating, by the one or more processors, based on an initial service request from a client of the one or more client of the service requests from the client, a data structure; and

applying, by the one or more processors, the data structure to subsequent service requests from the one or more clients of the service requests from the one or more clients, wherein applying the data structure tracks multilingual switching operations for the service requests from the one or more clients.

6. The computer-implemented method of claim 1, wherein the given client translates the voice input stream comprising the new request in real-time.

7. The computer-implemented method of claim 1, wherein the multilingual switches further comprise activities performed by the given user by utilizing applications executing on the one or more clients.

8. The computer-implemented method of claim 9, wherein the activities are selected from the group consisting of:

usage of the given user of a web browser, usage of the given user of a word processing programs, and usage of the given user of an accounting programs.

9. The computer-implemented method of claim 1, wherein generating the service profile for the user, further comprises:

determining, by the one or more processors, a location of the user.

10. The computer-implemented method of claim 9, wherein the location determining the priority order for the languages comprising the voice input streams is further based on the location of the user.

11. The computer-implemented method of claim 1, wherein the obtaining the new service request initiated by the given client further comprises:

identifying, by the one or more processors, the given user as an initiator of the new service request.

12. A computer program product comprising:

a computer readable storage medium readable by one or more processors and storing instructions for execution by the one or more processors for performing a method comprising;

monitoring, by the one or more processors, multilingual switches performed on one or more clients on behalf of a given user, wherein service requests initiated by the one or more clients on behalf of the given user comprise a portion of the multilingual switches, wherein a portion of the service requests each comprise a voice input stream;

based on the monitoring, identifying, by the one or more processors, based on the multilingual switches performed on the one or more clients, switch patterns of the given user;

generating, by the one or more processors, a service profile for the user, wherein the service profile comprises machine learned multilingual switch patterns for the given user;

determining, by the one or more processors, based on the machine learned multilingual switch patterns for the given user, a priority order for languages comprising the voice input streams, for the given user;

obtaining, by the one or more processors, a new service request comprising a voice input stream, initiated by a given client of the one or more clients, on behalf of the given user;

applying, by the one or more processors, the priority order to identify one or more languages spoken in the voice input stream of the new service request; and

transmitting, by the one or more processors, indicators of the identified one or more languages to the given client, wherein upon receiving the indicators, the given client translates the voice input stream from the identified one or more languages to one or more target languages.

13. The computer program product of claim 12, wherein the given user is selected from the group consisting of:

an individual user and a group of users.

14. The computer program product of claim 12, wherein identifying switch patterns of the given user to generate the service profile for the user further comprises:

identifying, by the one or more processors, for the monitored service requests, dependencies between factors in the contexts of the service requests and input languages of the voice input streams of the portion of the service requests; and

generating, by the one or more processors, a set of rules representing the identified dependencies, wherein the machine learned multilingual switch patterns comprise the set of rules.

15. The computer program product of claim 12, wherein the monitoring further comprises:

generating, by the one or more processors, based on an initial service request from a client of the one or more clients of the service requests from the client, a data structure; and

applying, by the one or more processors, the data structure to subsequent service requests from the one or more clients of the service requests from the client, wherein applying the data structure tracks multilingual switching operations for the service requests from the one or more clients.

16. The computer program product of claim 12, wherein the given client translates the voice input stream comprising the new request in real-time.

17. The computer program product of claim 12, wherein the multilingual switches further comprise activities performed by the given user by utilizing applications executing on the one or more clients.

18. The computer program product of claim 17, wherein the activities are selected from the group consisting of:

usage of the given user of a web browser, usage of the given user of a word processing programs, and usage of the given user of an accounting programs.

19. The computer program product of claim 17, wherein the portion of the service requests further comprise contextual data related to the voice input stream.

20. A system comprising:

a memory;

one or more processors in communication with the memory;

program instructions executable by the one or more processors via the memory to perform a method, the method comprising;

monitoring, by the one or more processors, multilingual switches performed on one or more clients on behalf of a given user, wherein service requests initiated by the one or more clients on behalf of the given user comprise a portion of the multilingual switches, wherein a portion of the service requests each comprise a voice input stream;

based on the monitoring, identifying, by the one or more processors, based on the multilingual switches performed on the one or more clients, switch patterns of the given user;

generating, by the one or more processors, a service profile for the user, wherein the service profile comprises machine learned multilingual switch patterns for the given user;

determining, by the one or more processors, based on the machine learned multilingual switch patterns for the given user, a priority order for languages comprising the voice input streams, for the given user;

obtaining, by the one or more processors, a new service request comprising a voice input stream, initiated by a given client of the one or more clients, on behalf of the given user;

applying, by the one or more processors, the priority order to identify one or more languages spoken in the voice input stream of the new service request; and

transmitting, by the one or more processors, indicators of the identified one or more languages to the given client, wherein upon receiving the indicators, the given client translates the voice input stream from the identified one or more languages to one or more target languages.