Dynamic multilingual speech recognition

• US 10,565,320 B1
• Filed: 09/28/2018
• Issued: 02/18/2020
• Est. Priority Date: 09/28/2018
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Liu, Su, Hwang, Inseok, Rozner, Eric, Yoo, Chungkuk
• Primary Examiner(s): Vo, Huyen X
• Application Number:
  US16/145,611
• Time in Patent Office: 508 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

A method, computer program product, and a system where a processor(s), monitors multilingual switches performed on a client on behalf of a given user. Based on the monitoring, the processor(s) identifies switch patterns of the given user to generate a service profile for the user of machine learned multilingual switch patterns for the given user. The processor(s) determines a priority order for languages comprising the voice input streams, for the given user. The processor(s) obtains a new translation request initiated by the client, on behalf of the given user and applies the priority order to identify one or more languages spoken in a voice input stream of the new translation request. The processor(s) transmits indicators of the identified one or more languages to the client, where upon receiving the indicators, the client translates the voice input stream from the identified one or more languages to one or more target languages.

Claims

1. A computer-implemented method, comprising:

monitoring, by one or more processors, multilingual switches performed on a client on behalf of a given user, wherein service requests initiated by the client on behalf of the given user comprise a portion of the multilingual switches, wherein translation requests comprise a portion of the service requests, and wherein the translation requests each comprise a voice input stream and contextual data related to the voice input stream;

based on the monitoring, identifying, by the one or more processors, based on the multilingual switches performed on the client, switch patterns of the given user to generate a service profile for the user, wherein the service profile comprises machine learned multilingual switch patterns for the given user;

determining, by the one or more processors, based on the machine learned multilingual switch patterns for the given user, a priority order for languages comprising the voice input streams, for the given user;

obtaining, by the one or more processors, a new translation request initiated by the client, on behalf of the given user;

applying, by the one or more processors, the priority order to identify one or more languages spoken in a voice input stream comprising the new translation request; and

transmitting, by the one or more processors, indicators of the identified one or more languages to the client, wherein upon receiving the indicators, the client translates the voice input stream from the identified one or more languages to one or more target languages.

2. The computer-implemented method of claim 1, wherein the given user is selected from the group consisting of:

an individual user and a group of users.

3. The computer-implemented method of claim 1, wherein identifying switch patterns of the given user to generate the service profile for the user further comprises:

identifying, by the one or more processors, for the monitored translation requests, dependencies between factors in the contexts of the translation requests and input languages of the voice input streams of the translation requests; and

generating, by the one or more processors, a set of rules representing the identified dependencies, wherein the machine learned multilingual switch patterns comprise the set of rules.

4. The computer implemented method of claim 1, wherein the contexts of each request of the requests are selected from the group consisting of:

an entry application for the request on the client, a location of a client upon entry of the request, and an element of a digital wardrobe of the client.

5. The computer implemented method of claim 1, wherein the contexts of each request of the requests are selected from the group consisting of:

the given user changing input methods from one language to another, the given user browsing web-enabled interfaces in different languages, and the given user using one or more applications in a given geographic location.

6. The computer-implemented method of claim 1, wherein the monitoring further comprises:

generating, by the one or more processors, based on an initial translation request from the client of the translation requests from the client, a data structure; and

applying, by the one or more processors, the data structure to subsequent translation requests from the client of the translation requests from the client, wherein applying the data structure tracks multilingual switching operations for the translation requests from the client.

7. The computer-implemented method of claim 1, wherein the client translates the voice input stream comprising the new request in real-time.

8. The computer-implemented method of claim 1, wherein the multilingual switches further comprise activities performed by the given user by utilizing applications executing on the client.

9. The computer-implemented method of claim 8, wherein the activities are selected from the group consisting of:

usage of the given user of a web browser, usage of the given user of a word processing programs, and usage of the given user of an accounting programs.

10. The computer-implemented method of claim 1, wherein the obtaining the new translation request initiated by the client further comprises:

identifying, by the one or more processors, the given user as an initiator of the new translation request.

11. A computer program product comprising:

a computer readable storage medium readable by one or more processors and storing instructions for execution by the one or more processors for performing a method comprising;

monitoring, by the one or more processors, multilingual switches performed on a client on behalf of a given user, wherein service requests initiated by the client on behalf of the given user comprise a portion of the multilingual switches, wherein translation requests comprise a portion of the service requests, and wherein the translation requests each comprise a voice input stream and contextual data related to the voice input stream;

based on the monitoring, identifying, by the one or more processors, based on the multilingual switches performed on the client, switch patterns of the given user to generate a service profile for the user, wherein the service profile comprises machine learned multilingual switch patterns for the given user;

determining, by the one or more processors, based on the machine learned multilingual switch patterns for the given user, a priority order for languages comprising the voice input streams, for the given user;

obtaining, by the one or more processors, a new translation request initiated by the client, on behalf of the given user;

applying, by the one or more processors, the priority order to identify one or more languages spoken in a voice input stream comprising the new translation request; and

transmitting, by the one or more processors, indicators of the identified one or more languages to the client, wherein upon receiving the indicators, the client translates the voice input stream from the identified one or more languages to one or more target languages.

12. The computer program product of claim 11, wherein the multilingual switches further comprise activities performed by the given user by utilizing applications executing on the client.

13. The computer program product of claim 12, wherein the activities are selected from the group consisting of:

usage of the given user of a web browser, usage of the given user of a word processing programs, and usage of the given user of an accounting programs.

14. The computer program product of claim 11, wherein the client translates the voice input stream comprising the new request in real-time.

15. The computer program product of claim 11, wherein the given user is selected from the group consisting of:

an individual user and a group of users.

16. The computer program product of claim 11, wherein identifying switch patterns of the given user to generate the service profile for the user further comprises:

identifying, by the one or more processors, for the monitored translation requests, dependencies between factors in the contexts of the translation requests and input languages of the voice input streams of the translation requests; and

generating, by the one or more processors, a set of rules representing the identified dependencies, wherein the machine learned multilingual switch patterns comprise the set of rules.

17. The computer program product of claim 11, wherein the contexts of each request of the requests are selected from the group consisting of:

an entry application for the request on the client, a location of a client upon entry of the request, and an element of a digital wardrobe of the client.

18. The computer program product of claim 11, wherein the contexts of each request of the requests are selected from the group consisting of:

the given user changing input methods from one language to another, the given user browsing web-enabled interfaces in different languages, and the given user using one or more applications in a given geographic location.

19. The computer program product of claim 11, wherein the monitoring further comprises:

generating, by the one or more processors, based on an initial translation request from the client of the translation requests from the client, a data structure; and

applying, by the one or more processors, the data structure to subsequent translation requests from the client of the translation requests from the client, wherein applying the data structure tracks multilingual switching operations for the translation requests from the client.

20. A system comprising:

a memory;

one or more processors in communication with the memory;

program instructions executable by the one or more processors via the memory to perform a method, the method comprising;

monitoring, by the one or more processors, multilingual switches performed on a client on behalf of a given user, wherein service requests initiated by the client on behalf of the given user comprise a portion of the multilingual switches, wherein translation requests comprise a portion of the service requests, and wherein the translation requests each comprise a voice input stream and contextual data related to the voice input stream;

based on the monitoring, identifying, by the one or more processors, based on the multilingual switches performed on the client, switch patterns of the given user to generate a service profile for the user, wherein the service profile comprises machine learned multilingual switch patterns for the given user;

determining, by the one or more processors, based on the machine learned multilingual switch patterns for the given user, a priority order for languages comprising the voice input streams, for the given user;

obtaining, by the one or more processors, a new translation request initiated by the client, on behalf of the given user;

applying, by the one or more processors, the priority order to identify one or more languages spoken in a voice input stream comprising the new translation request; and

transmitting, by the one or more processors, indicators of the identified one or more languages to the client, wherein upon receiving the indicators, the client translates the voice input stream from the identified one or more languages to one or more target languages.