Identification of voice inputs providing credentials

• US 10,446,143 B2
• Filed: 09/16/2016
• Issued: 10/15/2019
• Est. Priority Date: 03/14/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Akbacak, Murat, Hansen, Bryan, Evermann, Gunnar
• Primary Examiner(s): Azad, Abdul K
• Application Number:
  US15/268,360
• Publication Number:
  US 20170263249A1
• Time in Patent Office: 1,124 Days
• Field of Search: 704235, 704251-257
• CPC Class Codes: G06F 21/00 Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/086 Recognition of spelled words
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Systems and processes for identifying of a voice input providing one or more user credentials are provided. In one example process, a voice input can be received. A first character, a phrase identifying a second character, and a word can be identified based on the voice input. In response to the identification, the first character, the second character, and the word can be converted to text. The text can be caused to display, with a display, in a sequence corresponding to an order of the first character, the second character, and the word in the voice input.

Claims

1. An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a voice input including a combination of one or more characters, one or more phrases, and one or more words;

identifying, based on the voice input, a first character, a phrase identifying a second character, and a word,

wherein the phrase identifying the second character comprises two or more words, and

wherein the identifying of the first character, the phrase, or the word includes;

identifying an utterance of the voice input;

determining a type of the utterance; and

obtaining, based on the determined type of the utterance, a representation of the first character, the second character identified by the phrase, or the word corresponding to the utterance; and

in response to the identification, converting the first character, the second character, and the word to text.

2. The electronic device of claim 1, wherein the first character includes at least one of a letter, a numerical digit, a control character, or a space.

3. The electronic device of claim 2, wherein the control character comprises a punctuation mark identified by a punctuation word.

4. The electronic device of claim 1, wherein the phrase comprises a spelling word for assisting the identification of the second character.

5. The electronic device of claim 1, wherein the word comprises a plurality of characters.

6. The electronic device of claim 5, wherein the plurality of characters represent at least one of a domain extension or an acronym.

7. The electronic device of claim 1, wherein identifying, based on the voice input, the first character, the phrase identifying the second character, and the word comprises:

identifying at least one of the first character, the phrase identifying the second character, or the word based on one or more speech recognition models stored in the electronic device.

8. The electronic device of claim 1, wherein identifying, based on the voice input, the first character, the phrase identifying the second character, and the word comprises:

providing data representing the voice input to a server; and

receiving, from the server, data representing at least one of the first character, the phrase identifying the second character, or the word,

wherein identifying, based on the voice input, the first character, the phrase identifying the second character, and the word comprises

identifying at least one of the first character, the phrase identifying the second character, or the word based on one or more language models stored in the server.

9. The electronic device of claim 1, wherein determining the type of the utterance comprises performing at least one of:

determining whether the type of the utterance corresponds to a character;

determining whether the type of the utterance corresponds to a phrase identifying a character; and

determining whether the type of the utterance corresponds to a word.

10. The electronic device of claim 9, wherein determining whether the type of utterance corresponds to a character comprises:

determining, for the utterance, whether the utterance corresponds to a control character;

in accordance with a determination that the utterance corresponds to a control character, determining, for the utterance, whether the utterance corresponds to a punctuation word; and

wherein obtaining, based on the determined type of the utterance, the representation of the character comprises;

in accordance with a determination that the utterance corresponds to a punctuation word, obtaining a representation of a punctuation mark corresponding to the punctuation word,

in accordance with a determination that the utterance does not corresponds to a punctuation word, obtaining a representation of the control character other than a punctuation word.

11. The electronic device of claim 1, wherein identifying, based on the voice input, the first character, the phrase identifying the second character, and the word further comprises:

obtaining a confidence level for the representation;

determining whether the confidence level is greater than or equal to a threshold; and

wherein converting the first character, the second character, and the word to text comprises;

in accordance with a determination that the confidence level is greater than or equal to the threshold, converting the representation of the first character, the second character identified by the phrase, or the word corresponding to the utterance to text;

in accordance with a determination that the confidence level is less than the threshold, providing one or more candidate representations, an error message or a combination thereof.

12. The electronic device of claim 1, wherein the one or more programs further include instructions for:

identifying at least one command based on the voice input, wherein the at least one command enables editing of the displayed text.

13. The electronic device of claim 12, wherein identifying the at least one command comprises:

identifying at least one utterance;

determining whether a type of the utterance corresponds to a command; and

in accordance with a determination that the type of the utterance corresponds to a command, causing to initiate an action corresponding to the command for editing the displayed text.

14. The electronic device of claim 13, wherein the one or more programs further include instructions for:

storing the utterance corresponding to the at least one command and storing information associated with the utterance corresponding to the at least one command.

15. The electronic device of claim 1, wherein identifying, based on the voice input, the first character, the phrase identifying the second character, and the word is based on at least one of contextual data associated with a user or one or more recognized credential models.

16. The electronic device of claim 15, wherein the contextual data associated with the user comprises a user speech profile.

17. The electronic device of claim 15, wherein the one or more recognized credential models represent one or more recognized credential formats.

18. The electronic device of claim 1, wherein identifying, based on the voice input, the first character, the phrase identifying the second character, and the word is based on gaze detection.

19. The electronic device of claim 1, wherein the voice input corresponds to at least one of a first user credential representing a user name or a second user credential representing a password.

20. The electronic device of claim 1, wherein the one or more programs further include instructions for:

storing one or more utterances of the voice input, the one or more utterances corresponding to the first characters, the phrase identifying the second character, and the word recognized based on the identification.

21. The electronic device of claim 1, wherein identifying the utterance of the voice input comprises:

extracting one or more acoustic features from the voice input; and

identifying, based on the one or more acoustic features, a beginning and an end of the utterance of the voice input.

22. The electronic device of claim 21, wherein identifying, based on the one or more acoustic features, the beginning and the end of the utterance of the voice input comprises:

determining that at least one of the one or more acoustic features corresponds to a first period of silence before the utterance of the voice input; and

determining that at least one of the one or more acoustic features corresponds to a second period of silence after the utterance of the voice input.

23. The electronic device of claim 22, wherein the at least one of the one or more acoustic features corresponding to the first period of silence and the second period of silence is a frequency spectrum, and

wherein the frequency spectrum corresponding to the first period of silence and the frequency spectrum corresponding to the second period of silence are different from a frequency spectrum corresponding to the utterance of the voice input.

24. The electronic device of claim 1, wherein determining the type of the utterance comprises:

identifying, based on the utterance, one or more intermediate recognition results; and

determining, based on the one or more intermediate recognition results, the type of the utterance.

25. The electronic device of claim 24, wherein the one or more intermediate recognition results include a first sequence of phonemes, and wherein the one or more programs further include instructions for:

obtaining a second sequence of phonemes based on one or more stored intermediate recognition results corresponding to an utterance of a known type.

26. The electronic device of claim 25, wherein determining, based on the one or more intermediate recognition results, the type of the utterance comprises:

comparing the first sequence of phonemes to the second sequence of phonemes;

determining a confidence level based on a degree by which the first sequence matches the second sequence; and

determining, based on the determined confidence level, the type of the utterance.

27. The electronic device of claim 1, wherein the one or more programs further include instructions for:

causing to display, with a display, the text in a sequence corresponding to an order of the first character, the second character, and the word in the voice input.

28. The electronic device of claim 1, wherein the one or more programs further include instructions for:

after converting the first character, the second character, and the word to text, providing the text as a credential.

29. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive a voice input including a combination of one or more characters, one or more phrases, and one or more words;

identify, based on the voice input, a first character, a phrase identifying a second character, and a word,

wherein the phrase identifying the second character comprises two or more words, and

wherein the identifying of the first character, the phrase, or the word includes;

identifying an utterance of the voice input;

determining a type of the utterance; and

obtaining, based on the determined type of the utterance, a representation of the first character, the second character identified by the phrase, or the word corresponding to the utterance; and

in response to the identification, convert the first character, the second character, and the word to text.

30. The non-transitory computer-readable storage medium of claim 29, wherein determining the type of the utterance comprises performing at least one of:

determining whether the type of the utterance corresponds to a character;

determining whether the type of the utterance corresponds to a phrase identifying a character; and

determining whether the type of the utterance corresponds to a word.

31. The non-transitory computer-readable storage medium of claim 29, wherein the one or more programs further include instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

identify at least one command based on the voice input, wherein the at least one command enables editing of the displayed text.

32. The non-transitory computer-readable storage medium of claim 31, wherein identifying the at least one command comprises:

identifying at least one utterance;

determining whether a type of the utterance corresponds to a command; and

in accordance with a determination that the type of the utterance corresponds to a command, causing to initiate an action corresponding to the command for editing the displayed text.

33. The non-transitory computer-readable storage medium of claim 29, wherein identifying the utterance of the voice input comprises:

extracting one or more acoustic features from the voice input; and

identifying, based on the one or more acoustic features, a beginning and an end of the utterance of the voice input.

34. The non-transitory computer-readable storage medium of claim 33, wherein identifying, based on the one or more acoustic features, the beginning and the end of the utterance of the voice input comprises:

determining that at least one of the one or more acoustic features corresponds to a first period of silence before the utterance of the voice input; and

determining that at least one of the one or more acoustic features corresponds to a second period of silence after the utterance of the voice input.

35. The non-transitory computer-readable storage medium of claim 34, wherein the at least one of the one or more acoustic features corresponding to the first period of silence and the second period of silence is a frequency spectrum, and

wherein the frequency spectrum corresponding to the first period of silence and the frequency spectrum corresponding to the second period of silence are different from a frequency spectrum corresponding to the utterance of the voice input.

36. The non-transitory computer-readable storage medium of claim 29, wherein determining the type of the utterance comprises:

identifying, based on the utterance, one or more intermediate recognition results; and

determining, based on the one or more intermediate recognition results, the type of the utterance.

37. The non-transitory computer-readable storage medium of claim 36, wherein the one or more intermediate recognition results include a first sequence of phonemes, and wherein the one or more programs further include instructions for:

obtaining a second sequence of phonemes based on one or more stored intermediate recognition results corresponding to an utterance of a known type.

38. The non-transitory computer-readable storage medium of claim 37, wherein determining, based on the one or more intermediate recognition results, the type of the utterance comprises:

comparing the first sequence of phonemes to the second sequence of phonemes;

determining a confidence level based on a degree by which the first sequence matches the second sequence; and

determining, based on the determined confidence level, the type of the utterance.

39. The non-transitory computer-readable storage medium of claim 29, wherein the one or more programs further include instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

display, with a display, the text in a sequence corresponding to an order of the first character, the second character, and the word in the voice input.

40. The non-transitory computer-readable storage medium of claim 29, wherein the one or more programs further include instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

after converting the first character, the second character, and the word to text, provide the text as a credential.

41. The non-transitory computer-readable storage medium of claim 29, wherein the first character includes at least one of a letter, a numerical digit, a control character, or a space,

wherein the phrase comprises a spelling word for assisting the identification of the second character, and

wherein the word comprises a plurality of characters.

42. A method for identifying of a voice input providing one or more user credentials, comprising:

at an electronic device having one or more processors and memory;

receiving a voice input including a combination of one or more characters, one or more phrases, and one or more words;

identifying, based on the voice input, a first character, a phrase identifying a second character, and a word,

wherein the phrase identifying the second character comprises two or more words, and

wherein the identifying of the first character, the phrase, or the word includes;

identifying an utterance of the voice input;

determining a type of the utterance; and

obtaining, based on the determined type of the utterance, a representation of the first character, the second character identified by the phrase, or the word corresponding to the utterance; and

in response to the identification, converting the first character, the second character, and the word to text.

43. The method of claim 42, wherein determining the type of the utterance comprises performing at least one of:

determining whether the type of the utterance corresponds to a character;

determining whether the type of the utterance corresponds to a phrase identifying a character; and

determining whether the type of the utterance corresponds to a word.

44. The method of claim 42, further comprising:

identifying at least one command based on the voice input, wherein the at least one command enables editing of the displayed text.

45. The method of claim 44, wherein identifying the at least one command comprises:

identifying at least one utterance;

determining whether a type of the utterance corresponds to a command; and

in accordance with a determination that the type of the utterance corresponds to a command, causing to initiate an action corresponding to the command for editing the displayed text.

46. The method of claim 42, wherein identifying the utterance of the voice input comprises:

extracting one or more acoustic features from the voice input; and

identifying, based on the one or more acoustic features, a beginning and an end of the utterance of the voice input.

47. The method of claim 46, wherein identifying, based on the one or more acoustic features, the beginning and the end of the utterance of the voice input comprises:

determining that at least one of the one or more acoustic features corresponds to a first period of silence before the utterance of the voice input; and

determining that at least one of the one or more acoustic features corresponds to a second period of silence after the utterance of the voice input.

48. The method of claim 47, wherein the at least one of the one or more acoustic features corresponding to the first period of silence and the second period of silence is a frequency spectrum, and

wherein the frequency spectrum corresponding to the first period of silence and the frequency spectrum corresponding to the second period of silence are different from a frequency spectrum corresponding to the utterance of the voice input.

49. The method of claim 42, wherein determining the type of the utterance comprises:

identifying, based on the utterance, one or more intermediate recognition results; and

determining, based on the one or more intermediate recognition results, the type of the utterance.

50. The method of claim 49, wherein the one or more intermediate recognition results include a first sequence of phonemes, and wherein the one or more programs further include instructions for:

obtaining a second sequence of phonemes based on one or more stored intermediate recognition results corresponding to an utterance of a known type.

51. The method of claim 50, wherein determining, based on the one or more intermediate recognition results, the type of the utterance comprises:

comparing the first sequence of phonemes to the second sequence of phonemes;

determining a confidence level based on a degree by which the first sequence matches the second sequence; and

determining, based on the determined confidence level, the type of the utterance.

52. The method of claim 42, further comprising:

displaying, with a display, the text in a sequence corresponding to an order of the first character, the second character, and the word in the voice input.

53. The method of claim 42, further comprising:

after converting the first character, the second character, and the word to text, providing the text as a credential.

54. The method of claim 42, wherein the first character includes at least one of a letter, a numerical digit, a control character, or a space,

wherein the phrase comprises a spelling word for assisting the identification of the second character, and

wherein the word comprises a plurality of characters.