System and method for voice command context

• US 10,366,691 B2
• Filed: 01/19/2018
• Issued: 07/30/2019
• Est. Priority Date: 07/11/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Rochford, Ciaran, Harscoet, Philippe, Li, Xiaoguang, Bhavsar, Kinjal
• Primary Examiner(s): Islam, Mohammad K
• Application Number:
  US15/876,073
• Publication Number:
  US 20190019508A1
• Time in Patent Office: 557 Days
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A method, electronic device, and non-transitory computer readable medium for a mobile wallet remittance are provided. The method includes displaying, on a display, content including an object. The method also includes determining an area of an eye focus on the display, and associating the area of the eye focus with the object. The method additionally includes receiving a verbal command and deriving a command based on a detected set of lip movements. The method also includes extracting contextual information from at least one of the object associated with the eye focus, the received verbal command, or the derived command. The method also includes determining an intended command, based on the extracted contextual information and a reconciliation of the verbal command with the derived command. The method then performs the determined intended command

Claims

1. A multi-input method for controlling a head mounted display, the method comprising:

displaying, on a display of the head mounted display, content including an object;

tracking a position of an eye focus in relation to a position on the display;

maintaining a log of the position of the eye focus in relation to the display, and the content in proximity to the position of the eye focus on the display;

determining an area of the eye focus on the display, and associating the area of the eye focus with the object;

receiving a verbal command;

deriving a command based on a detected set of lip movements;

extracting contextual information from at least one of the object associated with the eye focus, the received verbal command, or the derived command;

determining an intended command, based on the extracted contextual information and at least one of the verbal command or the derived command; and

performing the determined intended command.

2. The method of claim 1, further comprising:

in response to receiving a command determining the content displayed on the display associated with the eye focus, has changed, wherein the command is at least one of the verbal command or the derived command; and

searching the log of the tracked position of the eye focus, to discover the object that is associated with the received command, to extract the contextual information associated with the object that is associated with the received command.

3. The method of claim 1, further comprising:

determining whether at least one of the verbal command or the derived command includes a generic term;

when at least one of the verbal command or the derived command includes the generic term, identifying at least one displayed object, associated with the eye focus in proximity to a time when at least one of the verbal command is received or the set of lip movements is detected;

determining whether the generic term is related to the identified at least one displayed object;

determining the intended command based on the extracted contextual information of the eye focus and at least one of the verbal command or the derived command, when the verbal command and the derived command are above a similarity threshold and the generic term is determined to be related to the identified at least one displayed object;

determining the intended command based on the extracted contextual information of the eye focus and the verbal command, when the generic term included in the verbal command is determined to be related to the identified at least one displayed object; and

determining the intended command based on the extracted contextual information of the eye focus and the derived command, when the generic term included in the derived command is determined to be related to the identified at least one displayed object.

4. The method of claim 1, wherein determining a requested operation comprises:

identifying an ambiguous term of either the verbal command or the derived command; and

replacing the identified ambiguous term with the extracted contextual information from the object associated with the eye focus.

5. The method of claim 1, further comprising:

when the verbal command and the derived command are below a similarity threshold, identifying the intended command based on an executable command included in either the verbal command or the derived command;

when either the verbal command or the derived command do not include the executable command, identifying a portion of at least one of the verbal command or the derived command that is ambiguous;

determining one or more replacement commands to clarify the portion of the verbal command or the derived command that is identified as ambiguous;

displaying the one or more replacement commands on the display;

identifying an additional eye focus on a replacement command of the one or more replacement commands, the additional eye focus on the replacement command indicating the replacement command is to replace the portion of the verbal command or the derived command that is ambiguous; and

replacing the portion of the verbal command or the derived command that is ambiguous with the replacement command based on the additional eye focus, to derive the intended command.

6. The method of claim 1, wherein:

the received verbal command is inaudible, wherein

determining the intended command is based on the extracted contextual information from the object associated with the eye focus and the derived command.

18. The method of claim 1 further comprising:

determining the intended command, when the verbal command and the derived command are above a similarity threshold indicating the verbal command and the derived command represent similar commands.

7. A electronic device comprising:

a display;

a communication interface;

a memory; and

at least one processor coupled to the display, communication interface, and the memory, the at least one processor is configured to;

display on the display, content including an object;

track a position of an eye focus in relation to a position on the display;

maintain a log of the position of the eye focus in relation to the display, and the content in proximity to the position of the eye focus on the display;

determine an area of the eye focus on the display, and associating the area of the eye focus with the object;

receive a verbal command;

derive a command based on a detected set of lip movements;

extract contextual information from at least one of the object associated with the eye focus, the received verbal command, or the derived command;

determine an intended command, based on the extracted contextual information and at least one of the verbal command or the derived command; and

perform the determined intended command.

8. The electronic device of claim 7, wherein the at least one processor is further configured to:

in response to receiving a command, determine the content displayed on the display associated with the eye focus, has changed, wherein the command is at least one of the verbal command or the derived command; and

search the log of the tracked position of the eye focus, to discover the object that is associated with the received command, to extract the contextual information associated with the object that is associated with the received command.

9. The electronic device of claim 7, wherein the at least one processor is further configured to:

determine whether at least one of the verbal command or the derived command includes a generic term;

when at least one of the verbal command or the derived command includes the generic term, identify at least one displayed object, associated with the eye focus in proximity to a time when at least one of the verbal command is received or the set of lip movements is detected;

determine whether the generic term is related to the identified at least one displayed object;

determine the intended command based on the extracted contextual information of the eye focus and at least one of the verbal command or the derived command, when the verbal command and the derived command are above a similarity threshold and the generic term is determined to be related to the identified at least one displayed object;

determine the intended command based on the extracted contextual information of the eye focus and the verbal command, when the generic term included in the verbal command is determined to be related to the identified at least one displayed object; and

determine the intended command based on the extracted contextual information of the eye focus and the derived command, when the generic term included in the derived command is determined to be related to the identified at least one displayed object.

10. The electronic device of claim 7, wherein the at least one processor is further configured to:

identify an ambiguous term of either the verbal command or the derived command; and

replace the identified ambiguous term with the extracted contextual information from the object associated with the eye focus.

11. The electronic device of claim 7, wherein the at least one processor is further configured to:

when the verbal command and the derived command are below a similarity threshold, identify the intended command based on an executable command included in either the verbal command or the derived command;

when either the verbal command or the derived command do not include the executable command, identify a portion of at least one of the verbal command or the derived command that is ambiguous;

determine one or more replacement commands to clarify the portion of the verbal command or the derived command that is identified as ambiguous;

display the one or more replacement commands on the display;

identify an additional eye focus on a replacement command of the one or more replacement commands, the additional eye focus on the replacement command indicating the replacement command is to replace the portion of the verbal command or the derived command that is ambiguous; and

replace the portion of the verbal command or the derived command that is ambiguous with the replacement command based on the additional eye focus, to derive the intended command.

12. The electronic device of claim 7, wherein:

the received verbal command is inaudible, and

the at least one processor is further configured to determine the intended command based on the extracted contextual information from the object associated with the eye focus and the derived command.

19. The electronic device of claim 7, wherein the at least one processor is further configured to:

determine the intended command, when the verbal command and the derived command are above a similarity threshold indicating the verbal command and the derived command represent similar commands.

13. A non-transitory computer readable medium embodying a computer program, the computer program comprising program code that when executed by at least one processor of an electronic device causes at least one processing device to:

display on a display, content including an object;

track a position of an eye focus in relation to a position on the display;

maintain a log of the position of the eye focus in relation to the display, and the content in proximity to the position of the eye focus on the display;

determine an area of the eye focus on the display, and associating the area of the eye focus with the object;

receive a verbal command;

derive a command based on a detected set of lip movements;

extract contextual information from at least one of the object associated with the eye focus, the received verbal command, or the derived command;

determine an intended command, based on the extracted contextual information and at least one of the verbal command or the derived command; and

perform the determined intended command.

14. The non-transitory computer readable medium of claim 13, where the program code that when executed further causes the at least one processor to:

in response to receiving a command, determine the content displayed on the display associated with the eye focus, has changed, wherein the command is at least one of the verbal command or the derived command; and

search the log of the tracked position of the eye focus, to discover the object that is associated with the received command, to extract the contextual information associated with the object that is associated with the received command.

15. The non-transitory computer readable medium of claim 13, wherein the program code that when executed further causes the at least one processor to:

determine whether at least one of the verbal command or the derived command includes a generic term;

when at least one of the verbal command or the derived command includes the generic term, identify at least one displayed object, associated with the eye focus in proximity to a time when at least one of the verbal command is received or the set of lip movements is detected;

determine whether the generic term is related to the identified at least one displayed object;

determine the intended command based on the extracted contextual information of the eye focus and at least one of the verbal command or the derived command, when the verbal command and the derived command are above a similarity threshold and the generic term is determined to be related to the identified at least one displayed object;

determine the intended command based on the extracted contextual information of the eye focus and the verbal command, when the generic term included in the verbal command is determined to be related to the identified at least one displayed object; and

determine the intended command based on the extracted contextual information of the eye focus and the derived command, when the generic term included in the derived command is determined to be related to the identified at least one displayed object.

16. The non-transitory computer readable medium of claim 13, wherein the program code that when executed further causes the at least one processor to:

identify an ambiguous term of either the verbal command or the derived command; and

replace the identified ambiguous term with the extracted contextual information from the object associated with the eye focus.

17. The non-transitory computer readable medium of claim 13, wherein the program code that when executed further causes the at least one processor to:

when the verbal command and the derived command are below a similarity threshold, identify the intended command based on an executable command included in either the verbal command or the derived command;

when either the verbal command or the derived command do not include the executable command, identify a portion of at least one of the verbal command or the derived command that is ambiguous;

determine one or more replacement commands to clarify the portion of the verbal command or the derived command that is identified as ambiguous;

display the one or more replacement commands on the display;

identify an additional eye focus on a replacement command of the one or more replacement commands, the additional eye focus on the replacement command indicating the replacement command is to replace the portion of the verbal command or the derived command that is ambiguous; and

replace the portion of the verbal command or the derived command that is ambiguous with the replacement command based on the additional eye focus, to derive the intended command.

20. The non-transitory computer readable medium of claim 13, where the program code that when executed further causes the at least one processor to:

determine the intended command, when the verbal command and the derived command are above a similarity threshold indicating the verbal command and the derived command represent similar commands.