VOICE RECOGNITION METHOD AND APPARATUS USING VIDEO RECOGNITION

• US 20140028826A1
• Filed: 07/25/2013
• Published: 01/30/2014
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Dongyeol, Suh, Sangbum
• Granted Patent: US 10,049,665 B2
• Time in Patent Office: Days
• US Class Current: 348/77
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal

Abstract

Provided are a method and an apparatus for performing exact start and end recognition of voice based on video recognition. The method includes determining whether a speech starts based on at least one of first video and audio data before conversion into a voice recognition mode, converting into the voice recognition mode and generating second audio data including a voice command, when it is determined that speech starts, and determining whether the speech is terminated based on at least one of second video and audio data after conversion into the voice recognition mode.

Claims

1. A method of recognizing a voice command by an electronic device, the method comprising:

determining whether a speech starts based on at least one of first video data and first audio data before conversion into a voice recognition mode;

converting the electronic device into the voice recognition mode and generating second audio data including a voice command of a user, when it is determined that speech starts; and

determining whether the speech is terminated based on at least one of second video data and second audio data after conversion into the voice recognition mode.

2. The method of claim 1, further comprising displaying a voice command for executing an icon displayed on a screen after determining whether the speech starts, and removing display of the voice command after determining whether the speech is terminated.

3. The method of claim 2, wherein determining whether the speech starts comprises identifying a lip of the user from the first video data and determining whether the speech starts based on motion of the lip.

4. The method of claim 3, further comprising attenuating a threshold for determining the speech start to re-determine whether the speech starts when at least one of result values of analyzing the first video data and the first audio data is less than or equal to the threshold, after determining whether the speech starts.

5. The method of claim 4, wherein determining whether the speech starts comprises determining that the speech does not start when a face of the user is not included in the first video data.

6. The method of claim 5, wherein determining whether the speech is terminated comprises determining that the speech is terminated when a voice of the user is not included in the second audio data for at least a preset time.

7. The method of claim 6, further comprising attenuating a threshold for determining the speech end to re-determine whether the speech is terminated when at least one of result values of analyzing the second video data and the second audio data is less than or equal to the threshold, after determining whether the speech is terminated.

8. The method of claim 4, wherein determining whether the speech is terminated comprises determining that the speech is terminated when a face of the user is not included in the second video data.

9. An electronic device which recognizes a voice command, the electronic device comprising:

an audio processor which collects and records a voice input;

a camera unit which collects and records a video input; and

a controller which controls to generate first video data or first audio data before conversion into a voice recognition mode, determine whether a speech starts based on at least one of the first video data and the first audio data, convert into the voice recognition mode when it is determined that the speech starts, control to generate second audio data or second video data including a voice command of a user, and determine whether the speech is terminated based on at least one of the second video data and the second audio data.

10. The electronic device of claim 9, further comprising a display unit which displays a screen,

wherein the controller controls the display unit to display a voice command for executing an icon displayed on the screen when the speech start, and controls the display unit to remove display of the voice command when the speech is terminated.

11. The electronic device of claim 10, wherein the controller identifies a lip of a user from the first video data, and determines whether the speech starts based on motion of the lip.

12. The electronic device of claim 11, wherein the controller attenuates a threshold for determining the speech start to re-determine whether the speech starts when at least one of result values of analyzing the first video data and the first audio data is less than or equal to the threshold.

13. The electronic device of claim 12, wherein the controller determines that the speech does not start when a face of the user is not included in the first video data.

14. The electronic device of claim 13, wherein the controller determines that the speech is terminated when a voice of the user is not included in the second audio data for at least a preset time.

15. The electronic device of claim 14, wherein the controller attenuates a threshold for determining the speech end to re-determine whether the speech is terminated when at least one of result values of analyzing the second video data and the second audio data is less than or equal to the threshold.

16. The electronic device of claim 15, wherein the controller determines that the speech is terminated when a face of the user is not included in the second video data.

17. A platform for recognizing a voice command, the platform comprising:

a multi-media framework which collects and records voice input or video input to generate first video data or first audio data before conversion into a voice recognition mode, and generates second video data or second audio data after conversion into the voice recognition mode; and

a voice framework which determines whether a speech starts based on at least one of the first video data or the first audio data, and determines whether the speech is terminated based on at least one of the second video data or the second audio data.

18. The platform of claim 17, further comprising a user interface framework which displays a screen,

wherein the user interface framework displays a voice command for executing an icon displayed on the screen when the voice framework determines that the speech starts, and removes display of the voice command when the voice framework determines that the speech is terminated.

19. The platform of claim 18, wherein the voice framework attenuates a threshold for determining the speech start to re-determine whether the speech starts when at least one of result values of analyzing the first video data and the first audio data is less than or equal to the threshold, and attenuates a preset threshold for determining the speech end to re-determine whether the speech is terminated when at least one of result values of analyzing the second video data and the second audio data is less than or equal to the threshold.

20. The platform of claim 19, wherein the voice framework identifies a lip of a user from the first video data to determine whether the speech starts based on motion of the lip, and determines that the speech is terminated when a voice of the user is not included in the second audio data for at least a preset time.

21. The platform of claim 20, wherein the voice framework determines that the speech is not started when a face of the user is not included in the first video data, and determines that the speech is terminated when a face of the user is not included in the second video data.