Server for processing speech, control method thereof, image processing apparatus, and control method thereof

• US 10,140,985 B2
• Filed: 05/07/2014
• Issued: 11/27/2018
• Est. Priority Date: 07/02/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Shin, Yong-wook, Chung, Ji-hye, Shin, Seung-min, Lee, Hee-ran, Choi, Sung-wook
• Primary Examiner(s): Mishra, Richa
• Application Number:
  US14/271,577
• Publication Number:
  US 20150012280A1
• Time in Patent Office: 1,665 Days
• Field of Search: None
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

A system includes a server and an image processing apparatus, and the server is provided that includes a communication interface, a storage, and a processor. The communication interface is configured to communicate with the image processing apparatus. The storage is configured to store data. The processor may provide a result of processing a first event that includes a speech of a user to the image processing apparatus in response to the first event being received from the image processing apparatus, store a record of the first event in the storage according to processing of the first event, determine a relation between the first and second events that includes a user input by a non-speech method in response to the second event being received from the image processing apparatus, and process the second event based on the record of the first event stored in the storage in response to the relation.

Claims

1. A system comprising:

a server; and

at least one image processing apparatus,

wherein the server comprises;

a communicator configured to communicate with the at least one image processing apparatus;

a processor configured to;

based on an input of a speech of a user being received from the at least one image processing apparatus, control the communicator to;

transmit, to the at least one image processing apparatus, a plurality of retrieval results corresponding to the input of the speech of the user,

wherein the at least one image processing apparatus comprises;

a display;

a user interface comprising;

a speech input interface configured to receive the input of a speech of a user; and

a non-speech input interface configured to receive a non-speech user input;

a communicator configured to communicate with the server; and

a processor configured to;

based on the input of the speech being received through the speech input interface,

control the communicator of the at least one image processing apparatus to transmit the input of the speech to the server;

control the display to display the plurality of retrieval results to be selected, based on the plurality of retrieval results corresponding to the input of the speech being received from the server;

based on the non-speech user input being received through the non-speech input interface,

identify whether the non-speech user input is related to the input of the speech of the user;

based on the non-speech user input not being related to the input of the speech of the user, perform an operation corresponding to the non-speech user input, independently of the input of the speech; and

based on the non-speech user input being related to the input of the speech of the user, perform an operation on one of the plurality of retrieval results selected by the non-speech user input, based on a result of processing the input of the speech.

2. The system of claim 1, wherein the processor of the server is further configured to:

perform retrieval from a database based on the input of the speech when the input of the speech is received, and

control the communicator of the server to transmit, to the at least one image processing apparatus, one final result selected from among a plurality of retrieval results with respect to the input of the speech, based on the non-speech user input being received when the retrieval results are being displayed.

3. The system of claim 1, wherein the processor of the server is further configured to:

control the communicator of the server to transmit a request command to select one of the retrieval results when there is the plurality of retrieval results with respect to the input of the speech.

4. The system of claim 1, wherein the processor of the at least one image processing apparatus is further configured to convert the non-speech user input into a virtual voice command such that the server recognizes the non-speech user input, and control the communicator to transmit the virtual voice command to the server.

5. A control method of a server, the control method comprising:

receiving, from an image processing apparatus, an input of a speech of a user;

transmitting, to the image processing apparatus, a plurality of retrieval results corresponding to the input of the speech to be displayed on the image processing apparatus;

receiving, by the image processing apparatus, at least one image from the server the plurality of retrieval results corresponding to the input of the speech;

displaying, on the image processing apparatus, the plurality of retrieval results to be selected on a display of the image processing apparatus;

receiving a non-speech user input through a non-speech input interface of the image processing apparatus;

based on the non-speech user input being received through the non-speech input interface, identifying, by the image processing apparatus, whether the non-speech user input is related to the input of the speech of the user;

based on the non-speech user input not being related to the input of the speech of the user, performing, by the image processing apparatus, an operation corresponding to the non-speech user input, independently of the input of the speech; and

based on the non-speech user input being related to the input of the speech of the user, performing, by the image processing apparatus, an operation on one of the plurality of retrieval results selected by the non-speech user input, based on a result of processing the input of the speech.

6. The control method of claim 5, further comprising:

performing retrieval from a database based on the input of the speech, and

transmitting, to the image processing apparatus, one final result selected among a plurality of retrieval results with respect to the input of the speech, based on the non-speech user input being received when the retrieval results are being displayed.

7. The control method of claim 5, wherein the receiving of the non-speech user input comprises:

transmitting, to the image processing apparatus, a request command configured to select one of the retrieval results when there is the plurality of retrieval results with respect to the input of the speech; and

receiving, from the image processing apparatus, the non-speech user input corresponding to the request command.

8. The control method of claim 5, further comprising:

converting, by the image processing apparatus, the non-speech user input into a virtual voice command such that the server recognizes the non-speech user input, and

transmitting, by the image processing apparatus, the virtual voice command to the server.

9. An image processing apparatus comprising:

a display;

a user interface comprising;

a speech input interface configured to receive an input of a speech of a user; and

a non-speech input interface configured to receive a non-speech user input;

a communicator configured to communicate with a server that performs retrieval processing corresponding to the input of the speech; and

a processor configured to;

based on the input of the speech being received through the speech input interface,

control the communicator to transmit the input of the speech to the server;

control the display to display a received plurality of retrieval results to be selected, based on the plurality of retrieval results corresponding to the input of the speech being received from the server;

based on the non-speech user input being received through the non-speech input interface,

identify whether the non-speech user input is related to the input of the speech of the user;

based on the non-speech user input not being related to the input of the speech of the user, perform an operation corresponding to the non-speech user input, independently of the input of the speech; and

based on the non-speech user input being related to the input of the speech of the user, perform an operation on one of the plurality of retrieval results is selected by the non-speech user input, based on a result of processing the input of the speech.

10. The image processing apparatus of claim 9, wherein the processor is further configured to identify whether the non-speech user input gives an instruction to select one of the retrieval results based on the non-speech user input being received when the retrieval results are being displayed.

11. The image processing apparatus of claim 9, wherein the processor is further configured to convert the non-speech user input into a virtual voice command such that the server recognizes the non-speech user input, and control the communicator to transmit the virtual voice command to the server.

12. A control method of an image processing apparatus, the control method comprising:

receiving an input of a speech of a user;

communicating with a server that performs retrieval processing on the input of the speech;

transmitting the input of the speech to the server based on the input of the speech being received;

receiving from the server a plurality of retrieval results on the input of the speech;

displaying the plurality of retrieval results for selection on a display of the image processing apparatus;

receiving a non-speech user input through a non-speech input interface of the image processing apparatus;

based on the non-speech user input being received through the non-speech input interface, identifying whether the non-speech user input is related to the input of the speech of the user;

based on the non-speech user input not being related to the input of the speech of the user, performing an operation corresponding to the non-speech user input, independently of the input of the speech; and

based on the non-speech user input being related to the input of the speech of the user, performing an operation on one of the plurality of retrieval results selected by the non-speech user input, based on a result of processing the input of the speech.

13. The control method of claim 12, further comprising:

identifying whether the non-speech user input gives an instruction to select one of the retrieval results based on the non-speech user input being received when the retrieval results are being displayed.

14. The control method of claim 13, further comprising:

converting the non-speech user input into a virtual voice command that the server recognizes, and

transmitting the virtual voice command to the server.