Information processing system, information processing method, communication terminal, information processing apparatus, and control method and control program thereof

• US 9,489,951 B2
• Filed: 05/22/2013
• Issued: 11/08/2016
• Est. Priority Date: 05/30/2012
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Kobayashi, Yoshikazu
• Primary Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US14/403,268
• Publication Number:
  US 20150142437A1
• Time in Patent Office: 1,266 Days
• Field of Search: 704/275
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An apparatus of this invention is directed to an information processing apparatus that determines a search range of one of the pieces of instruction information based on the other of a plurality of different pieces of instruction information, and effectively narrows down manipulation instruction candidates corresponding to a user even if the manipulation instruction candidates are extended. The information processing apparatus includes an instruction information receiver that receives instruction voice information indicating the instruction voice of the user acquired from the voice of the user, and instruction operation information indicating the instruction operation of the user acquired from the operation of the user, a search range determining unit that determines a search range for recognizing the instruction operation information according to the instruction voice information, or determines a search range for recognizing the instruction voice information according to the instruction operation information, and a user instruction recognizer that recognizes an instruction of the user based on a search result obtained by searching for one of the instruction voice information and the instruction operation information within the search range determined by the search range determining unit.

Claims

1. An information processing apparatus comprising:

an instruction information receiver that receives a set of first instruction information acquired from a voice of a user, and second instruction information acquired from a finger gesture or a finger motion of the user;

a search range determining unit that determines, in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

a user instruction recognizer that recognizes an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range; and

an instruction history accumulator that accumulates, in association with each other, the set of the first and second instruction information and an appearance count of recognition failure indicating that said user instruction recognizer has erroneously recognized the instruction of the user,

wherein said search range determination unit adds a recognition result of the recognition failure to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to said instruction history accumulator.

2. The information processing apparatus according to claim 1, wherein said instruction information receiver further receives device information of a device as an instruction target of the user, and

further comprising an operation command transmitter that transmits an operation command of the device according to the instruction of the user recognized by said user instruction recognizer.

3. The information processing apparatus according to claim 1, wherein the second instruction information is acquired based on the gesture or the finger motion of the user from a camera that captures an instruction operation of the user.

4. The information processing apparatus according to claim 3, wherein the second instruction information is acquired, from a touch panel used to input the instruction operation of the user, based on the finger motion of the user on the touch panel.

5. The information processing apparatus according to claim 1, wherein said search range determining unit acquires an attribute of the user based on at least one of the voice of the user and the finger gesture or the finger motion of the user, and determines at least one of the first and second search ranges according to the attribute of the user.

6. The information processing apparatus according to claim 5, wherein the attribute of the user includes one of a nationality of the user, a spoken language of the user, a race of the user, an occupation of the user, an environment of the user, a scene that the user has encountered, a habit of the user, and a dialect of the user.

7. The information processing apparatus according to claim 1, wherein the user is a worker in a store, and said user instruction recognizer recognizes an instruction of the worker.

8. The information processing apparatus according to claim 1, wherein said user instruction recognizer recognizes a manipulation instruction of the user for a device installed in a room.

9. The information processing apparatus according to claim 1, wherein said information processing apparatus manages a conference system to which the user joins, and

said user instruction recognizer recognizes an instruction of the user to the conference system.

10. A communication terminal communicating with the information processing apparatus according to claim 1, comprising:

a voice detector that detects the voice of the user;

an operation detector that detects the finger gesture or the finger motion of the user;

an instruction information transmitter that acquires the first instruction information from the voice of the user via said voice detector, acquires the second instruction information from the finger gesture or the finger motion of the user via said operation detector, and transmits, to the information processing apparatus, a set of the first instruction information and the second instruction information together with device information of a device as an instruction target of the user;

an operation command receiver that receives, from the information processing apparatus, an operation command of the device according to an instruction of the user recognized by the information processing apparatus based on the set of the first instruction information and the second instruction information; and

an operation command output unit that outputs, to the device as the instruction target of the user, the operation command received by said operation command receiver.

11. A control method of a communication terminal communicating with the information processing apparatus according to claim 1, comprising:

acquiring the first instruction information from the voice of the user via a voice detector, acquiring the second instruction information from the finger gesture or the finger motion of the user via an operation detector, and transmitting, to the information processing apparatus, a set of the first instruction information and the second instruction information together with device information of a device as an instruction target of the user;

receiving, from the information processing apparatus, an operation command of the device according to an instruction of the user recognized by the information processing apparatus based on the set of the first instruction information and the second instruction information; and

outputting, to the device as the instruction target of the user, the operation command received in the receiving step.

12. A non-transitory computer-readable storage medium storing a control program of a communication terminal, communicating with the information processing apparatus according to claim 1, for causing a computer to execute a method, comprising:

acquiring the first instruction information from the voice of the user via a voice detector, acquiring the second instruction information from the finger gesture or the finger motion of the user via an operation detector, and transmitting, to the information processing apparatus, a set of the first instruction information and the second instruction information together with device information of a device as an instruction target of the user;

receiving, from the information processing apparatus, an operation command of the device according to an instruction of the user recognized by the information processing apparatus based on the set of the first instruction information and the second instruction information; and

outputting, to the device as the instruction target of the user, the operation command received in the receiving step.

13. A control method of an information processing apparatus, comprising:

receiving a set of first instruction information acquired from a voice of a user, and second instruction information acquired from a finger gesture or a finger motion of the user;

determining, in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

recognizing an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range; and

accumulating, as a recognition history in association with each other, the set of the first and second instruction information and an appearance count of recognition failure indicating that the user instruction has been erroneously recognized,

wherein, in said determining step, a recognition result of the recognition failure is added to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to the accumulated recognition history.

14. A non-transitory computer-readable storage medium storing a control program of an information processing apparatus for causing a computer to execute a method, comprising:

receiving a set of first instruction information acquired from a voice of a user, and second instruction information acquired from a finger gesture or a finger motion of the user;

determining, in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

recognizing an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range; and

accumulating, as a recognition history in association with each other, the set of the first and second instruction information and an appearance count of recognition failure indicating that the user instruction has been erroneously recognized,

wherein, in said determining step, a recognition result of the recognition failure is added to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to the accumulated recognition history.

15. A communication terminal comprising:

a voice detector that detects a voice of a user;

an operation detector that detects a finger gesture or a finger motion of the user;

an instruction information acquirer that acquires a set of first instruction information from the voice of the user, and second instruction information from a finger gesture or a finger motion of the user;

a search range determining unit that determines in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

a user instruction recognizer that recognizes an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range; and

an instruction history accumulator that accumulates, in association with each other, the set of the first and second instruction information and an appearance count of recognition failure indicating that said user instruction recognizer has erroneously recognized the instruction of the user,

wherein said search range determination unit adds a recognition result of the recognition failure to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to said instruction history accumulator.

16. A control method of a communication terminal, comprising:

detecting a voice of a user;

detecting a finger gesture or a finger motion of the user;

acquiring a set of first instruction information from a voice of the user, and second instruction information from the finger gesture or the finger motion of the user;

determining, in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

recognizing an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range determined in the determining step; and

accumulating, as a recognition history in association with each other, the set of the first and second instruction information and an appearance count of recognition failure indicating that the user instruction has been erroneously recognized, and

wherein, in said determining step, a recognition result of the recognition failure is added to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to the accumulated recognition history.

17. A non-transitory computer-readable storage medium storing a control program of a communication terminal for causing a computer to execute a method, comprising:

detecting a voice of a user;

detecting a finger gesture or a finger motion of the user;

acquiring a set of first instruction information from the voice of the user, and second instruction information from the finger gesture or the finger motion of the user;

determining, in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

recognizing an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range determined in the determining step; and

accumulating, as a recognition history in association with each other, the set of the first and second instruction information and an appearance count of recognition failure indicating that said user instruction recognizer has erroneously recognized the instruction of the user,

wherein, in said determining step, a recognition result of the recognition failure is added to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to the accumulated recognition history.

18. An information processing system including an information processing apparatus and a communication terminal which are connected via a network,

said information processing apparatus comprising:

an instruction information receiver that receives, together with device information of a device as an instruction target of a user, a set of first instruction information acquired from a voice of the user, and second instruction information acquired from a finger gesture or a finger motion of the user;

a search range determining unit that determines, in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

a user instruction recognizer that recognizes an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range determined by said search range determining unit;

an instruction history accumulator that accumulates, in association with each other, the set of the first and second instruction information and an appearance count of recognition failure indicating that said user instruction recognizer has erroneously recognized the instruction of the user; and

an operation command transmitter that transmits an operation command of the device according to the instruction of the user recognized by said user instruction recognizer,

wherein said search range determination unit adds a recognition result of the recognition failure to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to said instruction history accumulator; and

said communication terminal comprising;

a voice detector that detects the voice of the user,

an operation detector that detects the finger gesture or the finger motion of the user,

an instruction information transmitter that acquires the first instruction information via said voice detector, acquires the second instruction information via said operation detector, and transmits, to said information processing apparatus, the set of the first instruction information and the second instruction information together with the device information of the device;

an operation command receiver that receives, from said information processing apparatus, the operation command of the device according to the instruction of the user recognized by said information processing apparatus based on the set of the first instruction information and the second instruction information, and

an operation command output unit that outputs, to the device as the instruction target of the user, the operation command received by said operation command receiver.

19. An information processing method of an information processing system including an information processing apparatus and a communication terminal which are connected via a network, comprising:

by the information processing apparatus,

receiving, together with device information of a device as an instruction target of a user, a set of first instruction information acquired from a voice of the user, and second instruction information acquired from a finger gesture or a finger motion of the user;

determining, in accordance with the first instruction information, a second search range for recognizing the second instruction information, or, in accordance with the second instruction information, a first search range for recognizing the first instruction information;

recognizing an instruction of the user based on a first search result obtained by searching for the first instruction information within the first search range or a second search result obtained by searching for the second instruction information within the second search range determined in the determining step;

accumulating, in association with each other, a set of the first and second instruction information and an appearance count of recognition failure indicating that said user instruction recognizer has erroneously recognized the instruction of the user; and

transmitting an operation command of the device according to the instruction of the user recognized in the recognizing step,

wherein, in said determining step, a recognition result of the recognition failure is added to at least one of the first search range and the second search range, when it is determined that the appearance count of the recognition failure exceeds a threshold by referring to the accumulated recognition history, and

by the communication terminal

acquiring the first instruction information via a voice detector that detects the voice of the user, acquiring the second instruction information via an operation detector that detects the operation of the user, and transmitting, to said information processing apparatus, the set of the first instruction information and the second instruction information together with device information of the device;

receiving, from the information processing apparatus, the operation command of the device according to an instruction of the user recognized by the information processing apparatus based on the set of the first instruction information and the second instruction information; and

outputting, to the device as the instruction target of the user, the operation command received in the receiving step.