CONTROL METHOD, CONTROLLER, AND NON-TRANSITORY RECORDING MEDIUM

• US 20170186428A1
• Filed: 12/09/2016
• Published: 06/29/2017
• Est. Priority Date: 12/25/2015
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: KUNITAKE, YUJI, TANAKA, KEIICHI, OHTO, HIDETAKA
• Granted Patent: US 10,056,081 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

A method of controlling a plurality of equipment pieces by a controller including a microphone, a sensor, and a speaker includes collecting, with the microphone, sound around the controller, sensing, with the sensor, a location of a person with respect to the plurality of equipment pieces, generating sensing data based on the sensing, extracting an utterance for controlling the plurality of equipment pieces from the sound collected by the microphone, in accordance with the utterance, identifying a target equipment piece to be controlled among the plurality of equipment pieces, determining whether to control the target equipment piece based on the sensing data, when the control equipment piece is determined to be controlled, generating an audio response for verifying whether to control the target equipment piece in accordance with the utterance, and causing the speaker to output the audio response.

Claims

1. A method of controlling a plurality of equipment pieces by a controller,

the controller including a microphone, a sensor, and a speaker, the method comprising:

collecting, with the microphone, sound around the controller;

sensing, with the sensor, a location of a person with respect to the plurality of equipment pieces;

generating sensing data based on the sensing;

extracting an utterance for controlling the plurality of equipment pieces from the sound collected by the microphone;

in accordance with the utterance, identifying a target equipment piece to be controlled among the plurality of equipment pieces;

determining whether to control the target equipment piece based on the sensing data; and

when the target equipment piece is determined to be controlled,

generating an audio response for verifying whether to control the target equipment piece in accordance with the utterance, and

causing the speaker to output the audio response.

2. The method according to claim 1, further comprising:

when the target equipment piece is determined to be controlled,

identifying a control item of the target equipment piece based on the utterance; and

generating a command corresponding to the control item.

3. The method according to claim 1, wherein

the sensing data indicates whether or not a person is located within a predetermined range from the target equipment piece, and

when the sensing data indicates that the person is located outside of the first range from the target equipment piece, determining to control the target equipment piece.

4. The method according to claim 3, wherein

the sensor is a camera, and

the sensing data is an image of the predetermined range photographed by the camera.

5. The method according to claim 4, wherein

the controller further includes a memory, and

the method further comprises:

when the image indicates that the person is located within the predetermined range,

identifying the person by comparing a feature of a face or a body part of the person in the image with image information stored in the memory in advance, the image information including images of features of faces or body parts of a plurality of people;

determining whether a voiceprint corresponding to the identified person matches a voiceprint extracted from the utterance by comparing audio information stored in the memory in advance with the voiceprint extracted from the utterance, the audio information indicating relationships between each of the plurality of people and each of corresponding voiceprints;

when the voiceprint corresponding to the identified person is determined to match the voiceprint extracted from the utterance, determining whether a line of sight of the identified person is directed towards the target equipment piece based on the image; and

when the line of the sight of the identified person is determined to be directed towards the target equipment piece, determining to control the target equipment piece.

6. The method according to claim 4, wherein

the controller further includes a memory, and

the method further comprises:

when the image indicates that the person is located within the predetermined range,

identifying the person by comparing a feature of a face or a body part of the person in the image with image information stored in the memory in advance, the image information including images of features of faces or body parts of a plurality of people;

determining whether a voiceprint corresponding to the identified person matches a voiceprint extracted from the utterance by comparing audio information stored in the memory in advance with the voiceprint extracted from the utterance, the audio information indicating relationship between each of the plurality of people and each of corresponding voiceprints;

when the voiceprint corresponding to the identified person is determined to match the voiceprint extracted from the utterance, determining whether a face of the identified person is directed towards the target equipment piece based on the image; and

when the face of the identified person is determined to be directed towards the target equipment piece, determining to control the target equipment piece.

7. The method according to claim 4, wherein

the controller further includes a memory, and

the method further comprises:

when the image indicates that the person is located within the predetermined range,

identifying the person by comparing a feature of a face or a body part of the person extracted from the image with image information stored in the memory in advance, the image information including images of features of faces or body parts of a plurality of people;

determining whether a voiceprint corresponding to the identified person matches a voiceprint extracted from the utterance by comparing audio information stored in the memory in advance with the voiceprint extracted from the utterance, the audio information indicating relationships between each of the plurality of people and each of corresponding voiceprints;

when the voiceprint corresponding to the identified person is determined to match the voiceprint extracted from the utterance, determining whether a trunk of the identified person is directed towards the target equipment piece based on the image; and

when the body part of the identified person is determined to be directed towards target equipment piece, determining to control the target equipment piece.

8. The method according to claim 1, wherein

the utterance is a first utterance, and

the method further comprises:

when, after the audio response is output, further collecting sound by the microphone;

extracting a second utterance corresponding to the audio response from the sound collected by the microphone after the audio response is output;

determining whether the first utterance and the second utterance are from a same person; and

when the first utterance and the second utterance are determined to be from the same person, generating a command for controlling the target equipment piece.

9. The method according to claim 8, wherein

the determining of whether the first utterance and the second utterance are from the same person includes comparing a first voiceprint extracted from the first utterance with a second voiceprint extracted from the second utterance.

10. The method according to claim 2, wherein

the utterance is a first utterance and the command is a first command, and the method further comprises:

when a second utterance for controlling the target equipment piece is extracted from sound collected by the microphone within predetermined time after the first command is generated, determining whether the first utterance and the second utterance are from a same person; and

when the first utterance and the second utterance are determined to be from the same person, generating a second command corresponding to the second utterance without generating the audio response for verifying whether to control the target equipment piece in response to the second utterance.

11. The method according to claim 10, wherein

the determining of whether the first utterance and the second utterance are from the same person includes comparing a first voiceprint extracted from the first utterance with a second voiceprint extracted from the second utterance.

12. The method according to claim 1, further comprising:

regularly determining whether the utterance extracted from the sound collected by the microphone includes a predetermined trigger word for controlling the target equipment piece;

when the utterance extracted from the sound collected by the microphone is determined to include the predetermined trigger word, identifying a control item of the target equipment piece regardless of the determination of whether to control the target equipment piece; and

generating a command corresponding to the control item.

13. The method according to claim 1, further comprising:

when the target equipment piece is determined not to be controlled, regularly determining whether to control the target equipment piece in accordance with the sensing data; and

when the target equipment piece is determined to be controlled within a predetermined time after the utterance for controlling the plurality of equipment pieces is extracted from the sound collected by the microphone, generating a command that corresponds to the utterance and controls the target equipment piece.

15. A non-transitory recording medium storing a program for causing a processor to execute the method according to claim 1.

14. A controller comprising:

a microphone that collects sound around the controller;

a sensor that senses a location of a person with respect to a plurality of equipment pieces, and generates sensing data;

a speaker; and

a processor, the processor that

extracts an utterance for controlling the plurality of equipment pieces from the sound collected by the microphone,

in accordance with the utterance, identifying a target equipment piece to be controlled among the plurality of equipment pieces,

determining whether to control the target equipment piece based on the sensing data, and

when the target equipment piece is determined to be controlled,

generating an audio response for verifying whether to control the target equipment piece in accordance with the utterance, and

cause the speaker to output the audio response.