Control method, controller, and non-transitory recording medium

• US 10,056,081 B2
• Filed: 12/09/2016
• Issued: 08/21/2018
• Est. Priority Date: 12/25/2015
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Kunitake, Yuji, Tanaka, Keiichi, Ohto, Hidetaka
• Primary Examiner(s): ALBERTALLI, BRIAN LOUIS
• Application Number:
  US15/375,075
• Publication Number:
  US 20170186428A1
• Time in Patent Office: 620 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/06 Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

A method of controlling a plurality of equipment pieces by a controller including a microphone, a sensor, and a speaker includes collecting, with the microphone, sound around the controller, sensing, with the sensor, a location of a person with respect to the plurality of equipment pieces, generating sensing data based on the sensing, extracting an utterance for controlling the plurality of equipment pieces from the sound collected by the microphone, in accordance with the utterance, identifying a target equipment piece to be controlled among the plurality of equipment pieces, determining whether to control the target equipment piece based on the sensing data, when the control equipment piece is determined to be controlled, generating an audio response for verifying whether to control the target equipment piece in accordance with the utterance, and causing the speaker to output the audio response.

Claims

1. A method of controlling a plurality of equipment pieces by a controller,

the controller including a microphone, a sensor, and a speaker, the method comprising:

collecting, with the microphone, sound around the controller;

sensing, with the sensor, a location of a person with respect to the plurality of equipment pieces;

generating sensing data based on the sensing;

extracting an utterance for controlling the plurality of equipment pieces from the sound collected by the microphone;

in accordance with the utterance, identifying a target equipment piece to be controlled for opening its door among the plurality of equipment pieces;

in accordance with the utterance, determining whether the person is providing the utterance for opening a door of the target equipment piece;

determining whether the person is located within a predetermined range from the target equipment piece or the person is located outside of the predetermine range based on the sensing data;

generating a command for opening the door of the target equipment piece when the person is determined to provide the utterance for opening the door of the target equipment piece, and the person is determined to be located outside the predetermined range;

outputting the command to the target equipment piece;

generating an audio response for verifying whether to open the door of the target equipment piece when (i) the person is determined to provide the utterance for opening the door of the target equipment piece, (ii) the person is determined to be located within the predetermined range, and (iii) a line of sight of the person, a face of the person, and a trunk of the person are not directed towards the target equipment piece; and

causing the speaker to output the audio response to the person.

2. The method according to claim 1, wherein

the sensor is a camera, and

the sensing data is an image of the predetermined range photographed by the camera.

3. The method according to claim 2, wherein

the controller further includes a memory, and

the method further comprises:

when the image indicates that the person is located within the predetermined range,

identifying the person by comparing a feature of the face or a body part of the person in the image with image information stored in the memory in advance, the image information including images of features of faces or body parts of a plurality of people;

determining whether a voiceprint corresponding to the identified person matches a voiceprint extracted from the utterance by comparing audio information stored in the memory in advance with the voiceprint extracted from the utterance, the audio information indicating relationships between each of the plurality of people and each of corresponding voiceprints; and

when the voiceprint corresponding to the identified person is determined to match the voiceprint extracted from the utterance, determining whether the line of sight of the identified person is directed towards the target equipment piece based on the image.

4. The method according to claim 2, wherein

the controller further includes a memory, and

the method further comprises:

when the image indicates that the person is located within the predetermined range,

identifying the person by comparing a feature of the face or a body part of the person in the image with image information stored in the memory in advance, the image information including images of features of faces or body parts of a plurality of people;

determining whether a voiceprint corresponding to the identified person matches a voiceprint extracted from the utterance by comparing audio information stored in the memory in advance with the voiceprint extracted from the utterance, the audio information indicating relationship between each of the plurality of people and each of corresponding voiceprints; and

when the voiceprint corresponding to the identified person is determined to match the voiceprint extracted from the utterance, determining whether the face of the identified person is directed towards the target equipment piece based on the image.

5. The method according to claim 2, wherein

the controller further includes a memory, and

the method further comprises:

when the image indicates that the person is located within the predetermined range,

identifying the person by comparing a feature of the face or a body part of the person extracted from the image with image information stored in the memory in advance, the image information including images of features of faces or body parts of a plurality of people;

determining whether a voiceprint corresponding to the identified person matches a voiceprint extracted from the utterance by comparing audio information stored in the memory in advance with the voiceprint extracted from the utterance, the audio information indicating relationships between each of the plurality of people and each of corresponding voiceprints; and

when the voiceprint corresponding to the identified person is determined to match the voiceprint extracted from the utterance, determining whether the trunk of the identified person is directed towards the target equipment piece based on the image.

6. The method according to claim 1, wherein

the utterance is a first utterance, and

the method further comprises:

when, after the audio response is output, further collecting sound by the microphone;

extracting a second utterance corresponding to the audio response from the sound collected by the microphone after the audio response is output; and

determining whether the first utterance and the second utterance are from a same person,

when the first utterance and the second utterance are determined to be from the same person, the command is generated.

7. The method according to claim 6, wherein

the determining of whether the first utterance and the second utterance are from the same person includes comparing a first voiceprint extracted from the first utterance with a second voiceprint extracted from the second utterance.

8. The method according to claim 1, wherein

the utterance is a first utterance and the command is a first command, and

the method further comprises:

when a second utterance for controlling the target equipment piece is extracted from sound collected by the microphone within predetermined time after the first command is generated, determining whether the first utterance and the second utterance are from a same person; and

when the first utterance and the second utterance are determined to be from the same person, generating a second command corresponding to the second utterance without generating the audio response for verifying whether to control the target equipment piece in response to the second utterance.

9. The method according to claim 8, wherein

the determining of whether the first utterance and the second utterance are from the same person includes comparing a first voiceprint extracted from the first utterance with a second voiceprint extracted from the second utterance.

10. The method according to claim 1, further comprising:

when the person is determined to be located within the predetermined range, regularly determining whether the person is located within the predetermined range or outside of the predetermined range in accordance with the sensing data; and

when the person is determined to be located outside of the predetermined range within a predetermined time after the utterance for controlling the plurality of equipment pieces is extracted from the sound collected by the microphone, the command is generated.

12. A non-transitory recording medium storing a program for causing a processor to execute the method according to claim 1.

11. A controller comprising:

a microphone that collects sound around the controller;

a sensor that senses a location of a person with respect to a plurality of equipment pieces, and generates sensing data;

a speaker; and

a processor, the processor that

extracts an utterance for controlling the plurality of equipment pieces from the sound collected by the microphone,

in accordance with the utterance, identifies a target equipment piece to be controlled among the plurality of equipment pieces,

in accordance with the utterance, determines whether the person is providing the utterance for opening a door of the target equipment piece,

determines whether the person is located within a predetermined range from the target equipment piece or the person is located outside of the predetermine range based on the sensing data,

generates a command for opening the door of the equipment when the person is determined to request opening the door of the target equipment piece and the person is determined to be located outside the predetermined range,

outputs the command to the target equipment piece,

generates an audio response for verifying whether to open the door of the target equipment piece when (i) the person is determined to request opening the door of the target equipment piece, (ii) the person is determined to be located within the predetermined range, and (iii) a line of sight of the person, a face of the person, and a trunk of the person are not directed towards the target equipment piece; and

causes the speaker to output the audio response to the person.