Control System and Control Method

• US 20200134297A1
• Filed: 12/30/2019
• Published: 04/30/2020
• Est. Priority Date: 07/22/2016
• Status: Active Grant

• Current Assignee: Yamaha Corporation
• Original Assignee: Yamaha Corporation
• Inventors: MAEZAWA, Akira
• Granted Patent: US 10,846,519 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

A control system includes: at least one processor; and a memory that is operatively coupled to the at least one processor and that is configured to store instructions executable by the processor, where upon execution of the instructions the processor is caused to: acquire image information including images of a user captured over time; determine whether a preliminary gesture relating to a cue gesture is performed based on a motion of a face of the user and a gaze direction of the user, where the cue gesture is indicative of a timing for generating an event and the motion of the face and the gaze direction are detected from the image information; estimate a timing for generating an event in a case where it is determined that the preliminary gesture is performed; and output a result of the estimation.

Claims

1. A control system comprising:

at least one processor; and

a memory that is operatively coupled to the at least one processor and that is configured to store instructions executable by the processor, wherein upon execution of the instructions the processor is caused to;

acquire image information including images of a user captured over time;

determine whether a preliminary gesture relating to a cue gesture is performed based on a motion of a face of the user and a gaze direction of the user, where the cue gesture is indicative of a timing for generating an event and the motion of the face and the gaze direction are detected from the image information;

estimate a timing for generating an event in a case where it is determined that the preliminary gesture is performed; and

output a result of the estimation.

2. The control system according to claim 1,

wherein the processor is configured to estimate the timing for generating the event using a result of an output of a cue gesture estimation model that outputs whether a cue gesture is performed in an input image, where the cue gesture estimation model is trained using as training data a data set in which an image for learning obtained by capturing a face including human eyes is associated with a result of determination for a cue gesture indicative of a timing for generating the event in the image for learning.

3. The control system according to claim 1, wherein:

an event indicated by a cue gesture indicative of a timing for generating the event is a start of sound production, and

the processor is configured to, in estimating the timing for generating the event, estimate a timing indicative of a start of sound production using a cue gesture estimation model, wherein the cue gesture estimation model is indicative of a result of learning relations among images and cue gestures, where each of the cue gestures is a motion of a face including human eyes, the motion indicating a start of sound production.

4. The control system according to claim 1, wherein:

an event indicated by a cue gesture indicative of a timing for generating the event is a tempo cycle in playing music, and

the processor is configured to, in estimating the timing for generating the event, estimate a timing indicative of a tempo cycle in playing music using a cue gesture estimation model, wherein the cue gesture estimation model is indicative of a result of learning relations between images and cue gestures, where each of the cue gestures is a motion of a face including human eyes, the motion indicating a tempo cycle in playing music.

5. The control system according to claim 1, wherein

the processor is configured to, in determining whether the preliminary gesture is performed, determine, based on the image information, that the preliminary gesture is performed in a case where it is determined that the motion of the face including human eyes is in a specific first direction and also that the gaze direction is a specific second direction.

6. The control system according to claim 1,

wherein in determining whether the preliminary gesture is performed, the processor is configured to:

extract the face of the user from each of the captured images indicated by the image information by using a result of an output of a face extract model that outputs a face of a person in an input image, where the face extract model is trained by learning as training data a data set in which an image for learning obtained by capturing a face including human eyes and a result of determination for the face in the image for learning; and

detect the motion of the face based on images of the extracted face.

7. The control system according to claim 1, wherein:

the image information includes depth information indicative of a distance to an object for each pixel in each of the captured images, and

in determining whether the preliminary gesture is performed, the processor is configured to;

for each captured image,

separate and remove based on the depth information a background from the captured image indicated by the image information, and

extract a face including human eyes from the captured image, based on an image obtained by separating and removing the background.

8. A control method comprising:

acquiring image information including images of a user captured over time;

determining whether a preliminary gesture relating to a cue gesture is performed based on a motion of a face of the user and a gaze direction of the user, where the cue gesture is indicative of a timing for generating an event and the motion of the face and the gaze direction are detected from the image information;

estimating a timing for generating an event in a case where it is determined that the preliminary gesture is performed; and

outputting a result of the estimation.

9. The control method according to claim 8,

wherein the estimating includes estimating the timing for generating the event using a result of an output of a cue gesture estimation model that outputs whether a cue gesture is performed in an input image, where the cue gesture estimation model is trained using as training data a data set in which an image for learning obtained by capturing a face including human eyes is associated with a result of determination for a cue gesture indicative of a timing for generating the event in the image for learning.

10. The control method according to claim 8, wherein:

an event indicated by a cue gesture indicative of a timing for generating the event is a start of sound production, and

the estimating the timing for generating the event includes estimating a timing indicative of a start of sound production using a cue gesture estimation model, wherein the cue gesture estimation model is indicative of a result of learning relations among images and cue gestures, where each of the cue gestures is a motion of a face including human eyes, the motion indicating a start of sound production.

11. The control method according to claim 8, wherein:

an event indicated by a cue gesture indicative of a timing for generating the event is a tempo cycle in playing music, and

the estimating the timing for generating the event includes estimating a timing indicative of a tempo cycle in playing music using a cue gesture estimation model, wherein the cue gesture estimation model is indicative of a result of learning relations between images and cue gestures, where each of the cue gestures is a motion of a face including human eyes, the motion indicating a tempo cycle in playing music.

12. The control method according to claim 8, wherein the determining whether the preliminary gesture is performed includes determining, based on the image information, that the preliminary gesture is performed in a case where it is determined that the motion of the face including human eyes is in a specific first direction and also that the gaze direction is a specific second direction.

13. The control method according to claim 8,

wherein the determining whether the preliminary gesture is performed includes:

extracting the face from each of the captured images indicated by the image information by using a result of an output of a face extract model that outputs a face of a person in an input image, where the face extract model is trained by learning as training data a data set in which an image for learning obtained by capturing a face including human eyes and a result of determination for the face in the image for learning; and

detecting the motion of the face based on images of the extracted face.

14. The control method according to claim 8, wherein:

the image information includes depth information indicative of a distance to an object for each pixel in each of the captured images, and

the determining whether the preliminary gesture is performed includes;

for each captured image,

separating and removing based on the depth information a background from the captured image indicated by the image information, and

extracting a face including human eyes from the captured image, based on an image obtained by separating and removing the background.