Gesture detection system, gesture detection apparatus, and mobile communication terminal

• US 9,720,509 B2
• Filed: 10/29/2014
• Issued: 08/01/2017
• Est. Priority Date: 11/05/2013
• Status: Active Grant

• Current Assignee: Moff, Inc.
• Original Assignee: Moff, Inc.
• Inventors: Takahagi, Akinori, Yonesaka, Motohiro
• Primary Examiner(s): Lee, Benjamin C
• Assistant Examiner(s): Liang, Dong Hui
• Application Number:
  US14/526,737
• Publication Number:
  US 20150123897A1
• Time in Patent Office: 1,007 Days
• CPC Class Codes: A63H 30/04 using wireless transmission
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A gesture detection system having a gesture detection apparatus to detect a gesture of a user, and a mobile communication terminal that can communicate with the gesture detection apparatus, includes a storage unit to store first gesture data defining the gesture of the user, and audio or visual data associated with the first gesture data; an obtainment unit to obtain second gesture data representing the gesture of the user; a transmission unit to transmit the second gesture data to the mobile communication terminal; a determination unit to determine whether the gesture defined by the first gesture data is the same as the gesture represented by the second gesture data; a selection unit to select the audio or visual data associated with the first gesture data depending on the determination result; and an output unit to output the audio or visual data.

Claims

1. A gesture detection system including a gesture detection apparatus for detecting a gesture, and a mobile communication terminal for communicating with the gesture detection apparatus,

wherein the gesture detection apparatus includes

a sensor configured to detect acceleration data in an X-axis direction, acceleration data in a Y-axis direction, and acceleration data in a Z-axis direction;

and

a transmitter configured to transmit, to the mobile communication terminal, detection data including the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, and the acceleration data in the Z-axis direction, wherein the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, and the acceleration data in the Z-axis direction are detected by the sensor,

wherein the mobile communication terminal includes

a receiver configured to receive the detection data transmitted by the transmitter of the gesture detection apparatus;

an output unit configured to output audio and/or video based on output data;

a storage unit configured to store threshold data, a flag value, association data, and the output data; and

a processor;

wherein the threshold data includes a threshold value in the X-axis direction, a threshold value in the Y-axis direction, and a threshold value in the Z-axis direction,

wherein the flag value takes one of binary values,

wherein the output data includes a plurality of output data patterns for the audio and/or the video,

wherein the association data includes a table that defines one-to-one correspondence between a plurality of combinations and the plurality of output data patterns, each of the plurality of combinations being a combination of a threshold condition, which corresponds to a particular gesture among a plurality of stored gestures, and the flag value, wherein each threshold condition is determined by relation between an acceleration value in the X-axis direction and the threshold value in the X-axis direction, relation between an acceleration value in the Y-axis direction and the threshold value in the Y-axis direction, and relation between an acceleration value in the Z-axis direction and the threshold value in the Z-axis direction,

wherein the processor detects one threshold condition of the plurality of threshold conditions by comparing a maximum value of the acceleration data in the X-axis direction with the threshold value in the X-axis direction, comparing a maximum value of the acceleration data in the Y-axis direction with the threshold value in the Y-axis direction, and comparing a maximum value of the acceleration data in the Z-axis direction with the threshold value in the Z-axis direction,

wherein the processor retrieves the flag value stored in the storage unit,

wherein the processor selects, from the plurality of output data patterns, one output data pattern corresponding to a combination of the one threshold condition and the flag value,

wherein the processor causes the output unit to output the audio and/or the video based on the selected one output data pattern, and

wherein, after the output unit outputs the audio and/or the video, upon detecting that the one threshold condition is a predetermined threshold condition with a predetermined combination of threshold conditions in the X-axis, Y-axis and Z-axis corresponding to a predetermined gesture, the processor updates the flag value with the other one of the binary values.

2. The gesture detection system according to claim 1, wherein the gesture detection apparatus includes

a housing, and

a belt attached to the housing on two opposite side surfaces of the housing.

3. The gesture detection system according to claim 2, wherein the gesture detection apparatus is attached to a toy operated by a user, or a wrist of the user, by the belt.

4. A gesture detection apparatus attached to a toy operated by a user, or a wrist of the user, the gesture detection apparatus comprising:

a sensor configured to detect acceleration data in an X-axis direction, acceleration data in a Y-axis direction, and acceleration data in a Z-axis direction; and

a transmitter configured to transmit, to a mobile communication terminal, detection data including the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, and the acceleration data in the Z-axis direction, wherein the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, and the acceleration data in the Z-axis direction are detected by the sensor,

wherein the mobile communication terminal includes

a receiver configured to receive the detection data transmitted by the transmitter of the gesture detection apparatus;

an output unit configured to output audio and/or video based on output data;

a storage unit configured to store threshold data, a flag value, association data, and the output data; and

a processor,

wherein the threshold data includes a threshold value in the X-axis direction, a threshold value in the Y-axis direction, and a threshold value in the Z-axis direction,

wherein the flag value takes one of binary values,

wherein the output data includes a plurality of output data patterns for the audio and/or the video,

wherein the association data includes a table that defines one-to-one correspondence between a plurality of combinations and the plurality of output data patterns, each of the plurality of combinations being a combination of a threshold condition, which corresponds to a particular gesture among a plurality of stored gestures, and the flag value, wherein each threshold condition is determined by relation between an acceleration value in the X-axis direction and the threshold value in the X-axis direction, relation between an acceleration value in the Y-axis direction and the threshold value in the Y-axis direction, and relation between an acceleration value in the Z-axis direction and the threshold value in the Z-axis direction,

wherein the processor detects one threshold condition of the plurality of threshold conditions by comparing a maximum value of the acceleration data in the X-axis direction with the threshold value in the X-axis direction, comparing a maximum value of the acceleration data in the Y-axis direction with the threshold value in the Y-axis direction, and comparing a maximum value of the acceleration data in the Z-axis direction with the threshold value in the Z-axis direction,

wherein the processor retrieves the flag value stored in the storage unit,

wherein the processor selects, from the plurality of output data patterns, one output data pattern corresponding to a combination of the one threshold condition and the flag value,

wherein the processor causes the output unit to output the audio and/or the video based on the selected one output data pattern, and

wherein, after the output unit outputs the audio and/or the video, upon detecting that the one threshold condition is a predetermined threshold condition with a predetermined combination of threshold conditions in the X-axis, Y-axis and Z-axis corresponding to a predetermined gesture, the processor updates the flag value with the other one of the binary values.

5. A mobile communication terminal for communicating with a gesture detection apparatus for detecting a gesture, the mobile communication terminal comprising:

a receiver configured to receive detection data transmitted by the gesture detection apparatus, wherein the detection data includes acceleration data in an X-axis direction, acceleration data in a Y-axis direction, and acceleration data in a Z-axis direction, wherein the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, and the acceleration data in the Z-axis direction are detected by the gesture detection apparatus;

an output unit configured to output audio and/or video based on output data;

a storage unit configured to store threshold data, a flag value, association data, and the output data; and

a processor,

wherein the threshold data includes a threshold value in the X-axis direction, a threshold value in the Y-axis direction, and a threshold value in the Z-axis direction,

wherein the flag value takes one of binary values,

wherein the output data includes a plurality of output data patterns for the audio and/or the video,

wherein the association data includes a table that defines one-to-one correspondence between a plurality of combinations and the plurality of output data patterns, each of the plurality of combinations being a combination of a threshold condition, which corresponds to a particular gesture among a plurality of stored gestures, and the flag value, wherein each threshold condition is determined by relation between an acceleration value in the X-axis direction and the threshold value in the X-axis direction, relation between an acceleration value in the Y-axis direction and the threshold value in the Y-axis direction, and relation between an acceleration value in the Z-axis direction and the threshold value in the Z-axis direction,

wherein the processor detects one threshold condition of the plurality of threshold conditions by comparing a maximum value of the acceleration data in the X-axis direction with the threshold value in the X-axis direction, comparing a maximum value of the acceleration data in the Y-axis direction with the threshold value in the Y-axis direction, and comparing a maximum value of the acceleration data in the Z-axis direction with the threshold value in the Z-axis direction,

wherein the processor retrieves the flag value stored in the storage unit,

wherein the processor selects, from the plurality of output data patterns, one output data pattern corresponding to a combination of the one threshold condition and the flag value,

wherein the processor causes the output unit to output the audio and/or the video based on the selected one output data pattern, and

wherein, after the output unit outputs the audio and/or the video, upon detecting that the one threshold condition is a predetermined threshold condition with a predetermined combination of threshold conditions in the X-axis, Y-axis and Z-axis corresponding to a predetermined gesture, the processor updates the flag value with the other one of the binary values.

6. A non-transitory computer-readable recording medium storing a program executed by a computer for communicating with a gesture detection apparatus for detecting a gesture,

wherein the computer comprises:

a receiver configured to receive detection data transmitted by the gesture detection apparatus, wherein the detection data includes acceleration data in an X-axis direction, acceleration data in a Y-axis direction, and acceleration data in a Z-axis direction, wherein the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, and the acceleration data in the Z-axis direction are detected by the gesture detection apparatus;

an output unit configured to output audio and/or video based on output data;

a storage unit configured to store threshold data, a flag value, association data, and the output data; and

a processor,

wherein the threshold data includes a threshold value in the X-axis direction, a threshold value in the Y-axis direction, and a threshold value in the Z-axis direction,

wherein the flag value takes one of binary values,

wherein the output data includes a plurality of output data patterns for the audio and/or the video,

wherein the association data includes a table that defines one-to-one correspondence between a plurality of combinations and the plurality of output data patterns, each of the plurality of combinations being a combination of a threshold condition, which corresponds to a particular gesture among a plurality of stored gestures, and the flag value, wherein each threshold condition is determined by relation between an acceleration value in the X-axis direction and the threshold value in the X-axis direction, relation between an acceleration value in the Y-axis direction and the threshold value in the Y-axis direction, and relation between an acceleration value in the Z-axis direction and the threshold value in the Z-axis direction,

wherein, when the program is executed by the computer, the program causes the processor of the computer to execute a process including;

detecting one threshold condition of the plurality of threshold conditions by comparing a maximum value of the acceleration data in the X-axis direction with the threshold value in the X-axis direction, comparing a maximum value of the acceleration data in the Y-axis direction with the threshold value in the Y-axis direction, and comparing a maximum value of the acceleration data in the Z-axis direction with the threshold value in the Z-axis direction;

retrieving the flag value stored in the storage unit;

selecting, from the plurality of output data patterns, one output data pattern corresponding to a combination of the one threshold condition and the flag value,

causing the output unit to output the audio and/or the video based on the selected one output data pattern; and

after the output unit outputs the audio and/or the video, upon detecting that the one threshold condition is a predetermined threshold condition with a predetermined combination of threshold conditions in the X-axis, Y-axis and Z-axis corresponding to a predetermined gesture, updating the flag value with the other one of the binary values.

7. A gesture detection system including a gesture detection apparatus for detecting a gesture, and a mobile communication terminal for communicating with the gesture detection apparatus,

wherein the gesture detection apparatus includes

a sensor configured to detect acceleration data in an X-axis direction, acceleration data in a Y-axis direction, and acceleration data in a Z-axis direction;

a calculation unit configured to calculate a relative angle of the gesture detection apparatus around the Z-axis, wherein the relative angle is an angle between a direction of the gesture detection apparatus and a calculated reference direction; and

a transmitter configured to transmit, to the mobile communication terminal, posture data including the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, the acceleration data in the Z-axis direction, and the relative angle, wherein the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, and the acceleration data in the Z-axis direction are detected by the sensor, and the relative angle is calculated by the calculation unit,

wherein the mobile communication terminal includes

a receiver configured to receive the posture data transmitted by the transmitter of the gesture detection apparatus;

an output unit configured to output audio and/or video based on output data;

a storage unit configured to store threshold data, association data, and the output data; and

a processor,

wherein the threshold data includes a threshold value in the X-axis direction, a threshold value in the Y-axis direction, a threshold value in the Z-axis direction, and a plurality of threshold values with respect to an angle around the Z-axis,

wherein the output data includes a plurality of output data patterns for the audio and/or the video,

wherein the association data includes a table that defines one-to-one correspondence between a plurality of threshold conditions and the plurality of output data patterns, wherein each of the plurality of threshold conditions, which corresponds to a particular gesture among a plurality of stored gestures, is determined by relation between an acceleration value in the X-axis direction and the threshold value in the X-axis direction, relation between an acceleration value in the Y-axis direction and the threshold value in the Y-axis direction, relation between an acceleration value in the Z-axis direction and the threshold value in the Z-axis direction, and relation between the angle around the Z-axis and the plurality of threshold values with respect to the angle around the Z-axis,

wherein the storage unit further stores a plurality of operational directions in which a plurality of percussion instruments in a set are respectively placed with respect to the calculated reference direction,

one of the plurality of threshold conditions corresponds to a gesture of beating a percussion instrument in the set,

wherein the reference direction is calculated based on the posture data detected by the gesture detection apparatus corresponding to a gesture of beating a predetermined one of the percussion instruments in the set,

wherein the processor detects said one of the plurality threshold conditions by comparing a maximum value of the acceleration data in the X-axis direction with the threshold value in the X-axis direction, comparing a maximum value of the acceleration data in the Y-axis direction with the threshold value in the Y-axis direction, comparing a maximum value of the acceleration data in the Z-axis direction with the threshold value in the Z-axis direction, and comparing the relative angle calculated by the calculation unit and the plurality of threshold values with respect to the angle around the Z-axis,

wherein the processor selects, from the plurality of output data patterns, one output data pattern corresponding to said one of the threshold conditions, and

wherein the processor causes the output unit to output the audio and/or the video based on the selected one output data pattern.

8. The gesture detection system according to claim 7, wherein the storage unit stores

audio data to be generated when the gesture of beating one of the percussion instruments is performed, the audio data being associated with the gesture and the plurality of operational directions.

9. A non-transitory computer-readable recording medium storing a program executed by a computer for communicating with a gesture detection apparatus,

wherein the computer includes

a receiver configured to receive posture data transmitted by the gesture detection apparatus, the posture data including acceleration data in an X-axis direction, acceleration data in a Y-axis direction, acceleration data in a Z-axis direction, and a relative angle of the gesture detection apparatus around the Z-axis, wherein the acceleration data in the X-axis direction, the acceleration data in the Y-axis direction, the acceleration data in the Z-axis direction are detected by the gesture detection apparatus, and the relative angle is an angle between a direction of the gesture detection apparatus and a calculated reference direction;

an output unit configured to output audio and/or video based on output data;

a storage unit configured to store threshold data, association data, and the output data; and

a processor,

wherein the threshold data includes a threshold value in the X-axis direction, a threshold value in the Y-axis direction, a threshold value in the Z-axis direction, and a plurality of threshold values with respect to an angle around the Z-axis,

wherein the output data includes a plurality of output data patterns for the audio and/or the video,

wherein the association data includes a table that defines one-to-one correspondence between a plurality of threshold conditions and the plurality of output data patterns, wherein each of the plurality of threshold conditions, which corresponds to a particular gesture among a plurality of stored gestures, is determined by relation between an acceleration value in the X-axis direction and the threshold value in the X-axis direction, relation between an acceleration value in the Y-axis direction and the threshold value in the Y-axis direction, relation between an acceleration value in the Z-axis direction and the threshold value in the Z-axis direction, and relation between the angle around the Z-axis and the plurality of threshold values with respect to the angle around the Z-axis,

wherein the storage unit further stores a plurality of operational directions in which a plurality of percussion instruments in a set are respectively placed with respect to the calculated reference direction,

one of the plurality of threshold conditions corresponds to a gesture of beating a percussion instrument in the set,

wherein the reference direction is calculated based on the posture data detected by the gesture detection apparatus corresponding to a gesture of beating a predetermined one of the percussion instruments in the set,

wherein, when the program is executed by the computer, the program causes the processor of the computer to execute a process including;

detecting said one of the plurality threshold conditions by comparing a maximum value of the acceleration data in the X-axis direction with the threshold value in the X-axis direction, comparing a maximum value of the acceleration data in the Y-axis direction with the threshold value in the Y-axis direction, comparing a maximum value of the acceleration data in the Z-axis direction with the threshold value in the Z-axis direction, and comparing the relative angle and the plurality of threshold values with respect to the angle around the Z-axis,

selecting, from the plurality of output data patterns, one output data pattern corresponding to said one of the threshold conditions, and

causing the output unit to output the audio and/or the video based on the selected one output data pattern.