Sensor driven enhanced visualization and audio effects

• US 10,102,835 B1
• Filed: 04/28/2017
• Issued: 10/16/2018
• Est. Priority Date: 04/28/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Shah, Saurin, Kar, Swarnendu, Krishnamurthy, Lakshman, Webb, Arthur D, Goel, Manan
• Primary Examiner(s): Warren, David
• Application Number:
  US15/581,957
• Publication Number:
  US 20180315405A1
• Time in Patent Office: 536 Days
• Field of Search: 84615
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/0066 using a MIDI interface
• CPC Class Codes: G10H 1/0075 with translation or conversion means for unvailable commands, e.g. special tone colors
• CPC Class Codes: G10H 1/0083 using wireless transmission, e.g. radio, light, infrared
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data graphical user interfaces specifically adapted for electrophonic musical instruments G10H2220/091; fingering displays G10H2220/041
• CPC Class Codes: G10H 2220/185 Stick input, e.g. drumsticks with position or contact sensors stick for music conducting applications, e.g. conductor baton movement detection G10H2220/206
• CPC Class Codes: G10H 2220/191 Plectrum or pick sensing, e.g. for detection of string striking or plucking Plectra in general, e.g. for stringed musical instruments G10D3/173
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/206 Conductor baton movement detection used to adjust rhythm, tempo or expressivity of, e.g. the playback of musical pieces
• CPC Class Codes: G10H 2220/365 Bow control in general, i.e. sensors or transducers on a bow; Input interface or controlling process for emulating a bow, bowing action or generating bowing parameters, e.g. for appropriately controlling a specialised sound synthesiser bowed string instrument sound synthesis per se G10H2250/445; electrophonic stringed instrument details G10H2230/075
• CPC Class Codes: G10H 2220/395 Acceleration sensing or accelerometer use, e.g. 3D movement computation by integration of accelerometer data, angle sensing with respect to the vertical, i.e. gravity sensing. conductor baton movement sensing G10H2220/206, angle sensing without reference to gravity G10H2220/391; player body joint angle sensing G10H2220/321
• CPC Class Codes: G10H 2230/281 Spint drum assembly, i.e. mimicking two or more drums or drumpads assembled on a common structure, e.g. drum kit multi-toned percussion instruments G10H2230/255

Abstract

Systems and methods may be used to provide effects corresponding to movement of instrument objects or other objects. A method may include receiving sensor data from an object based on movement of the object, recognizing a gesture from the sensor data, and determining an effect, such as a visualization or audio effect corresponding to the gesture. The method may include causing the effect to be output in response to the determination.

Claims

1. A server in communication with a pair of drum sticks, the server comprising:

a processor to;

receive sensor data from a sensor of at least one drum stick of the pair of drum sticks, the sensor data based on movement of the at least one drum stick;

recognize a gesture from the sensor data;

determine, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and

cause the visualization effect and the audio effect to be output in response to the determination, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect.

2. The server of claim 1, wherein to determine the visualization effect, the processor is to use a visualization engine.

3. The server of claim 1, wherein the sensor data includes data from sensors of both of the pair of drum sticks and wherein the gesture includes movement of the pair of drum sticks in coordination with each other.

4. The server of claim 1, wherein to cause the visualization effect to be output, the processor is to send the visualization effect to a virtual reality headset of a user controlling the pair of drum sticks to be displayed on the virtual reality headset.

5. The server of claim 1, wherein to cause the audio effect to be output, the processor is to send the audio effect to a speaker to play the audio effect.

6. The server of claim 1, wherein the output to the display includes captured video of a person performing the movement with the at least one drumstick.

7. The server of claim 1, wherein the processor is further to receive data from the sensor indicating an initial position of the at least one drum stick, and wherein to recognize the gesture, the processor is to determine a final position of the at least one drum stick.

8. The server of claim 7, wherein the drum sound is determined based on the initial position and the final position.

9. The server of claim 1, wherein the visualization effect corresponding to the gesture and the audio effect including the drum sound corresponding to the gesture are determined based on an orientation of the at least one drum stick identified in the sensor data.

10. The server of claim 1, wherein to determine the visualization effect, the processor is to determine the visualization effect based on a series of previously recognized gestures.

11. The server of claim 1, wherein the processor is further to:

receive additional sensor data from a second sensor attached to an ankle or a foot of a user controlling the at least one drum stick;

recognize a second gesture from the additional sensor data;

determine, from the second gesture, a second audio effect including a second drum sound corresponding to the second gesture; and

cause the second audio effect to be output with the visualization effect and the audio effect.

12. The server of claim 1, wherein the processor is further to:

receive wearable sensor data from a plurality of wearable devices within a predetermined proximity of the at least one drum stick; and

modify the visualization effect based on the wearable sensor data.

13. The server of claim 1, wherein the visualization effect includes a lighting effect, and wherein to cause the visualization effect to be output, the processor is to send the lighting effect to a plurality of wearable devices within a predetermined proximity of the at least one drum stick to be displayed at the plurality of wearable devices.

14. The server of claim 1, wherein the gesture includes at least one of a linear movement, a tapping movement, a sweeping movement, a minimum acceleration, or a minimum deceleration.

15. The server of claim 1, wherein the audio effect includes Multidimensional Polyphonic Expression instructions for a musical instrument digital interface (MIDI) player.

16. A method for providing effects corresponding to movement of drum sticks, the method comprising:

receiving sensor data from a sensor of at least one drum stick of a pair of drum sticks, the sensor data based on movement of the at least one drum stick;

recognizing a gesture from the sensor data;

determining, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and

causing the visualization effect and the audio effect to be output in response to the determination, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect.

17. The method of claim 16, wherein determining the visualization effect includes using a visualization engine.

18. The method of claim 16, wherein the sensor data includes data from sensors of both of the pair of drum sticks and wherein the gesture includes movement of the pair of drum sticks in coordination with each other.

19. The method of claim 16, wherein causing the visualization effect to be output includes sending the visualization effect to a virtual reality headset of a user controlling the pair of drum sticks to be displayed on the virtual reality headset.

20. At least one non-transitory machine-readable medium including instructions for providing effects corresponding to movement of drum sticks, which when executed by a machine, cause the machine to:

receive sensor data from a sensor of at least one drum stick of a pair of drum sticks, the sensor data based on movement of the at least one drum stick;

recognize a gesture from the sensor data;

determine, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and

cause the visualization effect and the audio effect to be output in response to the determination, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect.

21. The at least one non-transitory machine-readable medium of claim 20, further comprising instructions to:

receive wearable sensor data from a plurality of wearable devices within a predetermined proximity of the at least one drum stick; and

modify the visualization effect based on the wearable sensor data.

22. The at least one non-transitory machine-readable medium of claim 20, wherein the visualization effect includes a lighting effect, and wherein the instructions to cause the visualization effect to be output include instructions to send the lighting effect to a plurality of wearable devices within a predetermined proximity of the at least one drum stick to be displayed at the plurality of wearable devices.

23. A virtual drum set system comprising:

a pair of drum sticks each including;

a sensor to provide data based on movement of a drum stick of the pair of drum sticks; and

a transceiver to transmit the sensor data;

a device including a processor to;

recognize a gesture from the sensor data; and

determine, from the gesture, a visualization effect corresponding to the gesture and an audio effect including a drum sound corresponding to the gesture; and

a display device to display the visualization effect, wherein the visualization effect includes a virtual drum set and is output to a display to be displayed in coordination with the audio effect; and

a speaker to play the audio effect.

24. The virtual drum set system of claim 23, wherein the display device is a virtual realty headset and the visualization effect includes a virtual drum set.

25. The virtual drum set system of claim 23, wherein the sensor includes a nine-axis sensor including a magnetometer, an accelerometer, and a gyroscope.