Free-space gesture musical instrument digital interface (MIDI) controller

• US 8,618,405 B2
• Filed: 12/09/2010
• Issued: 12/31/2013
• Est. Priority Date: 12/09/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tansley, Dennis Stewart
• Primary Examiner(s): Donels, Jeffrey
• Application Number:
  US12/963,866
• Publication Number:
  US 20120144979A1
• Time in Patent Office: 1,118 Days
• Field of Search: 847/23, 846/45, 847/24
• US Class Current: 84/645
• CPC Class Codes: G10H 1/0066 using a MIDI interface
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/455 Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes

Abstract

The free-space gesture MIDI controller technique described herein marries the technologies embodied in a free-space gesture controller with MIDI controller technology, allowing a user to control an infinite variety of electronic musical instruments through body gesture and pose. One embodiment of the free-space gesture MIDI controller technique described herein uses a human body gesture recognition capability of a free-space gesture control system and translates human gestures into musical actions. Rather than directly connecting a specific musical instrument to the free-space gesture controller, the technique generalizes its capability and instead outputs standard MIDI signals, thereby allowing the free-space gesture control system to control any MIDI-capable instrument.

Claims

1. A computer-implemented process for using free-space gesture recognition to control a MIDI-capable electronic device, comprising:

using a depth camera, capturing free-space gestures of a first human being simulating playing a musical device;

mapping each gesture captured to a standard MIDI control signal for operating the musical device;

capturing audio of the first human being, or vocal or audio from another instrument, and any additional human beings present;

using the mapped MIDI control signals to control a MIDI-capable musical device while playing back the captured audio.

2. The computer-implemented process of claim 1 wherein the mapping further comprises:

mapping each gesture captured to a standard MIDI control signal using a game console.

3. The computer-implemented process of claim 1 wherein the mapping further comprises:

mapping each gesture captured to a standard MIDI control signal using a computing device.

4. The computer-implemented process of claim 1 wherein the audio is captured by a microphone array that can also perform sound source localization.

5. The computer-implemented process of claim 1 wherein the MIDI-capable electronic device that can be controlled using the mapped MIDI control signals is a musical instrument.

6. The computer-implemented process of claim 1, further comprising:

capturing gestures of at least one additional human being playing at least one additional electronic device;

mapping each gesture captured by the at least one additional human being to a standard MIDI control signal for operating each of the at least one additional electronic device;

using the mapped MIDI control signals to control each of the at least one additional MIDI-capable electronic device.

7. The computer-implemented process of claim 1 wherein at least one of the additional human beings are at a different location of the first human being.

8. The computer-implemented process of claim 1 wherein any MIDI-capable electronic device can be controlled with the mapped gestures.

9. The computer-implemented process of claim 1 wherein the mapping of each gesture to a standard MIDI control signal is fixed to a certain control signal meaning.

10. The computer-implemented process of claim 1 wherein the mapping of each gesture to a standard MIDI control signal is editable by a user to allocate certain gestures to certain control signal meanings.

11. A computer-implemented process for using free-space gesture recognition to control a MIDI-capable electronic musical instrument, comprising:

using one or more depth cameras, capturing free-space gestures of more than one human simulating playing an electronic musical instrument, each of the one or more human beings simulating playing an electronic musical instrument using the captured gestures in a different location;

mapping each free-space gesture of each human being captured to a standard MIDI control signal for a standard MIDI-capable musical instrument;

using the mapped MIDI control signals to play the one or more standard MIDI-capable musical instruments;

sending audio of at least one human being playing an electronic musical instrument at a first location to the location of at least one other human being playing an electronic musical instrument over a network; and

playing the sent audio with the at least one other human being playing the electronic musical instrument.

12. The computer-implemented process of claim 11 wherein the mapping of each free-space gesture to a standard MIDI control signal is fixed.

13. The computer-implemented process of claim 11 wherein the mapping of each gesture to a standard MIDI control signal is editable by a user to allocate certain gestures to certain control signal meanings.

14. The computer-implemented process of claim 11, further comprising:

sending video of one or more human beings playing an electronic musical instrument at the first location to the location of at least one other human being playing an electronic musical instrument over a network.

15. A system for playing a musical device using gestures, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

capture gestures of a human being simulating playing an electronic musical device using a depth camera, wherein the module to capture gestures further comprises sub-modules to;

transmit encoded information on infrared light patterns in a space where the human being is gesturing; and

capture changes to the encoded infrared light patterns with the depth camera to determine which gestures the human being is making;

map each gesture captured to a standard control signal for operating an electronic musical device;

use the mapped control signals to play an electronic musical device; and

capture audio of the human being, or vocal or audio from another instrument along with audio from the electronic musical device played using the mapped control signals.

16. The system of claim 15, wherein the module to map each gesture captured to a standard control signal for operating the musical device further comprising modules to:

prompt a human being to make a gesture representing a musical note or sequence;

record a gesture made by the prompted human being; and

map the recorded gesture to the musical note or sequence.

17. The system of claim 15, wherein each standard control signal is a MIDI control signal.