Computerized percussion instrument

• US 9,224,377 B2
• Filed: 11/09/2012
• Issued: 12/29/2015
• Est. Priority Date: 11/11/2011
• Status: Active Grant

• Current Assignee: Fictitious Capital Limited
• Original Assignee: Fictitious Capital Limited
• Inventors: Lee, Richard Jason, Morvan, Yann Pierre
• Primary Examiner(s): Uhlir, Christopher
• Application Number:
  US13/672,797
• Publication Number:
  US 20130118339A1
• Time in Patent Office: 1,145 Days
• Field of Search: 847/23
• US Class Current: 1/1
• CPC Class Codes: G10H 1/14 during execution voice controlled instruments G10H5/005
• CPC Class Codes: G10H 1/32 Constructional details
• CPC Class Codes: G10H 1/34 Switch arrangements, e.g. keyboards or mechanical switches specially adapted for electrophonic musical instruments
• CPC Class Codes: G10H 2220/185 Stick input, e.g. drumsticks with position or contact sensors stick for music conducting applications, e.g. conductor baton movement detection G10H2220/206
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/455 Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
• CPC Class Codes: G10H 2230/281 Spint drum assembly, i.e. mimicking two or more drums or drumpads assembled on a common structure, e.g. drum kit multi-toned percussion instruments G10H2230/255
• CPC Class Codes: G10H 3/146 using a membrane, e.g. a drum; Pick-up means for vibrating surfaces, e.g. housing of an instrument
• CPC Class Codes: G10H 7/00 Instruments in which the tones are synthesised from a data store, e.g. computer organs

Abstract

A computerized musical percussion instrument is disclosed.

Markers carried by the musician are observed by an imager to produce a series of two dimensional images over the time of the performance.

A processor receives the images and distinguishes between markers (e.g. left hand, right hand) by comparing the position and size of unidentified markers in the current image to the position and size of identified markers in preceding images.

The processor analyzes each markers'"'"' movements and detects a drum hit when a marker undergoes a sharp reversal of its motion direction after reaching sufficient speed. The processor determines which drum the musician intends to hit by comparing the position and size of the marker at the instant of the hit to the position and size attributes of each drum. The processor outputs an audio signal for each hit, corresponding to the drum hit, with a volume determined by marker speed.

Claims

1. A musical instrument comprising

an imager arranged to provide a series of two dimensional images of an operator of the musical instrument;

a processor, coupled to the imager to receive the images, the processor configured to;

store an indication of a size and/or position of a first marker in a first selected image of the series of two dimensional images;

store an indication of a size and/or position of a second marker in the first selected image of the series of two dimensional images;

use the stored indication of the size and/or position of the first marker and the stored indication of the size and/or position of the second marker in the first selected image of the series of two dimensional images to distinguish between the first marker and the second marker in a second selected image of the series of two dimensional images independent of the proximity of the first marker relative to the second marker based on at least;

the size and/or position of the first marker and the size and/or position of the second marker in the second selected image of the series of two dimensional images, and

the size and/or position of the first marker and the size and/or position of the second marker in at least one preceding image of the series of two dimensional images that was captured before the first selected image of the series of two dimensional images and the second selected image of the series of two dimensional images; and

trigger an audio output signal based on at least one of;

the size and/or position of the first marker,

the size and/or position of the second marker,

movements of the first marker, and

movements of the second marker.

2. The musical instrument of claim 1 wherein the processor is configured to respond to at least one of the first marker and the second marker completing a selected sequence of movements by selecting an audio signal for output based on at least one of the size and/or position of the first marker and the size and/or position of the second marker.

3. The musical instrument of claim 1 in which the processor is configured to:

determine, for each of the first marker and the second marker that was present in a first one of the at least one preceding image of the series of two dimensional images, whether that marker was also present in a second one of the at least one preceding image of the series of two dimensional images,

if present, determine a change in size and/or a change in position of the marker between the first preceding image of the series of two dimensional images and the second preceding image of the series of two dimensional images, and

distinguish between the first marker and the second marker based on at least one of the determined changes in size and/or position of the first marker and the determined changes in size and/or position of the second marker between the first preceding image of the series of two dimensional images and the second preceding image of the series of two dimensional images independent of the proximity of the first marker relative to the second marker.

4. The musical instrument of claim 2 in which the selected sequence of movements comprises at least one reversal in the movement of at least one of the first and second marker, in which a reversal comprises the at least one of the first marker and the second marker moving in a first direction at a speed superior to a selected speed for at least a selected first number of images of the series of two dimensional images, followed by a movement in a second direction opposite to the first direction, or by an absence of movement, for at least a selected second number of images of the series of two dimensional images.

5. The musical instrument of claim 2 in which the processor is configured to control a volume of the audio signal based on a speed of at least one of the first marker and the second marker.

6. The musical instrument of claim 1 in which the imager comprises only a single camera and the images consist solely of a series of two dimensional images collected from that single camera.

7. The musical instrument of claim 1 in which the first marker and the second marker each comprises a retro-reflector carried by the operator and in which the instrument further comprises a lamp positioned in proximity to the imager so as to illuminate the imager by reflecting light from the retro-reflectors when, in use, the retro-reflectors are arranged to direct light towards the imager.

8. The musical instrument of claim 2, in which the processor is configured to communicate an indication of an audio signal to a user, and to store an association between the audio signal and at least one of the size and/or position of the first and and/or position of the second marker in response to at least one of the first marker and the second marker completing a selected sequence of movements.

9. The musical instrument of claim 8 in which the indication of an audio signal comprises a name and/or another visual indication of a musical instrument.

10. The musical instrument of claim 9 in which selecting an audio signal for output comprises selecting the audio signal based on the stored association.

19. The musical instrument of claim 1 in which the processor is further configured to:

store an indication of a size and/or position of a third marker in the first selected image of the series of two dimensional images, and

use the stored indication of the size and/or position of the first marker, the stored indication of the size and/or position of the second marker, and the stored indication of the third marker in the first selected image of the series of two dimensional images to distinguish between each of the first marker, the second marker and the third marker in the second selected image of the series of two dimensional images independent of the proximity of the first marker relative to the second marker and the third marker based on at least;

the size and/or position of the first marker, the size and/or position of the second marker, and the size and/or position of the third marker in the first selected image of the series of two dimensional images, and

the size and/or position of the first marker, the size and/or position of the second marker, and the size and/or position of the third marker in the second selected image of the series of two dimensional images; and

trigger an audio output signal based on at least one of;

the size and/or position of the first marker,

the size and/or position of the second marker,

the size and/or position of the third marker,

movements of the first marker,

movements of the second marker, and

movements of the third marker.

11. A computer implemented method of processing images to control audio signals so as to simulate a musical instrument, the method comprising:

receiving a series of two dimensional images of an operator of the musical instrument;

storing an indication of a size and/or position of a first marker in a first selected image of the series of two dimensional images;

storing an indication of a size and/or position of a second marker in the first selected image of the series of two dimensional images;

using the stored indication of the size and/or position of the first marker and the stored indication of the size and/or position of the second marker in the first selected image of the series of two dimensional images to distinguish between the first marker and the second marker in a second selected image of the series of two dimensional images independent of the proximity of the first marker relative to the second marker based on at least;

the size and/or position of the first marker and the size and/or position of the second marker in the second selected image, and

the size and/or position of the first marker and the size and/or position of the second marker in at least one preceding image of the series of two dimensional images that was captured before the first selected image of the series of two dimensional images and the second selected image of the series of two dimensional images; and

triggering an audio output signal based on at least one of;

the size and/or position of the first marker,

the size and/or position of the second marker,

movements of the first marker, and

movements of the second marker.

12. The computer implemented method of claim 11 further comprising:

storing an indication of at least one of the size and/or position of the first marker and the size and/or position of the second marker in the at least one preceding image of the series of two dimensional images, for use in distinguishing between the first marker and the second marker in at least one of the first selected image and second selected image of the series of two dimensional images or a subsequent image of the series of two dimensional images.

13. The computer implemented method of claim 12 further comprising:

determining, for each of the first marker and the second marker that was present in a first one of the at least one preceding image of the series of two dimensional images, whether that marker was also present in a second one of the at least one preceding image of the series of two dimensional images;

if present, determining a change in size and/or a change in position of the marker between the first preceding image of the series of two dimensional images and the second preceding image of the series of two dimensional images, and

distinguishing between the first marker and the second marker based on at least one of the determined changes in size and/or position of the first marker and the determined changes in size and/or position of the second marker between the first preceding image of the series of two dimensional images and the second preceding image of the series of two dimensional images independent of the proximity of the first marker relative to the second marker.

14. The computer implemented method of claim 11 in which the selected sequence of movements comprises at least one reversal in the movement of a marker at least one of the first marker and the second marker, in which a reversal comprises the at least one of the first marker and the second marker moving in a first direction at a speed superior to a selected speed for at least a selected first number of images of the series of two dimensional images, followed by a movement in a second direction opposite to the first direction, or by an absence of movement, for at least a selected second number of images of the series of two dimensional images.

15. The computer implemented method of claim 11 in which a volume of the triggered audio output signal is determined based on a speed of at least one of the first marker and the second marker.

16. The computer implemented method of claim 11 in which the series of two dimensional images comprises images collected solely from a single camera.

17. A computer program product, comprising a computer readable medium storing program instructions for causing a processor to perform the method of claim 11.

20. The computer implemented method of claim 11 further comprising:

identifying a third marker by determining a size and/or position of the third marker in the first selected image of the series of two dimensional images;

storing an indication of the size and/or position of the third marker in the first selected image of the series of two dimensional images;

using the stored indication of the size and/or position of the first marker, the stored indication of the size and/or position of the second marker, and size and/or position of the third marker in the first selected image of the series of two dimensional images to distinguish between each of the first marker, the second marker and the third marker independent of the proximity of the first marker relative to the second marker and the third marker in a second selected image of the series of two dimensional images based on at least;

the size and/or position of the first marker, the size and/or position of the second marker and the size and/or position of the third marker in the first selected image of the series of two dimensional images, and

the size and/or position of the first marker, the size and/or position of the second marker and the size and/or position of the third marker in the second selected image of the series of two dimensional images; and

triggering an audio output signal based on at least one of;

the size and/or position of the first marker,

the size and/or position of the second marker,

the size and/or position of the third marker,

movements of the first marker,

movements of the second marker, and

movements of the third marker.

18. A kit for adapting a computer to provide a musical instrument, the kit comprising:

a wide angle lens adapter for a digital camera;

at least one retro-reflector to be carried by a user; and

a lamp, coupled to the wide angle lens adapter so as to illuminate the wide angle lens adapter by reflecting light from the retro-reflector when, in use, the retro-reflector is arranged to direct light towards the wide angle lens adapter;

a computer program product storing program instructions for causing a processor to perform a method that comprises;

receiving a series of two dimensional images of an operator of the musical instrument;

storing an indication of a size and/or position of a first marker in a first selected image of the series of two dimensional images;

storing an indication of a size and/or position of a second marker in the first selected image of the series of two dimensional images;

using the stored indication of the size and/or position of the first marker and the stored indication of the second marker in the first selected image of the series of two dimensional images to distinguish between the first marker and the second marker in a second selected image of the series of two dimensional images independent of the proximity of the first marker relative to the second marker based on at least;

the size and/or position of the first marker and the size and/or position of the second marker in the second selected image of the series of two dimensional images, and

the size and/or position of the first marker and the size and/or position of the second marker in at least one preceding image of the series of two dimensional images that was captured before the first selected image of the series of two dimensional images and the second selected image of the series of two dimensional images; and

triggering an audio output signal based on at least one of;

the size and/or position of the first marker,

the size and/or position of the second marker,

movements of the first marker, and

movements of the second marker.