Systems and methods for playing virtual music instrument through tracking of fingers with coded light

• US 9,830,894 B1
• Filed: 05/25/2016
• Issued: 11/28/2017
• Est. Priority Date: 05/25/2016
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Ma, Shang, Liu, Qiong
• Primary Examiner(s): Fletcher, Marlon
• Application Number:
  US15/164,548
• Publication Number:
  US 20170345403A1
• Time in Patent Office: 552 Days
• CPC Class Codes: G10H 1/34 Switch arrangements, e.g. keyboards or mechanical switches specially adapted for electrophonic musical instruments
• CPC Class Codes: G10H 2220/041 Remote key fingering indicator, i.e. fingering shown on a display separate from the instrument itself or substantially disjoint from the keys
• CPC Class Codes: G10H 2220/066 Colour, i.e. indications with two or more different colours
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/326 Control glove or other hand or palm-attached control device
• CPC Class Codes: G10H 2220/411 Light beams key actuation detection using light G10H2220/305
• CPC Class Codes: G10H 2230/065 Spint piano, i.e. mimicking acoustic musical instruments with piano, cembalo or spinet features, e.g. with piano-like keyboard; Electrophonic aspects of piano-like acoustic keyboard instruments; MIDI-like control therefor
• CPC Class Codes: G10H 2230/351 Spint bell, i.e. mimicking bells, e.g. cow-bells bells in general G10K1/06
• CPC Class Codes: G10H 3/00 Instruments in which the tones are generated by electromechanical means

Abstract

Finger-tracking systems and methods for virtual instruments playback. The described system tracks the position of user'"'"'s ten fingers on a projection surface and can be used to play virtual instruments such as virtual piano, drums, and bells. The system tracks the movement of user'"'"'s ten fingers while keeping them free of encumbrance or excessive postural constraints. More specifically, in one or more embodiments, a coded light based projector is used to send out location signal onto a flat surface, and ten light sensors are mounted on user'"'"'s fingers to receive these signals and locate user'"'"'s fingers. Based on their locations and relative distance to a fixed point, a printed music instrument can be used for virtual instrument music playback. With the described tracking system, various embodiments of virtual music instruments may be implemented, including a system and method for virtual piano playing as well as virtual Chinese bell playing on printed keyboard and printed Chinese bell set.

Claims

1. A finger-tracking system comprising:

a. a projector configured to project a temporal projector light signal, wherein the temporal projector light signal is encoded, for each pixel of the projector, with an information segment comprising the pixel coordinates of the each pixel of the projector;

b. a plurality of light sensors, each of the plurality of light sensors being attached to a finger of a user, wherein each of the plurality of light sensors is configured to detect the temporal projector light signal and generate a sensor signal; and

c. a processing unit operatively coupled to each of the plurality of light sensors and configured to receive the sensor signal from each of the plurality of light sensors, to determine a location information of each finger of the user and to issue a command based on the detected location of at least one finger of the user.

2. The finger-tracking system of claim 1, wherein the processing unit determines the location information of each finger of the user by identifying a projector pixel corresponding to the sensor signal from a light sensor attached to the corresponding finger of the user.

3. The finger-tracking system of claim 1, wherein the issued command causes a sound or a musical note to be synthesized.

4. The finger-tracking system of claim 1, wherein the location information of each finger of the user is determined in relation to an image of a piano keyboard comprising a plurality of piano keys.

5. The finger-tracking system of claim 4, wherein the processing unit uses the location information of each finger of the user to determine which piano key of the plurality of piano keys has been pressed.

6. The finger-tracking system of claim 5, wherein the processing unit causes a sound or a musical note corresponding to pressed piano key to be synthesized.

7. The finger-tracking system of claim 5, wherein the processing unit causes a sequence of pressed piano keys to be recorded.

8. The finger-tracking system of claim 7, wherein the recorded sequence of pressed piano keys is compared with a reference sequence to determine a difference and a feedback to the user is generated based on the determined difference.

9. The finger-tracking system of claim 1, wherein the location information of each finger of the user is determined in relation to an image of a plurality of Chinese bells.

10. The finger-tracking system of claim 9, wherein the processing unit uses the location information of each finger of the user to determine which bell of the plurality of Chinese bells has been struck.

11. The finger-tracking system of claim 10, wherein the processing unit causes a sound or a musical note corresponding to the struck bell to be synthesized.

12. The finger-tracking system of claim 1, wherein the temporal projector light signal projected by the project comprises a plurality of sequential light pulses encoding pixel coordinates of the each pixel of the projector.

13. The finger-tracking system of claim 1, further comprising a computer system comprising a display unit and operatively coupled with the processing unit and configured to receive from the processing unit the determined location information of each finger of the user and to display the received location information of each finger of the user on the display unit.

14. The finger-tracking system of claim 13, wherein the location information of each finger of the user is displayed on the display unit in a different color.

15. The finger-tracking system of claim 1, wherein the projector is a DLP projector.

16. A method for tracking fingers of a user, the method comprising:

a. using a projector to project a temporal projector light signal, wherein the temporal projector light signal is encoded, for each pixel of the projector, with an information segment comprising the pixel coordinates of the each pixel of the projector;

b. detecting the temporal projector light signal using a plurality of light sensors, each of the plurality of light sensors being attached to a finger of a user, wherein each of the plurality of light sensors is configured to detect the temporal projector light signal and generate a sensor signal; and

c. using a processing unit operatively coupled to each of the plurality of light sensors to receive the sensor signal from each of the plurality of light sensors, to determine a location information of each finger of the user and to issue a command based on the detected location of at least one finger of the user.

17. The method for tracking fingers of a user of claim 16, wherein the processing unit determines the location information of each finger of the user by identifying a projector pixel corresponding to the sensor signal from a light sensor attached to the corresponding finger of the user.

18. The method for tracking fingers of a user of claim 16, wherein the issued command causes a sound or a musical note to be synthesized.

19. The method for tracking fingers of a user of claim 16, wherein the location information of each finger of the user is determined in relation to an image of a piano keyboard comprising a plurality of piano keys.

20. A computer-readable medium embodying a set of instructions implementing a method for tracking fingers of a user, the method comprising:

a. using a projector to project a temporal projector light signal, wherein the temporal projector light signal is encoded, for each pixel of the projector, with an information segment comprising the pixel coordinates of the each pixel of the projector;

b. detecting the temporal projector light signal using a plurality of light sensors, each of the plurality of light sensors being attached to a finger of a user, wherein each of the plurality of light sensors is configured to detect the temporal projector light signal and generate a sensor signal; and

c. using a processing unit operatively coupled to each of the plurality of light sensors to receive the sensor signal from each of the plurality of light sensors, to determine a location information of each finger of the user and to issue a command based on the detected location of at least one finger of the user.