INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM

• US 20100211200A1
• Filed: 12/04/2009
• Published: 08/19/2010
• Est. Priority Date: 12/05/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: KOBAYASHI, Yoshiyuki
• Granted Patent: US 8,706,274 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/8047 Music games
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/051 for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data
• CPC Class Codes: G10H 2220/135 Musical aspects of games or videogames; Musical instrument-shaped game input interfaces
• CPC Class Codes: G10H 2220/141 Games on or about music, i.e. based on musical knowledge, e.g. musical multimedia quizzes
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

An information processing apparatus is provided which includes a metadata extraction unit for analyzing an audio signal in which a plurality of instrument sounds are present in a mixed manner and for extracting, as a feature quantity of the audio signal, metadata changing along with passing of a playing time, and a player parameter determination unit for determining, based on the metadata extracted by the metadata extraction unit, a player parameter for controlling a movement of a player object corresponding to each instrument sound.

Claims

1. An information processing apparatus comprising:

a metadata extraction unit for analyzing an audio signal in which a plurality of instrument sounds are present in a mixed manner and for extracting, as a feature quantity of the audio signal, metadata changing along with passing of a playing time; and

a player parameter determination unit for determining, based on the metadata extracted by the metadata extraction unit, a player parameter for controlling a movement of a player object corresponding to each instrument sound.

2. The information processing apparatus according to claim 1, wherein

the metadata extraction unit extracts, as the metadata, one or more pieces of data selected from among a group formed from a beat of the audio signal, a chord progression, a music structure, a melody line, a bass line, a presence probability of each instrument sound, a solo probability of each instrument sound and a voice feature of vocals.

3. The information processing apparatus according to claim 2, wherein

the metadata extraction unit extracts, as the metadata, one or more pieces of data selected from among a group formed from a genre of music to which the audio signal belongs, age of the music to which the audio signal belongs, information of the audio signal relating to a player, types of the instrument sounds included in the audio signal and tone of music of the audio signal.

4. The information processing apparatus according to claim 3, wherein

the player parameter determination unit

determines, in case information on height and weight of a player is extracted as the information relating to the player, a player parameter indicating a size of the player object based on the information on height and weight, and

determines, in case information on a sex of the player is extracted as the information relating to the player, a player parameter indicating a hairstyle and clothing of the player object based on the information on a sex.

5. The information processing apparatus according to claim 3, further comprising:

a lighting parameter determination unit for determining, based on the metadata extracted by the metadata extraction unit, a lighting parameter for controlling lighting on a stage on which the player object is placed,

wherein

the lighting parameter determination unit determines the lighting parameter so that the lighting changes in sync with the beat detected by the metadata extraction unit.

6. The information processing apparatus according to claim 5, wherein

the lighting parameter determination unit determines, based on the presence probability of each instrument sound extracted by the metadata extraction unit, a lighting parameter indicating a brightness of a spotlight shining on the player object corresponding to the each instrument sound.

7. The information processing apparatus according to claim 6, wherein

the lighting parameter determination unit refers to the music structure extracted by the metadata extraction unit, and determines the lighting parameter so that the lighting changes according to a type of a structure of music being played.

8. The information processing apparatus according to claim 7, wherein

the lighting parameter determination unit determines the lighting parameter so that a colour of the lighting changes based on the age of the music extracted by the metadata extraction unit.

9. The information processing apparatus according to claim 2, further comprising:

an audience parameter determination unit for determining, based on the metadata extracted by the metadata extraction unit, an audience parameter for controlling a movement of audience objects placed in audience seats provided in a location different from the stage,

wherein

the audience parameter determination unit determines the audience parameter so that the movement of the audience objects changes in sync with the beat detected by the metadata extraction unit.

10. The information processing apparatus according to claim 9, wherein

the audience parameter determination unit refers to the music structure extracted by the metadata extraction unit, and determines the audience parameter so that the movement of the audience objects changes according to a type of a structure of music being played.

11. The information processing apparatus according to claim 2, wherein

the player parameter determination unit determines, based on the solo probability of each instrument sound extracted by the metadata extraction unit, a player parameter indicating a posture and an expression of the player object corresponding to the each instrument sound.

12. The information processing apparatus according to claim 11, wherein

the player parameter determination unit determines, based on the presence probability of each instrument sound extracted by the metadata extraction unit, a player parameter indicating a moving extent of a playing hand of the player object corresponding to the each instrument sound.

13. The information processing apparatus according to claim 2, wherein

the player parameter determination unit determines, based on the presence probability of vocals extracted by the metadata extraction unit, a player parameter indicating a size of an open mouth of the player object corresponding to the vocals or a distance between a hand holding a microphone and the mouth.

14. The information processing apparatus according to claim 13, wherein

the player parameter determination unit determines, based on a difference between an average pitch of the melody line extracted by the metadata and a pitch of the melody line for each frame or based on the voice feature of vocals extracted by the metadata extraction unit, a player parameter indicating a movement of an expression of the player object corresponding to the vocals.

15. The information processing apparatus according to claim 14, wherein

the player parameter determination unit determines, based on the melody line extracted by the metadata extraction unit, a player parameter indicating a movement of a hand not holding a microphone, the hand being of the player object corresponding to the vocals.

16. The information processing apparatus according to claim 2, wherein

the player parameter determination unit determines, based on the chord progression extracted by the metadata extraction unit, a player parameter indicating a position of a hand of the player object, the player parameter corresponding to one or more sections selected from among a group formed from a guitar, a keyboard and strings.

17. The information processing apparatus according to claim 2, wherein

the player parameter determination unit determines, based on the bass line extracted by the metadata extraction unit, a position of a hand holding a neck, the hand being of the player object corresponding to a bass.

18. The information processing apparatus according to claim 1, wherein

the player object is an externally connected robot or a player image realized by computer graphics, and

the information processing apparatus further includes an object control unit for controlling a movement of the externally connected robot by using the player parameter determined by the player parameter determination unit or for controlling a movement of the player image by using the player parameter determined by the player parameter determination unit.

19. An information processing method comprising the steps of:

analyzing an audio signal in which a plurality of instrument sounds are present in a mixed manner and extracting, as a feature quantity of the audio signal, metadata changing along with passing of a playing time; and

determining, based on the metadata extracted by the step of analyzing and extracting, a player parameter for controlling a movement of a player object corresponding to each instrument sound.

20. A program for causing a computer to realize:

a metadata extraction function for analyzing an audio signal in which a plurality of instrument sounds are present in a mixed manner and for extracting, as a feature quantity of the audio signal, metadata changing along with passing of a playing time; and

a player parameter determination function for determining, based on the metadata extracted by the metadata extraction function, a player parameter for controlling a movement of a player object corresponding to each instrument sound.