INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM

• US 20140297012A1
• Filed: 04/01/2014
• Published: 10/02/2014
• Est. Priority Date: 12/05/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: KOBAYASHI, Yoshiyuki
• Granted Patent: US 9,557,956 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/8047 Music games
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/051 for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data graphical user interfaces specifically adapted for electrophonic musical instruments G10H2220/091; fingering displays G10H2220/041
• CPC Class Codes: G10H 2220/135 Musical aspects of games or videogames; Musical instrument-shaped game input interfaces game background music G10H2210/026; musical game scoring or performance evaluation G10H2210/091
• CPC Class Codes: G10H 2220/141 Games on or about music, i.e. based on musical knowledge, e.g. musical multimedia quizzes teaching of music per se G09B15/00
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

An information processing apparatus is provided which includes a metadata extraction unit for analyzing an audio signal in which a plurality of instrument sounds are present in a mixed manner and for extracting, as a feature quantity of the audio metadata changing along with passing of a playing time, and a player parameter determination unit for determining, based on the metadata extracted by the metadata extraction unit, a player parameter for controlling a movement of a player object corresponding to each instrument sound.

Claims

1. An information processing apparatus comprising:

a signal determining unit for determining signal data;

a metadata extracting unit for analyzing the signal data comprising a plurality of signals present in a mixed manner, and for extracting, as a feature quantity of the signal data, metadata that changes along with passage of playing time;

a determining unit for determining one or more object parameters based on the extracted metadata, and

a switching unit for switching between one or more objects based on the one or more object parameters in a time sequence.

2. The information processing apparatus according to claim 1, wherein the object parameter is associated with switching from displaying the object to not displaying the object, the switching being performed based on a calculated presence probability of a signal.

3. The information processing apparatus according to claim 1, wherein the object is an externally connected robot or an object image realized by computer graphics, and the information processing apparatus further includes an object control unit for controlling a movement of the externally connected robot by using the object parameter determined by the determining unit or for controlling a movement of the object image by using the object parameter.

4. The information processing apparatus of claim 1, further comprising:

determining, based on the metadata, an audience parameter for controlling, movement of audience objects placed in audience seats provided in a location different from a stage,

wherein the audience parameter determines changes in the synchronicity of the movement of the audience objects with a beat determined by the metadata.

5. The information processing apparatus of claim 1, wherein the audience parameter comprises at least a music structure extracted from the metadata to determine the movement of the audience objects according to a type of a structure of music being played.

6. The information processing apparatus of claim 1, wherein the metadata comprises one or more pieces of data comprising:

a beat of the signal data, a chord progression, a music structure, a melody line, a bass line, a presence probability of each signal, a solo probability of each signal, and a voice feature of vocals.

7. An information processing method comprising the steps of:

analyzing signal data comprising a plurality of signals present in a mixed manner and extracting, as a feature quantity of the signal data, metadata that changes along with passage of playing time;

determining, based on the extracted metadata, one or more object parameters for controlling movement of objects corresponding to the plurality of signals; and

switching between one or more objects based on the one or more object parameters in a time sequence.

8. The information processing method of claim 7, wherein the metadata comprises one or more pieces of data comprising:

a beat of the signal data, a chord progression, a music structure, a melody line, a bass line, a presence probability of each signal, a solo probability of each signal, and a voice feature of vocals.

9. The information processing method of claim 8, wherein based on a solo probability of each signal, an object parameter can indicate a posture and an expression of the object corresponding to each signal.

10. The information processing method of claim 8, wherein based on a presence probability of each signal, the object parameter can indicate a moving extent of a playing hand of the object corresponding to each signal.

11. The information processing method of claim 8, wherein based on a presence probability of vocals, an object parameter can indicate a size of an open mouth of the object corresponding to vocals or a distance between a hand holding a microphone and the mouth.

12. The information processing method of claim 11, wherein based on a difference between an average pitch of the melody line and a pitch of the melody line for each frame, or based on the voice feature of vocals, an object parameter can indicate a movement of an expression of the object corresponding to the vocals.

13. The information processing method of claim 12, wherein based on the melody line, an object parameter can indicate a movement of a hand not holding a microphone, the hand being of the object corresponding to the vocals.

14. The information processing method of claim 8, wherein based on a chord progression, an object parameter, corresponding to one or more sections selected from among a group formed from a guitar, a keyboard and strings, can indicate a position of a hand of the object.

15. The information processing method of claim 8, wherein based on a bass line, a position of a hand holding a neck, the hand being of the object corresponding to a bass.

16. A non-transitory computer readable medium comprising program code for causing a computer to perform steps comprising:

analyzing signal data comprising a plurality of signals present in a mixed manner and extracting, as a feature quantity of signal data, metadata that changes along with passage of playing time;

determining, based on the extracted metadata, one or more object parameters for controlling movement of objects corresponding to the plurality of signals; and

switching between one or more objects based on the one or more object parameters in a time sequence.

17. The non-transitory computer readable medium of claim 16, wherein the metadata comprises one or more pieces of data comprising:

a genre of music to which the signal data belongs, an age of the music to which the signal data belongs, information about the signal data relating to an object, types of signals included in the signal data, and tone of music of the signal data.

18. The non-transitory computer readable medium of claim 17, further comprising:

determining, in case information on height and weight of an object is extracted as the information relating to an object, an object parameter indicating a size of the object based on the information on height and weight, and

determining, in case information on a sex of the object is extracted as the information relating to an object, an object parameter indicating a hairstyle and clothing of the object based on the information on a sex.

19. The non-transitory computer readable medium of claim 17, further comprising:

determining a lighting parameter, based on the extracted metadata, for controlling lighting on a stage on which the object is placed,

wherein the lighting parameter is determined such that the lighting on the stage changes synchronously with a beat detected by the extracted metadata.

20. The non-transitory computer readable medium of claim 19, wherein the lighting parameter, based on the presence probability of each signal of the plurality of signals, can indicate a brightness of a spotlight shining on an object corresponding to each signal.

21. The non-transitory computer readable medium of claim 20, wherein the lighting parameter comprises at least a music structure extracted from the metadata, to determine the lighting changes according to a type of a structure of music being played.

22. The non-transitory computer readable medium of claim 21, wherein the lighting parameter determines a color of the lighting changes based on the age of the music extracted from the metadata.

23. The non-transitory computer readable medium of claim 16, wherein the metadata comprises one or more pieces of data comprising:

a beat of the signal data, a chord progression, a music structure, a melody line, a bass line, a presence probability of each signal, a solo probability of each signal, and a voice feature of vocals.