Information processing apparatus, information processing method, and program

• US 9,557,956 B2
• Filed: 04/01/2014
• Issued: 01/31/2017
• Est. Priority Date: 12/05/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kobayashi, Yoshiyuki
• Primary Examiner(s): Flanders, Andrew C
• Application Number:
  US14/242,538
• Publication Number:
  US 20140297012A1
• Time in Patent Office: 1,036 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 2300/8047 Music games
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/051 for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/076 for extraction of timing, tempo; Beat detection tempo display G10H2220/081; tempo control G10H2210/375
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data graphical user interfaces specifically adapted for electrophonic musical instruments G10H2220/091; fingering displays G10H2220/041
• CPC Class Codes: G10H 2220/135 Musical aspects of games or videogames; Musical instrument-shaped game input interfaces game background music G10H2210/026; musical game scoring or performance evaluation G10H2210/091
• CPC Class Codes: G10H 2220/141 Games on or about music, i.e. based on musical knowledge, e.g. musical multimedia quizzes teaching of music per se G09B15/00
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

An information processing apparatus is provided which includes a metadata extraction unit for analyzing an audio signal in which a plurality of instrument sounds are present in a mixed manner and for extracting, as a feature quantity of the audio signal, metadata changing along with passing of a playing time, and a player parameter determination unit for determining, based on the metadata extracted by the metadata extraction unit, a player parameter for controlling a movement of a player object corresponding to each instrument sound.

Claims

1. An information processing apparatus, comprising:

a signal determining unit configured to determine signal data, wherein the signal data comprises a plurality of signals present in a mixed manner;

a log spectrum analysis unit configured to output a log spectrum of the signal data, wherein the log spectrum represents an intensity distribution of the signal data for each pitch;

a metadata extracting unit configured to analyze the signal data and extract, as a feature quantity of the signal data, metadata that changes along with passage of time of play of a music based on the log spectrum;

a determining unit configured to determine one or more object parameters and one or more audience parameters based on the extracted metadata; and

a switching unit configured to switch between one or more objects based on the one or more object parameters in a time sequence,

wherein the one or more audience parameters control movement of audience objects placed in audience seats provided in a location different from a stage, wherein the one or more audience parameters comprise at least a music structure extracted from the metadata to determine the movement of the audience objects based on a type of a structure of the music being played.

2. The information processing apparatus according to claim 1, wherein the one or more object parameters are associated with switch between display of the one or more objects and non-display of the one or more objects, wherein the switch is performed based on a calculated presence probability of a signal of the plurality of signals.

3. The information processing apparatus according to claim 1, wherein at least one of the one or more objects is an externally connected robot or an object image realized by computer graphics, and the information processing apparatus further comprises an object control unit configured to control a movement of the externally connected robot based on the one or more object parameters determined by the determining unit or to control a movement of the object image based on the one or more object parameters.

4. The information processing apparatus of claim 1, wherein the one or more audience parameters determine changes in synchronicity of the movement of the audience objects with a beat determined by the metadata.

5. The information processing apparatus of claim 1, wherein the metadata comprises one or more pieces of data, wherein the one or more pieces of data comprises a beat of the signal data, a chord progression, the music structure, a melody line, a bass line, a presence probability of each signal of the plurality of signals, a solo probability of each signal of the plurality of signals, or a voice feature of vocals.

23. The information processing apparatus of claim 1, wherein the music structure is determined based on the log spectrum of the signal data.

6. An information processing method, comprising:

analyzing signal data comprising a plurality of signals present in a mixed manner;

obtaining a log spectrum of the signal data, wherein the log spectrum represents an intensity distribution of the signal data for each pitch;

extracting, as a feature quantity of the signal data, metadata that changes along with passage of playing time of a music based on the log spectrum;

determining, based on the extracted metadata, one or more object parameters for controlling movement of one or more objects corresponding to the plurality of signals and one or more audience parameters for controlling movement of audience objects placed in audience seats provided in a location different from a stage; and

switching between the one or more objects based on the one or more object parameters in a time sequence,

wherein the one or more audience parameters comprise at least a music structure extracted from the metadata to determine the movement of the audience objects according to a type of a structure of the music being played.

7. The information processing method of claim 6, wherein the metadata comprises one or more pieces of data, wherein the one or more pieces of data comprises a beat of the signal data, a chord progression, the music structure, a melody line, a bass line, a presence probability of each signal of the plurality of signals, a solo probability of each signal of the plurality of signals, or a voice feature of vocals.

8. The information processing method of claim 7, wherein based on the solo probability of each signal of the plurality of signals, one of the one or more object parameters indicates a posture and an expression of an object of the one or more objects corresponding to the each signal.

9. The information processing method of claim 7, wherein based on the presence probability of each signal of the plurality of signals, one of the one or more object parameters indicates a moving extent of a playing hand of an object of the one or more objects corresponding to the each signal.

10. The information processing method of claim 7, wherein based on a presence probability of the vocals, one of the one or more object parameters indicates a size of an open mouth of an object of the one or more objects corresponding to the vocals or a distance between a hand holding a microphone and the mouth.

11. The information processing method of claim 10, wherein based on a difference between an average pitch of the melody line and a pitch of the melody line for each frame, or based on the voice feature of the vocals, one of the one or more object parameters indicates a movement of an expression of an object of the one or more objects corresponding to the vocals.

12. The information processing method of claim 7, wherein based on the melody line, one of the one or more object parameters indicates a movement of a hand other than a hand holding a microphone, the hand being of an object of the one or more objects corresponding to the vocals.

13. The information processing method of claim 7, wherein based on the chord progression, one of the one or more object parameters indicates a position of a hand of an object of the one or more objects, wherein the one or more object parameters correspond to one or more sections selected from a guitar, a keyboard or strings.

14. The information processing method of claim 7, wherein based on the bass line, one of the one or more object parameters indicates a position of a hand holding a neck, the hand being of an object of the one or more objects corresponding to a bass.

15. A non-transitory computer-readable medium having stored thereon a set of computer-executable instructions for causing a computer to perform an operation, comprising:

analyzing signal data comprising a plurality of signals present in a mixed manner;

obtaining a log spectrum of the signal data, wherein the log spectrum represents an intensity distribution of the signal data for each pitch;

extracting, as a feature quantity of signal data, metadata that changes along with passage of playing time of a music based on the log spectrum;

determining, based on the extracted metadata, one or more object parameters for controlling movement of one or more objects corresponding to the plurality of signals and one or more audience parameters for controlling movement of audience objects placed in audience seats provided in a location different from a stage; and

switching between the one or more objects based on the one or more object parameters in a time sequence,

wherein the one or more audience parameters comprise at least a music structure extracted from the metadata to determine the movement of the audience objects according to a type of a structure of the music being played.

16. The non-transitory computer-readable medium of claim 15, wherein the metadata comprises one or more pieces of data, wherein the one or more pieces of data comprises a genre of the music to which the signal data belongs, an age of the music to which the signal data belongs, information about the signal data relating to an object of the one or more objects, types of the plurality of signals included in the signal data, and tone of the music of the signal data.

17. The non-transitory computer-readable medium of claim 16, further comprising:

determining, in an event information on height and weight of the object is extracted as the information relating to the object, one of the one or more object parameters indicating a size of the object based on the information on the height and the weight, and

determining, in an event information on a gender of the object is extracted as the information relating to the object, one of the one or more object parameters indicating a hairstyle and clothing of the object based on the information on the gender.

18. The non-transitory computer-readable medium of claim 16, further comprising:

determining a lighting parameter, based on the extracted metadata, for controlling lighting on the stage on which the object is placed,

wherein the lighting parameter is determined such that the lighting on the stage changes synchronously with a beat detected by the extracted metadata.

19. The non-transitory computer-readable medium of claim 18, wherein the lighting parameter, based on a presence probability of each signal of the plurality of signals, indicates a brightness of a spotlight shining on the object corresponding to each signal.

20. The non-transitory computer-readable medium of claim 19, wherein the lighting parameter comprises at least the music structure extracted from the metadata, wherein the lighting parameter determines the lighting changes according to the type of the structure of the music being played.

21. The non-transitory computer-readable medium of claim 20, wherein the lighting parameter determines a color of the lighting changes based on the age of the music extracted from the metadata.

22. The non-transitory computer-readable medium of claim 15, wherein the metadata comprises one or more pieces of data, wherein the one or more pieces of data comprises a beat of the signal data, a chord progression, the music structure, a melody line, a bass line, a presence probability of each signal of the plurality of signals, a solo probability of each signal of the plurality of signals, and a voice feature of vocals.