PICTURE PROCESSING APPARATUS, PROCESSING METHOD FOR USE THEREWITH, AND PROGRAM

• US 20100111499A1
• Filed: 12/17/2008
• Published: 05/06/2010
• Est. Priority Date: 01/21/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Tsurumi, Shingo
• Granted Patent: US 8,599,320 B2
• Time in Patent Office: Days
• US Class Current: 386/224
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/607 for more than one sound signal, e.g. stereo, multilanguages H04N5/602 takes precedence
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

A picture conversion information supply section 130 calculates per frame an affine transformation parameter for picture conversion based on motion information about a moving picture. With reference to a reference picture, a picture conversion section 140 affine-transforms pictures making up the moving picture per frame using the calculated affine transformation parameters. Based on information indicating the center position, angle or scaling factor about the transformed pictures coming from the picture conversion section 140, a sound conversion information calculation section 190 calculates sound conversion information for converting the sound corresponding to the pictures. Based on the sound conversion information, a sound conversion processing section 200 controls the volume of each of the channels making up the sound, adds up the controlled sound of each channel, and outputs the result as output sound to speakers 220.

Claims

1. A picture processing apparatus comprising:

content acquisition means for acquiring content data including a moving picture and sound corresponding to the moving picture;

picture conversion information supply means which, based on a first picture and a second picture making up said moving picture, supplies picture conversion information about said second picture relative to said first picture;

picture conversion means for converting said second picture based on said picture conversion information with reference to said first picture;

picture synthesis means for synthesizing the converted second picture and a background picture serving as a background of said second picture into a synthesized picture;

display control means for causing display means to display said synthesized picture;

sound conversion information calculation means for calculating sound conversion information about the sound regarding said second picture based on said picture conversion information;

sound conversion processing means for generating output sound by performing a conversion process on said sound based on said sound conversion information; and

sound output control means for causing sound output means to output said output sound.

2. The picture processing apparatus according to claim 1, wherein said picture conversion information includes elements regarding a movement of said second picture relative to said first picture.

3. The picture processing apparatus according to claim 1, wherein said picture conversion information includes elements regarding a rotation of said second picture relative to said first picture.

4. The picture processing apparatus according to claim 1, wherein said picture conversion information includes elements regarding a scaling factor of said second picture relative to said first picture.

5. The picture processing apparatus according to claim 1, wherein

said sound conversion processing means includes volume control means and sound addition means;

said volume control means controls the volumes of a plurality of channels making up said sound based on said sound conversion information; and

said sound addition means adds up the controlled sound for each of the channels.

6. The picture processing apparatus according to claim 1, wherein said sound conversion processing means generates sound on a right channel and a left channel making up said output sound through said conversion process.

7. The picture processing apparatus according to claim 1, wherein said sound conversion processing means generates sound on a center channel making up said output sound through said conversion process.

8. The picture processing apparatus according to claim 1, wherein

said sound includes the sound on a right channel and a left channel;

and

said sound conversion processing means generates said output sound by performing said conversion process on the sound on said right channel and said left channel.

9. The picture processing apparatus according to claim 1, wherein

said sound includes the sound on a center channel;

and

said sound conversion processing means generates said output sound by performing said conversion process on the sound on said center channel.

10. The picture processing apparatus according to claim 1, further comprising

picture holding means for holding pictures including said first picture as historical pictures;

wherein

said first picture is positioned chronologically before said second picture in a timeline of said moving picture;

said picture conversion means converts at least either said second picture or said historical pictures held in said picture holding means based on said picture conversion information; and

said picture synthesis means synthesizes said second picture and said historical pictures at least either of which have been converted by said picture conversion means, into said synthesized picture and causes said picture holding means to hold said synthesized picture as a new historical picture.

11. The picture processing apparatus according to claim 10, further comprising

display area extraction means for determining a display area targeted for display on said display means from said new historical picture held in said picture holding means, before extracting the picture included in said display area as a display picture;

wherein

said picture synthesis means overwrites said display picture with the converted second picture to obtain a new display picture by synthesis;

said display control means causes said display means to display said new display picture;

said display area extraction means generates display area extraction information about the position, angle, or size of said display area in a holding area of said picture holding means; and

said sound conversion information calculation means calculates said sound conversion information based on said picture conversion information and said display area extraction information.

12. The picture conversion apparatus according to claim 1, wherein said picture conversion means converts said second picture based on template information indicating a display area in which to display said moving picture on said display means.

13. A sound conversion processing method for use with a picture processing apparatus having a sound output means for outputting sound corresponding to a moving picture, said sound conversion processing method comprising:

a content acquisition step of acquiring content data including said moving picture and said sound;

a picture conversion information supply step which, based on a first picture and a second picture making up said moving picture, supplies picture conversion information about said second picture relative to said first picture;

a picture conversion step of converting said second picture based on said picture conversion information with reference to said first picture;

a sound conversion information calculation step of calculating sound conversion information about said sound based on said picture conversion information;

a sound conversion processing step of generating output sound by performing a conversion process on said sound based on said sound conversion information; and

a sound output control step of causing said sound output means to output said output sound.

14. A program for causing a computer to execute steps on a picture processing apparatus having a sound output means for outputting sound corresponding to a moving picture, said steps comprising:

a content acquisition step of acquiring content data including said moving picture and said sound;

a picture conversion information supply step which, based on a first picture and a second picture making up said moving picture, supplies picture conversion information about said second picture relative to said first picture;

a picture conversion step of converting said second picture based on said picture conversion information with reference to said first picture;

a sound conversion information calculation step of calculating sound conversion information about said sound based on said picture conversion information;

a sound conversion processing step of generating output sound by performing a conversion process on said sound based on said sound conversion information; and

a sound output control step of causing said sound output means to output said output sound.