Picture processing apparatus, processing method for use therewith, and program

• US 8,599,320 B2
• Filed: 12/17/2008
• Issued: 12/03/2013
• Est. Priority Date: 01/21/2008
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Tsurumi, Shingo
• Primary Examiner(s): Harod, Jefferey
• Assistant Examiner(s): HAIEM, SEAN N
• Application Number:
  US12/530,477
• Publication Number:
  US 20100111499A1
• Time in Patent Office: 1,812 Days
• Field of Search: None
• US Class Current: 348/738
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/607 for more than one sound signal, e.g. stereo, multilanguages H04N5/602 takes precedence
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

A picture conversion information supply section 130 calculates per frame an affine transformation parameter for picture conversion based on motion information about a moving picture. With reference to a reference picture, a picture conversion section 140 affine-transforms pictures making up the moving picture per frame using the calculated affine transformation parameters. Based on information indicating the center position, angle or scaling factor about the transformed pictures coming from the picture conversion section 140, a sound conversion information calculation section 190 calculates sound conversion information for converting the sound corresponding to the pictures. Based on the sound conversion information, a sound conversion processing section 200 controls the volume of each of the channels making up the sound, adds up the controlled sound of each channel, and outputs the result as output sound to speakers 220.

Claims

1. A picture processing apparatus comprising:

a content acquisition section to acquire content data including a moving picture and sound corresponding to the moving picture;

a picture conversion information supply section to supply, based on a picture of a first frame and a picture of a second frame making up said moving picture, picture conversion information about said second picture relative to said first picture, the picture conversion information including information on an angle of the picture in the second frame relative to the picture in the first frame, a center of the picture in the second frame relative to a center of the picture in the first frame, and a size of the picture in the second frame relative to the picture in the first frame;

a picture conversion section to convert said picture in the second frame based on said picture conversion information with reference to said picture in the first frame;

a picture synthesis section to synthesize the converted picture of the second frame and a background picture serving as a background of said picture in the second frame into a synthesized picture;

a display control to cause a display to display said synthesized picture;

a sound conversion information calculation section to calculate sound conversion information about the sound regarding said picture in the second frame based on said picture conversion information;

a sound conversion processing section to generate output sound by performing a conversion process on said sound based on said sound conversion information; and

a sound output control section to cause a sound output section to output said output sound.

2. The picture processing apparatus according to claim 1, wherein

said sound conversion processing section includes a volume control section and a sound addition section;

said volume control section controls volumes of a plurality of channels making up said sound based on said sound conversion information; and

said sound addition section adds up the controlled sound for each of the channels.

3. The picture processing apparatus according to claim 1, wherein said sound conversion processing section generates sound on a right channel and a left channel making up said output sound through said conversion process.

4. The picture processing apparatus according to claim 1, wherein said sound conversion processing section generates sound on a center channel making up said output sound through said conversion process.

5. The picture processing apparatus according to claim 1, wherein

said sound includes the sound on a right channel and a left channel;

and

said sound conversion processing section generates said output sound by performing said conversion process on the sound on said right channel and said left channel.

6. The picture processing apparatus according to claim 1, wherein

said sound includes the sound on a center channel;

and

said sound conversion processing section generates said output sound by performing said conversion process on the sound on said center channel.

7. The picture processing apparatus according to claim 1, further comprising

a picture holding section to hold pictures including said picture in the first frame as historical pictures;

wherein

said picture in the first frame is positioned chronologically before said picture in the second frame in a timeline of said moving picture;

said picture conversion section converts at least either said picture in the first frame or said historical pictures held in said picture holding section based on said picture conversion information; and

said picture synthesis section synthesizes said picture in the second frame and said historical pictures at least either of which have been converted by said picture conversion section, into said synthesized picture and causes said picture holding section to hold said synthesized picture as a new historical picture.

8. The picture processing apparatus according to claim 7, further comprising:

a display area extraction section to extract a display area targeted for display on said display from said new historical picture held in said picture holding section, before extracting the picture included in said display area as a display picture;

wherein

said picture synthesis section overwrites said display picture with the converted second picture to obtain a new display picture by synthesis;

said display control section causes said display to display said new display picture;

said display area extraction section generates display area extraction information about the position, angle, or size of said display area in a holding area of said picture holding section; and

said sound conversion information calculation section calculates said sound conversion information based on said picture conversion information and said display area extraction information.

9. The picture conversion apparatus according to claim 1, wherein said picture conversion section converts said picture in the second frame based on template information indicating a display area in which to display said moving picture on said display means.

12. The picture processing apparatus according to claim 1, wherein the picture conversion information supply section determines the picture conversion information based on an affine transformation of the pictures in the first and second frames.

13. The picture processing apparatus according to claim 1, wherein the sound output section includes a plurality of speakers, and the sound output control section controls an output level of each of the plurality of speakers.

10. A sound conversion processing method for use with a picture processing apparatus having a sound output section to output sound corresponding to a moving picture, said sound conversion processing method comprising:

acquiring content data including said moving picture and said sound;

supplying, based on a picture in a first frame and a picture in a second frame making up said moving picture, picture conversion information about said picture in the second frame relative to said picture in the first frame, the picture conversion information including information on an angle of the picture in the second frame relative to the picture in the first frame, a center of the picture in the second frame relative to a center of the picture in the first frame, and a size of the picture in the second frame relative to the picture in the first frame;

converting said picture in the second frame based on said picture conversion information with reference to said picture in the first frame;

calculating sound conversion information about said sound based on said picture conversion information;

generating output sound by performing a conversion process on said sound based on said sound conversion information; and

causing said sound output section to output said output sound.

11. A non-transitory computer-readable medium encoded with computer-readable instructions that, when executed by a computer, cause the computer to execute a method a picture processing apparatus having a sound output section to output sound corresponding to a moving picture, said method comprising:

acquiring content data including said moving picture and said sound;

supplying, based on a picture in a first frame and a picture in a second frame making up said moving picture, picture conversion information about said picture in the second frame relative to said picture in the first frame, the picture conversion information including information on an angle of the picture in the second frame relative to the picture in the first frame, a center of the picture in the second frame relative to a center of the picture in the first frame, and a size of the picture in the second frame relative to the picture in the first frame;

converting said picture in the second frame based on said picture conversion information with reference to said first picture;

calculating sound conversion information about said sound based on said picture conversion information;

generating output sound by performing a conversion process on said sound based on said sound conversion information; and

causing said sound output means to output said output sound.

14. A picture processing apparatus for processing a moving picture including a first picture and a second picture which is positioned chronologically after said first picture in a timeline of said moving picture, said picture processing apparatus comprising:

a picture conversion section configured to convert said second picture based on picture conversion information with reference to said first picture, said picture conversion information including at least one element regarding a movement of said second picture relative to said first picture;

a picture synthesis section configured to synthesize the converted second picture to a synthesized picture which has been output by the picture synthesis section previously based on said picture conversion information; and

a sound conversion information calculation section configured to calculate right channel sound conversion information and left channel sound conversion information based on display area extraction information regarding a display area which is extracted from a whole area of the synthesized picture and said picture conversion information,

wherein the right channel sound conversion information differs from the left channel sound conversion information according to the movement of the second picture relative to the first picture.

15. The picture processing apparatus according to claim 14, wherein the display area extraction information includes rotation of the second picture relative to the synthesized picture.

16. The picture processing apparatus according to claim 14, wherein the display area extraction information includes scaling of the second picture relative to the synthesized picture.

17. The picture processing apparatus according to claim 14 wherein the display area extraction information includes the position, angle or size of the second picture relative to the synthesized picture.

18. A method of processing a moving picture including a first picture and a second picture which is positioned chronologically after said first picture in a timeline of said moving picture, the method comprising:

converting said second picture based on picture conversion information with reference to said first picture, said picture conversion information including at least one element regarding a movement of said second picture relative to said first picture;

synthesizing the converted second picture to a synthesized picture which has been output by the picture synthesis section previously based on said picture conversion information; and

calculating right channel sound conversion information and left channel sound conversion information based on display area extraction information regarding a display area which is extracted from a whole area of the synthesized picture and said picture conversion information,

wherein the right channel sound conversion information differs from the left channel sound conversion information according to the movement of the second picture relative to the first picture.

19. The method according to claim 18, wherein the display area extraction information includes rotation of the second picture relative to the synthesized picture.

20. The method according to claim 18, wherein the display area extraction information includes scaling of the second picture relative to the synthesized picture.

21. The method according to claim 18 wherein the display area extraction information includes the position, angle or size of the second picture relative to the synthesized picture.

22. A non-transitory computer-readable medium storing computer-readable instructions thereon, the computer-readable instructions when executed by a computer cause the computer to perform a method for processing a moving picture including a first picture and a second picture which is positioned chronologically after said first picture in a timeline of said moving picture, comprising:

converting said second picture based on picture conversion information with reference to said first picture, said picture conversion information including at least one element regarding a movement of said second picture relative to said first picture;

synthesizing the converted second picture to a synthesized picture which has been output by the picture synthesis section previously based on said picture conversion information; and

calculating right channel sound conversion information and left channel sound conversion information based on display area extraction information regarding a display area which is extracted from a whole area of the synthesized picture and said picture conversion information,

wherein the right channel sound conversion information differs from the left channel sound conversion information according to the movement of the second picture relative to the first picture.

23. The non-transitory computer-readable medium according to claim 22, wherein the display area extraction information includes rotation of the second picture relative to the synthesized picture.

24. The non-transitory computer-readable medium according to claim 22, wherein the display area extraction information includes scaling of the second picture relative to the synthesized picture.

25. The non-transitory computer-readable medium according to claim 22 wherein the display area extraction information includes the position, angle or size of the second picture relative to the synthesized picture.