METHOD AND APPARATUS FOR GENERATING AN IMAGE CUT-OUT

• US 20140300644A1
• Filed: 02/11/2014
• Published: 10/09/2014
• Est. Priority Date: 04/04/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: GILLARD, Clive Henry, PORTER, Robert Mark Stefan, KEATING, Stephen Mark
• Granted Patent: US 9,706,135 B2
• Time in Patent Office: Days
• US Class Current: 345/660
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A method of generating a cut-out from an image of a scene which has been captured by a camera is described. The method comprises: defining the position of a virtual camera, the image plane of the virtual camera being the cut out, with respect to the position of the camera capturing the scene;

defining a set of pixel positions for the cut out of the captured image; generating a virtual camera rotation matrix to be applied to the defined set of pixel positions, the virtual camera rotation matrix expressing a combination of at least one of the yaw, pitch and roll of the virtual camera; applying the virtual camera rotation matrix to the defined set of pixel positions to determine a corresponding transformed pixel position for each of the defined pixel positions, the transformed pixel positions defining a shape and position of a portion of the captured image of the scene; mapping each of the transformed pixel positions in the set of transformed pixel positions to an associated pixel in the portion of the image of the scene; and establishing a value for each one of pixels of the image cut out in the defined pixel positions corresponding to the transformed pixel positions using the value of the corresponding associated pixel in the portion of the image of the scene which has been determined.

Claims

1. A method of generating a cut-out from an image of a scene which has been captured by a camera, the method comprising:

defining the position of a virtual camera, the image plane of the virtual camera being the cut out, with respect to the position of the camera capturing the scene;

defining a set of pixel positions for the cut out of the captured image;

generating a virtual camera rotation matrix to be applied to the defined set of pixel positions, the virtual camera rotation matrix expressing a combination of at least one of the yaw, pitch and roll of the virtual camera;

applying the virtual camera rotation matrix to the defined set of pixel positions to determine a corresponding transformed pixel position for each of the defined pixel positions, the transformed pixel positions defining a shape and position of a portion of the captured image of the scene;

mapping each of the transformed pixel positions in the set of transformed pixel positions to an associated pixel in the portion of the image of the scene; and

establishing a value for each one of pixels of the image cut out in the defined pixel positions corresponding to the transformed pixel positions using the value of the corresponding associated pixel in the portion of the image of the scene which has been determined.

2. The method according to claim 1, wherein the virtual camera rotation matrix is calculated in accordance with a product of one or more of matrices associated with one or more of the pitch, yaw and roll of the virtual camera.

3. The method according to claim 2, wherein the virtual camera rotation matrix, V, is given by:

V=P_V*Y_V*R_V

4. The method according to claim 2, wherein the yaw θ

_V and pitch φ

_V of the virtual camera is controlled by a user operated controller.

5. The method according to claim 2, wherein the yaw θ

_V and pitch φ

_V of the virtual camera is controlled by an object tracking algorithm configured to track an object in the captured image of the scene.

6. The method according to claim 1, wherein the roll ρ

_V of the virtual camera is calculated in accordance with the angle of pitch of the camera capturing the scene about a predetermined pitch axis and the angle of roll of the camera capturing the scene about a predetermined camera roll axis.

7. The method according to claim 6, wherein the angle of the pitch and/or roll of the camera is determined from a rig upon which the camera capturing the scene is mounted.

8. A method according to claim 1, further comprising:

generating the captured image of the scene as a stitched image captured by a first camera in a first position and a second camera in a second position, the second position being different from the first position.

9. A method according to claim 1, wherein the cut-out is corrected for barrel and/or pin cushion distortion in the captured image using lens metadata associated with the camera capturing the image.

10. A method according to claim 1, wherein:

the captured image has been captured using a fish-eye lens and is an image comprising fish-eye distortion; and

the transformed pixel positions defining the shape and position of the portion of the captured image of the scene are further transformed so that the shape of the portion of the captured image more closely matches a shape of an object in the fish-eye distorted captured image.

20. A non-transitory computer readable medium including computer program instructions, which when executed by a computer causes the computer to perform the method of claim 1.

11. A device for generating a cut-out from an image of a scene which has been captured by a camera, the device comprising circuitry configured to:

define the position of a virtual camera, the image plane of the virtual camera being the cut out, with respect to the position of the camera capturing the scene;

define a set of pixel positions for the cut out of the captured image;

generate a virtual camera rotation matrix to be applied to the defined set of pixel positions, the virtual camera rotation matrix expressing a combination of at least one of the yaw, pitch and roll of the virtual camera;

apply the virtual camera rotation matrix to the defined set of pixel positions to determine a corresponding transformed pixel position for each of the defined pixel positions, the transformed pixel positions defining a shape and position of a portion of the captured image of the scene;

map each of the transformed pixel positions in the set of transformed pixel positions to an associated pixel in the portion of the image of the scene; and

establish a value for each one of pixels of the image cut out in the defined pixel positions corresponding to the transformed pixel positions using the value of the corresponding associated pixel in the portion of the image of the scene which has been determined.

12. The device according to claim 11, wherein the virtual camera rotation matrix is calculated in accordance with a product of one or more of matrices associated with one or more of the pitch, yaw and roll of the virtual camera.

13. The device according to claim 12, wherein the virtual camera rotation matrix, V, is given by:

V=P_V*Y_V*R_V

14. The device according to claim 12, wherein the yaw θ

_V and pitch φ

_V of the virtual camera is controlled by a user operated controller.

15. The device according to claim 12, wherein the yaw θ

_V and pitch φ

_V of the virtual camera is controlled by an object tracking algorithm configured to track an object in the captured image of the scene.

16. The device according to claim 11, wherein the roll ρ

_V of the virtual camera is calculated in accordance with the angle of pitch of the camera capturing the scene about a predetermined pitch axis and the angle of roll of the camera capturing the scene about a predetermined camera roll axis.

17. The device according to claim 16, wherein the angle of the pitch and/or roll of the camera is determined from a rig upon which the camera capturing the scene is mounted.

18. The device according to claim 16, wherein the roll ρ

_V of the virtual camera is calculated using the equation;

ρ

_V=sin<sup>−

1</sup>(sin(θ

_V)*sin(φ

_V+φ

_rig))+ρ

_rig

wherein φ

_rig defines an angle of pitch of the camera capturing the scene about a predetermined camera pitch axis and ρ

_rig defines an angle of roll of the camera capturing the scene about a predetermined camera roll axis, the predetermined first camera pitch and roll axes defining a horizontal plane in the scene.

19. A device according to claim 11, wherein:

the captured image has been captured using a fish-eye lens and is an image comprising fish-eye distortion; and

the transformed pixel positions defining the shape and position of the portion of the captured image of the scene are further transformed so that the shape of the portion of the captured image more closely matches a shape of an object in the fish-eye distorted captured image.