IMAGE PROCESSING SYSTEM

• US 20100134634A1
• Filed: 09/11/2009
• Published: 06/03/2010
• Est. Priority Date: 11/28/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: WITT, Sarah Elizabeth
• Granted Patent: US 8,264,546 B2
• Time in Patent Office: Days
• US Class Current: 348/181
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

An image processing system includes a camera, an image processor and a calibration surface. The calibration surface includes thereon a calibration pattern, the calibration pattern comprising a plurality of alternately coloured elements which provide a corresponding plurality of corners at locations on the calibration pattern at which more than two of the coloured elements adjoin. The corner locations define a first group of lines and a second group of lines, the lines in each of the group of lines are parallel with respect to each other and with respect to the calibration surface. The first group of lines is orthogonal to the second group of lines and the camera is operable to generate and communicate to the image processor a video signal representative of a scene as viewed by the camera, the scene including the calibration surface. The image processor is operable to detect from the video signal at least some of the corner locations and thereby identify the first and second group of lines of the calibration pattern, to extrapolate the lines from each group of lines to determine a presence and a location of a first intersection point on a plane of the scene as viewed by the camera at which the extrapolated lines from the first group intersect and a second intersection point on the plane of scene as viewed by the camera at which the extrapolated lines from the second group intersect, and to estimate from the presence and the location of the first and second intersection points one or more of an estimated roll angle value, an estimated pitch angle value and an estimated yaw angle value corresponding to a roll angle, pitch angle and yaw angle of the camera relative to the calibration surface. The detection of the corner locations by the image processor comprises identifying from the video signal a plurality of potential corner locations, defining an area of the scene as viewed by the camera centred around each potential corner location, analysing a colour of points lying on a periphery of each area, thereby estimating a likelihood that a corner is located at each potential corner location.

Claims

1. An image processing system comprising a camera, an image processor and a calibration surface, the calibration surface including thereon a calibration pattern, the calibration pattern comprising a plurality of alternately coloured elements which provide a corresponding plurality of corners at locations on the calibration pattern at which more than two of the coloured elements adjoin, the corner locations defining a first group of lines and a second group of lines, the lines in each of the group of lines being parallel with respect to each other on calibration surface, the first group of lines being orthogonal to the second group of lines, the camera being operable to generate a video signal representative of a scene as viewed by the camera, the scene including the calibration surface, the image processor being operable

to detect from the video signal at least some of the corner locations to identify the first and second group of lines of the calibration pattern,

to extrapolate the lines from each group of lines to determine a presence and a location of a first intersection point on a plane of the scene as viewed by the camera at which the extrapolated lines from the first group intersect and a second intersection point on the plane of scene as viewed by the camera at which the extrapolated lines from the second group intersect, and

to estimate from the presence and the location of the first and second intersection points one or more of an estimated roll angle value, an estimated pitch angle value and an estimated yaw angle value corresponding to a roll angle, pitch angle and yaw angle of the camera relative to the calibration surface, wherein

the detection of the corner locations by the image processor comprises identifying from the video signal a plurality of potential corner locations, defining an area of the scene as viewed by the camera centred around each potential corner location, analysing a colour of points lying on a periphery of each area, to estimate a likelihood that a corner is located at each potential corner location.

2. An image processing system according to claim 1, wherein each point on the video signal is identified as a potential corner location.

3. An image processing system according to claim 1, wherein the analysing of the colour of the points lying on the periphery of the area comprises dividing the periphery into a first part and a second part, comparing a distribution of colour of the points on the first half with a distribution of colour of the points on the second half and if the distribution of colour in the first half and second half are substantially similar, identifying an estimated corner location at the potential corner location.

4. An image processing system according to claim 3, wherein the detection of the corner locations includes refining the estimated corner location by

defining a further area around the potential corner location,

determining edge points around the periphery of the further area at which two of the alternately coloured elements meet, and

identifying a refined corner location at a point in the centre of the area at which lines plotted between the edge points intersect.

5. An image processing system according to claim 1, wherein the corner locations are a predetermined distance apart on the calibration surface, the image processor being operable to determine an estimated centre point of the calibration surface from a distribution of the corner locations over the calibration pattern and to determine an estimated X displacement value, an estimated Y displacement value and an estimated Z displacement, the estimated X, Y and Z displacement values corresponding to a position in three dimensional space of a position of the camera relative to the calibration surface.

6. An image processing system according to claim 5, wherein the image processor is operable to detect from the video signal representative of the scene at least one of a plurality of markers, each marker being uniquely identifiable by the image processor and occupying a predetermined position on the calibration surface, the image processor being operable to refine at least one of the estimate of the centre point and the estimate of the yaw angle based on the at least one detected marker.

7. An image processing system according to claim 1, wherein the calibration pattern is a checkerboard pattern comprising alternately coloured squares.

8. An image processing system according to claim 1, wherein if the extrapolated lines from one of the first or second groups of lines do not all intersect at a point, the image processing apparatus is operable to determine that one of either the roll angle is zero with respect to the calibration surface, or the pitch angle of the camera is normal with respect to the calibration surface.

9. An image processing system according to claim 1, wherein if the extrapolated lines from the first group of lines do not all intersect at a point and the extrapolated lines from the second group of lines do not all intersect at a point, the image processing apparatus is operable to determine that the roll angle is zero and the pitch angle of the camera is normal with respect to the calibration surface.

10. An image processing system according to claim 1, wherein the image processor is operable to generate a composite image comprising image data derived from the video signal combined with a rendering of a computer generated object, the computer generated object rendered in accordance with the estimated roll angle value, pitch angle value and yaw angle value.

11. An image processing system according to claim 1, wherein the video signal comprises a plurality of video frames and the image processor is operable to estimate one or more of the estimated roll angle value, the estimated pitch angle value and the estimated yaw angle value for individual video frames of the video signal.

12. An image processor operable to receive a video signal from a camera representative of a scene as viewed by the camera, the scene including a calibration surface including thereon a calibration pattern, the calibration pattern comprising a plurality of alternately coloured elements which provide a corresponding plurality of corners at locations on the calibration pattern at which more than two of the coloured elements adjoin, the corner locations defining a first group of lines and a second group of lines, the lines in each of the group of lines being parallel with respect to each other and on the calibration surface, the first group of lines being orthogonal to the second group of lines, the image processor being operable

to detect from the video signal at least some of the corner locations to identify the first and second group of lines of the calibration pattern,

to extrapolate the lines from each group of lines to determine a presence and a location of a first intersection point on a plane of the scene as viewed by the camera at which the extrapolated lines from the first group intersect and a second intersection point on the plane of scene as viewed by the camera at which the extrapolated lines from the second group intersect, and

to estimate from the presence and the location of the first and second intersection points one or more of an estimated roll angle value, an estimated pitch angle value and an estimated yaw angle value corresponding to a roll angle, pitch angle and yaw angle of the camera relative to the calibration surface, wherein the detection of the corner locations by the image processor comprises identifying from the video signal a plurality of potential corner locations, defining an area of the scene as viewed by the camera centred around each potential corner location, analysing a colour of points lying on a periphery of each area to estimate a likelihood that a corner is located at each potential corner location.

13. A method of estimating camera parameters from a captured image, the method comprising

generating a video signal representative of a scene as viewed by a camera, the scene including a calibration surface including thereon a calibration pattern, the calibration pattern comprising a plurality of alternately coloured elements which provide a corresponding plurality of corners at locations on the calibration pattern at which more than two of the coloured elements adjoin, the corner locations defining a first group of lines and a second group of lines, the lines in each of the group of lines being parallel with respect to each other and with respect to the calibration surface, the first group of lines being orthogonal to the second group of lines,

detecting from the video signal at least some of the corner locations and to identify the first and second group of lines of the calibration pattern,

extrapolating the lines from each group of lines,

determining from the extrapolated lines a presence and a location of a first intersection point on a plane of the scene as viewed by the camera at which the extrapolated lines from the first group intersect and a second intersection point on the plane of the scene as viewed by the camera at which the extrapolated lines from the second group intersect,

estimating from the presence and the location of the first and second intersection points one or more of an estimated roll angle value, an estimated pitch angle value and an estimated yaw angle value corresponding to a roll angle, pitch angle and yaw angle of the camera relative to the calibration surface, wherein the detecting the corner locations, comprises

identifying from the video signal a plurality of potential corner locations,

defining an area of the scene as viewed by the camera centred around each potential corner location, and

analysing a colour of points lying on a periphery of each area to estimate a likelihood that a corner is located at each potential corner location.

14. A computer graphics generation system, comprising

a camera for generating video images of a scene captured by the camera, the scene including a calibration surface including thereon a calibration pattern, the calibration pattern comprising a plurality of alternately coloured elements which provide a corresponding plurality of corners at locations on the calibration pattern at which more than two of the coloured elements adjoin, the corner locations defining a first group of lines and a second group of lines, the lines in each of the group of lines being parallel with respect to each other and on the calibration surface, the first group of lines being orthogonal to the second group of lines, and

an image processor arranged in operation to receive the video signal from the camera, and to combine the video images with rendered computer generated objects, the computer generated objects being rendered in accordance with a position and orientation of the camera determined by

detecting from the video signal at least some of the corner locations to identify the first and second group of lines of the calibration pattern,

extrapolating the lines from each group of lines to determine a presence and a location of a first intersection point on a plane of the scene as viewed by the camera at which the extrapolated lines from the first group intersect and a second intersection point on the plane of scene as viewed by the camera at which the extrapolated lines from the second group intersect, and

estimating from the presence and the location of the first and second intersection points one or more of an estimated roll angle value, an estimated pitch angle value and an estimated yaw angle value corresponding to a roll angle, pitch angle and yaw angle of the camera relative to the calibration surface, wherein

the detection of the corner locations by the image processor comprises identifying from the video signal a plurality of potential corner locations, defining an area of the scene as viewed by the camera centred around each potential corner location, analysing a colour of points lying on a periphery of each area to estimate a likelihood that a corner is located at each potential corner location.

15. A computer program product having computer executable instructions stored on a storage medium, which when loaded on to a computer causes the computer to perform a method of estimating camera parameters from a captured image, the method comprising

generating a video signal representative of a scene as viewed by a camera, the scene including a calibration surface including thereon a calibration pattern, the calibration pattern comprising a plurality of alternately coloured elements which provide a corresponding plurality of corners at locations on the calibration pattern at which more than two of the coloured elements adjoin, the corner locations defining a first group of lines and a second group of lines, the lines in each of the group of lines being parallel with respect to each other and with respect to the calibration surface, the first group of lines being orthogonal to the second group of lines,

detecting from the video signal at least some of the corner locations and to identify the first and second group of lines of the calibration pattern,

extrapolating the lines from each group of lines,

determining from the extrapolated lines a presence and a location of a first intersection point on a plane of the scene as viewed by the camera at which the extrapolated lines from the first group intersect and a second intersection point on the plane of the scene as viewed by the camera at which the extrapolated lines from the second group intersect,

estimating from the presence and the location of the first and second intersection points one or more of an estimated roll angle value, an estimated pitch angle value and an estimated yaw angle value corresponding to a roll angle, pitch angle and yaw angle of the camera relative to the calibration surface, wherein the detecting the corner locations comprises

identifying from the video signal a plurality of potential corner locations,

defining an area of the scene as viewed by the camera centred around each potential corner location, and

analysing a colour of points lying on a periphery of each area to estimate a likelihood that a corner is located at each potential corner location.

16. An apparatus for estimating camera parameters from a captured image, the apparatus comprising

means for generating a video signal representative of a scene as viewed by a camera, the scene including a calibration surface including thereon a calibration pattern, the calibration pattern comprising a plurality of alternately coloured elements which provide a corresponding plurality of corners at locations on the calibration pattern at which more than two of the coloured elements adjoin, the corner locations defining a first group of lines and a second group of lines, the lines in each of the group of lines being parallel with respect to each other and with respect to the calibration surface, the first group of lines being orthogonal to the second group of lines,

means for detecting from the video signal at least some of the corner locations and to identify the first and second group of lines of the calibration pattern,

means for extrapolating the lines from each group of lines,

means for determining from the extrapolated lines a presence and a location of a first intersection point on a plane of the scene as viewed by the camera at which the extrapolated lines from the first group intersect and a second intersection point on the plane of the scene as viewed by the camera at which the extrapolated lines from the second group intersect,

means for estimating from the presence and the location of the first and second intersection points one or more of an estimated roll angle value, an estimated pitch angle value and an estimated yaw angle value corresponding to a roll angle, pitch angle and yaw angle of the camera relative to the calibration surface, wherein the means for detecting the corner locations comprises

means for identifying from the video signal a plurality of potential corner locations,

means for defining an area of the scene as viewed by the camera centred around each potential corner location, and

means for analysing a colour of points lying on a periphery of each area to estimate a likelihood that a corner is located at each potential corner location.