Image processing device and method

• US 8,477,995 B2
• Filed: 12/08/2009
• Issued: 07/02/2013
• Est. Priority Date: 02/19/2009
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Porter, Robert Mark Stefan, Gillard, Clive Henry
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US12/633,126
• Publication Number:
  US 20100208942A1
• Time in Patent Office: 1,302 Days
• Field of Search: 382/100, 382/103
• US Class Current: 382/103
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/207 using a single 2D image sensor

Abstract

An image processing device comprises receiving means operable to receive, from a camera, a captured image corresponding to an image of a scene captured by the camera. The scene contains at least one object. The device comprises determining means operable to determine a distance between the object within the scene and a reference position defined with respect to the camera, and generating means operable to detect a position of the object within the captured image, and to generate a modified image from the captured image based on image features within the captured image which correspond to the object in the scene. The generating means is operable to generate the modified image by displacing the position of the captured object within the modified image with respect to the determined position of the object within the captured image by an object offset amount which is dependent on the distance between the reference position and the object in the scene so that, when the modified image and the captured image are viewed together as a pair of images on a display, the captured object appears to be positioned at a predetermined distance from the display.

Claims

1. An image processing device comprising:

a receiving unit configured to receive, from a camera, a captured image corresponding to an image of a scene captured by the camera, the scene containing at least one object;

a determining unit configured to determine a distance between the object within the scene and a reference position defined with respect to the camera; and

a generating unit configured to detect a position of the object within the captured image, and to generate a modified image from the captured image based on image features within the captured image which correspond to the object in the scene,

in which;

the generating unit is operable to generate the modified image by displacing the position of the captured object within the modified image with respect to the determined position of the object within the captured image by an object offset amount which is dependent on the distance between the reference position and the object in the scene so that, when the modified image and the captured image are viewed together as a pair of images on a display, the captured object appears to be positioned at a predetermined distance from the display,

the reference position lies on a reference line, the reference line being orthogonal to an optical axis of the camera and passing through a position of the camera, and

the reference position is located on the reference line at a point where an object location line and the reference line intersect, the object location line being orthogonal to the reference line and passing through the object.

2. A device according to claim 1, in which the determining unit is operable to determine the distance between the reference position and the object in the scene in dependence upon distance data, the distance data being indicative of the distance between the reference position and the object in the scene.

3. A device according to claim 2, in which the distance data comprises distance map data which indicates, for each pixel of the captured image, a respective distance between the reference position and a scene feature within the scene coinciding with that pixel.

4. A device according to claim 3, in which the generating unit is operable to detect object pixels within the captured image which correspond to the object within the scene.

5. A device according to claim 4, in which:

the object pixels together form a foreground image, and those pixels which are detected as not corresponding to the object within the scene are designated as background pixels which form a background image;

the generating unit is operable to displace the position of the object pixels with respect to the position of the object within the captured image by the object offset amount so as to generate a modified foreground image;

the determining unit is operable to determine the distance between the reference position and scene features in the scene which coincide with respective background pixels in dependence upon the distance data associated in the distance data map with those background pixels;

the generating unit is operable to generate the modified image by displacing the position of each background pixel with respect to the position of that background pixel in the background image by a respective background pixel offset amount which is dependent upon the distance between the reference position and the scene feature within the scene which corresponds to that background pixel so as to generate a modified background image, and to combine the modified background image with the modified foreground image so that, when the modified image and the captured image are viewed together as a pair of images on the display, the modified image and the captured image together appear as if three-dimensional.

6. A device according to claim 4, in which the determining unit is operable to determine the distance between the reference position and the object within the scene in dependence upon the distance data associated in the distance map data with those object pixels.

7. A device according to claim 4, in which the generating unit is operable to generate the modified image by displacing the position of each object pixel with respect to the position of that pixel in the captured image by a respective object pixel offset amount which is dependent upon the distance between the reference position and the scene feature within the scene which corresponds to that object pixel.

8. A device according to claim 2, in which a predetermined three-dimensional model of at least part of the scene is mapped to known feature points within the scene;

and

the determining unit is operable to determine the distance between the reference position and the object in the scene in dependence upon the detected position of the object in the scene with respect to the predetermined three-dimensional model.

9. A device according to claim 8, in which the determining unit is operable to detect predetermined image features within the captured image which correspond to positions of the known feature points within the scene so as to map the three-dimensional model to the known feature points.

10. A device according to claim 2, in which:

the determining unit is operable to detect a position of the object in the scene with respect to the camera, and to determine the distance between the object in the scene and the reference position in dependence upon the detected position of the object in the scene with respect to the camera.

11. A device according to claim 10, in which:

the receiving unit is operable to receive a sequence of images captured by the camera; and

the determining unit is operable to;

track the position of the captured object through the sequence of images in dependence upon the detected position of the object in the scene with respect to the camera so as to generate tracking data; and

determine the distance between the object in the scene and the reference position in dependence upon the tracking data.

12. A device according to claim 2, in which the determining unit is operable to generate the distance data by analysis of the captured image.

13. A device according to claim 2, comprising a distance data receiving unit configured to receive the distance data from a distance sensor, in which the determining unit is operable to determine the distance between the reference position and the object in the scene in dependence upon the distance data received from the distance sensor.

14. A device according to claim 1, in which:

the scene comprises a substantially planar surface;

the determining unit is operable to detect an intersection point at which the object within the scene coincides with the planar surface, and to determine the distance between the reference position and the intersection point; and

the object offset amount is dependent on the distance between the reference position and the intersection point.

15. A device according to claim 1, in which the generating unit is operable to detect a plurality of objects within the scene by analysis of the captured image;

the generating unit is operable to detect whether the plurality of objects within the scene are within a threshold distance of each other, those objects which are detected as being within the threshold distance of each other being designated as belonging to a group of objects within the scene;

the determining unit is operable to detect a respective distance between the reference position and each of the objects in the group of objects within the scene so as to generate an average distance value indicative of the average distance between the reference position and each of the of objects in the group of objects within the scene; and

if the plurality of objects in the scene are detected as being within the threshold distance of each other, the generating unit is operable to generate the modified image by displacing, with respect to detected positions of the plurality of objects within the scene, each captured object in the group of objects by a group offset amount, the group offset amount being dependent on the average distance value.

16. A device according to claim 1, in which the reference position is located at a position of the camera.

17. A device according to claim 1, in which the reference line is parallel to a scene feature within the scene.

18. A device according to claim 1, in which:

the receiving unit is operable to receive a plurality of captured images from different respective cameras having optical axes which intersect with each other, each captured image corresponding to a different image of the scene; and

the determining unit is operable to determine the distance between the object within the scene and the reference position by analysis of the captured images received from the cameras.

19. A device according to claim 1, in which the scene comprises a sports stadium having a sports pitch, and the object within the scene is a player on the sports pitch or a sports object used in that sport.

20. An image processing system comprising:

one or more cameras configured to capture an image of a scene so as to generate a captured image; and

an image processing device according to claim 1.

21. A system according to claim 20, comprising a display operable to display the modified image together with the captured image as a pair of images on the display.

22. An image processing method comprising:

receiving, from a camera, a captured image corresponding to an image of a scene captured by the camera, the scene comprising at least one object;

determining a distance between the object within the scene and a reference position defined with respect to the camera;

detecting a position of the object within the captured image; and

generating a modified image from the captured image based on image features within the captured image which correspond to the object in the scene by displacing the position of the captured object within the modified image with respect to the determined position of the object within the captured image by an object offset amount which is dependent on the distance between the reference position and the object in the scene so that, when the modified image and the captured image are viewed together as a pair of images on a display, the captured object appears to be positioned at a predetermined distance from the display, wherein

the reference position lies on a reference line, the reference line being orthogonal to an optical axis of the camera and passing through a position of the camera, and

the reference position is located on the reference line at a point where an object location line and the reference line intersect, the object location line being orthogonal to the reference line and passing through the object.

23. A non-transitory computer-readable storage medium comprising computer-readable instructions thereon, which, when executed by a computer, cause the computer to execute the method of claim 22.