System and method of imaging

• US 9,480,917 B2
• Filed: 03/01/2013
• Issued: 11/01/2016
• Est. Priority Date: 03/21/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Raghoebardayal, Sharwin Winesh
• Primary Examiner(s): Anderson, II, James M
• Application Number:
  US13/782,249
• Publication Number:
  US 20130250042A1
• Time in Patent Office: 1,341 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

An imaging system includes a video camera having a wide field of view lens, an imaging sensor operable to form a first image having a first pixel area, and an interface operable to receive commands from a host entertainment device. The video camera is operable to receive a command requesting at least one secondary image having a second pixel area for a 3D mapping of a portion of the first image. The system includes a graphics processor operable to perform the 3D mapping to generate the at least one each secondary image from the first image. The 3D mapping transforms a section of the first image depicting a surface having a normal in a first direction to form a secondary image in which the normal of that surface faces directly out of the secondary image.

Claims

1. An imaging system, comprising a video camera, the video camera comprising:

a wide field of view lens;

an imaging sensor operable to form a first image having a first pixel area; and

an interface operable to receive commands from a host entertainment device;

in which

the video camera is operable to receive a command requesting at least one secondary image, the command identifying a second pixel area of the at least one secondary image for a 3D mapping of a portion of the first image; and

the imaging system further comprises a graphics processor operable to perform the 3D mapping to generate the at least one secondary image from the first image,

in which the 3D mapping transforms a section of the first image depicting a surface having a normal in a first direction to form a given one of the at least one secondary image in which the normal of that surface faces directly out of the given secondary image, and the 3D mapping includes rotating an estimated normal of the depicted surface in 3D space so that its normal substantially coincides with the normal of a 2D plane of the given secondary image.

2. An imaging system according claim 1 in which the first image depicts a captured scene, and the 3D mapping comprises rotating the estimated normal of the depicted surface in the scene in 3D space so that its normal substantially coincides with the normal of a 2D plane of the given secondary image.

3. An imaging system according claim 1, in which the 3D mapping comprises

estimating the plane of a surface from the position and estimated depth of at least three points in the image;

and

estimating the normal of that surface.

4. An imaging system according to claim 1, in which at least two of a plurality of requested secondary images have a different respective pixel area.

5. An imaging system according to claim 1, in which if the pixel area of a requested secondary image is not the same as the pixel area of the portion of the first image, the graphics processor resamples the corresponding portion of the first image to generate the at least one secondary image.

6. An imaging system according to claim 1, in which the wide field of view lens is a fisheye lens, and the graphics processor is operable to substantially remove from an image curvilinear distortion caused by the fisheye lens.

7. An imaging system according to claim 1, in which the video camera comprises a structured light emitter operable to emit a pattern of light onto a scene that can be captured by the imaging sensor.

8. An imaging system according to claim 7 in which the graphics processor is operable to derive a depth map from the pattern of light captured by the imaging sensor.

9. An imaging system according to claim 8 in which the graphics processor is operable to derive the normal of one or more surfaces in the first image based upon the depth map.

10. An imaging system according to claim 1, comprising:

a host entertainment device;

and in which the host entertainment device comprises

an interface for connection to the video camera; and

a processor operable to generate commands requesting from the video camera one or more of the secondary images based upon the first image captured by the video camera.

11. An imaging system according to claim 10, in which the host entertainment device comprises the graphics processor.

12. An imaging system according to claim 1, in which the video camera comprises the graphics processor.

13. A method of imaging for a camera in connection with a host entertainment device, comprising the steps of:

capturing with the camera a first image having a first pixel area using a lens with a wide field of view;

receiving at the camera a command from the host entertainment device requesting at least one secondary image having a second pixel area and comprising a 3D mapping of a portion of the first image; and

performing the 3D mapping to generate the at least one secondary image from the first image;

in which the 3D mapping transforms a section of the first image depicting a surface having a normal in a first direction to form a given one of the at least one secondary image in which the normal of that surface faces directly out of the given secondary image, and the 3D mapping includes rotating an estimated normal of the depicted surface in 3D space so that its normal substantially coincides with the normal of a 2D plane of the given secondary image.

14. A method of imaging according to claim 13, in which:

the step of generating the at least one secondary image comprises implementing the 3D mapping at the camera, and

the method further comprising the step of;

providing the at least one secondary image to the host entertainment device.

15. A non-transitory computer-readable recording medium storing computer implementable instructions thereon, the instructions, when implemented on a computer, cause the computer to perform the steps of:

capturing with the camera a first image having a first pixel area using a lens with a wide field of view;

receiving at the camera a command from a host device requesting at least one secondary image having a second pixel area and comprising a 3D mapping of a portion of the first image; and

performing the 3D mapping to generate the at least one secondary image from the first image;

in which the 3D mapping transforms a section of the first image depicting a surface having a normal in a first direction to form a given one of the at least one secondary image in which the normal of that surface faces directly out of the secondary image, and the 3D mapping includes rotating an estimated normal of the depicted surface in 3D space so that its normal substantially coincides with the normal of a 2D plane of the given secondary image.