IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM

• US 20160054793A1
• Filed: 03/14/2014
• Published: 02/25/2016
• Est. Priority Date: 04/04/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: KASAHARA, Shunichi, REKIMOTO, Junichi
• Granted Patent: US 9,823,739 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

Provided is an image processing device including: an image processing unit configured to generate an output image using a first image obtained by imaging a real space from a first viewpoint as an input image. Based on a position and a posture of the first viewpoint in the real space, the image processing unit generates, as the output image, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space.

Claims

1. An image processing device comprising:

an image processing unit configured to generate an output image using a first image obtained by imaging a real space from a first viewpoint as an input image,

wherein, based on a position and a posture of the first viewpoint in the real space, the image processing unit generates, as the output image, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space.

2. The image processing device according to claim 1, wherein the position and the posture of the second viewpoint are set using the position and the posture of the first viewpoint as a criterion.

3. The image processing device according to claim 2, further comprising:

a display control unit configured to control a display unit of a terminal device to display the output image,

wherein, based on the position and the posture of the first viewpoint in the real space, the image processing unit generates, as the output image, a third image obtained by virtually imaging the real space from a third viewpoint at which a position and a posture are freely set in the real space, and

wherein the display control unit causes the display unit to display one or both of the second and third images.

4. The image processing device according to claim 3,

wherein the display control unit causes the display unit to display at least the third image, and

wherein, when the second viewpoint is included in a range of the third image, the image processing unit displays an icon indicating the second viewpoint in the third image.

5. The image processing device according to claim 4, further comprising:

a manipulation acquisition unit configured to acquire information based on a user manipulation on an image displayed on the display unit,

wherein, when information based on a user manipulation of selecting the icon indicating the second viewpoint is acquired, the image processing unit presents the second image as the output image along with the third image or instead of the third image.

6. The image processing device according to claim 2, wherein the second viewpoint is a viewpoint set back from the first viewpoint.

7. The image processing device according to claim 2,

wherein the image processing unit continuously generates the second image based on the continuously captured first image, and

wherein the position and the posture of the second viewpoint are changed to track the position and the posture of the first viewpoint.

8. The image processing device according to claim 7, wherein the image processing unit smooths a change in the position and the posture of the first viewpoint to reflect the change in the position and the posture of the second viewpoint.

9. The image processing device according to claim 8, wherein the image processing unit sets a restriction on a change amount of the position and the posture of the second viewpoint and releases the restriction when the first image displayed in the second image based on a relation between the position and the posture of the first viewpoint and the position and the posture of the second viewpoint is deviated by a predetermined proportion or more from a range of the second image.

10. The image processing device according to claim 1, wherein the position and the posture of the second viewpoint are set using a position and a posture of an object disposed in the real space as a criterion.

11. The image processing device according to claim 1, wherein, when the first viewpoint is included in a range of the second image, the image processing unit displays an icon indicating the first viewpoint in the second image.

12. The image processing device according to claim 11, further comprising:

a display control unit configured to control a display unit of a terminal device to display the output image; and

a manipulation acquisition unit configured to acquire information based on a user manipulation on an image displayed on the display unit,

wherein, when information based on a user manipulation of selecting the icon indicating the first viewpoint is acquired, the image processing unit presents the first image as the output image along with the second image or instead of the second image.

13. The image processing device according to claim 1, wherein the image processing unit generates the second image using, as the second viewpoint, one viewpoint among a plurality of viewpoints at which positions and postures in the real space are different, and displays an icon indicating another viewpoint among the plurality of viewpoints in the second image when the other viewpoint is included in a range of the second image.

14. The image processing device according to claim 13, further comprising:

a display control unit configured to control a display unit of a terminal device to display the output image; and

a manipulation acquisition unit configured to acquire information based on a user manipulation on an image displayed on the display unit,

wherein, when information based on a user manipulation of selecting the icon indicating the other viewpoint is acquired, the image processing unit generates the second image using the other viewpoint corresponding to the selected icon as the second viewpoint and presents the newly generated second image as the output image along with the original second image or instead of the original second image.

15. The image processing device according to claim 1, wherein the image processing unit displays the first image deformed based on a relation between the position and the posture of the first viewpoint and the position and the posture of the second viewpoint in the second image.

16. The image processing device according to claim 15, wherein the image processing unit continuously generates the second image based on the continuously captured first image, displays the latest first image in the second image, and generates a portion of the second image other than the latest first image based on the previous first image.

17. The image processing device according to claim 16, wherein the image processing unit generates the portion of the second image other than the latest first image by stitching between the latest first image and the previous first image or between the previous first images.

18. The image processing device according to claim 15, wherein the image processing unit generates the output image using, as input images, a plurality of the first images obtained by imaging the real space from a plurality of the first viewpoints at which positions and postures in the real space are different and displays, in the second image, the plurality of first images deformed based on a relation between the position and the posture of each of the plurality of first viewpoints and the position and the posture of the first viewpoint.

19. An image processing method comprising:

generating, by a processor configured to generate an output image using a first image obtained by imaging a real space from a first viewpoint as an input image, based on a position and a posture of the first viewpoint in the real space, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space as the output image.

20. A program causing a computer configured to generate an output image using a first image obtained by imaging a real space from a first viewpoint as an input image to realize:

a function of generating, based on a position and a posture of the first viewpoint in the real space, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space as the output image.