Image processing device, image processing method, and program

• US 9,823,739 B2
• Filed: 03/14/2014
• Issued: 11/21/2017
• Est. Priority Date: 04/04/2013
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kasahara, Shunichi, Rekimoto, Junichi
• Primary Examiner(s): Mazumder, Saptarshi
• Application Number:
  US14/779,452
• Publication Number:
  US 20160054793A1
• Time in Patent Office: 1,348 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

Provided is an image processing device including: an image processing unit configured to generate an output image using a first image obtained by imaging a real space from a first viewpoint as an input image. Based on a position and a posture of the first viewpoint in the real space, the image processing unit generates, as the output image, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space.

Claims

1. An image processing device comprising:

an image processing unit configured to generate an output image using a first image obtained by imaging a real space from a first viewpoint as an input image,

wherein, based on a position and a posture of the first viewpoint in the real space, the image processing unit generates, as the output image, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space,

wherein the first image is displayed together with the second image based on a relation between the position and the posture of the first viewpoint,

wherein the image processing unit sets a restriction on a change amount of the position and the posture of the second viewpoint, based on a change of the position and the posture of the first viewpoint,

wherein, when the display of the first image is deviated by a predetermined proportion or more from a range of the second image, the image processing unit releases the restriction, and

wherein the image processing unit is implemented via at least one processor.

2. The image processing device according to claim 1, wherein the position and the posture of the second viewpoint are set using the position and the posture of the first viewpoint as a criterion.

3. The image processing device according to claim 2, further comprising:

a display control unit configured to control a display unit of a terminal device to display the output image,

wherein, based on the position and the posture of the first viewpoint in the real space, the image processing unit generates, as the output image, a third image obtained by virtually imaging the real space from a third viewpoint at which a position and a posture are freely set in the real space, and

wherein the display control unit is implemented via at least one processor and causes the display unit to display one or both of the second and third images.

4. The image processing device according to claim 3,

wherein the display control unit causes the display unit to display at least the third image, and

wherein, when the second viewpoint is included in a range of the third image, the image processing unit displays an icon indicating the second viewpoint in the third image.

5. The image processing device according to claim 4, further comprising:

a manipulation acquisition unit configured to acquire information based on a user manipulation on an image displayed on the display unit,

wherein, when information based on a user manipulation of selecting the icon indicating the second viewpoint is acquired, the image processing unit presents the second image as the output image along with the third image or instead of the third image,

wherein the manipulation acquisition unit is implemented via at least one processor.

6. The image processing device according to claim 2, wherein the second viewpoint is a viewpoint set back from the first viewpoint.

7. The image processing device according to claim 2,

wherein the image processing unit continuously generates the second image based on the continuously captured first image, and

wherein the position and the posture of the second viewpoint are changed to track the position and the posture of the first viewpoint.

8. The image processing device according to claim 7, wherein the image processing unit smooths a change in the position and the posture of the first viewpoint to reflect the change in the position and the posture of the second viewpoint.

9. The image processing device according to claim 1, wherein the position and the posture of the second viewpoint are set using a position and a posture of an object disposed in the real space as a criterion.

10. The image processing device according to claim 1, wherein, when the first viewpoint is included in a range of the second image, the image processing unit displays an icon indicating the first viewpoint in the second image.

11. The image processing device according to claim 10, further comprising:

a display control unit configured to control a display unit of a terminal device to display the output image; and

a manipulation acquisition unit configured to acquire information based on a user manipulation on an image displayed on the display unit,

wherein, when information based on a user manipulation of selecting the icon indicating the first viewpoint is acquired, the image processing unit presents the first image as the output image along with the second image or instead of the second image,

wherein the display control unit and the manipulation acquisition unit are each implemented via at least one processor.

12. The image processing device according to claim 1, wherein the image processing unit generates the second image using, as the second viewpoint, one viewpoint among a plurality of viewpoints at which positions and postures in the real space are different, and displays an icon indicating another viewpoint among the plurality of viewpoints in the second image when the other viewpoint is included in a range of the second image.

13. The image processing device according to claim 12, further comprising:

a display control unit configured to control a display unit of a terminal device to display the output image; and

a manipulation acquisition unit configured to acquire information based on a user manipulation on an image displayed on the display unit,

wherein, when information based on a user manipulation of selecting the icon indicating the other viewpoint is acquired, the image processing unit generates the second image using the other viewpoint corresponding to the selected icon as the second viewpoint and presents the newly generated second image as the output image along with the original second image or instead of the original second image,

wherein the display control unit and the manipulation acquisition unit are each implemented via at least one processor.

14. The image processing device according to claim 1, wherein the image processing unit displays the first image deformed based on a relation between the position and the posture of the first viewpoint and the position and the posture of the second viewpoint in the second image.

15. The image processing device according to claim 14, wherein the image processing unit continuously generates the second image based on the continuously captured first image, displays the latest first image in the second image, and generates a portion of the second image other than the latest first image based on the previous first image.

16. The image processing device according to claim 15, wherein the image processing unit generates the portion of the second image other than the latest first image by stitching between the latest first image and the previous first image or between the previous first images.

17. The image processing device according to claim 14, wherein the image processing unit generates the output image using, as input images, a plurality of the first images obtained by imaging the real space from a plurality of the first viewpoints at which positions and postures in the real space are different and displays, in the second image, the plurality of first images deformed based on a relation between the position and the posture of each of the plurality of first viewpoints and the position and the posture of the first viewpoint.

18. An image processing method executed using at least one processor, the method comprising:

generating, by a processor configured to generate an output image using a first image obtained by imaging a real space from a first viewpoint as an input image, based on a position and a posture of the first viewpoint in the real space, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space as the output image;

displaying the first image together with the second image based on a relation between the position and the posture of the first viewpoint; and

setting a restriction on a change amount of the position and the posture of the second viewpoint, based on a change of the position and the posture of the first viewpoint,

releasing the restriction when the display of the first image is deviated by a predetermined proportion or more from a range of the second image.

19. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:

generating an output image using a first image obtained by imaging a real space from a first viewpoint as an input image, based on a position and a posture of the first viewpoint in the real space, a second image obtained by virtually imaging the real space from a second viewpoint having a position and a posture different from the position and the posture of the first viewpoint in the real space as the output image,

displaying the first image together with the second image based on a relation between the position and the posture of the first viewpoint; and

setting a restriction on a change amount of the position and the posture of the second viewpoint, based on a change of the position and the posture of the first viewpoint,

releasing the restriction when the display of the first image is deviated by a predetermined proportion or more from a range of the second image.