IMAGE PROCESSING APPARATUS AND METHOD, AND PROGRAM

• US 20160320863A1
• Filed: 12/05/2014
• Published: 11/03/2016
• Est. Priority Date: 12/19/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: SHIMODA, RYUICHI
• Granted Patent: US 10,133,364 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0489 using dedicated keyboard keys or combinations thereof
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The present technique relates to an image processing apparatus, method, and program which can improve operability. A short distance image separation unit generates a short distance image indicating a short distance object located at the side of an imaging unit, on the basis of a clipped image which is clipped from an input image captured by the imaging unit, and a distance image indicating the distance to an object in each region of the clipped image. A contour extraction unit extracts, as an adjustment image, a contour of the short distance image, and a display image generation unit generates an output image on the basis of the adjustment image and displays the output image on an image display unit. The user moves his/her hand while viewing the output image displayed on the image display unit so that his/her hand overlaps with the region of the hand in the output image. An XY calculation unit calculates the amount of correction of the display position of the image on the basis of adjustment images of different time points. The technique may be applied to a head mount display.

Claims

1. An image processing apparatus comprising:

a generation unit configured to generate, on a basis of an input image captured by an imaging unit disposed in a vicinity of a user'"'"'s viewpoint and distance information indicating a distance to an object in each region of the input image, an image indicating the object in the input image located closer to the imaging unit than a predetermined distance in real space;

an image display unit configured to display the image, allowing the user to view the object via a display screen; and

an amount-of-correction calculation unit configured to calculate, on a basis of the image at mutually different time points, an amount of correction for correcting a display position or display size of an image to be displayed on the image display unit.

2. The image processing apparatus according to claim 1, wherein the amount-of-correction calculation unit calculates the amount of correction for correcting the display position, on a basis of an amount of movement, in the image, of a short distance object, which is the object located closer to the imaging unit than the predetermined distance.

3. The image processing apparatus according to claim 2, wherein, in a state where the image as a reference, or the image corrected by the amount of correction and succeeding the image as the reference is being displayed on the image display unit, the amount-of-correction calculation unit calculates the amount of correction, on a basis of an amount of movement of an operation instruction unit between the image captured when the user moves the operation instruction unit so that the operation instruction unit as the short distance object overlaps with the operation instruction unit on the image being displayed, and the image as the reference, or the image succeeding the image as the reference.

4. The image processing apparatus according to claim 1, wherein the amount-of-correction calculation unit calculates the amount of correction for correcting the display size, on a basis of a change of size, in the image, of a short distance object, which is the object located closer to the imaging unit than the predetermined distance.

5. The image processing apparatus according to claim 4, wherein, in a state where the image as a reference, or the image corrected by the amount of correction and succeeding the image as the reference is being displayed on the image display unit, the amount-of-correction calculation unit calculates the amount of correction, on a basis of a change of size of an operation instruction unit between the image captured when the user moves the operation instruction unit so that the operation instruction unit as the short distance object has a same size as the operation instruction unit on the image being displayed, and the image as the reference.

6. The image processing apparatus according to claim 1, wherein the amount-of-correction calculation unit calculates the amount of correction for correcting the display size, on a basis of an amount of movement, in the image, of a specific part of a short distance object, which is the object located closer to the imaging unit than the predetermined distance.

7. The image processing apparatus according to claim 1,

wherein the image display unit further displays a virtual image to be operated by the user, and

wherein the image processing apparatus further includes a correction unit configured to correct the display position or the display size of the virtual image, on a basis of the amount of correction.

8. The image processing apparatus according to claim 7, further comprising:

a mask generation unit configured to generate a masked image indicating an unmasked region, the unmasked region being a region of the object located closer to the imaging unit than a distance of the virtual image in space;

a mask correction unit configured to correct a position or size of the unmasked region in the masked image, on a basis of the amount of correction; and

a mask processing unit configured to remove the unmasked region from the virtual image by performing transparent processing on the virtual image, on the basis of the masked image corrected by the mask correction unit and the amount of correction,

wherein the image display unit displays the virtual image subjected to the transparent processing.

9. The image processing apparatus according to claim 7, further comprising:

an operation correction unit configured to correct, on a basis of the amount of correction, a position on the virtual image operated by the user.

10. The image processing apparatus according to claim 1,

wherein the image display unit is provided for each of the user'"'"'s left and right eyes, and

wherein the amount-of-correction calculation unit calculates the amount of correction for each of the user'"'"'s left and right eyes.

11. An image processing method comprising:

generating, on a basis of an input image captured by an imaging unit disposed in a vicinity of a user'"'"'s viewpoint and distance information indicating a distance to an object in each region of the input image, an image indicating the object in the input image located closer to the imaging unit than a predetermined distance in real space;

displaying the image on an image display unit allowing the user to view the object via a display screen; and

calculating, on a basis of the image at mutually different time points, an amount of correction for correcting a display position or display size of an image to be displayed on the image display unit.

12. A program causing a computer to perform processing comprising:

generating, on a basis of an input image captured by an imaging unit disposed in a vicinity of a user'"'"'s viewpoint and distance information indicating a distance to an object in each region of the input image, an image indicating the object in the input image located closer to the imaging unit than a predetermined distance in real space;

displaying the image on an image display unit allowing the user to view the object via a display screen; and

calculating, on a basis of the image at mutually different time points, an amount of correction for correcting a display position or display size of an image to be displayed on the image display unit.