Image display device and image display method

• US 10,134,189 B2
• Filed: 11/11/2014
• Issued: 11/20/2018
• Est. Priority Date: 01/23/2014
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Yamamoto, Yuki, Kawamoto, Kenta, Kon, Takayasu
• Primary Examiner(s): Craddock, Robert
• Application Number:
  US15/038,950
• Publication Number:
  US 20160379413A1
• Time in Patent Office: 1,470 Days
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/36 characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

To provide an excellent image display device worn on a head or face and used to view images, and an image display method.

When displaying an image of a virtual world, such as a movie or computer graphics, with a display unit 309, a head-mounted display 100 temporarily removes the image in a portion specified with a touch panel, and display an image of the real world captured with an outward-facing camera 312. By looking at the image of the real world mixed in with the image of the virtual world, the user is able to perceive the real world and suitably interact with the real world, thereby making it possible to lessen psychological resistance while also avoiding physical harm due to events such as collisions with objects in the real world.

Claims

1. An image display device worn on a head or face, comprising:

a display configured to be disposed at an eye position of a user wearing the image display device on his or her head or face;

a processing device configured to control;

acquiring an image of a real world;

specifying a region in which to display the image of the real world;

mixing the image of the real world into a virtual image on a basis of the region specified; and

generating an image to display on the display by mixing the image of the real world, in which the image of the real world corresponds to the region specified, into the virtual image,

wherein the image of the real world is positioned in an area of the virtual image corresponding to the region specified.

2. The image display device according to claim 1, wherein

the processing device is configured to control:

capturing an image of a gaze direction of the user, and

acquiring a captured image of the real world.

3. The image display device according to claim 1, wherein the processing device is configured to control:

generating an image of a virtual world;

calculating an image R_I of the real world and an image R_V of the virtual world corresponding to the region specified, and

generating the image to display on the display by replacing the image R_V of a corresponding region of a virtual world rendering result V with the image R_I of the corresponding region of the real world.

4. The image display device according to claim 3, wherein the processing device is configured to control specifying the region in which to display the image of the real world based on an operation by the user.

5. The image display device according to claim 4, wherein the processing device is configured to control specifying the region in which to display the image of the real world based on a gesture operation by the user.

6. The image display device according to claim 4, wherein the processing device is configured to control:

accepting an input operation by the user, and

specifying the region in which to display the image of the real world based on an operation by the user with respect to an input device.

7. The image display device according to claim 6, wherein

the input device is a touch panel, and

the processing device is configured to control specifying a region corresponding to a location where the user touched the touch panel.

8. The image display device according to claim 7, wherein

the touch panel is disposed behind a display screen of the display.

9. The image display device according to claim 1, wherein the processing device is configured to control:

detecting an object in the real world; and

generating a three-dimensional model M_I of a certain range including the object on a basis of the image of the real world acquired, and also calculating a region R_V of a virtual space V corresponding to a location where the object was detected; and

calculating a three-dimensional model M_M in which the three-dimensional model M_I of the object is placed in the corresponding region R_V of a three-dimensional model V of the virtual world, and conducting a rendering process.

10. The image display device according to claim 9, wherein the processing device is configured to control detecting a part of the user'"'"'s body, or alternatively, a predetermined physical body in the real world.

11. The image display device according to claim 9, wherein the processing device is configured to control detecting a physical body in the real world within a fixed distance from the image display device.

12. The image display device according to claim 9, wherein the processing device is configured to control operating a boundary on a condition that a current position of the object overlaps with a boundary between the three-dimensional model M_I of the real world and the virtual world V.

13. The image display device of claim 1,

wherein the processing device is configured to control:

detecting an object in the real world;

generating a three-dimensional model M_I of a certain range including the object on a basis of the image of the real world acquired; and

when a virtual object is determined to exist in the virtual image at a location in the virtual image corresponding to a location in front of the object in the real world, generating the image to display on the display the three-dimensional model M_I of the object displayed obscured behind the virtual object in the virtual image.

14. An image display device worn on a head or face, comprising:

a display;

a processing device configured to control;

acquiring an image of a real world;

specifying a region in which to display the image of the real world;

mixing the image of the real world into a virtual image on a basis of the specified region, and generating an image to display on the display;

generating an image of a virtual world;

calculating an image R_I of the real world and an image R_V of the virtual world corresponding to the region specified; and

generating the image to display on the display by superimposing, onto the image R_V of a corresponding region of a virtual world rendering result V, the image R_I of the corresponding region of the real world,

wherein the processing device is configured to control;

specifying the region in which to display the image of the real world based on an operation by the user, and

a mixture ratio of the image of the real world based on an elapsed time from when the user performed the operation to specify the region.

15. The image display device according to claim 14, wherein

after a fixed time elapses from when the user performed the operation to specify the region, the processing device is configured to control switching the corresponding region from the image R_I of the real world to the image R_V of the virtual world, or alternatively, gradually reverting the corresponding region back to the image R_V of the virtual world.

16. An image display device worn on a head or face, comprising:

a display;

a processing device configured to;

acquiring an image of a real world;

specifying a region in which to display the image of the real world;

mixing the image of the real world into a virtual image on a basis of the specified region, and generating an image to display on the display;

generating an image of a virtual world;

calculating an image R_I of the real world and an image R_V of the virtual world corresponding to the region specified; and

generating the image to display on the display by superimposing, onto the image R_V of a corresponding region of a virtual world rendering result V, the image R_I of the corresponding region of the real world,

wherein the processing device is configured to control;

specifying the region in which to display the image of the real world based on an operation by the user, and

changing the region in which to display the image of the real world according to an operation by the user on a boundary between the image of the real world and the image of the virtual world.

17. An image display device worn on a head or face, comprising:

a display;

a processing device configured to control;

acquiring an image of a real world;

specifying a region in which to display the image of the real world;

mixing the image of the real world into a virtual image on a basis of the specified region, and generating an image to display on the display;

detecting an object in the real world;

generating a three-dimensional model M_I of a certain range including the object on a basis of the image of the real world acquired, and calculating a region R_V of a virtual space V corresponding to a location where the object was detected;

calculating a three-dimensional model M_M in which the three-dimensional model M_I of the object is placed in the corresponding region R_V of a three-dimensional model V of the virtual world, and conducting a rendering process; and

a mixture ratio of the image of the real world based on an elapsed time from when the object is detected.

18. An image display method comprising:

controlling, by a processing device,

acquiring an image of a real world;

specifying a region in which to display the image of the real world;

mixing the image of the real world into a virtual image on a basis of the region specified;

generating an image based on the mixing the image of the real world, in which the image of the real world corresponds to the region specified, into the virtual image,

wherein the image of the real world is positioned in an area of the virtual image corresponding to the region specified; and

displaying the generated image on a display disposed at an eye position of theft user.

19. An image display method comprising:

controlling, by a processing device,

acquiring an image of a real world;

specifying a region in which to display the image of the real world;

generating an image of a virtual world;

calculating an image R_I of the real world and an image R_V of the virtual world, each of the image R_I and the image R_V corresponding to the region specified;

generating an image by superimposing, onto the image R_V of a corresponding region of a virtual world rendering result V, the image R_I of the corresponding region of the real world, wherein the image R_I of the real world is positioned in an area of the virtual world rendering result V corresponding to the region specified; and

displaying the generated image on a display disposed at an eye position of the user.

20. An image display method comprising:

controlling, by a processing device,

acquiring an image of a real world;

detecting an object in the real world; and

specifying a corresponding region R_V of a three-dimensional model V of the virtual world of the detected object;

generating a three-dimensional model M_I of a certain range including the object on a basis of the image of the real world, and also calculating the region R_V of a virtual space V corresponding to a location where the object was detected;

calculating a three-dimensional model M_M;

placing the three-dimensional model M_I of the object in the corresponding region R_V of the three-dimensional model V of the virtual world;

conducting a rendering process to thereby generate an image based on the placing the three-dimensional model M_I of the object in the corresponding region R_V of the three-dimensional model V of the virtual world, wherein the three-dimensional model M_I of the object is positioned in an area of the virtual world corresponding to the corresponding region R_V specified; and

displaying the generated image on a display disposed at an eye position of the user.