Image display device, method of controlling image display device, computer program, and image display system

• US 10,451,874 B2
• Filed: 08/28/2014
• Issued: 10/22/2019
• Est. Priority Date: 09/25/2013
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Kimura, Fusashi
• Primary Examiner(s): Nguyen, Chanh D
• Assistant Examiner(s): Kiyabu, Karin
• Application Number:
  US14/471,333
• Publication Number:
  US 20150084857A1
• Time in Patent Office: 1,881 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form

Abstract

An image display device includes a generating unit configured to generate an integrated image including first display regions where at least a part of a plurality of identification images for distinguishing a plurality of external devices connected to the image display device from one another are displayed as a list and a second display region where a display image, which is an image displayed by one external device selected out of the plurality of external devices, is displayed and an image display unit configured to cause a user of the image display device to visually recognize the integrated image.

Claims

1. A head-mounted display device comprising:

a camera that picks up an outside scene image;

a transmissive-type display configured to display virtual imagery while allowing real world scenery to be seen through the transmissive-type display; and

a processor or application specific integrated circuit (ASIC) configured to;

establish connections between the head-mounted display device and a plurality of external devices, respectively, the plurality of external devices being provided respective markers;

for each one of the plurality of external devices that a connection is established for;

acquire, from the one of the plurality of external devices that the connection is established for, information identifying the one of the plurality of external devices that the connection is established for;

generate, based on the information identifying the one of the plurality of external devices that the connection is established for, an identification image for distinguishing the one of the plurality of external devices that the connection is established for from the other plurality of external devices connected to the head-mounted display device, wherein the identification image comprises a frame image of a screen including multiple contents for operating the one of the plurality of external devices displayed by the one of the plurality of external devices; and

cut the connection between the head-mounted display device and the one of the plurality of external devices that the connection was established for;

generate a first integrated image including a first display region including the identification images generated for each of the plurality of external devices that were connected to the head-mounted display device organized as a list, each identification image identifying one of the plurality of external devices that were connected to the head-mounted display device;

cause the transmissive-type display to display the first integrated image;

detect one of the markers in the outside scene image captured by the camera;

make a selection of one of the plurality of external devices based on the detected marker;

generate a second integrated image including another first display region including another plurality of identification images organized as another list, and a second display region where a display image is displayed by;

requesting the display image from the selected one of the plurality of external devices;

receiving the display image, which is a frame image obtained by the selected one of the plurality of external devices by rendering an image of a display screen of the selected one of the plurality of external devices that is currently displayed, wherein the respective identification images are superimposed on the display image in the second integrated image; and

cause the transmissive-type display to display the second integrated image.

2. The head-mounted display device according to claim 1, wherein the processor or ASIC is further configured to:

repeatedly acquire the display image from the one external device corresponding to the selection of the one of the plurality of external devices until a different one of the plurality of external devices is selected; and

repeatedly update the second display region of the second integrated image using the acquired display image until a different one of the plurality of external devices is selected.

3. The head-mounted display device according to claim 1, wherein the identification images include at least one of:

an image displayed at one point in time on the one external device that the connection is established for, an image representing information for identifying the one external device that the connection is established for, and an image representing a symbol allocated to the one external device that the connection is established for.

4. The head-mounted display device according to claim 1, wherein the processor or ASIC is further configured to:

acquire distances between the head-mounted display device and each of the plurality of external devices, and

determine, according to the acquired respective distances, at least one of;

a display order of the plurality of identification images in the first display regions and a size of the identification images in the first display regions.

5. The head-mounted display device according to claim 1, wherein the processor or ASIC is further configured to:

cause the transmissive-type display to update the first display regions.

6. The head-mounted display device according to claim 1, wherein the processor or ASIC is further configured to cause the transmissive-type display to:

generate image light representing the first integrated image, and

emit the generated image light to eyes of the user to cause the user to visually recognize the first integrated image.

7. The head-mounted display device according to claim 6, wherein the processor or ASIC is further configured to:

set an aspect ratio of the second display region as an aspect ratio of a generable region of image light to be generated by the transmissive-type display,

generate the second integrated image in which the another first display region is arranged on an outer side of the second display region,

when there is no operation by the user, cause the transmissive-type display to generate the image light of a portion of the second integrated image equivalent to the second display region, and

when there is an operation by the user, cause the display to generate at least the image light of a portion of the integrated image equivalent to the another first display region.

8. The head-mounted display device according to claim 6, wherein the processor or ASIC is further configured to:

generate image light representing information for additional presentation to be added to the one external device corresponding to the selection of the one of the plurality of external devices and for providing the user with augmented reality, and

cause the transmissive-type display to emit the generated image light to the eyes of the user to cause the user to visually recognize the information for additional presentation.

9. The head-mounted display device according to claim 1, wherein the processor or ASIC is further configured to:

dynamically update the display image of the one external device corresponding to the selection of the one of the plurality of external devices.

10. A method of controlling a head-mounted display device including a camera that picks up an outside scene image, the method comprising:

establishing connections between the head-mounted display device and a plurality of external devices, respectively, the plurality of external devices being provided respective markers;

for each one of the plurality of external devices that a connection is established for;

acquiring, from the one of the plurality of external devices that the connection is established for, information identifying the one of the plurality of external devices that the connection is established for;

generating, based on the information identifying the one of the plurality of external devices that the connection is established for, an identification image for distinguishing the one of the plurality of external devices that the connection is established for from the other plurality of external devices connected to the head-mounted display device, wherein the identification image comprises a frame image of a screen including multiple contents for operating the one of the plurality of external devices displayed by the one of the plurality of external devices; and

cutting the connection between the head-mounted display device and the one of the plurality of external devices that the connection was established for;

generating a first integrated image including a first display region including the identification images generated for each of the plurality of external devices that were connected to the head-mounted display device organized as a list, each identification image identifying one of the plurality of external devices that were connected to the head-mounted display device;

causing the transmissive-type display to display the first integrated image;

detecting one of the markers in the outside scene image captured by the camera;

making a selection of one of the plurality of external devices based on the detected marker;

generating a second integrated image including another first display region including another plurality of identification images organized as another list, and a second display region where a display image is displayed by;

requesting the display image from the selected one of the plurality of external devices;

receiving the display image, which is a frame image obtained by the selected one of the plurality of external devices by rendering an image of a display screen of the selected one of the plurality of external devices that is currently displayed, wherein the respective identification images are superimposed on the display image in the second integrated image; and

causing the transmissive-type display to display the second integrated image.

11. A non-transitory computer readable medium comprising computer-executable instructions that, when executed by a computer, cause the computer to:

establish connections between a head-mounted display device and a plurality of external devices, respectively, the plurality of external devices being provided respective markers;

for each one of the plurality of external devices that a connection is established for;

acquire, from the one of the plurality of external devices that the connection is established for, information identifying the one of the plurality of external devices that the connection is established for;

generate, based on the information identifying the one of the plurality of external devices that the connection is established for, an identification image for distinguishing the one of the plurality of external devices that the connection is established for from the other plurality of external devices connected to the head-mounted display device, wherein the identification image comprises a frame image of a screen including multiple contents for operating the one of the plurality of external devices displayed by the one of the plurality of external devices; and

cut the connection between the head-mounted display device and the one of the plurality of external devices that the connection was established for;

generate a first integrated image including a first display region including the identification images generated for each of the plurality of external devices that were connected to the head-mounted display device organized as a list, each identification image identifying one of the plurality of external devices that were connected to the head-mounted display device;

cause the transmissive-type display to display the first integrated image;

detect one of the markers in an outside scene image captured by a camera of the head-mounted display device;

make a selection of one of the plurality of external devices based on the detected marker;

generate a second integrated image including another first display region including another plurality of identification images organized as another list, and a second display region where a display image is displayed by;

requesting the display image from the selected one of the plurality of external devices;

receiving the display image, which is a frame image obtained by the selected one of the plurality of external devices by rendering an image of a display screen of the selected one of the plurality of external devices that is currently displayed, wherein the respective identification images are superimposed on the display image in the second integrated image; and

cause the transmissive-type display to display the second integrated image.

12. A head-mounted display system comprising:

an information processing device;

a head-mounted display device connected to the information processing device, the head-mounted display device comprising a transmissive-type display configured to display virtual imagery while allowing real world scenery to be seen through the transmissive-type display, and including a camera that picks up an outside scene image; and

a plurality of external devices connected to the information processing device, wherein

the information processing device includes;

a processor or application specific integrated circuit (ASIC) configured to;

establish connections between the head-mounted display device and the plurality of external devices, respectively, the plurality of external devices being provided respective markers;

for each one of the plurality of external devices that a connection is established for;

 

acquire, from the one of the plurality of external devices that the connection is established for, information identifying the one of the plurality of external devices that the connection is established for;

 

generate, based on the information identifying the one of the plurality of external devices that the connection is established for, an identification image for distinguishing the one of the plurality of external devices that the connection is established for from the other plurality of external devices connected to the head-mounted display device, wherein the identification image comprises a frame image of a screen including multiple contents for operating the one of the plurality of external devices displayed by the one of the plurality of external devices; and

 

cut the connection between the head-mounted display device and the one of the plurality of external devices that the connection was established for;

generate a first integrated image including a first display region including the identification images generated for each of the plurality of external devices that were connected to the head-mounted display device organized as a list, each identification image identifying one of the plurality of external devices that were connected to the head-mounted display device;

cause the transmissive-type display to display the first integrated image;

detect one of the markers in the outside scene image captured by the camera;

make a selection of one of the plurality of external devices based on the detected marker;

generate a second integrated image including another first display region including another plurality of identification images organized as another list, and a second display region where a display image is displayed by;

 

requesting the display image from the selected one of the plurality of external devices;

 

receiving the display image, which is a frame image obtained by the selected one of the plurality of external devices by rendering an image of a display screen of the selected on of the plurality of external devices that is currently displayed, wherein the respective identification images are superimposed on the display image in the second integrated image; and

 

cause the transmissive-type display to display the second integrated image.