Head-mounted display device and control method for the head-mounted display device

• US 9,557,566 B2
• Filed: 03/04/2013
• Issued: 01/31/2017
• Est. Priority Date: 03/07/2012
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Fujimaki, Yutaka
• Primary Examiner(s): Mehmood, Jennifer
• Assistant Examiner(s): Azongha, Sardis F
• Application Number:
  US13/783,746
• Publication Number:
  US 20130234914A1
• Time in Patent Office: 1,429 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A head-mounted display device with which a user can visually recognize a virtual image and an outside scene simultaneously includes an image pickup unit configured to pick up an image of the outside scene and acquire an outside scene image, an augmented-reality processing unit configured to analyze the acquired outside scene image and generate additional image data for augmented reality according to a result of the analysis, a display element configured to generate image light using the generated additional image data, and a light guide unit configured to form a virtual image before the eyes of the user using the generated image light. The augmented-reality processing unit acquires an overlapping region where a display region for the virtual image and an image pickup region of the image pickup unit overlap in the outside scene image and generates the additional image data using the overlapping region.

Claims

1. A head-mounted display device with which a user can visually recognize a virtual image and an outside scene simultaneously, the head-mounted display device comprising:

a first image pickup unit configured to acquire a first outside scene image, the first image pickup unit being arranged in a position between a right eye and a left eye of the user;

a second image pickup unit configured to acquire a second outside scene image and arranged in a position corresponding to the right eye or a right temple of the user or a periphery of the right eye or the right temple during wearing of the head-mounted display device by the user;

a third image pickup unit configured to acquire a third outside scene image and arranged in a position corresponding to the left eye or a left temple of the user or a periphery of the left eye or the left temple during wearing of the head-mounted display device by the user;

an augmented-reality processing unit configured to analyze the acquired first, second, and third outside scene images and generate additional image data for augmented reality according to a result of the analysis;

a display element configured to generate image light using the generated additional image data;

a light guide unit configured to form a virtual image before the eyes of the user using the generated image light; and

a storing unit having stored therein guide image data for showing a display region for the virtual image to the user, the guide image data having a rectangular shape and including a plurality of guide marks arranged in each of a plurality of corners of the guide image data,

wherein

the augmented-reality processing unit acquires an overlapping region where the display region for the virtual image and an image pickup region of the first image pickup unit overlap in the first outside scene image and generates the additional image data using the overlapping region,

in acquiring the overlapping region, the augmented-reality processing unit causes the display element to generate image light using the guide image data, guides the user to indicate positions of each of the plurality of guide marks in a given sequence, causes the first image pickup unit to acquire the first outside scene image, and acquires, as a boundary between the overlapping region and other regions, the positions indicated by the user obtained by analyzing the acquired first outside scene image,

the augmented-reality processing unit generates the additional image data for augmented reality display in both the overlapping region and a region outside of the overlapping region based on the analysis of the first, second, and third outside scene images,

the additional image data generated for display in the overlapping region by the display element is displayed to be recognized by a user of the head-mounted display device, and

the additional image data generated for the region outside of the overlapping region is indicated to be present outside the overlapping region by displaying an icon indicating presence of the additional image data outside of the overlapping region.

2. The head-mounted display device according to claim 1, wherein

the plurality of guide marks includes a guide mark arranged in each of a plurality of corners of the guide image data and a guide mark arranged in a center of the guide image data.

3. The head-mounted display device according to claim 1, wherein

the augmented-reality processing unit guides the user to indicate positions of each of the plurality of guide marks in a given sequence by causing the display element to display a message instructing the user to indicate positions of each of the plurality of guide marks in order of the given sequence displayed in the message.

4. The head-mounted display device according to claim 1, further comprising:

a sound processing unit that supplies a sound signal to the user,

wherein the augmented-reality processing unit guides the user to indicate positions of each of the plurality of guide marks in a given sequence by causing the sound processing unit to supply a sound signal to the user instructing the user to indicate positions of each of the plurality of guide marks in order of the given sequence provided in the sound signal.

5. The head-mounted display device according to claim 1, wherein

the display region is an entire region of the head-mounted display device where a virtual image can be displayed to the user wearing the head-mounted display device,

a non-display region surrounds the display region, and

the plurality of guide marks are arranged in each of the plurality of corners of the guide image data such that the plurality of guide marks are positioned at the boundary between the display region and the non-display region.

6. The head-mounted display device according to claim 1, wherein

the augmented-reality processing unit further specifies a center point of a specific object in an external environment as a reference point and calculates a shift amount between the center point of the specific object included in the second outside scene image and the center point of the specific object included in the third outside scene image and acquires information concerning depth of the specific object in the external environment based on the calculated shift amount.

7. A control method for a head-mounted display device with which a user can visually recognize a virtual image and an outside scene simultaneously, the control method comprising:

acquiring a first outside scene image from a first image pickup unit arranged in a position between a right eye and a left eye of the user;

acquiring a second outside scene image from a second image pickup unit arranged in a position corresponding to the right eye or a right temple of the user or a periphery of the right eye or the right temple during wearing of the head-mounted display device by the user;

acquiring a third outside scene image from a third image pickup unit arranged in a position corresponding to the left eye or a left temple of the user or a periphery of the left eye or the left temple during wearing of the head-mounted display device by the user;

analyzing the acquired first, second, and third outside scene images and generating additional image data for augmented reality according to a result of the analysis;

generating image light using the generated additional image data;

forming a virtual image before the eyes of the user using the generated image light; and

storing guide image data for showing a display region for the virtual image to the user, the guide image data having a rectangular shape and including a plurality of guide marks arranged in each of a plurality of corners of the guide image data,

wherein

in the generating the additional image data, an overlapping region where the display region for the virtual image and an image pickup region for acquiring the first outside scene image overlap in the first outside scene image is acquired and the additional image data is generated using the overlapping region,

the overlapping region is acquired by generating the guide image data, guiding the user to indicate positions of each of the plurality of guide marks in a given sequence, causing the first image pickup unit to acquire the first outside scene image, and acquiring, as the boundary between the overlapping region and other regions, the positions indicated by the user obtained by analyzing the acquired first outside scene image,

in generating the additional image data, the additional image data is generated for augmented reality display in both the overlapping region and a region outside of the overlapping region based on the analysis of the first, second, and third outside scene images,

the additional image data generated for display in the overlapping region is displayed to be recognized by a user of the head-mounted display device, and

the additional image data generated for the region outside of the overlapping region is indicated to be present outside the overlapping region by displaying an icon indicating presence of the additional image data outside of the overlapping region.

8. A head-mounted display device with which a user can visually recognize a virtual image and an outside scene simultaneously, the head-mounted display device comprising:

a first image pickup unit configured to acquire a first outside scene image, the first image pickup unit being arranged in a position between a right eye and a left eye of the user;

a second image pickup unit configured to acquire a second outside scene image and arranged in a position corresponding to the right eye or a right temple of the user or a periphery of the right eye or the right temple during wearing of the head-mounted display device by the user;

a third image pickup unit configured to acquire a third outside scene image and arranged in a position corresponding to the left eye or a left temple of the user or a periphery of the left eye or the left temple during wearing of the head-mounted display device by the user;

an augmented-reality processing unit configured to analyze the acquired first, second, and third outside scene images and generate additional image data for augmented reality according to a result of the analysis;

a display element configured to generate image light using the generated additional image data;

a light guide unit configured to form a virtual image before the eyes of the user using the generated image light; and

a storing unit having stored therein guide image data for showing a display region for the virtual image to the user, the guide image data having a rectangular shape and including frame-like guide marks arranged in at least two places among four corners of the guide image data,

wherein the augmented-reality processing unit

acquires an overlapping region where the display region for the virtual image and an image pickup region of the first image pickup unit overlap in the first outside scene image and generates the additional image data using the overlapping region,

causes the display element to generate image light using the guide image data, guides the user to hold a rectangular-shaped object in front of the head-mounted display device in an area where the rectangular-shaped object may be picked-up by the first image pickup unit, and guides the user to arrange the rectangular-shaped object to appear inside the frame-like guide marks in the display region for the virtual image, such that four corners of the rectangular shaped object correspond with the frame-like guide marks from the perspective of the user,

causes the first image pickup unit to acquire the first outside scene image, and

acquires, as a boundary between the overlapping region and other regions, a position of the rectangular-shaped object positioned inside the frame-like guide marks obtained by analyzing the acquired first outside scene image.