Head-mounted display device, control method for head-mounted display device, and work supporting system

• US 9,448,407 B2
• Filed: 12/03/2013
• Issued: 09/20/2016
• Est. Priority Date: 12/13/2012
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Kimura, Fusashi
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US14/095,827
• Publication Number:
  US 20140168266A1
• Time in Patent Office: 1,022 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A head-mounted display device for enabling a user to simultaneously visually recognize a virtual image and an outside scene includes a warning-information generating unit configured to generate warning information, which is an image for calling the user'"'"'s attention and an image display unit configured to cause the user to visually recognize the warning information as the virtual image.

Claims

1. A head-mounted display device for enabling a user to simultaneously visually recognize a virtual image and an outside scene of the user'"'"'s environment, the head-mounted display device comprising:

an image acquiring unit that acquires an image in a visual field direction of the user in a wearing state of the head-mounted display device, the image including an external object;

a warning-information generating unit that;

recognizes a specified part of the external object having an outer shape that coincides with a pattern stored in advance;

determines that the specified part of the external object poses a bodily danger to the user based on the recognition that the outer shape coincides with the pattern; and

generates warning information, which is an image for calling the user'"'"'s attention, on the basis of the image acquired by the image acquiring unit; and

an image display unit that causes the user to visually recognize the warning information as the virtual image,

wherein the warning information generated by the warning-information generating unit includes a predetermined image that is arranged at the specified part of the external object, the predetermined image being an image formed by a figure having a shape that surrounds the outer shape of the specified part of the external object such that portions of the external object other than the specified part are not surrounded by the figure.

2. The head-mounted display device according to claim 1, wherein the warning-information generating unit specifies, from the image acquired by the image acquiring unit, a place where the user needs to be careful and generates warning information in which the predetermined image is arranged near the specified place.

3. The head-mounted display device according to claim 2, wherein

the image acquiring unit detects infrared light radiated from an object and acquires a temperature distribution image representing a distribution of temperature, and

the warning-information generating unit specifies a place where the temperature is equal to or higher than a first threshold in the temperature distribution image acquired by the image acquiring unit and sets the place as the place where the user needs to be careful.

4. The head-mounted display device according to claim 2, wherein

the image acquiring unit detects infrared light radiated from an object and acquires a temperature distribution image representing a distribution of temperature, and

the warning-information generating unit specifies a place where the temperature is equal to or lower than a second threshold in the temperature distribution image acquired by the image acquiring unit and sets the place as the place where the user needs to be careful.

5. The head-mounted display device according to claim 2, wherein

the image acquiring unit acquires an image representing the shape of an object from visible light radiated from the object, and

the warning-information generating unit recognizes the image acquired by the image acquiring unit, specifies the specified part of the external object having the outer shape that coincides with the pattern stored in advance, and sets the specified part of the external object as the place where the user needs to be careful.

6. The head-mounted display device according to claim 1, wherein the outer shape of the specified part of the external object is determined to include an acute angle part.

7. The head-mounted display device according to claim 1, wherein the predetermined image is an image formed by a character, a pattern, or a figure that reminds the user of a warning or a combination of the character, the pattern, and the figure.

8. The head-mounted display device according to claim 1, wherein the figure has a shape corresponding to the outer shape of the specified part of the external object.

9. The head-mounted display device according to claim 3, wherein the predetermined image is an image obtained by extracting, from the temperature distribution image acquired by the image acquiring unit, a place where temperature is equal to or higher than the first threshold or a place where temperature is equal to or lower than the second threshold.

10. The head-mounted display device according to claim 1, wherein the warning-information generating unit generates the warning information for an overlapping area where a display area of the virtual image and an acquisition area of the image acquiring unit overlap each other in the image acquired by the image acquiring unit.

11. The head-mounted display device according to claim 1, further comprising a sounding body, wherein

the warning-information generating unit causes, on the basis of the image acquired by the image acquiring unit, the sounding body to output sound for calling the user'"'"'s attention.

12. The head-mounted display device according to claim 1, further comprising a tactile stimulation unit configured to apply stimulation through a tactile sense to the user, wherein

the warning-information generating unit actuates, on the basis of the image acquired by the image acquiring unit, the tactile stimulation unit in order to call the user'"'"'s attention.

28. The head-mounted display device according to claim 1, wherein the outer shape of the specified part of the external object is determined to include a plurality of acute angle parts.

29. The head-mounted display device according to claim 1, wherein the specified part of the external object is determined to include two or more of an acute angle part, a catching part, and a roll-in part.

13. A control method for a head-mounted display device for enabling a user to simultaneously visually recognize a virtual image and an outside scene of the user'"'"'s environment, the control method comprising:

acquiring an image in a visual field direction of the user in a wearing state of the head-mounted display device, the image including an external object;

recognizing a specified part of the external object having an outer shape that coincides with a pattern stored in advance;

determining that the specified part of the external object poses a bodily danger to the user based on the recognition that the outer shape coincides with the pattern;

generating warning information, which is an image for calling the user'"'"'s attention, on the basis of the acquired image; and

causing the user to visually recognize the warning information as the virtual image,

wherein the warning information includes a predetermined image that is arranged at the specified part of the external object, the predetermined image being an image formed by a figure having a shape that surrounds the outer shape of the specified part of the external object such that portions of the external object other than the specified part are not surrounded by the figure.

14. A head-mounted display device for enabling a user to simultaneously visually recognize a virtual image and an outside scene of the user'"'"'s environment, the head-mounted display device comprising:

a procedure managing unit that generates procedure information, which is an image representing content that the user should carry out, concerning a present procedure;

an image acquiring unit that acquires an image in a visual field direction of the user in a wearing state of the head-mounted display device, the image including an external object;

a warning-information generating unit that;

recognizes a specified part of an external object having an outer shape that coincides with a pattern stored in advance;

determines that the specified part of the external object poses a bodily danger to the user based on the recognition that the outer shape coincides with the pattern; and

generates warning information, which is an image for calling the user'"'"'s attention, in the present procedure; and

an image display unit that causes the user to visually recognize the procedure information and the warning information as the virtual image,

wherein

the procedure managing unit specifies, on the basis of the image acquired by the image acquiring unit, the present procedure from a plurality of procedures included in work,

the procedure managing unit generates the procedure information for an overlapping area where a display area of the virtual image and an acquisition area of the image acquiring unit overlap each other in the image acquired by the image acquiring unit,

the warning-information generating unit generates the warning information for the overlapping area, and

the warning-information generating unit specifies, from the overlapping area in the image acquired by the image acquiring unit, that the specified part of the external object poses a bodily danger to the user and generates warning information in which a predetermined image is arranged at the specified part of the external object.

15. The head-mounted display device according to claim 14, wherein the warning-information generating unit generates the warning information when warning point data stored in advance is present in the present procedure.

16. The head-mounted display device according to claim 14, wherein the procedure managing unit generates the procedure information corresponding to the present procedure using association of information representing the procedure included in the work and information representing content that the user should carry out in the procedure.

17. The head-mounted display device according to claim 15, wherein

the warning point data stored in advance is data associating information representing the procedure included in the work and information concerning at least one of an accident in the past and a near miss in the past in the procedure, and

the warning-information generating unit generates, using the warning point data, the warning information corresponding to the present procedure.

18. The head-mounted display device according to claim 14, wherein the procedure managing unit limits generation of the procedure information corresponding to a next procedure until the procedure managing unit succeeds in predetermined check processing after generating the procedure information corresponding to the present procedure.

19. The head-mounted display device according to claim 18, wherein the predetermined check processing is acquisition of an indication of intension for check from the user.

20. The head-mounted display device according to claim 18, wherein the predetermined check processing is processing for recognizing the image acquired by the image acquiring unit and checking whether a result of the image recognition satisfies a condition set in advance.

21. The head-mounted display device according to claim 20, wherein the condition set in advance varies according to characteristics of the user including at least any one of age, sex, physique, and proficiency in work of the user.

22. The head-mounted display device according to claim 14, wherein the warning-information generating unit generates different kinds of the warning information when an accident occurred in the past in the present procedure and when a near miss occurred in the past in the present procedure.

23. The head-mounted display device according to claim 14, wherein the warning-information generating unit generates different kinds of the warning information according to characteristics of the user including at least any one of age, sex, physique, and proficiency in work of the user.

24. The head-mounted display device according to claim 17, further comprising a sounding body, wherein

the warning-information generating unit causes, on the basis of the information concerning at least one of an accident in the past and a near miss in the past corresponding to the present procedure, the sounding body to output sound for calling the user'"'"'s attention.

25. The head-mounted display device according to claim 17, further comprising a tactile stimulation unit configured to apply stimulation through a tactile sense to the user, wherein

the warning-information generating unit actuates, on the basis of the information concerning at least one of an accident in the past and a near miss in the past corresponding to the present procedure, the tactile stimulation unit in order to call the user'"'"'s attention.

27. A work supporting system, comprising:

the head-mounted display device according to claim 16; and

a server, wherein

the server includes;

a first table configured to store association of the information representing the procedure included in the work and the information representing content that the user should carry out in the procedure;

a second table configured to store association of the information representing the procedure included in the work and the information concerning at least one of an accident in the past and a near miss in the past in the procedure; and

a transmitting unit configured to transmit, according to a request from the head-mounted display device, at least one of the information stored in the first table and the information stored in the second table.

26. A control method for a head-mounted display device for enabling a user to simultaneously visually recognize a virtual image and an outside scene of the user'"'"'s environment, the control method comprising:

generating procedure information, which is an image representing content that the user should carry out, concerning a present procedure;

acquiring an image in a visual field direction of the user in a wearing state of the head-mounted display device, the image including an external object;

recognizing a specified part of an external object having an outer shape that coincides with a pattern stored in advance;

determining that the specified part of the external object poses a bodily danger to the user based on the recognition that the outer shape coincides with the pattern;

generating warning information, which is an image for calling the user'"'"'s attention, in the present procedure; and

causing the user to visually recognize the procedure information and the warning information as the virtual image,

wherein

the present procedure is specified, on the basis of the image acquired, from a plurality of procedures included in work;

the procedure information is generated for an overlapping area where a display area of the virtual image and an acquisition area of the image acquiring unit overlap each other in the image acquired by the image acquiring unit,

the warning information is generated for the overlapping area, and

based on the overlapping area in the image acquired by the image acquiring unit, the specified part of the external object is specified to pose a bodily danger to the user and warning information in which a predetermined image is arranged at the specified part of the external object is generated.