Apparatus and method for virtual retinal display capable of controlling presentation of images to viewer in response to viewer's motion

• US 7,825,996 B2
• Filed: 04/28/2006
• Issued: 11/02/2010
• Est. Priority Date: 10/30/2003
• Status: Active Grant

• Current Assignee: Brother Kogyo Kabushiki Kaisha (Brother Industries Limited)
• Original Assignee: Brother Kogyo Kabushiki Kaisha (Brother Industries Limited)
• Inventors: Yamada, Shoji, Mase, Yasufumi
• Primary Examiner(s): LEE, MICHAEL
• Application Number:
  US11/413,046
• Publication Number:
  US 20060197832A1
• Time in Patent Office: 1,649 Days
• Field of Search: 348/744, 348/51, 348/52, 348/53, 348/115, 348/750, 348/755, 348/757, 348/759, 345/8, 600/558, 359/630, 359/632
• US Class Current: 348/744
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: H04N 5/7491 of head mounted projectors

Abstract

An apparatus is disclosed which projects light onto a retina of a viewer, to thereby allow the viewer to perceive a display object via a virtual image. The apparatus includes: an emitter emitting light; a modulator modulating the light; a display unit outputting the modulated light to the retina of the viewer through an exit of the display unit, to thereby display in an image display region an image representative of the display object in the form of a virtual image; and a controller controlling the emitter and the modulator so that the image is displayed in the image display region. The controller detects a relative motion of the viewer with respect to a position of the viewer, and performs at least one of control of a display/non-display status of the image, and control of a display position of the image, in accordance with the detected relative motion.

Claims

1. An apparatus for projecting light onto a retina of a viewer, to thereby allow the viewer to perceive a display object via a virtual image, the apparatus comprising:

an emitter emitting light;

a modulator modulating the light;

a display unit outputting the modulated light to the retina of the viewer through an exit of the display unit, to thereby display in an image display region an image representative of the display object in the form of a virtual image, the display unit being mounted on a head of the viewer in operation;

a detector detecting a relative position of a to-be-detected segment of a human body of the viewer, with respect to a reference segment of the human body, the to-be-detected segment being selected from a plurality of segments constituting the human body which are displaceable relative to each other, for detection of a motion of the to-be-detected segment, and the reference segment being selected from the plurality of segments for establishment of a reference position related to the human body; and

a controller controlling the emitter and the modulator so that the image is displayed in the image display region, the controller performing display control which includes at least one of control of a display/non-display status of the image, and control of a display position of the image, based on the detected relative position.

2. The apparatus according to claim 1, wherein the detector includes first and second portions which are displaceable relative to each other, the first portion being attached to the to-be-detected segment for allowing integral movement therewith, the second portion being attached to the reference segment for allowing integral movement therewith, and

wherein the detector further includes a detecting device detecting a relative position between the first and second portions.

3. The apparatus according to claim 2, wherein one of the first and second portions functions as a signal generator generating a detection signal for use in detection of the relative position between the first and second portions, and

wherein a remainder of the first and second portions functions as a receiver receiving the generated detection signal which propagates through space.

5. The apparatus according to claim 1, wherein the plurality of segments include the head, and an upper limb, a lower limb, and a middle, of the viewer, wherein the head, optionally with at least one of the upper limb, and the lower limb, is assigned as the to-be-detected segment, and wherein the middle is assigned as the reference segment.

6. The apparatus according to claim 1, wherein the controller controls the emitter and the modulator so as to allow the viewer to perceive the image at a position predetermined with respect to the reference segment.

14. The apparatus according to claim 1, wherein the controller selects any one of a plurality of display modes including a viewer-based display mode for use in displaying the image such that the viewer perceives the image at a position predetermined with respect to the reference segment, and a display-unit-based display mode for use in displaying the image such that the viewer perceives the image at a position predetermined with respect to a position of the display unit, and

wherein the controller displays the image in accordance with the selected display mode.

15. The apparatus according to claim 14, wherein the controller selects any one of the plurality of display modes in response to a command from the viewer.

16. The apparatus according to claim 1, of a see-through type allowing the viewer to perceive the image representative of the display object such that the image is overlaid onto a real outside world.

17. The apparatus according to claim 1, of a retinal scanning type in which a beam of light is scanned on the retina of the viewer two-dimensionally, to thereby allow the viewer to perceive the image,

wherein the modulator includes a wavefront-curvature modulating unit modulating a curvature of wavefront of the beam of light entering from the apparatus into a pupil of the viewer, on a frame-by-frame basis, for an image to be displayed which is made up of a succession of image frames, or on a sub-area-by-sub-area basis, for an image to be displayed which is made up of a succession of image frames, each of which is divided into a plurality of sub-areas.

18. The apparatus according to claim 1, wherein the image includes a plurality of objects which are displayed together in a stereoscopic manner, using a plurality of sets of image information including depth-position information for specifying a depth position of each object, and position-on-plane information for specifying a position of each object on a plane corresponding to the depth position of the same object, the apparatus further comprising:

a selector selecting one of the plurality of objects as a specified object; and

an image processor performing image processing for modifying a display format in which the image is displayed, based on the depth-position information for the specified object which has been selected by the selector.

19. The apparatus according to claim 18, wherein the image processor performs the image processing such that the specified object which has been selected by the selector is displayed in a display format different from that of at least one non-specified object which is at least one remainder of the plurality of objects.

20. The apparatus according to claim 18, wherein the selector includes a specified-object detector detecting as the specified object a focused object which is one of the plurality of objects that the viewer is focusing on.

21. The apparatus according to claim 20, wherein the specified-object detector includes:

a view-direction detector detecting a view direction of the viewer; and

a determining unit retrieving one of the plurality sets of image information which corresponds to the detected view direction, based on a detection result of the view-direction detector, the determining unit further determining the specified object based on the retrieved set of image information.

22. The apparatus according to claim 20, wherein the specified-object detector includes:

a view-direction detector detecting view directions of the viewer for left and right eyes of the viewer, respectively; and

a calculator calculating a point at which the viewer has gazed, based on both the view directions for the left and right eyes detected by the view-direction detector, and a distance between the left and right eyes, the calculator further detecting the specified object based on a calculation result of the calculator.

23. The apparatus according to claim 18, further comprising a depth-position detector detecting the depth position of each object, based on the depth-position information,

wherein the image processor performs the image processing such that the specified object, optionally with at least one separate object which is displayed at the same depth position as that of the specified object, is displayed in a display format different from that of at least one non-specified object which is displayed at a depth position different from that of the specified object.

24. The apparatus according to claim 18, wherein the image processor includes a section of varying a luminance of the specified object.

25. The apparatus according to claim 18, wherein the image processor includes a section of highlighting a boundary of the specified object.

26. The apparatus according to claim 18, wherein the image processor includes a section of adding a separate boundary image to the specified object along an outline of the specified object.

27. The apparatus according to claim 23, wherein the image processor includes a section of displaying transparently or semi-transparently a non-specified object located in front of the specified object.

4. An apparatus for projecting light onto a retina of a viewer, to thereby allow the viewer to perceive a display object via a virtual image, the apparatus comprising:

an emitter emitting light;

a modulator modulating the light;

a display unit outputting the modulated light to the retina of the viewer through an exit of the display unit, to thereby display in an image display region an image representative of the display object in the form of a virtual image, the display unit being mounted on a head of the viewer in operation; and

a controller controlling the emitter and the modulator so that the image is displayed in the image display region;

wherein the controller detects a relative motion of the viewer with respect to a position of the viewer by treating the position of the viewer as a position reference for display, performs display control which includes at least one of control of a display/non-display status of the image, and control of a display position of the image, in accordance with the detected relative motion, and activates the display control, upon satisfaction of the motion of the viewer with a predetermined condition, based on the detected relative.

7. An apparatus for projecting light onto a retina of a viewer, to thereby allow the viewer to perceive a display object via a virtual image, the apparatus comprising:

an emitter emitting light;

a modulator modulating the light;

a display unit outputting the modulated light to the retina of the viewer through an exit of the display unit, to thereby display in an image display region an image representative of the display object in the form of a virtual image, the display unit being mounted on a head of the viewer in operation; and

a controller that;

controls the emitter and the modulator so that the image is displayed in the image display region;

detects a relative motion of the viewer with respect to a position of the viewer by treating the position of the viewer as a position reference for display, and performs display control which includes at least one of control of a display/non-display status of the image, and control of a display position of the image, in accordance with the detected relative motion;

uses, for allowing the viewer to perceive the image in the image display region, (a) a definition coordinate system for use in defining the display object, and (b) a display coordinate system for use in defining the image to be displayed in the image display region, the display coordinate system being fixed with respect to the display unit;

detects a relative relationship between a position and an orientation of the head of the viewer and a reference coordinate system which is predetermined so as to be related to the human body;

converts definition data which defines the display object with respect to the definition coordinate system, into display data for displaying the image with respect to the display coordinate system, so as to reflect the detected relative relationship; and

controls the emitter and the modulator based on the display data.

8. The apparatus according to claim 7, wherein the controller converts display-object data defining the display object with respect to the definition coordinate system, into the display data defined in the display coordinate system, and

wherein the definition coordinate system is fixed with respect to the display coordinate system.

9. The apparatus according to claim 7, wherein the controller converts display-object data defining the display object with respect to the definition coordinate system, into the display data defined in the display coordinate system, and

wherein the definition coordinate system is fixed with respect to the reference coordinate system.

10. The apparatus according to claim 7, wherein the controller converts display-object data defining the display object with respect to the definition coordinate system, into the display data defined in the display coordinate system, and

wherein the definition coordinate system is fixed with respect to a third coordinate system which is different from the reference coordinate system and the display coordinate system, and which is related to any one of a plurality of segments constituting the human body of the viewer.

11. The apparatus according to claim 7, wherein the reference coordinate system is defined using any one of (a) a combination of at least one set of position information indicative of at least one position of at least one point related to the human body, and at least two sets of orientation information indicative of orientations related to the at least one point;

(b) a combination of at least two sets of position information indicative of positions of at least two points related to the human body, and at least one set of orientation information indicative of at least one orientation related to at least one of the at least two points; and

(c) a combination of at least three sets of position information indicative of positions of at least three points related to the human body.

12. The apparatus according to claim 7, wherein the reference coordinate system is defined using three sets of position information indicative of positions of three points related to a unitary segment of the human body.

13. The apparatus according to claim 7, wherein the reference coordinate system is defined using both two sets of position information indicative of positions of two points related to a unitary segment of the human body, and orientation information indicative of a direction in which gravity acts on one point related to the unitary segment.

28. A method of projecting light onto a retina of a viewer, to thereby allow the viewer to perceive a display object via a virtual image, the method comprising the steps of:

emitting light by an emitter;

modulating the light by a modulator;

causing a display unit to output the modulated light to the retina of the viewer through an exit of the display unit, to thereby display in an image display region an image representative of the display object in the form of a virtual image, the display unit being mounted on a head of the viewer in operation;

detecting a relative position of a to-be-detected segment of a human body of the viewer, with respect to a reference segment of the human body, the to-be-detected segment being selected from a plurality of segments constituting the human body which are displaceable relative to each other, for detection of a motion of the to-be-detected segment, and the reference segment being selected from the plurality of segments for establishment of a reference position related to the human body;

controlling the emitter and the modulator so that the image is displayed in the image display region; and

performing display control which includes at least one of control of a display/non-display status of the image, and control of a display position of the image, based on the detected relative position.