Methods, systems, and computer readable media for generating an augmented scene display

• US 9,858,721 B2
• Filed: 01/15/2014
• Issued: 01/02/2018
• Est. Priority Date: 01/15/2013
• Status: Active Grant

• Current Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Original Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Inventors: Maimone, Andrew, Fuchs, Henry
• Primary Examiner(s): Chu, David H
• Application Number:
  US14/761,229
• Publication Number:
  US 20150363978A1
• Time in Patent Office: 1,448 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

The subject matter described herein includes systems, methods, and computer readable media for generating an augmented scene display. An exemplary method includes forming, using a display device operating in a first stage, an augmented virtual image by emitting light rays through a plurality of spatial light modulation layers included in a display device. The method also includes forming, using the display device operating in a second stage, an occluded real image by opening a shutter element of the display device to receive light rays from a real object and utilizing the plurality of spatial light modulation layers to block any light ray from the real object which coincides with the augmented virtual image. The method further includes generating an augmented scene display that includes both the occluded real image and the augmented virtual image by alternating the operation of the display device between the first stage and the second stage.

Claims

1. A method for generating an augmented scene display, the method comprising:

forming, using a display device operating in a first stage, an augmented virtual image by emitting light rays through a plurality of spatial light modulation layers included in the display device;

forming, using the display device operating in a second stage, an occluded real image by opening a shutter element of the display device to receive light rays from a real object and utilizing the plurality of spatial light modulation layers to block any light ray from the real object which coincides with the augmented virtual image; and

generating an augmented scene display that includes both the occluded real image and the augmented virtual image by alternating the operation of the display device between the first stage and the second stage.

2. The method of claim 1 wherein the plurality of spatial light modulation layers includes a plurality of transparent liquid crystal displays (LCDs).

3. The method of claim 1 wherein the display device includes a head-worn device.

4. The method of claim 1 wherein the plurality of spatial light modulation layers is configured to attenuate the emitted light rays.

5. The method of claim 1 wherein the emitted light rays used to form an augmented virtual image are generated by a backlight element included in the display device.

6. The method of claim 5 wherein the first stage includes a virtual image formation phase wherein the light rays are emitted from a backlight element in the display device and passes through patterns displayed on the plurality of spatial light modulation layers to form the augmented virtual image while the light rays from the real object are blocked by the shutter element.

7. The method of claim 6 wherein the second stage includes an occluded real-image formation phase wherein the shutter element is opened and a backlight element in the display device is disabled.

8. The method of claim 1 wherein the shutter element is configured to be switched from a globally opaque state in the first stage to a globally transparent state in the second stage.

9. The method of claim 8 wherein the shutter element includes a single liquid crystal cell.

10. The method of claim 5 wherein the backlight includes a transparent organic light emitting diode (OLED) panel.

11. The method of claim 1 wherein the first stage and second state are alternated beyond a flicker fusion threshold.

12. The method of claim 1 wherein the plurality of spatial light modulation layers are constrained to allow a focused image to be formed on an eye or a camera when the display device is positioned closer than the focal distance of the eye or the camera.

13. A system for generating an augmented scene display, the system comprising:

a display device that includes;

a backlight element configured to emit light rays in a first stage;

a shutter element configured to open and allow light rays from a real object to be received by the display device in a second stage; and

a plurality of spatial light modulation layers configured to form, in the first stage, an augmented virtual image by receiving and attenuating the emitted light rays and to form, in the second stage, an occluded real object image by blocking any light ray from the real object that coincides with the augmented virtual image, and to generate an augmented scene display that includes both the occluded real image and the augmented virtual image by alternating between the first stage and the second stage.

14. The system of claim 13 wherein the plurality of spatial light modulation layers includes a plurality of transparent liquid crystal displays (LCDs).

15. The system of claim 13 wherein the display device includes a head-worn device.

16. The system of claim 13 wherein the emitted light rays used to form an augmented virtual image are generated by the backlight element.

17. The system of claim 16 wherein the first stage includes a virtual image formation phase wherein the light rays emitted from the backlight element pass through patterns displayed on the plurality of spatial light modulation layers to form the augmented virtual image while the light rays from the real object are blocked by the shutter element.

18. The system of claim 17 wherein the second stage includes an occluded real-image formation phase wherein the shutter element is opened and the backlight element is disabled.

19. The system of claim 13 wherein the shutter element is configured to be switched from a globally opaque state in the first stage to a globally transparent state in the second stage.

20. The system of claim 19 wherein the shutter element includes a single liquid crystal cell.

21. The system of claim 13 wherein the backlight includes a transparent organic light emitting diode (OLED) panel.

22. The system of claim 13 wherein the first stage and second state are alternated beyond a flicker fusion threshold.

23. The system of claim 13 wherein the plurality of spatial light modulation layers are constrained to allow a focused image to be formed on an eye or a camera when the display device is positioned closer than the focal distance of the eye or the camera.

24. A method for generating an augmented scene display, the method comprising:

forming, using a display device operating in a first stage, an augmented virtual image by emitting light rays from a transparent light emissive display through a first plurality of spatial light modulation layers included in front of the transparent light emissive display, wherein the light rays are emitted by the transparent light emissive display only in display regions corresponding to the augmented virtual image;

forming, using the display device operating in a second stage, an occluded real image by setting to an initial clear state a second plurality of spatial light modulation layers of the display device located behind the transparent light emissive display to receive light rays from a real object and utilizing at least one of the second plurality of spatial light modulation layers to block any light ray from the real object which coincides with the augmented virtual image; and

generating an augmented scene display that includes both the occluded real image and the augmented virtual image by conducting the first stage and the second stage contemporaneously.

25. The method of claim 24 wherein the transparent light emissive display includes a transparent organic light emitting diode (OLED) panel.

26. A system for generating an augmented scene display, the system comprising:

a display device that includes;

a transparent light emissive display configured to emit light rays in a first stage;

a first plurality of spatial light modulation layers configured to form, in the first stage, an augmented virtual image by receiving and attenuating the emitted light rays;

a second plurality of spatial light modulation layers configured to, in a second stage, an initial clear state to allow light rays from a real object to be received by the display device in a second stage and to form an occluded real object image by blocking any light ray from the real object that coincides with the augmented virtual image; and

wherein the display device generates an augmented scene display that includes both the occluded real image and the augmented virtual image by conducting the first stage and the second stage contemporaneously.

27. The system of claim 26 wherein the transparent light emissive display includes a transparent organic light emitting diode (OLED) panel.

28. A non-transitory computer readable medium having stored thereon executable instructions that when executed by the processor of a computer control the computer to perform steps comprising:

forming, using a display device operating in a first stage, an augmented virtual image by emitting light rays through a plurality of spatial light modulation layers included in the display device;

forming, using the display device operating in a second stage, an occluded real image by opening a shutter element of the display device to receive light rays from a real object and utilizing the plurality of spatial light modulation layers to block any light ray from the real object which coincides with the augmented virtual image; and

generating an augmented scene display that includes both the occluded real image and the augmented virtual image by alternating the operation of the display device between the first stage and the second stage.

29. A non-transitory computer readable medium having stored thereon executable instructions that when executed by the processor of a computer control the computer to perform steps comprising:

forming, using a display device operating in a first stage, an augmented virtual image by emitting light rays from a transparent light emissive display through a first plurality of spatial light modulation layers included in front of the transparent light emissive display, wherein the light rays are emitted by the transparent light emissive display only in display regions corresponding to the augmented virtual image;

forming, using the display device operating in a second stage, an occluded real image by setting to an initial clear state a second plurality of spatial light modulation layers of the display device located behind the transparent light emissive display to receive light rays from a real object and utilizing at least one of the second plurality of spatial light modulation layers to block any light ray from the real object which coincides with the augmented virtual image; and

generating an augmented scene display that includes both the occluded real image and the augmented virtual image by conducting the first stage and the second stage contemporaneously.