Virtual Window

• US 20150319429A1
• Filed: 01/25/2015
• Published: 11/05/2015
• Est. Priority Date: 08/04/2012
• Status: Active Grant

• Current Assignee: Vergent Research Pty., Ltd.
• Original Assignee: Paul Lapstun
• Inventors: Lapstun, Paul
• Granted Patent: US 10,311,768 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 26/08 for controlling the direction of light in light guides G02B6/35
• CPC Class Codes: G02B 26/0833 the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD G02B26/0825 takes precedence; micromechanical devices in general B81B
• CPC Class Codes: G02B 26/085 the reflecting means being moved or deformed by electromagnetic means
• CPC Class Codes: G02B 26/0875 by means of one or more refracting elements
• CPC Class Codes: G02B 26/10 Scanning systems
• CPC Class Codes: G02B 26/101 with both horizontal and vertical deflecting means, e.g. raster or XY scanners colour television using laser beams scanning a display screen H04N9/3129
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 30/10 using integral imaging methods
• CPC Class Codes: G02B 30/27 involving lenticular arrays
• CPC Class Codes: G02F 1/11 based on acousto-optical elements, e.g. using variable diffraction by sound or like mechanical waves acousto-optical deflection G02F1/33
• CPC Class Codes: G02F 1/29 for the control of the position or the direction of light beams, i.e. deflection
• CPC Class Codes: G02F 1/294 Variable focal length devices
• CPC Class Codes: G02F 1/33 Acousto-optical deflection devices circuit or control arrangements therefor G02F1/113
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2320/064 by time modulation of the brightness of the illumination source
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/141 the light conveying information used for selecting or modulating the light emitting or modulating element
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 3/02 by tracing or scanning a light beam on a screen
• CPC Class Codes: G09G 5/10 : Intensity circuits
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/302 : for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/56 : provided with illuminating means
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/67 : Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 25/745 : Circuitry for generating timing or clock signals

Abstract

A virtual window system, the system comprising a light field camera device and a light field display device arranged back-to-back, the light field camera device configured to capture an input light field video stream, the light field display device configured to display an output light field video stream based on the input light field video stream.

Claims

1. A virtual window system, the system comprising a light field camera device and a light field display device arranged back-to-back, the light field camera device configured to capture an input light field video stream, the light field display device configured to display an output light field video stream based on the input light field video stream.

2. The system of claim 1, wherein the light field camera device comprises at least one input focus modulator, whereby the focus of at least part of the input light field video stream is modulated according to a fixation depth of a viewer of the light field display device.

3. The system of claim 1, wherein the light field display device comprises at least one output focus modulator, whereby the focus of at least part of the output light field video stream is modulated according to a fixation depth of a viewer of the light field display device.

4. The system of claim 1, wherein the light field camera device comprises at least one input focus modulator and the light field display device comprises at least one output focus modulator, whereby the focus of at least part of the input light field video stream and the focus of at least part of the output light field video stream are modulated, respectively, according to a fixation depth of a viewer of the light field display device.

5. The system of claim 1, wherein the input light field video stream is resampled to translate its light field boundary to compensate for the separation of the light field camera device and the light field display device, to produce the output light field video stream.

6. The system of claim 1, wherein the input light field video stream is incorporated into a rendered light field video stream to produce the output light field video stream, the rendered light field video stream comprising at least one virtual 3D object.

7. The system of claim 6, wherein the rendered light field video stream comprises at least one virtual 3D object which interacts with part of the input light field video stream, the interaction selected from the group comprising:

reflection and refraction.

8. The system of claim 1, wherein the light field camera device comprises an array of light field camera elements, each camera element comprising:

(a) an array of radiance sensors for sensing the radiance of a set of input beams over time, the input beams spanning a two-dimensional angular field; and

(b) an input focus modulator for modulating the focus of the input beams over time.

9. The system of claim 1, wherein the light field camera device comprises an array of light field camera elements, each camera element comprising:

(a) a scanner for scanning an input beam across a two-dimensional angular field;

(b) an input focus modulator for modulating the focus of the input beam over time; and

(c) a radiance sensor for sensing the radiance of the input beam over time.

10. The system of claim 1, wherein the light field display device comprises an array of light field display elements, each display element comprising:

(a) an array of beam generators and radiance modulators for generating and modulating the radiance of a set of output beams over time, the output beams spanning a two-dimensional angular field; and

(b) an output focus modulator for modulating the focus of the output beams over time.

11. The system of claim 1, wherein the light field display device comprises an array of light field display elements, each display element comprising:

(a) a beam generator for generating an output beam of light;

(b) a radiance modulator for modulating the radiance of the output beam over time;

(c) an output focus modulator for modulating the focus of the output beam over time; and

(d) a scanner for scanning the output beam across a two-dimensional angular field.

12. A virtual window system, the system comprising two two-way light field camera and display devices arranged back-to-back, each device configured to capture an input light field video stream and to display an output light field video stream based on the input light field video stream from the other device.

13. The system of claim 12, wherein at least one of the two-way devices comprises at least one input focus modulator, whereby the focus of at least part of its input light field video stream is modulated according to a fixation depth of a viewer of the other two-way device.

14. The system of claim 12, wherein at least one of the two-way devices comprises at least one output focus modulator, whereby the focus of at least part of its output light field video stream is modulated according to a fixation depth of a viewer of the two-way device.

15. The system of claim 12, wherein at least one of the two-way devices comprises at least one input focus modulator and the other two-way device comprises at least one output focus modulator, whereby the focus of at least part of its input light field video stream and the focus of at least part of the output light field video stream of the other device are modulated, respectively, according to a fixation depth of a viewer of the other two-way device.

16. The system of claim 12, wherein at least one of the input light field video streams is resampled to translate its light field boundary to compensate for the separation of the two-way devices, to produce the corresponding output light field video stream.

17. The system of claim 12, wherein at least one of the input light field video streams is incorporated into a rendered light field video stream to produce the corresponding output light field video stream, the rendered light field video stream comprising at least one virtual 3D object.

18. The system of claim 17, wherein the rendered light field video stream comprises at least one virtual 3D object which interacts with part of the corresponding input light field video stream, the interaction selected from the group comprising:

reflection and refraction.

19. The system of claim 12, wherein at least one of the two-way devices comprises an array of two-way camera and display elements, each two-way element comprising:

(a) an array of radiance sensors for sensing the radiance of a corresponding set of input beams over time, the input beams spanning a two-dimensional angular field;

(b) an input focus modulator for modulating the focus of the input beams over time;

(c) an array of beam generators and radiance modulators for generating and modulating the radiance of a set of output beams over time, the output beams spanning a two-dimensional angular field; and

(d) an output focus modulator for modulating the focus of the output beams over time.

20. The system of claim 12, wherein at least one of the two-way devices comprises an array of two-way camera and display elements, each two-way element comprising:

(a) a scanner for scanning an input beam and an output beam across a two-dimensional angular field;

(b) an input focus modulator for modulating the focus of the input beam over time;

(c) a radiance sensor for sensing the radiance of the input beam over time;

(d) a beam generator for generating the output beam of light;

(e) a radiance modulator for modulating the radiance of the output beam over time; and

(f) an output focus modulator for modulating the focus of the output beam over time.