Virtual window

• US 10,311,768 B2
• Filed: 01/25/2015
• Issued: 06/04/2019
• Est. Priority Date: 08/04/2012
• Status: Active Grant

• Current Assignee: Vergent Research Pty., Ltd.
• Original Assignee: Paul Lapstun
• Inventors: Lapstun, Paul
• Primary Examiner(s): Zhou, Hong
• Application Number:
  US14/604,750
• Publication Number:
  US 20150319429A1
• Time in Patent Office: 1,591 Days
• CPC Class Codes: G02B 26/08 for controlling the direction of light in light guides G02B6/35
• CPC Class Codes: G02B 26/0833 the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD G02B26/0825 takes precedence; micromechanical devices in general B81B
• CPC Class Codes: G02B 26/085 the reflecting means being moved or deformed by electromagnetic means
• CPC Class Codes: G02B 26/0875 by means of one or more refracting elements
• CPC Class Codes: G02B 26/10 Scanning systems
• CPC Class Codes: G02B 26/101 with both horizontal and vertical deflecting means, e.g. raster or XY scanners colour television using laser beams scanning a display screen H04N9/3129
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 30/10 using integral imaging methods
• CPC Class Codes: G02B 30/27 involving lenticular arrays
• CPC Class Codes: G02F 1/11 based on acousto-optical elements, e.g. using variable diffraction by sound or like mechanical waves acousto-optical deflection G02F1/33
• CPC Class Codes: G02F 1/29 for the control of the position or the direction of light beams, i.e. deflection
• CPC Class Codes: G02F 1/294 Variable focal length devices
• CPC Class Codes: G02F 1/33 Acousto-optical deflection devices circuit or control arrangements therefor G02F1/113
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2320/064 by time modulation of the brightness of the illumination source
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/141 the light conveying information used for selecting or modulating the light emitting or modulating element
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 3/02 by tracing or scanning a light beam on a screen
• CPC Class Codes: G09G 5/10 : Intensity circuits
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/302 : for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/56 : provided with illuminating means
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/67 : Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 25/745 : Circuitry for generating timing or clock signals

Abstract

A virtual window system, the system comprising a light field camera device and a light field display device arranged back-to-back, the light field camera device configured to capture an input light field video stream, the light field display device configured to display an output light field video stream based on the input light field video stream.

Claims

1. A virtual window system, the system comprising a light field camera device and a light field display device arranged back-to-back, the light field camera device configured to sample a continuous input light field at an input light field boundary to produce a discrete input light field video stream, the light field display device configured to reconstruct a continuous output light field at an output light field boundary from a discrete output light field video stream, wherein the system is configured to resample the discrete input light field video stream to translate it from the input light field boundary to the output light field boundary to produce the discrete output light field video stream, as if the discrete output light field video stream had been sampled from the continuous input light field at the output light field boundary.

2. The system of claim 1, wherein the light field camera device comprises at least one input focus modulator, whereby the focus of at least part of the input light field video stream is modulated according to a fixation depth of a viewer of the light field display device.

3. The system of claim 1, wherein the light field display device comprises at least one output focus modulator, whereby the focus of at least part of the output light field video stream is modulated according to a fixation depth of a viewer of the light field display device.

4. The system of claim 1, wherein the light field camera device comprises at least one input focus modulator and the light field display device comprises at least one output focus modulator, whereby the focus of at least part of the input light field video stream and the focus of at least part of the output light field video stream are modulated, respectively, according to a fixation depth of a viewer of the light field display device.

5. The system of claim 1, wherein the input light field video stream is incorporated into a rendered light field video stream to produce the output light field video stream, the rendered light field video stream comprising at least one virtual 3D object.

6. The system of claim 5, wherein the rendered light field video stream comprises at least one virtual 3D object which interacts with part of the input light field video stream, the interaction selected from the group comprising:

reflection and refraction.

7. The system of claim 1, wherein the light field camera device comprises an array of light field camera elements, each camera element comprising:

(a) an array of radiance sensors for sensing the radiance of a set of input beams over time, the input beams spanning a two-dimensional angular field; and

(b) an input focus modulator for modulating the focus of the input beams over time.

8. The system of claim 1, wherein the light field camera device comprises an array of light field camera elements, each camera element comprising:

(a) a scanner for scanning an input beam across a two-dimensional angular field;

(b) an input focus modulator for modulating the focus of the input beam over time; and

(c) a radiance sensor for sensing the radiance of the input beam over time.

9. The system of claim 1, wherein the light field display device comprises an array of light field display elements, each display element comprising:

(a) an array of beam generators and radiance modulators for generating and modulating the radiance of a set of output beams over time, the output beams spanning a two-dimensional angular field; and

(b) an output focus modulator for modulating the focus of the output beams over time.

10. The system of claim 1, wherein the light field display device comprises an array of light field display elements, each display element comprising:

(a) a beam generator for generating an output beam of light;

(b) a radiance modulator for modulating the radiance of the output beam over time;

(c) an output focus modulator for modulating the focus of the output beam over time; and

(d) a scanner for scanning the output beam across a two-dimensional angular field.

11. A method of providing a virtual window through a space, the window having an input light field boundary and an output light field boundary, the light field boundaries separated by the space, the method comprising:

(a) sampling, using a light field camera device, a continuous input light field at the input light field boundary to produce a discrete input light field video stream;

(b) resampling the discrete input light field video stream to translate it from the input light field boundary to the output light field boundary to produce a discrete output light field video stream, as if the discrete output light field video stream had been sampled from the continuous input light field at the output light field boundary; and

(c) reconstructing, using a light field display device, a continuous output light field at the output light field boundary from the discrete output light field video stream.