METHOD AND APPARATUS TO REALIZE VIRTUAL REALITY

• US 20170264879A1
• Filed: 05/22/2017
• Published: 09/14/2017
• Est. Priority Date: 01/24/2013
• Status: Active Grant

• Current Assignee: Yuchen Zhou
• Original Assignee: Yuchen Zhou
• Inventors: Zhou, Yuchen
• Granted Patent: US 10,178,367 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 26/0825 the reflecting element being a flexible sheet or membrane, e.g. for varying the focus flexible mirrors for cosmetic use A45D42/24
• CPC Class Codes: G02B 26/0833 the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD G02B26/0825 takes precedence; micromechanical devices in general B81B
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/322 using varifocal lenses or mirrors
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/341 : using temporal multiplexing
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

The current invention relates to the method and apparatus to determine the focus point of a viewer from a single eye of the viewer in a viewing space. The claimed method detects the focus depth and the line of eye sight from said single eye. It further relates to the method to use the determined focus point to achieve virtual reality and augmented reality.

Claims

1. A method to realize virtual reality by detecting focus point of a first eye of a viewer in a viewing space comprising the steps of:

a probing light being emitted towards said first eye by at least one optical emitter;

said probing light being reflected by said first eye into at least one reflection light that is received by at least one optical detector;

said at least one optical detector detecting a focus depth and a pupil position of said first eye from said at least one reflection light;

a computing device calculating a line of eye sight from said pupil position and determining a spatial position on said line of eye sight as said focus point, wherein distance between said spatial position to said viewer being equal to said focus depth;

an image generator using said focus depth and said focus point to produce a first image; and

an image display displaying said first image to said first eye.

2. The method according to claim 1, wherein said probing light is reflected by cornea of said first eye;

and wherein said focus depth is detected from shape change of said cornea by said at least one optical detector from intensity and angle of said at least one reflection light.

3. The method according to claim 1, wherein said image display is one of:

a screen of an electronic device;

a screen for image projection;

a head-mounted display;

an micro-electro-mechanical-system (MEMS) mirror array reflecting at least one incident light to project image onto retina of said first eye.

4. The method according to claim 1, wherein said image generator includes one of:

a camera, a camcorder, a computer based image generator, a stereo vision generation device.

5. The method according to claim 1, wherein said at least one optical emitter and said at least one optical detector are included in a see-through substrate, wherein said see-through substrate is one of:

embedded in a supporting frame disposed in front of said first eye; and

contacting the pupil of said first eye.

6. The method according to claim 1, wherein said at least one optical detector detects property of said at least one reflection light including one of:

the dispersion of said at least one reflection light;

duration of said at least one reflection light;

delay between pulses of said at least one reflection light; and

direction of said at least one reflection light.

7. The method according to claim 1, wherein said image display is fixed in distance to said first eye;

wherein an optical system is disposed between said image display and said first eye;

wherein an effective optical focus depth of said optical system varies according to a change of said focus depth, thereby making said first image appear focused on retina of said first eye after said change of said focus depth.

8. The method according to claim 7, wherein said optical system includes at least one of:

optical component having electrically controlled effective optical path;

optical component having variable focus depth and being composed of electrically controlled refractive index material;

mechanically positioned single optical lens;

mechanically positioned array of optical lenses;

micro-electro-mechanical-system actuated lens;

mirror array; and

prism array.

9. The method according to claim 1, wherein augmented reality is further achieved by:

an image capture device capturing a second image of said viewing space;

said computing device using said focus point to identify from said second image a first object in said viewing space;

displaying at least one imaginary object by said image display to said first eye exhibiting at least one property of said first object;

said viewer generating a first command and said computing device interpreting said first command into first information;

said computing device transmitting said first information through a communication means; and

said first object receiving said first information and performing at least one action.

10. The method according to claim 9, wherein said first command is generated by one of:

(i) eye gesture being one of;

time of stare by said first eye;

movement of said first eye;

change of eye lips open width of said first eye; and

shift of said focus point in said viewing space;

(ii) body gestures; and

(iii) vocal commands.

11. The method according to claim 1, wherein said spatial position of said focus point is determined by:

said computing device calculating at least one focus circle in said viewing space, wherein said focus circle centers on said viewer and has a radius equal to said focus depth; and

said computing device determining said spatial position as being the intersection point of said line of eye sight and said focus circle in said viewing space.

12. The method according to claim 1, wherein said first image includes a first object and a second object;

wherein said computing device uses said focus point to identify said first object from said first image;

said image generator generating a new image from said first image by defocusing said second object in said new image; and

said image display displays said new image to said first eye.

13. A method to realize virtual reality by detecting focus point of a first eye of a viewer in a viewing space comprising the steps of:

an eye sensor detecting a focus depth and a pupil position of said first eye;

a computing device determining said focus point on a line of eye sight of said first eye by utilizing said focus depth and said pupil position;

an image generator using said focus point to produce a first image;

an image display displaying said first image to said first eye through an optical system disposed in front of said first eye; and

said optical system operating to obtain an effective optical focus depth according to said focus depth of said first eye, thereby making said first image appear focused on retina of said first eye.

14. The method according to claim 13, wherein said optical system includes at least one of:

optical component having electrically controlled effective optical path;

optical component having variable focus depth and being composed of electrically controlled refractive index material;

mechanically positioned single optical lens;

mechanically positioned array of optical lenses;

micro-electro-mechanical-system actuated lens;

mirror array; and

prism array.

15. The method according to claim 13, wherein said first image includes a first object and a second object;

wherein said computing device uses said focus point to identify said first object from said first image;

said image generator generating a new image from said first image by defocusing said second object in said new image; and

said image display displays said new image to said first eye.

16. A method to detect a focus point of a first eye of a viewer in a viewing space comprising the steps of:

an eye sensor detecting a focus depth and a pupil position of said first eye;

a computing device calculating a line of eye sight from said pupil position;

said computing device determining a spatial position on said line of eye sight, wherein distance of said spatial position from said viewer equals to said focus depth; and

said computing device assigning said spatial position as said focus point.

17. The method according to claim 16, wherein augmented reality is realized by:

an image capture device capturing a first image of said viewing space;

said computing device identifying a first object in said viewing space from said first image by said focus point;

said eye sensor sensing a first eye gesture from said first eye;

an image display displaying a first description of a first action by said first object to said first eye after said first eye gesture;

said eye sensor sensing a second eye gesture from said first eye;

said computing device interpreting said first description into first information after said second eye gesture, and sending said first information through a communication means; and

said first object performing said first action after receiving said first information.

18. The method according to claim 17, wherein said first and second eye gestures include at least one of:

time of stare of said first eye;

movement of said first eye;

change of eye lips open width of said first eye; and

shift of said focus point in said viewing space.

19. The method according to claim 16, wherein said eye sensor is included in a see-through substrate, wherein said see-through substrate is one of:

embedded in a supporting frame disposed in front of said first eye; and

contacting the pupil of said first eye.

20. The method according to claim 16, wherein said eye sensor includes an optical emitter and an optical detector;

wherein a probing light is emitted towards said first eye by said optical emitter;

wherein said optical detector receives said probing light being reflected from said first eye by at least one of;

cornea;

outside surface of eye lens;

inside surface of eye lens; and

retina.