COMBINING VIDEO-BASED AND OPTIC-BASED AUGMENTED REALITY IN A NEAR EYE DISPLAY

• US 20170178375A1
• Filed: 03/23/2016
• Published: 06/22/2017
• Est. Priority Date: 03/24/2015
• Status: Active Grant

• Current Assignee: Augmedics, Ltd.
• Original Assignee: Augmedics, Ltd.
• Inventors: Benishti, Nessi, Elimelech, Nissan, Wolf, Stuart, Gerland, Elazar
• Granted Patent: US 9,928,629 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Apparatus, including a retaining structure (54), positioned near a subject'"'"'s eye that has a pupil with a diameter, an optical combiner (52A) mounted on the structure before the eye, and a pixelated screen (60A) having an array of variably transparent pixels coating the combiner. There is an image capturing device (68A) mounted on the structure to capture an image of a scene viewed by the eye, and a projector (64A) is mounted on the structure to project at least one of a portion of the captured image and a stored image onto a section of the screen at a selected location thereof. A processor (26) renders the screen section at least partially opaque, selects the section location in response to a region of interest in the scene identified by analysis of the captured image, and determines a dimension of the section in response to the pupil diameter.

Claims

1. Apparatus, comprising:

a retaining structure, configured to be positioned in proximity to an eye of a subject, the eye of the subject having a pupil with a pupil diameter;

an optical combiner mounted on the structure in front of the eye;

a pixelated screen, comprising an array of variably transparent pixels, coating the optical combiner;

at least one image capturing device mounted on the structure configured to capture an image of a scene viewed by the eye;

a projector mounted on the structure and configured to project at least one of a portion of the captured image and a stored image onto a section of the screen at a selected location thereof; and

a processor, configured to render the section of the screen at least partially opaque, to select the location of the section in response to a region of interest in the scene identified by analysis of the captured image, and to determine a dimension of the section in response to the pupil diameter.

2. The apparatus according to claim 1, wherein the processor is configured to identify the region of interest in response to radiation received by the image capturing device from at least one marker located at the region of interest.

3. The apparatus according to claim 1, and comprising a further image capturing device configured to identify the region of interest in response to received radiation from at least one marker located at the region of interest.

4. The apparatus according to claim 3, wherein the at least one image capturing device is configured to operate in the visible spectrum, and wherein the further image capturing device is configured to operate in the non-visible spectrum.

5. The apparatus according to claim 4, and comprising a radiator configured to radiate radiation in the non-visible spectrum towards the region of interest.

6. The apparatus according to claim 1, and comprising at least one marker positioned in proximity to the region of interest, and wherein the processor is configured to detect the marker in the captured image so as to identify the region of interest.

7. The apparatus according to claim 1, wherein the processor is configured to determine an initial pupil diameter in response to the dimension of the section being set by the subject to occlude an object of known size while the subject gazes at the object in a known ambient light brightness.

8. The apparatus according to claim 7, wherein the processor is configured to determine a brightness of the scene in response to the captured image of the scene, and wherein the processor is configured to determine the pupil diameter in response to the initial pupil diameter and the brightness of the scene.

9. The apparatus according to claim 1, wherein the processor is configured to determine an initial pupil diameter in response to analysis of a reflected image of the subject while the subject gazes into a mirror in a known ambient light brightness.

10. The apparatus according to claim 9, wherein the processor is configured to determine a brightness of the scene in response to the captured image of the scene, and wherein the processor is configured to determine the pupil diameter in response to the initial pupil diameter and the brightness of the scene.

11. The apparatus according to claim 1, wherein the dimension of the section is determined so that the region of interest is occluded.

12. The apparatus according to claim 11, wherein a region surrounding the region of interest is partially occluded.

13. The apparatus according to claim 12, wherein a fraction of occlusion in the region surrounding the region of interest is determined in response to the pupil diameter.

14. The apparatus according to claim 12, wherein the processor is configured to derive from the captured image an image corresponding to the region surrounding the region of interest, and wherein the projector is configured to project the derived image onto an area of the screen surrounding the at least partially opaque section of the screen.

15. The apparatus according to claim 14, wherein an intensity of the projected derived image is determined in response to the fraction of occlusion.

16. The apparatus according to claim 1, wherein the dimension of the section is determined in response to a size of the region of interest.

17. The apparatus according to claim 1, wherein the dimension of the section is determined so that an area greater than the region of interest is occluded.

18. The apparatus according to claim 1, wherein the dimension of the section is determined so that an area less than the region of interest is occluded.

19. The apparatus according to claim 1, wherein the retaining structure comprises a spectacle frame.

20. The apparatus according to claim 1, wherein the retaining structure comprises a helmet having a head-up display.

21. The apparatus according to claim 1, wherein the at least one image capturing device comprises two image capturing devices capturing respective images of the scene, and wherein the processor is configured to identify the region of interest by analysis of the respective images.

22. A method, comprising:

positioning a retaining structure in proximity to an eye of a subject, the eye of the subject having a pupil with a pupil diameter;

mounting an optical combiner on the structure in front of the eye;

coating the optical combiner with a pixelated screen, comprising an array of variably transparent pixels;

mounting at least one image capturing device on the structure so as to capture an image of a scene viewed by the eye;

mounting a projector on the structure the projector being configured to project at least one of a portion of the captured image and a stored image onto a section of the screen at a selected location thereof;

rendering the section of the screen at least partially opaque;

selecting the location of the section in response to a region of interest in the scene identified by analysis of the captured image; and

determining a dimension of the section in response to the pupil diameter.

23. The method according to claim 22, and comprising identifying the region of interest in response to radiation received by the image capturing device from at least one marker located at the region of interest.

24. The method according to claim 22, and comprising configuring a further image capturing device to identify the region of interest in response to received radiation from at least one marker located at the region of interest.

25. The method according to claim 24, wherein the at least one image capturing device is configured to operate in the visible spectrum, and wherein the further image capturing device is configured to operate in the non-visible spectrum.

26. The method according to claim 25, and comprising radiating radiation in the non-visible spectrum towards the region of interest.

27. The method according to claim 22, and comprising positioning at least one marker in proximity to the region of interest, and detecting the marker in the captured image so as to identify the region of interest.

28. The method according to claim 22, and comprising determining an initial pupil diameter in response to the dimension of the section being set by the subject to occlude an object of known size while the subject gazes at the object in a known ambient light brightness.

29. The method according to claim 28, and comprising determining a brightness of the scene in response to the captured image of the scene, and determining the pupil diameter in response to the initial pupil diameter and the brightness of the scene.

30. The method according to claim 22, and comprising determining an initial pupil diameter in response to analysis of a reflected image of the subject while the subject gazes into a mirror in a known ambient light brightness.

31. The method according to claim 30, and comprising determining a brightness of the scene in response to the captured image of the scene, determining the pupil diameter in response to the initial pupil diameter and the brightness of the scene.

32. The method according to claim 22, wherein the dimension of the section is determined so that the region of interest is occluded.

33. The method according to claim 22, wherein a region surrounding the region of interest is partially occluded.

34. The method according to claim 33, wherein a fraction of occlusion in the region surrounding the region of interest is determined in response to the pupil diameter.

35. The method according to claim 33, and comprising deriving from the captured image an image corresponding to the region surrounding the region of interest, and projecting the derived image onto an area of the screen surrounding the at least partially opaque section of the screen.

36. The method according to claim 35, wherein an intensity of the projected derived image is determined in response to the fraction of occlusion.

37. The method according to claim 22, wherein the dimension of the section is determined in response to a size of the region of interest.

38. The method according to claim 22, wherein the dimension of the section is determined so that an area greater than the region of interest is occluded.

39. The method according to claim 22, wherein the dimension of the section is determined so that an area less than the region of interest is occluded.

40. The method according to claim 22, wherein the retaining structure comprises a spectacle frame.

41. The method according to claim 22, wherein the retaining structure comprises a helmet having a head-up display.

42. The method according to claim 22, wherein the at least one image capturing device comprises two image capturing devices capturing respective images of the scene, and comprising identifying the region of interest by analysis of the respective images.

43. Apparatus, comprising:

a retaining structure, configured to be positioned in proximity to an eye of a subject;

an optical combiner mounted on the structure in front of the eye;

a pixelated screen, comprising an array of variably transparent pixels, coating the optical combiner;

at least one image capturing device mounted on the structure configured to capture an image of a scene viewed by the eye;

a processor, configured to render a section of the screen at least partially opaque, and

a projector mounted on the structure and configured to project at least one of a portion of the captured image and a stored image onto the section of the screen so that there is misalignment between the scene viewed by the eye through the combiner and the at least one portion of the captured image and the stored image.

44. The apparatus according to claim 43, wherein for a scene at 50 cm from the eye, the misalignment is no more than 2 cm.

45. The apparatus according to claim 43, wherein the projector is configured to project the portion of the captured image and the stored image, in registration with each other, onto the section of the screen.

46. Apparatus, comprising:

a retaining structure, configured to be positioned in proximity to an eye of a subject;

an optical combiner mounted on the structure in front of the eye;

a rotator connected to the optical combiner and configured to rotate the optical combiner about an axis;

a pixelated screen, comprising an array of variably transparent pixels, coating the optical combiner;

at least one image capturing device mounted on the structure configured to capture an image of a scene viewed by the eye; and

a processor, configured to render a section of the screen at least partially opaque, and to activate the rotator so that the optical combiner is oriented to be orthogonal to a region of interest in the scene.

47. The apparatus according to claim 46, wherein the processor is configured to select the section of the screen so as to occlude the region of interest.

48. The apparatus according to claim 46, wherein the axis is a vertical axis.

49. A method, comprising:

positioning a retaining structure in proximity to an eye of a subject;

mounting an optical combiner on the structure in front of the eye;

coating the optical combiner with a pixelated screen comprising an array of variably transparent pixels;

mounting at least one image capturing device on the structure, the device being configured to capture an image of a scene viewed by the eye;

rendering a section of the screen at least partially opaque;

mounting a projector on the structure; and

configuring the projector to project at least one of a portion of the captured image and a stored image onto the section of the screen so that there is misalignment between the scene viewed by the eye through the combiner and the at least one portion of the captured image and the stored image.

50. The method according to claim 49, wherein for a scene at 50 cm from the eye, the misalignment is no more than 2 cm.

51. The method according to claim 49, wherein the projector is configured to project the portion of the captured image and the stored image, in registration with each other, onto the section of the screen.

52. A method, comprising:

positioning a retaining structure in proximity to an eye of a subject;

mounting an optical combiner on the structure in front of the eye;

connecting a rotator to the optical combiner, the rotator being configured to rotate the optical combiner about an axis;

coating the optical combiner with a pixelated screen comprising an array of variably transparent pixels;

mounting at least one image capturing device on the structure, the device being configured to capture an image of a scene viewed by the eye;

rendering a section of the screen at least partially opaque; and

activating the rotator so that the optical combiner is oriented to be orthogonal to a region of interest in the scene.

53. The method according to claim 52, the section of the screen occludes the region of interest.

54. The method according to claim 52, wherein the axis is a vertical axis.