Content projection system and content projection method

• US 10,048,750 B2
• Filed: 12/04/2013
• Issued: 08/14/2018
• Est. Priority Date: 08/30/2013
• Status: Active Grant

• Current Assignee: Beijing Zhigu RUI TUO Tech Company Limited
• Original Assignee: Beijing Zhigu RUI TUO Tech Company Limited
• Inventors: Du, Lin, Zhang, Hongjiang
• Primary Examiner(s): Wilson, Nicholas R
• Application Number:
  US14/783,503
• Publication Number:
  US 20160179193A1
• Time in Patent Office: 1,714 Days
• Field of Search: None
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

This application discloses a content projection system and a content projection method. The system comprises: a gaze point detection apparatus, configured to detect a gaze point location of an eye; an image generating apparatus, configured to generate a virtual image according to visual information and the gaze point location; and a virtual image projection apparatus, configured to project the virtual image to the fundus according to the gaze point location. According to the system and method in the embodiments of the present application, the gaze point location of a user is detected in real time, an imaging parameter corresponding to the visual information is obtained with respect to the gaze point location, and therefore, the virtual image is generated and projected to the fundus, so that virtual graphic information is better fused with reality information, and the user obtains a better sense of immersion.

Claims

1. A system, comprising:

a gaze point detection apparatus that detects a gaze point location of an eye of a user, wherein the gaze point detection apparatus comprises;

an image collection device that collects an image presented at a fundus of the eye;

an adjustable imaging device that adjusts an imaging parameter of an optical path between the image collection device and the eye, enabling the image collection device to obtain a clearest image; and

an image processing device that processes the image obtained by the image collection device, and determines the gaze point location of the eye according to the imaging parameter for the clearest image obtained and an optical parameter of the eye;

an image generating apparatus that generates a virtual image based on visual information and gaze point information comprising an indication of the gaze point location; and

a virtual image projection apparatus that projects the virtual image to the fundus of the eye based on the gaze point information, wherein the gaze point information comprises distance information corresponding to a distance from the gaze point location to the eye of the user.

2. The system of claim 1, wherein the image generating apparatus comprises:

a visual information parameter determining module that determines at least one imaging parameter of the visual information based on the gaze point information.

3. The system of claim 2, wherein the image generating apparatus further comprises:

an image rendering module that renders the virtual image based on the visual information and the at least one imaging parameter of the visual information.

4. The system of claim 3, wherein the visual information corresponds to two eyes of the user.

5. The system of claim 3, wherein the at least one imaging parameter of the visual information corresponds to two eyes of the user.

6. The system of claim 3, wherein the gaze point detection apparatus further comprises:

a pupil detection module that detects a size of a pupil of the eye.

7. The system of claim 6, wherein the at least one imaging parameter of the visual information comprises depth-of-field information;

and

wherein the visual information parameter determining module further determines the depth-of-field information based on the size of the pupil of the eye.

8. The system of claim 7, wherein the image rendering module comprises:

a bokeh processing unit that performs corresponding bokeh imaging processing on the visual information based on the at least one imaging parameter of the visual information.

9. The system of claim 1, wherein the virtual image projection apparatus comprises:

a projection module that projects the virtual image to the fundus of the eye;

an adjustable imaging module, located on an optical path between the projection module and the fundus, generating, at the fundus, the virtual image projected by the projection module; and

an imaging parameter determining module that determines at least one projection imaging parameter of the adjustable imaging module based on the gaze point information.

10. The system of claim 9, wherein the adjustable imaging module further comprises:

a beam splitting unit that forms the optical path between the projection module and the fundus.

11. The system of claim 10, wherein the beam splitting unit comprises:

a curved-surface beam splitting unit that forms different optical paths between the projection module and the fundus separately corresponding to different locations of a pupil associated with different optical axis directions of the eye.

12. The system of claim 11, wherein the virtual image projection apparatus comprises:

a reverse deformation processing module that performs, on the virtual image, reverse deformation processing corresponding to the curved-surface beam splitting unit, to cause the fundus to receive the virtual image being presented.

13. The system of claim 1, wherein the adjustable imaging device comprises:

an adjustable lens unit located on the other optical path, wherein a focal length of the adjustable lens unit is adjustable or a location of the adjustable lens unit on the other optical path is adjustable.

14. The system of claim 1, wherein the adjustable imaging device comprises:

a curved-surface beam splitting unit that transmits the at least one image to the image collection device corresponding to different locations of the pupil associated with different optical axis directions of the eye.

15. The system of claim 1, wherein the gaze point detection apparatus further comprises:

a projection device that projects a light spot pattern to the fundus.

16. The system of claim 1, wherein the gaze point detection apparatus and the virtual image projection apparatus are part of a same apparatus.

17. The system of claim 1, wherein the content projection system is comprised in a pair of glasses.

18. A method, comprising:

detecting, by a system comprising a processor, a gaze point location of an eye of a user, wherein the detecting comprises;

collecting, by an image collection device, an image presented at a fundus of the eye;

adjusting, by an adjustable imaging device, an imaging parameter of an optical path between the image collection device and the eye, enabling the image collection device to obtain a clearest image; and

determining, by an image processing device and based on the image obtained by the image collection device, the gaze point location of the eye according to the imaging parameter for the clearest image obtained and an optical parameter of the eye;

generating a virtual image based on visual information and the gaze point location; and

projecting the virtual image to the fundus of the eye based on the gaze point location, wherein the gaze point location is at a determined distance from the eye of the user.

19. The method of claim 18, wherein the visual information comprises:

graphic information or image information.

20. The method of claim 19, wherein the generating the virtual image comprises:

determining at least one imaging parameter of the visual information based on the gaze point location and the determined distance.

21. The method of claim 20, wherein the generating the virtual image further comprises:

rendering the virtual image based on the visual information and the at least one imaging parameter of the visual information.

22. The method of claim 21, wherein the visual information corresponds to two eyes of the user.

23. The method of claim 21, wherein the at least one imaging parameter of the visual information corresponds to two eyes of the user.

24. The method of claim 19, further comprising:

detecting a size of a pupil of the eye.

25. The method of claim 18, wherein the projecting the virtual image to the fundus comprises:

projecting the virtual image to the fundus of the eye;

determining at least one projection imaging parameter of an optical path between a projection location and the fundus based on the gaze point location and the determined distance; and

imaging, at the fundus, the virtual image projected.

26. The method of claim 25, wherein the projecting the virtual image to the fundus further comprises:

imaging the virtual image at the fundus separately corresponding to different locations of a pupil associated with different optical axis directions of the eye.

27. The method of claim 26, wherein the projecting the virtual image to the fundus comprises:

performing, on the virtual image, reverse deformation processing corresponding to the locations of the pupil associated with different optical axis directions of the eye, to cause the fundus to receive the virtual image to be presented.

28. The method of claim 26, wherein the at least one imaging parameter of the visual information comprises depth-of-field information.

29. The method of claim 28, wherein determining the at least one imaging parameter further comprises:

determining the depth-of-field information based on a size of the pupil of the eye.

30. The method of claim 29, wherein the generating the virtual image comprises:

performing corresponding bokeh imaging processing on the visual information based on the at least one imaging parameter of the visual information.

31. The method of claim 18, wherein the adjusting the at least one other imaging parameter comprises:

adjusting a location of a lens unit on the other optical path or a focal length of the lens unit.

32. The method of claim 18, wherein the detecting the gaze point location further comprises:

transmitting the at least one image presented at the fundus to the image collection location separately corresponding to different locations of a pupil when an optical axis of the eye is in different directions.

33. The method of claim 18, wherein detecting the gaze point location further comprises:

projecting a light spot pattern to the fundus.

34. A computer readable storage device comprising executable instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:

determining a gaze point location of an eye of a user, wherein the determining comprises;

collecting, by an image collection device, an image presented at a fundus of the eye;

adjusting, by an adjustable imaging device, an imaging parameter of an optical path between the image collection device and the eye, enabling the image collection device to obtain a clearest image; and

determining, by an image processing device and based on the image obtained by the image collection device, the gaze point location of the eye according to the imaging parameter for the clearest image obtained and an optical parameter of the eye;

generating a virtual image based on visual information and the gaze point location; and

rendering the virtual image to the fundus of the eye based on the gaze point location and a distance between the gaze point location and the eye of the user.

35. The computer readable storage device of claim 34, wherein the generating a virtual image comprises:

determining at least one imaging parameter of the visual information based on the gaze point information.

36. The computer readable storage device of claim 35, wherein the generating a virtual image further comprises:

rendering the virtual image based on the visual information and the at least one imaging parameter of the visual information.