Interactive projection display

• US 9,870,050 B2
• Filed: 10/09/2014
• Issued: 01/16/2018
• Est. Priority Date: 10/10/2013
• Status: Active Grant

• Current Assignee: Beijing Zhigu RUI TUO Tech Company Limited
• Original Assignee: Beijing Zhigu RUI TUO Tech Company Limited
• Inventors: Du, Lin
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US15/028,019
• Publication Number:
  US 20160259406A1
• Time in Patent Office: 1,195 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G02B 27/10 Beam splitting or combining systems polarising systems G02B27/28; mixing and splitting light signals using optical waveguides G02B6/28; beam shaping, e.g. changing the cross-sectional area, by dividing or superposing multiple light beams G02B27/0905
• CPC Class Codes: G02B 3/14 of variable focal length
• CPC Class Codes: G02B 7/287 including a sight line detecting device
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

Disclosed are an interactive projection display method and an interactive projection display system. The method comprises: obtaining to-be-displayed location information corresponding to a coordination device; obtaining display information corres-ponding to the coordination device; generating, according to the display information, virtual display content corresponding to the coordination device; and projecting the virtual display content to a location corresponding to the to-be-displayed location information at a fundus of a user. According to the method and the system, virtual display content, used to interact with a user, of a coordination device is projected near the coordination device, to cause interaction between the user and the coordination device to be more convenient.

Claims

1. A method, comprising:

obtaining, by an interactive projection display system comprising a processor, to-be-displayed location information corresponding to a coordination device, wherein a to-be-displayed location corresponding to the to-be-displayed location information is located near the coordination device;

obtaining, by the interactive projection display system, display information corresponding to the coordination device;

generating, by the interactive projection display system according to the display information, virtual display content corresponding to the coordination device; and

projecting, by the interactive projection display system, the virtual display content to a location on a fundus of an eye corresponding to the to-be-displayed location information, such that the visual display content is perceived by the eye to be located at the to-be-displayed location.

2. The method of claim 1, wherein the obtaining the to-be-displayed location information comprises:

obtaining device location information of the coordination device; and

determining the to-be-displayed location information according to the device location information of the coordination device.

3. The method of claim 2, wherein the obtaining the device location information comprises:

receiving the device location information from the coordination device.

4. The method of claim 2, wherein the obtaining the device location information comprises:

receiving the device location information from an external device.

5. The method of claim 2, wherein the obtaining the device location information comprises:

detecting the device location information.

6. The method of claim 5, wherein the detecting the device location information comprises:

detecting a first location of a gaze point of the eye; and

determining the device location information according to the first location of the gaze point.

7. The method of claim 6, wherein the detecting the first location of the gaze point of the eye comprises:

acquiring at least one image at the fundus of the eye at a second location;

adjusting, to acquire an image of the at least one image that satisfies at least one defined clarity criterion, at least one imaging parameter of an optical path between the second location to acquire the at least one image and the eye; and

analyzing the at least one image to obtain the at least one imaging parameter and at least one optical parameter of the eye corresponding to the image that satisfies the at least one defined clarity criterion, and acquiring the first location of the gaze point of the eye relative to the eye.

8. The method of claim 7, wherein the adjusting the at least one parameter comprises:

adjusting a focal length of at least one optical device on the optical path or a third location of the at least one optical device on the optical path.

9. The method of claim 7, wherein the adjusting the at least one parameter further comprises:

transferring, separately corresponding to different locations of a pupil of the eye in response to different optical axis directions of the eye, the at least one image presented at the fundus to the second location.

10. The method of claim 7, wherein the detecting the first location of the gaze point of the eye further comprises:

projecting a light spot pattern to the fundus of the eye.

11. The method of claim 5, wherein the detecting the device location information comprises:

detecting interaction information associated with the coordination device; and

determining the device location information according to the interaction information.

12. The method of claim 2, wherein the determining the to-be-displayed location information comprises:

determining the to-be-displayed location information according to the device location information and a preset rule.

13. The method of claim 1, wherein the obtaining the to-be-displayed location information comprises:

obtaining the to-be-displayed location information through user input.

14. The method of claim 1, wherein the obtaining the display information comprises:

obtaining the display information from the coordination device.

15. The method of claim 1, wherein the obtaining the display information comprises:

obtaining the display information from an external device.

16. The method of claim 1, wherein the generating the virtual display content comprises:

generating the virtual display content according to the to-be-displayed location information and the display information.

17. The method of claim 16, wherein the virtual display content comprises perspective information corresponding to the to-be-displayed location information.

18. The method of claim 16, wherein the generating the virtual display content comprises:

generating the virtual display content separately corresponding to two eyes.

19. The method of claim 1, wherein the projecting the virtual display content comprises:

adjusting at least one projection imaging parameter of an optical path between a projection location and the fundus according to the to-be-displayed location information.

20. The method of claim 19, wherein the adjusting the at least one projection imaging parameter comprises:

adjusting at least one imaging parameter of at least one other optical device on another optical path or another location of the at least one another optical device on the other optical path.

21. The method of claim 19, wherein the projecting the virtual display content further comprises:

projecting, separately corresponding to different locations of a pupil in response to different optical axis directions of the eye related to the fundus, the virtual display content to the fundus.

22. The method of claim 21, wherein the projecting the virtual display content comprises:

performing, on the virtual display content, counter-deformation processing corresponding to the different locations of the pupil.

23. The method of claim 19, wherein the projecting the virtual display content comprises:

controlling a size of the virtual display content projected on the fundus.

24. The method of claim 1, further comprising:

obtaining, by the interactive projection display system, a trigger signal and triggering the obtaining of the to-be-displayed location information and the obtaining of the display information.

25. The method of claim 1, wherein the coordination device is a wearable device.

26. A system, comprising:

a memory that stores executable components; and

a processor, coupled to the memory, that executes the executable components to perform operations of the system, the executable components comprising;

a location information obtaining apparatus configured to obtain to-be-displayed location information corresponding to a coordination device, wherein a to-be-displayed location corresponding to the to-be-displayed location information is located near the coordination device;

a display information obtaining apparatus configured to obtain display information corresponding to the coordination device;

a content generating apparatus configured to generate, according to the display information, virtual display content corresponding to the coordination device; and

a projection apparatus configured to project the virtual display content to a location on a fundus of an eye corresponding to the to-be-displayed location information, such that the visual display content is perceived by the eye to be located at the to-be-displayed location.

27. The system of claim 26, wherein the location information obtaining apparatus comprises:

a device location obtaining module configured to obtain device location information of the coordination device; and

a to-be-displayed location determining module configured to determine the to-be-displayed location information according to the device location information.

28. The system of claim 27, wherein the device location obtaining module comprises:

a communications submodule configured to receive the device location information.

29. The system of claim 28, wherein the communications submodule is further configured to:

receive the device location information from the coordination device.

30. The system of claim 28, wherein the communications submodule is further configured to:

receive the device location information from an external device.

31. The system of claim 27, wherein the device location obtaining module comprises:

a detection submodule configured to detect the device location information.

32. The system of claim 31, wherein the detection submodule comprises:

a gaze point detection unit configured to detect a first location of a gaze point of the eye; and

a device location determining unit configured to determine the device location information according to the first location of the gaze point.

33. The system of claim 32, wherein the gaze point detection unit comprises:

an image acquisition subunit configured to acquire at least one image at the fundus of the eye at a second location;

an adjustable imaging subunit configured to adjust at least one imaging parameter of an optical path between the image acquisition subunit and the eye to acquire an image of the at least one image that satisfies at least one defined clarity criterion; and

an image processing subunit configured to analyze the at least one image to obtain the at least one imaging parameter and at least one optical parameter of the eye corresponding to the image that satisfies the at least one defined clarity criterion, and acquire the first location of the gaze point of the eye relative to the eye.

34. The system of claim 33, wherein the adjustable imaging subunit comprises:

an adjustable lens device, wherein at least one imaging parameter of the adjustable lens device being adjustable or a third location of the adjustable lens device on the optical path being adjustable.

35. The system of claim 33, wherein the adjustable imaging subunit further comprises:

a curved beam splitting device configured to transfer, separately corresponding to different locations of a pupil of the eye in response to different optical axis directions of the eye, the at least one image presented at the fundus to the image acquisition subunit.

36. The system of claim 33, wherein the gaze point detection unit further comprises:

a projection subunit configured to project a light spot pattern to the fundus of the eye.

37. The system of claim 31, wherein the detection submodule further comprises:

an interaction detection unit configured to detect interaction information associated with the coordination device; and

a device location determining unit configured to determine the device location information according to the interaction information.

38. The system of claim 27, wherein the to-be-displayed location determining module is further configured to determine the to-be-displayed location information according to the device location information and a preset rule.

39. The system of claim 26, wherein the device location information obtaining apparatus comprises:

an interaction obtaining module configured to obtain the to-be-displayed location information through user input.

40. The system of claim 26, wherein the display information obtaining apparatus comprises:

a communications module configured to obtain the display information corresponding to the coordination device from the coordination device.

41. The system of claim 26, wherein the display information obtaining apparatus comprises:

a communications module configured to obtain the display information corresponding to the coordination device from an external device.

42. The system of claim 26, wherein the content generating apparatus is further configured to generate the virtual display content according to the to-be-displayed location information and the display information.

43. The system of claim 42, wherein the content generating apparatus is further configured to generate the virtual display content comprising perspective information corresponding to the to-be-displayed location information.

44. The system of claim 42, wherein the content generating apparatus is further configured to generate the virtual display content separately corresponding to two eyes of the user.

45. The system of claim 26, wherein the projection apparatus comprises:

an imaging adjustment module configured to adjust at least one projection imaging parameter of an optical path between the projection module and the fundus according to the to-be-displayed location information.

46. The system of claim 45, wherein the imaging adjustment module comprises:

an adjustable lens device, wherein at least one imaging parameter of the adjustable lens device is adjustable or another location of the adjustable lens device on the optical path being adjustable.

47. The system of claim 45, wherein the imaging adjustment module comprises:

a curved beam splitting unit configured to project, separately corresponding to different locations of a pupil in response to different optical axis directions of the eye related to the fundus, the virtual display content to the fundus.

48. The system of claim 47, wherein the projection apparatus comprises:

a counter-deformation processing module configured to perform, on the virtual display content, counter-deformation processing corresponding to the different locations of the pupil.

49. The system of claim 45, wherein the projection apparatus comprises:

a projection control module configured to control a size of the virtual display content projected on the fundus.

50. The system of claim 26, wherein the executable components further comprise:

a triggering apparatus configured to obtain a trigger signal and trigger the location obtaining apparatus and the display information obtaining apparatus.

51. The system of claim 33, wherein functions of the gaze point detection unit and the projection apparatus are implemented by a same device.

52. The system of claim 26, wherein the coordination device is a wearable device.

53. The system of claim 26, wherein the system is a wearable device used near the eye.

54. A computer readable storage device, comprising at least one executable instruction, which, in response to execution, causes a system comprising a processor to perform operations, comprising:

receiving to-be-displayed location information corresponding to a coordination device and display information corresponding to the coordination device, wherein a to-be-displayed location corresponding to the to-be-displayed location information is located near the coordination device;

generating, according to the display information, virtual display content corresponding to the coordination device; and

projecting the virtual display content to a location on a fundus of an eye corresponding to the to-be-displayed location information, such that the visual display content is perceived by the eye to be located at the to-be-displayed location.