Display opacity control for preventing view occlusion in artificial reality

• US 10,679,410 B2
• Filed: 09/14/2018
• Issued: 06/09/2020
• Est. Priority Date: 09/14/2018
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Terrano, Mark
• Primary Examiner(s): Martello, Edward
• Application Number:
  US16/132,194
• Publication Number:
  US 20200090401A1
• Time in Patent Office: 634 Days
• Field of Search: None
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: H04M 1/6066 including a wireless connection
• CPC Class Codes: H04R 1/10 Earpieces; Attachments therefor ; Earphones; Monophonic headphones H04R1/28 takes precedence; stereophonic headphones H04R5/033
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

In one embodiment, the artificial reality system displays a virtual content to a first user. The virtual content has a first opacity. The system determines a first vergence distance of the first user and compare the first vergence distance of the first user to a first distance between the virtual content and the first user. The system adjusts the virtual content to have a second opacity that is less opaque than the first opacity in response to a determination that the first vergence distance of the first user is greater than the first distance by a first threshold distance.

Claims

1. A method comprising, by a computing system:

displaying a virtual content to a first user in a three-dimensional space, wherein the virtual content has a first opacity;

determining a first vergence distance of the first user;

comparing the first vergence distance of the first user to a first distance between the virtual content and the first user;

adjusting the virtual content to have a second opacity that is less opaque than the first opacity in response to a determination that the first vergence distance of the first user is greater than the first distance by a first threshold distance, wherein the virtual content having the second opacity is associated with a visual anchor displayed at a position of the virtual content in the three-dimensional space;

determining that the first user is looking at a visual anchor based on a second vergence distance of the first user; and

adjusting the virtual content to be more opaque than the second opacity in response to the determination that the first user is looking at the visual anchor.

2. The method of claim 1, wherein the visual anchor has a third opacity that is more opaque than the second opacity.

3. The method of claim 1, wherein the visual anchor comprises one or more of:

a corner fitting object;

a wireframe of a virtual panel;

a title bar of the virtual panel;

a line associated with the virtual panel;

a dot associated with the virtual panel;

a content outline of the virtual panel;

an opaque display element of the virtual panel;

a translucent display element of the virtual panel;

a text content of the virtual panel;

an icon;

a translucent virtual panel;

or

a virtual panel with a reduced size.

4. The method of claim 1, wherein the virtual content comprises one or more display elements on a virtual panel, and wherein adjusting the virtual content to have the second opacity comprises:

adjusting the virtual panel to have the second opacity; and

adjusting the one or more display elements on the virtual panel to have a fourth opacity, and wherein the fourth opacity is equal to or more opaque than the second opacity.

5. The method of claim 1, wherein the virtual content with the first opacity blocks the first user from seeing through the virtual content, and wherein the virtual content with the second opacity allows the first user to see through the virtual content.

6. The method of claim 1, further comprising determining that the virtual content interferes with a view of the first user.

7. The method of claim 6, wherein determining that the virtual content interferes with the view of the first user comprises:

determining a gazing direction of the first user using one or more sensors and an eye tracking system; and

determining that the virtual content is at least partially within a threshold angle to the gazing direction of the first user.

8. The method of claim 7, further comprising:

determining a gazing angle of the first user using the eye tracking system;

determining the view of the first user based on the gazing direction and the gazing angle of the first user; and

determining that the virtual content interferes with the view of the first user.

9. The method of claim 1, further comprising:

determining a moving direction of the first user using one or more sensors; and

determining that the virtual content interferes with a view of the first user along the moving direction of the first user, wherein adjusting of the virtual content to have the second opacity is based on the determination that the virtual content interferes with the view of the first user.

10. The method of claim 1, further comprising:

identifying a face of a second user in a field of view of the first user using one or more cameras and a face recognition model;

capturing a voice of the first or second user using one or more microphones; and

determining an interaction between the first and second users, wherein adjusting the virtual content to have the second opacity is based on the determined interaction between the first and second users.

11. The method of claim 1, further comprising:

identifying a face of a second user in a field of view of the first user using one or more cameras and a face recognition model; and

determining a social relevance score of the second user to the first user based on a social graph of a social network system, and wherein adjusting the virtual content to have the second opacity is based on a determination that the social relevance score is above a threshold.

12. The method of claim 1, wherein the first vergence distance of the first user is determined based on one or more factors associated with an object that the first user is looking at, and wherein the one or more factors comprise one or more of:

a gesture of the first user;

a voice command of the first user;

a correlation between the object and a behavior of the first user;

a selection of the first user;

or

an input of the first user.

13. The method of claim 1, further comprising relocating the virtual content to a new location to avoid interfering with a view of the first user.

14. The method of claim 1, further comprising resizing the virtual content to a smaller size to avoid interfering with a view of the first user, wherein the resized virtual content is displayed as a visual anchor associated with the virtual content.

15. The method of claim 1, wherein adjusting the virtual content to have the second opacity comprises adjusting the virtual content gradually from the first opacity to the second opacity based on a change of the first vergence distance of the first user.

16. The method of claim 1, wherein adjusting the virtual content to have a second opacity comprises adjusting the virtual content gradually from the first opacity to the second opacity in a speed corresponding to a changing speed of the first vergence distance of the first user.

17. The method of claim 1, wherein the first vergence distance of the first user is determined using an eye tracking system or using one or more cameras associated with a third-party system.

18. The method of claim 1, wherein the visual anchor is displayed persistently before and after the opacity of the virtual content is adjusted to be less opaque.

19. The method of claim 1, wherein the visual anchor is displayed after the opacity of the virtual content is adjusted to be less opaque.

20. The method of claim 1, further comprising:

determining that the first user is looking at an object in the three-dimensional space based on a determination that a first motion of a gazing point of the first user corresponds to a second motion of an object in the three-dimensional space, wherein at least at a portion of the virtual content is between the first user and the object, and wherein the virtual content is adjusted to be less opaque in response to the determination that the first user is looking at the object in the three-dimensional space.

21. One or more non-transitory computer-readable storage media embodying software that is operable when executed by a server computing device to:

display a virtual content to a first user in a three-dimensional space, wherein the virtual content has a first opacity;

determine a first vergence distance of the first user;

comparing the first vergence distance of the first user to a first distance between the virtual content and the first user;

adjust the virtual content to have a second opacity that is less opaque than the first opacity in response to a determination that the first vergence distance of the first user is greater than the first distance by a first threshold distance, wherein the virtual content having the second opacity is associated with a visual anchor displayed at a position of the virtual content in the three-dimensional space;

determine that the first user is looking at a visual anchor based on a second vergence distance of the first user; and

adjust the virtual content to be more opaque than the second opacity in response to the determination that the first user is looking at the visual anchor.

22. A system comprising:

one or more non-transitory computer-readable storage media embodying instructions;

one or more processors coupled to the storage media and operable to execute the instructions to;

display a virtual content to a first user in a three-dimensional space, wherein the virtual content has a first opacity;

determine a first vergence distance of the first user;

comparing the first vergence distance of the first user to a first distance between the virtual content and the first user;

adjust the virtual content to have a second opacity that is less opaque than the first opacity in response to a determination that the first vergence distance of the first user is greater than the first distance by a first threshold distance, wherein the virtual content having the second opacity is associated with a visual anchor displayed at a position of the virtual content in the three-dimensional space;

determine that the first user is looking at a visual anchor based on a second vergence distance of the first user; and

adjust the virtual content to be more opaque than the second opacity in response to the determination that the first user is looking at the visual anchor.