Correlating user reaction with at least an aspect associated with an augmentation of an augmented view

• US 9,671,863 B2
• Filed: 12/21/2012
• Issued: 06/06/2017
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Levien, Royce A., Malamud, Mark A., Fein, Gene, Lord, Richard T., Lord, Robert W., Rinaldo, Jr., John D., Tegreene, Clarence T.
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): PRINGLE-PARKER, JASON A
• Application Number:
  US13/723,610
• Publication Number:
  US 20140098134A1
• Time in Patent Office: 1,628 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computationally implemented methods and systems include detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations, and correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1. A system comprising:

circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment;

circuitry configured to determine a dwell path of the user relative to one or more augmentations based on one or more outputs of the circuitry configured to detect one or more eye characteristics;

circuitry configured to identify an indication of changed focus with respect to at least one of the one or more augmentations based on a determination that a dwell path has moved away from the at least one of the one or more augmentations;

circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations; and

circuitry configured to output one or more additional augmentations in the augmented view at one or more locations that are based on the dwell path.

2. The system of claim 1, wherein said circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to track eye focus-point movement of a user relative to one or more augmentations associated with display of an augmented view of a real environment.

3. The system of claim 1, wherein said circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to detect one or more eye characteristics of a user during and/or following display of one or more augmentations.

4. The system of claim 1, wherein said circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to detect one or more eye movements of a user away from one or more augmentations associated with display of an augmented view of a real environment.

5. The system of claim 1, wherein said circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to detect one or more eye movements of a user between two or more augmentations associated with display of an augmented view of a real environment.

6. The system of claim 1, wherein said circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to detect one or more pupil characteristics of a user relative to one or more augmentations associated with display of an augmented view of a real environment.

7. The system of claim 6, wherein said circuitry configured to detect one or more pupil characteristics of a user relative to one or more augmentations associated with display of an augmented view of a real environment comprises:

circuitry configured to detect pupil shape and/or pupil size of a user relative to one or more augmentations associated with display of an augmented view of a real environment.

8. The system of claim 1, further comprising:

circuitry configured to associate with one or more augmentations at least one of a cardiopulmonary characteristic, a skin characteristic and/or a brain characteristic of the user.

9. The system of claim 1, further comprising:

circuitry configured to associate with one or more augmentations at least one of a facial expression, a head position, and/or a head tilt of the user.

10. The system of claim 1, further comprising:

circuitry configured to associate one or more eye characteristics of the user with one or more forms of one or more augmentations.

11. The system of claim 10, wherein said circuitry configured to associate one or more eye characteristics of the user with one or more forms of one or more augmentations comprises:

circuitry configured to associate one or more eye characteristics of the user with one or more shapes or dimensions of one or more augmentations.

12. The system of claim 1, further comprising:

circuitry configured to associate one or more eye characteristics of the user with one or more locations of one or more augmentations.

13. The system of claim 1, wherein said circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations comprises:

circuitry configured to output one or more appearance changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations.

14. The system of claim 1, wherein said circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations comprises:

circuitry configured to output one or more location changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations.

15. The system of claim 1, wherein said circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations comprises:

circuitry configured to output one or more dimension or shape changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations.

16. The system of claim 1, wherein said circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations comprises:

circuitry configured to output one or more replacements to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations.

17. The system of claim 1, further comprising:

circuitry configured to acquire, in response to relay of one or more results, the one or more additional augmentations for presentation in the augmented view.

18. The system of claim 1, wherein the circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to detect dwell time of a user in association with display of an augmented view of a real environment.

19. The system of claim 1, wherein the circuitry configured to identify an indication of changed focus with respect to at least one of the one or more augmentations based on a determination that a dwell path has moved away from the at least one of the one or more augmentations comprises:

circuitry configured to identify an indication of changed focus with respect to at least one of the one or more augmentations based on a determination that a dwell path has moved away from the at least one of the one or more augmentations after at least a specified value of dwell time.

20. The system of claim 1, wherein the circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to detect one or more eye characteristics of a user spatially or temporally relative to one or more augmentations associated with display of an augmented view of a real environment.

21. The system of claim 1, wherein the circuitry configured to detect one or more eye characteristics of a user in association with display of an augmented view of a real environment comprises:

circuitry configured to detect using one or more sensors or cameras one or more eye characteristics of a user in association with display of an augmented view of a real environment.

22. The system of claim 1, wherein the circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations comprises:

circuitry configured to output one or more brightness, color, or hue changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations.

23. The system of claim 1, wherein the system is at least partly incorporated within one or more of the following devices:

tablet, smartphone, or glasses.

24. The system of claim 1, further comprising:

circuitry configured to detect one or more physiological values of the user using one or more sensors.

25. The system of claim 1, further comprising:

circuitry configured to detect one or more of the following parameters of the user using one or more sensors;

facial expression, head position, or head tilt.

26. The system of claim 1, further comprising:

circuitry configured to perform speech recognition based on audio received from one or more audio inputs or sensors.

27. The system of claim 1, further comprising:

circuitry configured to identify an indication of changed focus with respect to at least one of the one or more augmentations based on a determination that a dwell path has moved away from the at least one of the one or more augmentations and based on speech received via one or more audio inputs.

28. The system of claim 1, wherein the circuitry configured to output one or more additional augmentations in the augmented view at one or more locations that are based on the dwell path comprises:

circuitry configured to output one or more additional augmentations in the augmented view at one or more locations that are on the dwell path.

29. The system of claim 1, wherein the circuitry configured to output one or more additional augmentations in the augmented view at one or more locations that are based on the dwell path comprises:

circuitry configured to output one or more additional augmentations in the augmented view at one or more locations that are along the dwell path.

32. The system of claim 1, further comprising:

circuitry configured to identify one or more relationships between behavior of the user and one or more augmentations.

33. The system of claim 1, wherein the circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations comprises:

circuitry configured to output one or more temporal changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations.

34. The system of claim 1, wherein the circuitry configured to output one or more changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations comprises:

circuitry configured to output one or more location or format changes to the at least one of the one or more augmentations based on identification of an indication of changed focus with respect to the at least one of the one or more augmentations.

30. A wearable augmented reality (AR) device comprising:

a frame;

at least one sensor;

one or more displays;

one or more media bearing one or more executable instructions; and

one or more processing units operable to execute the one or more executable instructions to perform operations including at least;

detect one or more eye characteristics of a user in association with display of an augmented view of a real environment;

determine a dwell path of the user relative to one or more augmentations based on the one or more eye characteristics;

identify an indication of changed focus with respect to at least one of the one or more augmentations based on a determination that the dwell path has moved away from the at least one of the one or more augmentations;

output one or more changes to the at least one of the one or more augmentations based on the indication of changed focus with respect to the at least one of the one or more augmentations; and

output one or more additional augmentations in the augmented view at one or more locations that are based on the dwell path.

31. A method for controlling a processing component of an augmented reality device to perform operations comprising:

detecting one or more eye characteristics of a user in association with display of an augmented view of a real environment;

determining a dwell path of the user relative to one or more augmentations based on the one or more eye characteristics;

identifying an indication of changed focus with respect to at least one of the one or more augmentations based on a determination that the dwell path has moved away from the at least one of the one or more augmentations;

outputting one or more changes to the at least one of the one or more augmentations based on the indication of changed focus with respect to the at least one of the one or more augmentations; and

outputting one or more additional augmentations in the augmented view at one or more locations that are based on the dwell path.