Correlating user reactions with augmentations displayed through augmented views

• US 9,077,647 B2
• Filed: 12/28/2012
• Issued: 07/07/2015
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, John D. Jr., Tegreene, Clarence T.
• Primary Examiner(s): Zarka, David
• Assistant Examiner(s): PRINGLE-PARKER, JASON A
• Application Number:
  US13/729,278
• Publication Number:
  US 20140101555A1
• Time in Patent Office: 921 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]

Abstract

Computationally implemented methods and systems include receiving first data that at least identifies one or more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes, receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views; and correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1. A computationally-implemented method implemented via one or more computer processors, comprising:

receiving first data that at least identifies one or more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes;

receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views; and

correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views, wherein said correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views includes at least;

correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations, wherein said correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations includes at least;

determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations, wherein said determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations includes at least;

determining the one or more relationships between the one or more user reactions and the one or more forms of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more colors, brightness, and/or hue of the or more augmentations.

2. A computationally-implemented system, comprising:

means for receiving first data that at least identifies one or more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes;

means for receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views; and

means for or correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views,

wherein said means for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views includes at least;

means for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations, wherein said means for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations includes at least;

means for determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations, wherein said means for determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations includes at least;

means for determining the one or more relationships between the one or more user reactions and the one or more forms of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more colors, brightness, and/or hue of the or more augmentations.

3. The computationally-implemented system of claim 2, wherein said means for receiving first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes comprises:

means for receiving the first data including receiving first data that indicates one or more aspects of one or more non-augmentation scene elements that were included in the one or more actual scenes that were obfuscated in the one or more remotely displayed augmented views by the inclusion of the one or more augmentations in the one or more remotely displayed augmented views.

4. The computationally-implemented system of claim 2, wherein said means for receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views comprises:

means for receiving the second data by receiving second data that indicates one or more user reactions of the one or more users proximate to the remote display of the one or more remotely displayed augmented views.

5. The computationally-implemented system of claim 4, wherein said means for receiving the second data by receiving second data that indicates one or more user reactions of the one or more users proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for receiving the second data by receiving second data that indicates one or more physiological reactions of the one or more users proximate to the remote display of the one or more remotely displayed augmented views.

6. The computationally-implemented system of claim 5, wherein said means for receiving the second data by receiving second data that indicates one or more physiological reactions of the one or more users proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for receiving the second data that indicates the one or more physiological reactions of the one or more users by receiving second data that indicates one or more eye characteristics of the one or more users proximate to the remote display of the one or more remotely displayed augmented views.

7. The computationally-implemented system of claim 6, wherein said means for receiving the second data that indicates the one or more physiological reactions of the one or more users by receiving second data that indicates one or more eye characteristics of the one or more users proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for receiving the second data that indicates the one or more eye characteristics of the one or more users by receiving second data that indicates one or more pupil characteristics of one or more eyes of the one or more users proximate to the remote display of the one or more remotely displayed augmented views.

8. The computationally-system of claim 2, wherein said means for receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views comprises:

means for receiving the second data indicating the one or more user reactions of the one or more users by receiving second data that indicates one or more facial expressions, head positions, and/or head tilts of the one or more users proximate to the display of the one or more remotely displayed augmented views.

9. The computationally-implemented system of claim 2, wherein said means for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views comprises:

means for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and the one or more augmentations, the one or more user reactions having occurred proximate to the remote display of the one or more remotely displayed augmented views.

10. The computationally-implemented system of claim 9, wherein said means for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and the one or more augmentations, the one or more user reactions having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more user reactions and the one or more augmentations including determining one or more temporal relationships between the one or more user reactions and the one or more augmentations.

11. The computationally-implemented system of claim 9, wherein said means for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and the one or more augmentations, the one or more user reactions having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more user reactions and the one or more augmentations including determining one or more spatial relationships between the one or more user reactions and the one or more augmentations.

12. The computationally-implemented system of claim 9, wherein said means for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and the one or more augmentations, the one or more user reactions having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more user reactions and the one or more augmentations including determining one or more relationships between one or more physiological reactions of the one or more users and the one or more augmentations, the one or more physiological reactions having occurred proximate to the remote display of the one or more remotely displayed augmented views.

13. The computationally-implemented system of claim 12, wherein said means for determining the one or more relationships between the one or more user reactions and the one or more augmentations including determining one or more relationships between one or more physiological reactions of the one or more users and the one or more augmentations, the one or more physiological reactions having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between one or more physiological reactions of the one or more users and the one or more augmentations by determining one or more relationships between one or more eye characteristics of the one or more users and the one or more augmentations, the one or more eye characteristics having occurred proximate to the remote display of the one or more remotely displayed augmented views.

14. The computationally-implemented system of claim 13, wherein said means for determining the one or more relationships between one or more physiological reactions of the one or more users and the one or more augmentations by determining one or more relationships between one or more eye characteristics of the one or more users and the one or more augmentations, the one or more eye characteristics having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more eye characteristics of the one or more users and the one or more augmentations by determining one or more relationships between one or more eye movements of one or more eyes of the one or more users and the one or more augmentations, the one or more eye movements having occurred proximate to the remote display of the one or more remotely displayed augmented views.

15. The computationally-implemented system of claim 14, wherein said means for determining the one or more relationships between the one or more eye characteristics of the one or more users and the one or more augmentations by determining one or more relationships between one or more eye movements of one or more eyes of the one or more users and the one or more augmentations, the one or more eye movements having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more eye movements of the one or more eyes of the one or more users and the one or more augmentations by determining one or more relationships between one or more dwell paths of one or more eyes of the one or more users with respect to the one or more remotely displayed augmented views and the one or more augmentations, the one or more dwell paths of the one or more eyes of the one or more users having occurred proximate to the remote display of the one or more remotely displayed augmented views.

16. The computationally-implemented system of claim 15, wherein said means for determining the one or more relationships between the one or more eye movements of the one or more eyes of the one or more users and the one or more augmentations by determining one or more relationships between one or more dwell paths of one or more eyes of the one or more users with respect to the one or more remotely displayed augmented views and the one or more augmentations, the one or more dwell paths of the one or more eyes of the one or more users having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more dwell paths of one or more eyes of the one or more users and the one or more augmentations by determining one or more relationships between one or more dwell paths of the one or more eyes of the one or more users and the one or more augmentations, the one or more dwell paths having been as a result of eye focus of the one or more eyes of the one or more users moving away from the one or more augmentations included in the one or more remotely displayed augmented views.

17. The computationally-implemented system of claim 13, wherein said means for determining the one or more relationships between one or more physiological reactions of the one or more users and the one or more augmentations by determining one or more relationships between one or more eye characteristics of the one or more users and the one or more augmentations, the one or more eye characteristics having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more eye characteristics of the one or more users and the one or more augmentations by determining one or more relationships between eye focus of the one or more eyes of the one or more users with respect to the one or more remotely displayed augmented views and the one or more augmentations, the eye focus of the one or more eyes of the one or more users having occurred proximate to the remote display of the one or more remotely displayed augmented views.

18. The computationally-implemented system of claim 17, wherein said means for determining the one or more relationships between the one or more eye characteristics of the one or more users and the one or more augmentations by determining one or more relationships between eye focus of the one or more eyes of the one or more users with respect to the one or more remotely displayed augmented views and the one or more augmentations, the eye focus of the one or more eyes of the one or more users having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more eye focus of the one or more eyes of the one or more users with respect to the one or more remotely displayed augmented views and the one or more augmentations by determining one or more relationships between dwell time of the one or more eyes of the one or more users and the one or more augmentations, the dwell time of the one or more eyes of the one or more users being with respect to the one or more remotely displayed augmented views, and the dwell time having occurred proximate to the remote display of the one or more remotely displayed augmented views.

19. The computationally-implemented system of claim 13, wherein said means for determining the one or more relationships between one or more physiological reactions of the one or more users and the one or more augmentations by determining one or more relationships between one or more eye characteristics of the one or more users and the one or more augmentations, the one or more eye characteristics having occurred proximate to the remote display of the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more eye characteristics of the one or more users and the one or more augmentations by determining one or more relationships between one or more pupil characteristics of one or more eyes of the one or more users and the one or more augmentations, the one or more pupil characteristics having occurred proximate to the remote display of the one or more remotely displayed augmented views.

20. The computationally-implemented system of claim 2, wherein said means for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views comprises:

means for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between one or more facial expressions, and/or head positions of the one or more users and the one or more augmentations, the one or more facial expressions, and/or head positions of the one or more users having occurred proximate to the remote display of the one or more remotely displayed augmented views.

21. The computationally-implemented system of claim 2, wherein said means for determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations comprises:

means for determining the one or more relationships between the one or more user reactions and the one or more forms of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more shapes and/or sizes of the one or more augmentations.

22. The computationally-implemented system of claim 2, wherein said means for determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations comprises:

means for determining the one or more relationships between the one or more user reactions and the one or more forms of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more locations of the one or more augmentations in the one or more remotely displayed augmented views.

23. The computationally-implemented system of claim 2, wherein said means for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views comprises:

means for correlating the one or more user reactions with the one or more augmentations including determining one or more relationships between the one or more user reactions and one or more aspects of one or more non-augmentation scene elements included in the one or more remotely displayed augmented views.

24. The computationally-implemented system of claim 23, wherein said means for correlating the one or more user reactions with the one or more augmentations including determining one or more relationships between the one or more user reactions and one or more aspects of one or more non-augmentation scene elements included in the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more user reactions and the one or more aspects of one or more non-augmentation scene elements by determining one or more relationships between the one or more user reactions and one or more forms of the one or more non-augmentation scene elements included in the one or more remotely displayed augmented views.

25. The computationally-implemented system of claim 24, wherein said means for determining the one or more relationships between the one or more user reactions and the one or more aspects of one or more non-augmentation scene elements by determining one or more relationships between the one or more user reactions and one or more forms of the one or more non-augmentation scene elements included in the one or more remotely displayed augmented views comprises:

means for determining the one or more relationships between the one or more user reactions and the one or more forms of the one or more non-augmentation scene elements by determining one or more relationships between the one or more user reactions and one or more locations of the one or more non-augmentation scene elements in the one or more remotely displayed augmented views.

26. The computationally-implemented system of claim 25, wherein said means for determining the one or more relationships between the one or more user reactions and the one or more forms of the one or more non-augmentation scene elements by determining one or more relationships between the one or more user reactions and one or more locations of the one or more non-augmentation scene elements in the one or more remotely displayed augmented views comprises:

means for determining one or more relationships between the one or more user reactions and the one or more locations of the one or more non-augmentation scene elements in the one or more remotely displayed augmented views by determining one or more relationships between the one or more user reactions and one or more locations of the one or more non-augmentation scene elements relative to one or more locations of the one or more augmentations in the one or more remotely displayed augmented views.

27. The computationally-implemented system of claim 2, wherein said means for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views comprises:

means for correlating the one or more user reactions with the one or more augmentations including determining one or more relationships between the one or more user reactions and one or more non-augmentation scene elements obfuscated by the one or more augmentations in the one or more remotely displayed augmented views.

31. The computationally-implemented system of claim 2, wherein said means for receiving first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes comprises:

means for receiving the first data in response to transmitting the one or more augmentations to one or more augmented reality (AR) devices that remotely displayed the one or more remotely displayed augmented views.

32. The computationally-implemented system of claim 2, wherein said means for receiving first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes comprises:

receiving the first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes, the one or more augmentations having been displayed in the one or more remotely displayed augmented views by being displayed having a specific scaled size in the one or more remotely displayed augmented views in order to provide a distal or proximate view of the one or more augmentations.

33. The computationally-implemented system of claim 2, wherein said means for receiving first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes comprises:

means for receiving the first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes, the one or more augmentations having been remotely displayed by being projected away from one or more users viewing the one or more remotely displayed augmented views.

34. The computationally-implemented system of claim 33, wherein said means for receiving the first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes, the one or more augmentations having been remotely displayed by being projected away from one or more users viewing the one or more remotely displayed augmented views comprises:

means for receiving the first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes, the one or more augmentations having been remotely displayed by being projected away from one or more users and onto a see-through lens.

35. The computationally-implemented system of claim 2, wherein said means for receiving first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes comprises:

means for receiving the first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes, the one or more augmentations having been displayed by being projected into one or more eyes of one or more users viewing the one or more remotely displayed augmented views.

28. A computationally-implemented system, comprising:

means for receiving first data that at least identifies one more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes;

means for receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views;

means for or correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views;

means for developing one or more second augmentations for displaying through one or more augmented views, the one or more second augmentations being based, at least in part, the correlating, wherein said means for developing one or more second augmentations for displaying through one or more augmented views, the one or more second augmentations being based, at least in part, the correlating includes at least;

means for developing the one or more second augmentations for displaying through the one or more augmented views by developing one or more second augmentations having one or more forms that are based, at least in part, on the correlating, wherein said means for developing the one or more second augmentations for displaying through the one or more augmented views by developing one or more second augmentations having one or more forms that are based, at least in part, on the correlating includes at least;

means for developing the one or more second augmentations having the one or more forms by developing one or more second augmentations having one or more colors, brightness, and/or hue that are selected based, at least in part, on the correlating.

29. The computationally-implemented system of claim 28, wherein said means for developing the one or more second augmentations for displaying through the one or more augmented views by developing one or more second augmentations having one or more forms that are based, at least in part, on the correlating comprises:

means for developing the one or more second augmentations having the one or more forms by developing one or more second augmentations having one or more visual features, shapes, and/or dimensions that are selected, based at least in part, on the correlating.

30. The computationally-implemented system of claim 28, wherein said means for developing the one or more second augmentations for displaying through the one or more augmented views by developing one or more second augmentations having one or more forms that are based, at least in part, on the correlating comprises:

means for developing the one or more second augmentations having the one or more forms by developing one or more second augmentations designed to be located at one or more specific locations in the one or more augmented views, the one or more specific locations being based, at least in part, on the correlating.

36. A system, comprising:

circuitry for receiving first data that at least identifies one or more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes;

circuitry for receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views; and

circuitry for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views, wherein said circuitry for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views includes at least;

circuitry for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations, wherein said circuitry for correlating the one or more user reactions with the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations includes at least;

circuitry for determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations, wherein said circuitry for determining the one or more relationships between the one or more user reactions and the one or more aspects of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations includes at least;

circuitry for determining the one or more relationships between the one or more user reactions and the one or more forms of the one or more augmentations by determining one or more relationships between the one or more user reactions and one or more colors, brightness, and/or hue of the or more augmentations.

37. A system, comprising:

circuitry for receiving first data that at least identifies one or more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes;

circuitry for receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views; and

circuitry for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views, wherein said circuitry for correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views includes at least;

circuitry for determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations, wherein said circuitry for determining one or more relationships between the one or more user reactions and one or more aspects of the one or more augmentations includes at least;

circuitry for determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations, wherein said circuitry for determining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations includes at least;

circuitry for determining one or more relationships between the one or more user reactions and one or more colors, brightness, and/or hue of the or more augmentations.