Correlating user reactions with augmentations displayed through augmented views

• US 9,674,047 B2
• Filed: 12/31/2012
• Issued: 06/06/2017
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Levien, Royce A., Fein, Gene, Lord, Richard T., Lord, Robert W., Rinaldo, Jr., John D., Malamud, Mark A., Tegreene, Clarence T.
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): PRINGLE-PARKER, JASON A
• Application Number:
  US13/731,233
• Publication Number:
  US 20140098135A1
• Time in Patent Office: 1,618 Days
• Field of Search: None
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]

Abstract

Computationally implemented methods and systems include receiving first data that at least identifies one or more augmentations that were remotely displayed in one or more remotely displayed augmented views of one or more actual scenes, receiving second data indicating one or more user reactions of one or more users in response to the remote display of the one or more remotely displayed augmented views; and correlating the one or more user reactions with the one or more augmentations that were remotely displayed through the one or more remotely displayed augmented views. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1. A system, comprising:

at least one computing device; and

one or more instructions that, when implemented in the at least one computing device, program the at least one computing device for;

detecting an anchor visual cue in an actual scene;

displaying one or more first augmentations at a location in an augmented view of the actual scene responsive to the detection of the visual cue in the actual scene;

acquiring data indicating one or more user reactions of a user responsive to display of the one or more first augmentations, the one or more user reactions being indicative of at least one of lack of eye focus or lack of dwell time with respect to the one or more first augmentations;

associating the one or more user reactions with the one or more first augmentations; and

displaying one or more second augmentations in one or more augmented views based, at least in part, on the associating the one or more user reactions with the one or more first augmentations.

2. The system of claim 1, wherein said acquiring data indicating one or more user reactions of a user responsive to display of the one or more first augmentations includes:

acquiring data that indicates one or more user reactions of the user proximate to the displaying one or more first augmentations.

3. The system of claim 2, wherein said acquiring data that indicates one or more user reactions of the users proximate to the displaying one or more first augmentations includes:

acquiring data that indicates one or more physiological reactions of the user proximate to the displaying one or more first augmentations.

4. The system of claim 3, wherein said acquiring data that indicates one or more physiological reactions of the users proximate to the displaying one or more first augmentations includes:

acquiring one or more eye characteristics of the user proximate to the displaying one or more first augmentations.

5. The system of claim 4, wherein said acquiring one or more eye characteristics of the users proximate to the displaying one or more first augmentations includes:

acquiring data that indicates one or more pupil characteristics of one or more eyes of the user proximate to the displaying one or more first augmentations.

6. The system of claim 3, wherein said acquiring data that indicates one or more physiological reactions of the users proximate to the displaying one or more first augmentations includes:

acquiring data that indicates at least one of a cardiopulmonary characteristic, skin characteristic and/or brain characteristic of the user proximate to the displaying one or more first augmentations.

7. The system of claim 1, wherein said acquiring data indicating one or more user reactions of a user responsive to display of the one or more first augmentations includes:

acquiring data that indicates at least one of a facial expression, head position, and/or head tilt of the user proximate to the displaying one or more first augmentations.

8. The system of claim 1, further comprising:

identifying one or more relationships between the one or more user reactions and the one or more augmentations.

9. The system of claim 8, wherein said identifying one or more relationships between the one or more user reactions and the one or more augmentations includes:

identifying one or more temporal relationships between the one or more user reactions and the one or more augmentations.

10. The system of claim 8, wherein said identifying one or more relationships between the one or more user reactions and the one or more augmentations includes:

identifying one or more spatial relationships between the one or more user reactions and the one or more augmentations.

11. The system of claim 8, wherein said identifying one or more relationships between the one or more user reactions and the one or more augmentations includes:

identifying one or more relationships between one or more physiological reactions of the user and the one or more augmentations.

12. The system of claim 11, wherein said identifying one or more relationships between one or more physiological reactions of the users and the one or more augmentations includes:

identifying one or more relationships between one or more eye characteristics of the user and the one or more augmentations.

13. The system of claim 12, wherein said identifying one or more relationships between one or more eye characteristics of the users and the one or more augmentations includes:

identifying one or more relationships between one or more eye movements of one or more eyes of the user and the one or more augmentations.

14. The system of claim 13, wherein said identifying one or more relationships between one or more eye movements of one or more eyes of the users and the one or more augmentations includes:

identifying one or more relationships between one or more dwell paths of the one or more eyes of the user with respect to the one or more augmentations.

16. The system of claim 1, wherein said acquiring data indicating one or more user reactions of a user responsive to display of the one or more first augmentations includes:

acquiring data indicating eye focus of one or more eyes of the user with respect to one or more aspects of one or more non-augmentation scene elements included in the one or more displayed augmented views.

17. The system of claim 1, wherein said acquiring data indicating one or more user reactions of a user responsive to display of the one or more first augmentations includes:

acquiring data indicating dwell time of one or more eyes of the user with respect to the one or more displayed augmented views, the dwell time having occurred proximate to the display of the one or more displayed augmented views.

18. The system of claim 1, wherein said acquiring data indicating one or more user reactions of a user responsive to display of the one or more first augmentations includes:

acquiring data indicating one or more facial expressions and/or head positions of the user with respect to the one or more augmentations, the one or more facial expressions, head positions, and/or head positions of the user having occurred proximate to the display of the one or more displayed augmented views.

19. The system of claim 1, wherein said associating the one or more user reactions with the one or more first augmentations includes:

ascertaining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations.

20. The system of claim 19, wherein said ascertaining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations includes:

ascertaining one or more relationships between the one or more user reactions and one or more shapes, sizes, colors, brightness, and/or hue of the one or more augmentations.

21. The system of claim 19, wherein said ascertaining one or more relationships between the one or more user reactions and one or more forms of the one or more augmentations includes:

ascertaining one or more relationships between the one or more user reactions and one or more locations of the one or more augmentations in the one or more displayed augmented views.

22. The system of claim 1, wherein said further comprising:

selecting one or more forms for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations.

23. The system of claim 22, wherein said selecting one or more forms for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations includes:

selecting at least one of a visual feature, shape, dimension, color, brightness, and/or hue for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations.

24. The system of claim 22, wherein said selecting one or more forms for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations includes:

selecting one or more specific locations in the one or more augmented views for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations.

25. The system of claim 24, wherein said selecting one or more specific locations in the one or more augmented views for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations includes:

selecting one or more specific locations relative to one or more non-augmentation scene elements in the one or more augmented views for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations.

26. The system of claim 22, wherein said selecting one or more forms for the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations includes:

selecting one or more one or more predetermined amounts of time for display of the one or more second augmentations based, at least in part, on association of the one or more user reactions with the one or more augmentations.

15. A system comprising:

at least one computing device; and

one or more instructions that, when implemented in the at least one computing device, program the at least one computing device for;

displaying at least one augmented view of at least one actual scene including one or more augmentations at a first location in the at least one augmented view;

acquiring one or more user reactions responsive to display of the one or more augmented views including the one or more augmentations, the one or more user reactions being indicative of dwell paths of one or more eyes of a user with respect to the one or more augmentations displayed at the first location;

identifying one or more relationships between the one or more dwell paths of the one or more eyes of the user and the one or more augmentations, the one or more dwell paths having been as a result of eye focus of the one or more eyes of the user moving away from the one or more augmentations; and

displaying one or more second augmentations in at least one augmented view based, at least in part, on the one or more relationships between the one or more dwell paths of the one or more eyes of the user and the one or more augmentations.

27. An article of manufacture, comprising:

a non-transitory storage medium bearing;

one or more instructions for detecting an anchor visual cue in an actual scene;

one or more instructions for displaying one or more first augmentations at a location in an augmented view of the actual scene responsive to the detection of the visual cue in the actual scene;

one or more instructions for acquiring data indicating one or more user reactions of a user responsive to display of the one or more first augmentations, the one or more user reactions being indicative of at least one of lack of eye focus or lack of dwell time with respect to the one or more augmentations displayed at the first location;

one or more instructions for associating the one or more user reactions with the one or more first augmentations; and

one or more instructions for displaying one or more second augmentations in one or more augmented views based, at least in part, on the associating the one or more user reactions with the one or more first augmentations.

28. A system comprising:

circuitry configured to acquire first data that at least identifies one or more augmentations that were displayed at a first location in one or more augmented views of one or more actual scenes at least partly in response to detection of at least one anchor visual cue detected in the one or more actual scenes;

circuitry configured to determine at least one of lack of eye focus or lack of dwell time of a user with respect to the one or more augmentations displayed at the first location; and

circuitry configured to relay one or more second augmentations for display in one or more augmented views in at least one location that is selected based at least partly on the determined at least one of lack of eye focus or lack of dwell time of the user with respect to the one or more augmentations.

29. The system of claim 28, wherein the circuitry configured to relay one or more second augmentations for display in one or more augmented views in at least one location that is selected based at least partly on the determined at least one of lack of eye focus or lack of dwell time of the users with respect to the one or more augmentations includes:

circuitry configured to relay one or more second augmentations for display in one or more augmented views in at least one location and format that are selected based at least partly on the determined at least one of lack of eye focus or lack of dwell time of the user with respect to the one or more augmentations.

30. The system of claim 28, wherein the circuitry configured to relay one or more second augmentations for display in one or more augmented views in at least one location that is selected based at least partly on the determined at least one of lack of eye focus or lack of dwell time of the users with respect to the one or more augmentations includes:

circuitry configured to relay one or more second augmentations for display in one or more augmented views in at least one location that is selected based at least partly on the determined at least one of lack of eye focus or lack of dwell time of the user with respect to the one or more augmentations and based at least partly on one or more paths of eye focus relative to the one or more augmentations.

31. The system of claim 28, wherein the circuitry configured to relay one or more second augmentations for display in one or more augmented views in at least one location that is selected based at least partly on the determined at least one of lack of eye focus or lack of dwell time of the users with respect to the one or more augmentations includes:

circuitry configured to relay one or more second augmentations for display in one or more augmented views (i) at least partly in response to detection of the at least one anchor visual cue in one or more actual scenes and (ii) in a second location that is different from the first location and that is selected based at least partly on the determined at least one of lack of eye focus or lack of dwell time of the user with respect to the one or more augmentations that were displayed at the first location.