Correlating user reaction with at least an aspect associated with an augmentation of an augmented view

• US 10,180,715 B2
• Filed: 12/20/2012
• Issued: 01/15/2019
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, Jr., John D., Tegreene, Clarence T.
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US13/721,340
• Publication Number:
  US 20140098133A1
• Time in Patent Office: 2,217 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computationally implemented methods and systems include detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations, and correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1. A system, comprising:

circuitry configured for displaying an augmented view of a real environment to a user, the augmented view of the real environment including at least a first augmentation;

circuitry configured for detecting one or more user reactions, the one or more user reactions proximate to the display of the augmented view of the real environment to the user;

circuitry configured for correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment;

circuitry configured for determining one or more parties associated with one or more portions of the real environment;

circuitry configured for transmitting one or more results of the correlating to one or more network servers associated with the one or more parties; and

circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation.

2. The system of claim 1, wherein circuitry configured for detecting one or more user reactions, the one or more user reactions proximate to the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing one or more physiological characteristics of the user proximate to the display of the augmented view of the real environment to the user.

3. The system of claim 2, wherein circuitry configured for sensing one or more physiological characteristics of the user proximate to the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing one or more eye characteristics of one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user.

4. The system of claim 3, wherein circuitry configured for sensing one or more eye characteristics of one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing one or more eye movements of the one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user.

5. The system of claim 4, wherein circuitry configured for sensing one or more eye movements of the one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for tracking at least one dwell path of the one or more eyes of the user with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment to the user.

6. The system of claim 5, wherein circuitry configured for tracking at least one dwell path of the one or more eyes of the user with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for tracking at least one dwell path of the one or more eyes of the user as eye focus of the one or more eyes of the user moves at least one of (i) away from the first augmentation included in the augmented view of the real environment or (ii) between the first augmentation and at least one other augmentation included in the augmented view of the real environment.

7. The system of claim 3, wherein circuitry configured for sensing one or more eye characteristics of one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing eye focus of the one or more eyes of the user with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment to the user.

8. The system of claim 7, wherein circuitry configured for sensing eye focus of the one or more eyes of the user with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing eye focus of the one or more eyes of the user with respect to one or more aspects of at least one of the first augmentation or at least one other augmentation at least one of during or following the display of the augmented view of the real environment to the user.

9. The system of claim 7, wherein circuitry configured for sensing eye focus of the one or more eyes of the user with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing eye focus of the one or more eyes of the user with respect to one or more aspects of one or more non-augmentation scene elements included in the augmented view at least one of during or following the display of the augmented view of the real environment to the user.

10. The system of claim 7, wherein circuitry configured for sensing eye focus of the one or more eyes of the user with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing dwell time of the one or more eyes of the user with respect to one or more aspects of the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment to the user.

11. The system of claim 3, wherein circuitry configured for sensing one or more eye characteristics of one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing one or more pupil characteristics of the one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user.

12. The system of claim 11, wherein circuitry configured for sensing one or more pupil characteristics of the one or more eyes of the user at least one of during or following the display of the augmented view of the real environment to the user comprises:

circuitry configured for sensing at least one of shape or size of one or more pupils of the user at least one of during or following the display of the augmented view of the real environment to the user.

13. The system of claim 1, wherein circuitry configured for detecting one or more user reactions, the one or more user reactions proximate to the display of the augmented view of the real environment to the user comprises:

circuitry configured for detecting one or more user reactions, the one or more user reactions occurring prior to the display of the augmented view of the real environment to the user.

14. The system of claim 1, where circuitry configured for correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment comprises:

circuitry configured for correlating one or more eye movements of one or more eyes of the user that were sensed at least one of during or following the display of the augmented view of the real environment with one or more aspects of the first augmentation included in the augmented view of the real environment.

15. The system of claim 14, wherein circuitry configured for correlating one or more eye movements of one or more eyes of the user that were sensed at least one of during or following the display of the augmented view of the real environment with one or more aspects of the first included in the augmented view of the real environment comprises:

circuitry configured for correlating at least one dwell path of the one or more eyes of the user that was tracked with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment with one or more aspects of the first augmentation included in the augmented view of the real environment.

16. The system of claim 15, wherein circuitry configured for correlating at least one dwell path of the one or more eyes of the user that was tracked with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment with one or more aspects of the first augmentation included in the augmented view of the real environment comprises:

circuitry configured for correlating at least one dwell path of the one or more eyes of the user that was tracked as focus of the one or more eyes of the user moved away from the first augmentation included in the augmented view of the real environment with one or more aspects of the first augmentation included in the augmented view of the real environment.

17. The system of claim 15, wherein circuitry configured for correlating at least one dwell path of the one or more eyes of the user that was tracked with respect to the augmented view of the real environment at least one of during or following the display of the augmented view of the real environment with one or more aspects of the first augmentation included in the augmented view of the real environment comprises:

circuitry configured for correlating at least one dwell path of the one or more eyes of the user that was tracked as focus of the one or more eyes of the user moved between the first augmentation included in the augmented view of the real environment and at least one other augmentation included in the augmented view of the real environment with one or more aspects of the first augmentation included in the augmented view of the real environment.

18. The system of claim 1, wherein circuitry configured for correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment comprises:

circuitry configured for correlating the one or more user reactions with one or more non-augmentation scene elements included in the augmented view of the real environment.

19. The system of claim 18, wherein circuitry configured for correlating the one or more user reactions with one or more non-augmentation scene elements included in the augmented view of the real environment comprises:

circuitry configured for correlating the one or more user reactions with one or more locations of the one or more non-augmentation scene elements included in the augmented view of the real environment.

20. The system of claim 19, wherein circuitry configured for correlating the one or more user reactions with one or more locations of the one or more non-augmentation scene elements included in the augmented view of the real environment comprises:

circuitry configured for correlating the one or more user reactions with the one or more locations of the one or more non-augmentation scene elements included in the augmented view of the real environment relative to a location of the first augmentation included in the augmented view of the real environment, including at least determining whether the user has at least one of behaved or reacted differently depending on where one or more non-augmentation scene elements are located in a second augmented view of the real environment relative to a location of the first augmented view of the real environment.

21. The system of claim 1, wherein circuitry configured for correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment comprises:

circuitry configured for correlating the one or more user reactions with one or more non-augmentation scene elements that have been hidden in the augmented view of the real environment by the first augmentation included in the augmented view of the real environment, including at least correlating the one or more user reactions with detection that at least one portion of the first augmentation included in the augmented view is overlaid over one or more non-augmentation scene elements of interest to the user, hiding the one or more non-augmentation scene elements of interest to the user from view of the user to cause the one or more user reactions.

22. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation having one or more formats that were selected by the one or more parties, based at least in part, on the one or more results of the correlating that were transmitted to the one or more network servers associated with the one or more parties.

23. The system of claim 22, wherein circuitry configured for receiving a second augmentation having one or more formats that were selected by the one or more parties, based at least in part, on the one or more results of the correlating that were transmitted to the one or more network servers associated with the one or more parties comprises:

circuitry configured for receiving a second augmentation designed to be placed at one or more particular locations in the augmented view of the real environment, the second augmentation designed to be placed at one or more particular locations in the augmented view of the real environment based, at least in part, on the one or more results of the correlating that were transmitted to the one or more network servers associated with the one or more parties, the one or more particular locations in the augmented view of the real environment at which the second augmentation is to be placed determined by the one or more parties to be advantageous for the one or more parties.

24. The system of claim 22, wherein circuitry configured for receiving a second augmentation having one or more formats that were selected by the one or more parties, based at least in part, on the one or more results of the correlating that were transmitted to the one or more network servers associated with the one or more parties comprises:

circuitry configured for receiving a second augmentation designed to have at least one of one or more particular dimensions or one or more particular shapes, the second augmentation designed to have at least one of one or more particular dimensions or one or more particular shapes based, at least in part, on the one or more results of the correlating that were transmitted to the one or more network servers associated with the one or more parties.

25. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation including at least one or more visibility rules specified by the one or more parties, related to the displaying the augmented view of the real environment to the user in a manner advantageous to the one or more parties, and based, at least in part, on the one or more results of the correlating that were transmitted to the one or more network servers associated with the one or more parties.

26. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation having a different format than the first augmentation and including at least one or more visibility rules specified by the one or more parties and related to inclusion of the second augmentation in the augmented view of the real environment.

27. The system of claim 26, wherein circuitry configured for receiving a second augmentation having a different format than the first augmentation and including at least one or more visibility rules specified by the one or more parties and related to inclusion of the second augmentation in the augmented view of the real environment comprises:

circuitry configured for receiving a second augmentation having at least one of a size, shape, color, or length of visibility different than the first augmentation and including at least one or more visibility rules specified by the one or more parties and related to inclusion of the second augmentation in the augmented view of the real environment.

28. The system of claim 1, further comprising:

circuitry configured for displaying the second augmentation.

29. The system of claim 1, wherein circuitry configured for correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment comprises:

circuitry configured for linking at least two variables, including at least linking one or more augmentations displayed to the user with two separate user reactions including at least one eye movement and at least one dwell time detected proximate to display of the first augmentation included in the augmented view of the real environment.

30. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation including at least one or more of at least some instructions or at least some directives specified by the one or more parties defining when the second augmentation may be presented in association with the displaying the augmented view of the real environment to the user.

31. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation including at least one or more of at least some instructions or at least some directives defining how the second augmentation may be presented in response to a particular orientation of an augmented reality device displaying the augmented view of the real environment to the user being detected.

32. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation including at least one or more visibility rules relating to display of the second augmentation when a particular anchor visual cue is detected as being present in the real environment, including at least receiving a second augmentation operable to cause the augmented view of the real environment to present a visual image related to at least one of one or more goods or one or more services associated with the one or more parties in response to detecting a particular anchor visual cue relating to the one or more parties as being present in the real environment.

33. The system of claim 1, wherein circuitry configured for transmitting one or more results of the correlating to one or more network servers associated with the one or more parties comprises:

circuitry configured for transmitting, to the one or more network servers associated with the one or more parties, at least one indication of at least one inference that the user is bothered by at least one aspect of at least one augmentation associated with the one or more parties being placed at the center of a field of view of the user at least partially based on the user looking away quickly from the center of the field of view of the user.

34. The system of claim 1, wherein circuitry configured for determining one or more parties associated with one or more portions of the real environment comprises:

circuitry configured for using image recognition to determine the one or more parties at least partially based on recognizing at least one of a sign, a logo, an architectural cue, a building, or a structure associated with the one or more parties in the real environment.

35. The system of claim 1, wherein circuitry configured for determining one or more parties associated with one or more portions of the real environment comprises:

circuitry configured for using audio recognition to determine the one or more parties at least partially based on recognizing at least some audio input data including at least one sound associated with the one or more parties in the real environment.

36. The system of claim 1, wherein circuitry configured for correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment comprises:

circuitry configured for inferring that the user is bothered by at least one aspect of the augmented view being placed at the center of a field of view of the user at least partially based on the one or more user reactions.

37. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one posted tag associated with disabling of the at least one functional augmentation.

38. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation in response to a request made by the one or more parties to disable the at least one functional augmentation.

39. The system of claim 38, wherein circuitry configured for receiving a second augmentation in response to a request made by the one or more parties to disable the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation in response to a request made by the one or more parties to disable at least one functional calculator augmentation.

40. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation in response to a command transmitted by the one or more parties to an augmented reality (AR) device on which the at least one functional augmentation is displayed for the AR device to disable the at least one functional augmentation.

41. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for detecting the at least one signature associated with disabling of the at least one functional augmentation; and

circuitry configured for disabling the at least one functional augmentation responsive to detecting the at least one signature associated with disabling of the at least one functional augmentation.

42. The system of claim 41, wherein circuitry configured for detecting the at least one signature associated with disabling of the at least one functional augmentation comprises:

circuitry configured for detecting at least one tag posted by the one or more parties, that at least one tag associated with disabling of the at least one functional augmentation.

43. The system of claim 1, wherein circuitry configured for receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation comprises:

circuitry configured for receiving a second augmentation in response to a request made by the one or more parties to disable at least one functional calculator augmentation in response to detecting the first user activating the at least one functional calculator augmentation as at least one of the one or more user reactions.

44. A method, comprising:

displaying an augmented view of a real environment to a user, the augmented view of the real environment including at least a first augmentation;

detecting one or more user reactions, the one or more user reactions proximate to the display of the augmented view of the real environment to the user;

correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment;

determining one or more parties associated with one or more portions of the real environment;

transmitting one or more results of the correlating to one or more network servers associated with the one or more parties; and

receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation,

wherein at least one of the displaying, detecting, correlating, determining, transmitting, or receiving is at least partially implemented using at least one processing device.

45. A system, comprising:

at least one computing device; and

one or more instructions that, when executed by the at least one computing device, cause the at least one computing device to perform one or more operations including at least;

displaying an augmented view of a real environment to a user, the augmented view of the real environment including at least a first augmentation;

detecting one or more user reactions, the one or more user reactions proximate to the display of the augmented view of the real environment to the user;

correlating the one or more user reactions with the first augmentation included in the augmented view of the real environment;

determining one or more parties associated with one or more portions of the real environment;

transmitting one or more results of the correlating to one or more network servers associated with the one or more parties; and

receiving a second augmentation including at least one or more visibility rules related to disabling at least one functional augmentation included in the augmented view of the real environment in response to detection of at least one signature associated with disabling of the at least one functional augmentation, the one or more parties associated with one or more visibility rules requiring disabling of the at least one functional augmentation.