CORRELATING USER REACTION WITH AT LEAST AN ASPECT ASSOCIATED WITH AN AUGMENTATION OF AN AUGMENTED VIEW

• US 20140098133A1
• Filed: 12/20/2012
• Published: 04/10/2014
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, John D. Jr., Tegreene, Clarence T.
• Granted Patent: US 10,180,715 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computationally implemented methods and systems include detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations, and correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1. A computationally-implemented method, comprising:

detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations; and

correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented.

2-70. -70. (canceled)

71. A computationally-implemented system, comprising:

means for detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations; and

means for correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented.

72. The computationally-implemented system of claim 71, wherein said means for detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations comprises:

means for detecting the one or more user reactions of the user including sensing one or more physiological characteristics of the user proximate to the display of the augmented view.

73. The computationally-implemented system of claim 72, wherein said means for detecting the one or more user reactions of the user including sensing one or more physiological characteristics of the user proximate to the display of the augmented view comprises:

means for sensing the one or more physiological characteristics of the user including sensing one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view.

74. The computationally-implemented system of claim 73, wherein said means for sensing the one or more physiological characteristics of the user including sensing one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view comprises:

means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing one or more eye movements of the one or more eyes of the user during and/or following the display of the augmented view.

75. The computationally-implemented system of claim 74, wherein said means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing one or more eye movements of the one or more eyes of the user during and/or following the display of the augmented view comprises:

means for sensing the one or more eye movements of the one or more eyes of the user including tracking dwell path of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view.

76. The computationally-implemented system of claim 75, wherein said means for sensing the one or more eye movements of the one or more eyes of the user including tracking dwell path of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

means for tracking the dwell path of the one or more eyes of the user with respect to the augmented view including tracking dwell path of the one or more eyes of the user as focus of the one or more eyes of the user moves away from the one or more augmentations included in the augmented view.

77. The computationally-implemented system of claim 75, wherein said means for sensing the one or more eye movements of the one or more eyes of the user including tracking dwell path of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

means for tracking the dwell path of the one or more eyes of the user with respect to the augmented view including tracking dwell path of the one or more eyes of the user as focus of the one or more eyes of the user moves between multiple augmentations included in the augmented view.

78. The computationally-implemented system of claim 73, wherein said means for sensing the one or more physiological characteristics of the user including sensing one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view comprises:

means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view.

79. The computationally-implemented system of claim 78, wherein said means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

means for sensing the eye focus of the one or more eyes of the user including sensing eye focus of the one or more eyes of the user with respect to the one or more aspects of the one or more augmentations during and/or following the display of the augmented view.

80. The computationally-implemented system of claim 78, wherein said means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

means for sensing the eye focus of the one or more eyes of the user including sensing eye focus of the one or more eyes of the user with respect to the one or more aspects of one or more non-augmentation scene elements included in the augmented view during and/or following the display of the augmented view.

81. The computationally-implemented system of claim 78, wherein said means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

means for sensing the eye focus of the one or more eyes of the user including sensing dwell time of the one or more eyes of the user with respect to the one or more aspects of the augmented view during and/or following the display of the augmented view.

82. The computationally-implemented system of claim 73, wherein said means for sensing the one or more physiological characteristics of the user including sensing one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view comprises:

means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing one or more pupil characteristics of the one or more eyes of the user during and/or following the display of the augmented view.

83. The computationally-implemented system of claim 82, wherein said means for sensing the one or more eye characteristics of the one or more eyes of the user including sensing one or more pupil characteristics of the one or more eyes of the user during and/or following the display of the augmented view comprises:

means for sensing the one or more pupil characteristics of the one or more eyes of the user including sensing shape and/or size of one or more pupils of the user during and/or following the display of the augmented view.

89. The computationally-implemented system of claim 72, wherein said means for detecting the one or more user reactions of the user including sensing one or more physiological characteristics of the user proximate to the display of the augmented view comprises:

means for sensing the one or more physiological characteristics of the user proximate to the display of the augmented view including sensing the one or more physiological characteristics of the user at least just prior to the display of the augmented view.

93. The computationally-implemented system of claim 71, wherein said means for detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations comprises:

means for detecting the one or more user reactions of the user in response to the display of the augmented view including registering one or more aspects of the augmented view.

103. The computationally-implemented system of claim 71, wherein said means for correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented comprises:

means for correlating the detected one or more user reactions with the at least one or more aspects associated with the one or more augmentations by correlating one or more physiological characteristics of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

104. The computationally-implemented system of claim 103, wherein said means for correlating the detected one or more user reactions with the at least one or more aspects associated with the one or more augmentations by correlating one or more physiological characteristics of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the one or more physiological characteristics of the user with the one or more aspects of the one or more augmentations by correlating one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

105. The computationally-implemented system of claim 104, wherein said means for correlating the one or more physiological characteristics of the user with the one or more aspects of the one or more augmentations by correlating one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the one or more eye characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating one or more eye movements of the one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

106. The computationally-implemented system of claim 105, wherein said means for correlating the one or more eye characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating one or more eye movements of the one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the sensed one or more eye movements of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating dwell path of the one or more eyes of the user that was tracked with respect to the augmented view during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

107. The computationally-implemented system of claim 106, wherein said means for correlating the sensed one or more eye movements of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating dwell path of the one or more eyes of the user that was tracked with respect to the augmented view during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the dwell path of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating dwell path of the one or more eyes of the user that was tracked as focus of the one or more eyes of the user moved away from the one or more augmentations included in the augmented view with the one or more aspects of the one or more augmentations.

108. The computationally-implemented system of claim 106, wherein said means for correlating the sensed one or more eye movements of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating dwell path of the one or more eyes of the user that was tracked with respect to the augmented view during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the dwell path of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating dwell path of the one or more eyes of the user that was tracked as focus of the one or more eyes of the user moved between multiple augmentations included in the augmented view with the one or more aspects of the one or more augmentations.

109. The computationally-implemented system of claim 104, wherein said means for correlating the one or more physiological characteristics of the user with the one or more aspects of the one or more augmentations by correlating one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the one or more eye characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating eye focus of the one or more eyes of the user with respect to the augmented view that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

110. The computationally-implemented system of claim 109, wherein said means for correlating the one or more eye characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating eye focus of the one or more eyes of the user with respect to the augmented view that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the eye focus of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating eye focus of the one or more eyes of the user with the one or more aspects of the one or more augmentations, the eye focus of the one or more eyes of the user to be correlated being with respect to the one or more aspects of the one or more augmentations that were sensed during and/or following the display of the augmented view.

111. The computationally-implemented system of claim 109, wherein said means for correlating the one or more eye characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating eye focus of the one or more eyes of the user with respect to the augmented view that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the eye focus of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating the eye focus of the one or more eyes of the user with the one or more aspects of the one or more augmentations, the eye focus of the one or more eyes of the user being with respect to one or more aspects of one or more non-augmentation scene elements included in the augmented view that were sensed during and/or following the display of the augmented view.

113. The computationally-implemented system of claim 104, wherein said means for correlating the one or more physiological characteristics of the user with the one or more aspects of the one or more augmentations by correlating one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the one or more eye characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating one or more pupil characteristics of the one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

114. The computationally-implemented system of claim 113, wherein said means for correlating the one or more eye characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating one or more pupil characteristics of the one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the one or more pupil characteristics of the one or more eyes of the user with the one or more aspects of the one or more augmentations by correlating shape and/or size of one or more pupils of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

119. The computationally-implemented system of claim 103, wherein said means for correlating the detected one or more user reactions with the at least one or more aspects associated with the one or more augmentations by correlating one or more physiological characteristics of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

means for correlating the one or more physiological characteristics of the user with the one or more aspects of the one or more augmentations by correlating one or more cardiopulmonary characteristic, skin characteristic and/or brain characteristic of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

120. The computationally-implemented system of claim 71, wherein said means for correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented comprises:

means for correlating the detected one or more user reactions with at least the one or more aspects associated with the one or more augmentations by correlating at least one of a facial expression, a head position, or a head tilt of the user that was detected during and/or following the display of the augmented view with the one or more augmentations that were included in the augmented view that was displayed.

121. The computationally-implemented system of claim 71, wherein said means for correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented comprises:

means for correlating the detected one or more user reactions with one or more forms of the one or more augmentations included in the augmented view that was presented.

122. The computationally-implemented system of claim 121, wherein said means for correlating the detected one or more user reactions with one or more forms of the one or more augmentations included in the augmented view that was presented comprises:

means for correlating the detected one or more user reactions with one or more shapes and/or dimensions of the one or more augmentations.

123. The computationally-implemented system of claim 121, wherein said means for correlating the detected one or more user reactions with one or more forms of the one or more augmentations included in the augmented view that was presented comprises:

means for correlating the detected one or more user reactions with one or more locations of the one or more augmentations in the augmented view.

124. The computationally-implemented system of claim 71, wherein said means for correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented comprises:

means for correlating the detected one or more user reactions with the at least one or more aspects associated with the one or more augmentations including correlating the detected one or more user reactions with one or more aspects of one or more non-augmentation scene elements included in the augmented view.

125. The computationally-implemented system of claim 124, wherein said means for correlating the detected one or more user reactions with the at least one or more aspects associated with the one or more augmentations including correlating the detected one or more user reactions with one or more aspects of one or more non-augmentation scene elements included in the augmented view comprises:

means for correlating the detected one or more user reactions with the one or more aspects of the one or more non-augmentation scene elements included in the augmented view by correlating the detected one or more user reactions with one or more locations of the one or more non-augmentation scene elements in the augmented view.

126. The computationally-implemented system of claim 125, wherein said means for correlating the detected one or more user reactions with the one or more aspects of the one or more non-augmentation scene elements included in the augmented view by correlating the detected one or more user reactions with one or more locations of the one or more non-augmentation scene elements in the augmented view comprises:

means for correlating the detected one or more user reactions with the one or more locations of the one or more non-augmentation scene elements in the augmented view including correlating the detected one or more user reactions with the one or more locations of the one or more non-augmentation scene elements relative to one or more locations of the one or more augmentations in the augmented view.

127. The computationally-implemented system of claim 71, wherein said means for correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented comprises:

means for correlating the detected one or more user reactions with one or more non-augmentation scene elements that have been obfuscated in the augmented view by the one or more augmentations.

128. The computationally-implemented system of claim 71, further comprising:

means for transmitting one or more results of the correlating; and

means for receiving, in response to said transmitting, one or more second augmentations for displaying in a second augmented view of the actual scene or of a second actual scene from the real environment.

129. The computationally-implemented system of claim 128, wherein said means for transmitting one or more results of the correlating comprises:

means for transmitting the one or more results of the correlating via one or more wireless and/or wired networks.

130. The computationally-implemented system of claim 128, wherein said means for transmitting one or more results of the correlating comprises:

means for transmitting the one or more results of the correlating to one or more network servers associated with one or more independent parties.

131. The computationally-implemented system of claim 128 wherein said means for transmitting one or more results of the correlating comprises:

means for transmitting the one or more results of the correlating by transmitting one or more defined relationships between the detected one or more user reactions and the at least one or more aspects associated with the one or more augmentations.

132. The computationally-implemented system of claim 128, wherein said means for receiving, in response to said transmitting, one or more second augmentations for displaying in a second augmented view of the actual scene or of a second actual scene from the real environment comprises:

means for receiving the one or more second augmentations for the displaying including receiving one or more second augmentations having one or more formats that were selected based, at least in part, on the one or more results of the correlating that were transmitted.

133. The computationally-implemented system of claim 132, wherein said means for receiving the one or more second augmentations for the displaying including receiving one or more second augmentations having one or more formats that were selected based, at least in part, on the one or more results of the correlating that were transmitted comprises:

means for receiving the one or more second augmentations having the one or more formats including receiving one or more second augmentations that are designed to be placed at one or more particular locations in the second augmented view, the placement or placements of the one or more second augmentations at the one or more particular locations in the second augmented view being based, at least in part, on the one or more results of the correlating that were transmitted.

134. The computationally-implemented system of claim 132, wherein said means for receiving the one or more second augmentations for the displaying including receiving one or more second augmentations having one or more formats that were selected based, at least in part, on the one or more results of the correlating that were transmitted comprises:

means for receiving the one or more second augmentations having the one or more formats including receiving one or more second augmentations that are designed to have one or more particular dimensions or shapes, the one or more particular dimensions or shapes of the one or more second augmentations being based, at least in part, on the one or more results of the correlating that were transmitted.

135. The computationally-implemented system of claim 128, wherein said means for receiving, in response to said transmitting, one or more second augmentations for displaying in a second augmented view of the actual scene or of a second actual scene from the real environment comprises:

means for receiving the one or more second augmentations for the displaying including receiving one or more visibility rules for displaying the one or more second augmentations that are based, at least in part, on the one or more results of the correlating that were transmitted.

84-88. -88. (canceled)

90-92. -92. (canceled)

94-102. -102. (canceled)

112. (canceled)

115-118. -118. (canceled)

136-140. -140. (canceled)

141. A system, comprising:

circuitry for detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations; and

circuitry for correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented.