CORRELATING USER REACTION WITH AT LEAST AN ASPECT ASSOCIATED WITH AN AUGMENTATION OF AN AUGMENTED VIEW

• US 20140098134A1
• Filed: 12/21/2012
• Published: 04/10/2014
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, John D. Jr., Tegreene, Clarence T.
• Granted Patent: US 9,671,863 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computationally implemented methods and systems include detecting one or more user reactions of a user in response to a display to the user of an augmented view of an actual scene from a real environment, the augmented view that was displayed including one or more augmentations, and correlating the detected one or more user reactions with at least one or more aspects associated with the one or more augmentations that were included in the augmented view that was presented. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1-141. -141. (canceled)

142. A system, comprising:

a user reaction ascertaining module configured to ascertain one or more user reactions of a user reacting to a display of an augmented view of an actual scene from a real environment, the augmented view that was displayed having included one or more augmentations;

a user reaction associating module configured to associate the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view; and

an augmented view presenting module configured to present a second augmented view of the actual scene or of a second actual scene from the real environment, the second augmented view having one or more second augmentations that have been included into the second augmented view based, at least in part, on the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations.

143. The system of claim 142, wherein said user reaction ascertaining module configured to ascertain one or more user reactions of a user reacting to a display of an augmented view of an actual scene from a real environment, the augmented view that was displayed having included one or more augmentations comprises:

a physiological characteristic determining module configured to determine the one or more user reactions of the user including sensing one or more physiological characteristics of the user proximate to the display of the augmented view.

144. The system of claim 143, wherein said physiological characteristic determining module configured to determine the one or more user reactions of the user including sensing one or more physiological characteristics of the user proximate to the display of the augmented view comprises:

an eye characteristic determining module configured to determine one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view.

145. The system of claim 144, wherein said eye characteristic determining module configured to determine one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view comprises:

an eye movement determining module configured to determine one or more eye movements of the one or more eyes of the user during and/or following the display of the augmented view.

146. The system of claim 145, wherein said eye movement determining module configured to determine one or more eye movements of the one or more eyes of the user during and/or following the display of the augmented view comprises:

an eye dwell path detecting module configured to detect dwell path of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view.

147. The system of claim 146, wherein said eye dwell path detecting module configured to detect dwell path of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

an eye dwell path detecting module configured to detect dwell path of the one or more eyes of the user as focus of the one or more eyes of the user moves away from the one or more augmentations included in the augmented view.

148. The system of claim 146, wherein said eye dwell path detecting module configured to detect dwell path of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

an eye dwell path detecting module configured to detect dwell path of the one or more eyes of the user as focus of the one or more eyes of the user moves between multiple augmentations included in the augmented view.

149. The system of claim 144, wherein said eye characteristic determining module configured to determine one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view comprises:

an eye attention determining module configured to determine eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view.

150. The system of claim 149, wherein said eye attention determining module configured to determine eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

an eye attention determining module configured to determine eye focus of the one or more eyes of the user with respect to the one or more aspects of the one or more augmentations during and/or following the display of the augmented view.

151. The system of claim 149, wherein said eye attention determining module configured to determine eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

an eye attention determining module configured to determine eye focus of the one or more eyes of the user with respect to one or more aspects of one or more non-augmentation scene elements included in the augmented view during and/or following the display of the augmented view.

152. The system of claim 149, wherein said eye attention determining module configured to determine eye focus of the one or more eyes of the user with respect to the augmented view during and/or following the display of the augmented view comprises:

an eye attention time determining module configured to determine dwell time of the one or more eyes of the user with respect to one or more aspects of the augmented view during and/or following the display of the augmented view.

153. The system of claim 144, wherein said eye characteristic determining module configured to determine one or more eye characteristics of one or more eyes of the user during and/or following the display of the augmented view comprises:

a pupil characteristic detecting module configured to detect one or more pupil characteristics of the one or more eyes of the user during and/or following the display of the augmented view.

154. The system of claim 153, wherein said pupil characteristic detecting module configured to detect one or more pupil characteristics of the one or more eyes of the user during and/or following the display of the augmented view comprises:

a pupil characteristic detecting module configured to detect shape and/or size of one or more pupils of the user during and/or following the display of the augmented view.

160. The system of claim 143, wherein said physiological characteristic determining module configured to determine the one or more user reactions of the user including sensing one or more physiological characteristics of the user proximate to the display of the augmented view comprises:

a physiological characteristic determining module configured to determine the one or more physiological characteristics of the user at least just prior to the display of the augmented view.

164. The system of claim 142, wherein said user reaction ascertaining module configured to ascertain one or more user reactions of a user reacting to a display of an augmented view of an actual scene from a real environment, the augmented view that was displayed having included one or more augmentations comprises:

an augmented view aspect recording module configured to record one or more aspects of the augmented view.

174. The system of claim 142, wherein said user reaction associating module configured to associate the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view comprises:

a physiological characteristic associating module configured to associate one or more physiological characteristics of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

175. The system of claim 174, wherein said physiological characteristic associating module configured to associate one or more physiological characteristics of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

an eye characteristic associating module configured to associate one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

176. The system of claim 175, wherein said eye characteristic associating module configured to associate one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

an eye movement associating module configured to associate one or more eye movements of the one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

177. The system of claim 176, wherein said eye movement associating module configured to associate one or more eye movements of the one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

a dwell path associating module configured to associate dwell path of the one or more eyes of the user that was tracked with respect to the augmented view during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations.

178. The system of claim 177, wherein said dwell path associating module configured to associate dwell path of the one or more eyes of the user that was tracked with respect to the augmented view during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

a dwell path associating module configured to associate dwell path of the one or more eyes of the user that was tracked as focus of the one or more eyes of the user moved away from the one or more augmentations included in the augmented view with the one or more aspects of the one or more augmentations.

179. The system of claim 177, wherein said dwell path associating module configured to associate dwell path of the one or more eyes of the user that was tracked with respect to the augmented view during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

a dwell path associating module configured to associate dwell path of the one or more eyes of the user that was tracked as focus of the one or more eyes of the user moved between multiple augmentations included in the augmented view with the one or more aspects of the one or more augmentations.

180. The system of claim 175, wherein said eye characteristic associating module configured to associate one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

an eye characteristic associating module including an eye focus associating module configured to associate eye focus of the one or more eyes of the user with respect to the augmented view that were detected proximate to the display of the augmented view with the one or more aspects of the one or more augmentations.

181. The system of claim 180, wherein said eye characteristic associating module including an eye focus associating module configured to associate eye focus of the one or more eyes of the user with respect to the augmented view that were detected proximate to the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

an eye focus associating module configured to associate the eye focus of the one or more eyes of the user with the one or more aspects of the one or more augmentations, the eye focus of the one or more eyes of the user to be associated being with respect to the one or more aspects of the one or more augmentations that were detected proximate to the display of the augmented view.

182. The system of claim 180, wherein said eye characteristic associating module including an eye focus associating module configured to associate eye focus of the one or more eyes of the user with respect to the augmented view that were detected proximate to the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

an eye focus associating module configured to associate the eye focus of the one or more eyes of the user with the one or more aspects of the one or more augmentations, the eye focus of the one or more eyes of the user being with respect to one or more aspects of one or more non-augmentation scene elements included in the augmented view that were detected proximate to the display of the augmented view.

184. The system of claim 175, wherein said eye characteristic associating module configured to associate one or more eye characteristics of one or more eyes of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

a pupil characteristic associating module configured to associate one or more pupil characteristics of the one or more eyes of the user that were detected proximate to the display of the augmented view with the one or more aspects of the one or more augmentations.

185. The system of claim 184, wherein said pupil characteristic associating module configured to associate one or more pupil characteristics of the one or more eyes of the user that were detected proximate to the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

a pupil shape/size associating module configured to associate shape and/or size of one or more pupils of the user that were detected proximate to the display of the augmented view with the one or more aspects of the one or more augmentations.

190. The system of claim 174, wherein said physiological characteristic associating module configured to associate one or more physiological characteristics of the user that were sensed during and/or following the display of the augmented view with the one or more aspects of the one or more augmentations comprises:

a physiological characteristic associating module configured to associate one or more of cardiopulmonary characteristic, skin characteristic and/or brain characteristic of the user that were detected proximate to the display of the augmented view with the one or more aspects of the one or more augmentations.

191. The system of claim 142, wherein said user reaction associating module configured to associate the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view comprises:

a user reaction associating module configured to associate at least one facial expression, a head position, or a head tilt of the user that was detected proximate to the display of the augmented view with the one or more augmentations that were included in the augmented view that was displayed.

192. The system of claim 142, wherein said user reaction associating module configured to associate the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view comprises:

a user reaction associating module configured to associate the one or more user reactions with one or more forms of the one or more augmentations included in the augmented view that was displayed.

193. The system of claim 192, wherein said user reaction associating module configured to associate the one or more user reactions with one or more forms of the one or more augmentations included in the augmented view that was displayed comprises:

a user reaction associating module configured to associate the one or more user reactions with one or more shapes and/or dimensions of the one or more augmentations.

194. The system of claim 192, wherein said user reaction associating module configured to associate the one or more user reactions with one or more forms of the one or more augmentations included in the augmented view that was displayed comprises:

a user reaction associating module configured to associate the one or more user reactions with one or more locations of the one or augmentations in the augmented view.

195. The system of claim 142, wherein said user reaction associating module configured to associate the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view comprises:

a user reaction associating module configured to associate the one or more user reactions with one or more aspects of one or more non-augmentation scene elements included in the augmented view.

196. The system of claim 195, wherein said user reaction associating module configured to associate the one or more user reactions with one or more aspects of one or more non-augmentation scene elements included in the augmented view comprises:

a user reaction associating module configured to associate the one or more user reactions with one or more locations of the one or more non-augmentation scene elements in the augmented view.

197. The system of claim 196, wherein said user reaction associating module configured to associate the one or more user reactions with one or more locations of the one or more non-augmentation scene elements in the augmented view comprises:

a user reaction associating module configured to associate the one or more user reactions with the one or more locations of the one or more non-augmentation scene elements relative to one or more locations of the one or more augmentations in the augmented view.

198. The system of claim 142, wherein said user reaction associating module configured to associate the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view comprises:

a user reaction associating module configured to associate the one or more user reactions with one or more non-augmentation scene elements that have been obfuscated in the augmented view by the one or more augmentations.

199. The system of claim 142, wherein said augmented view presenting module configured to present a second augmented view of the actual scene or of a second actual scene from the real environment, the second augmented view having one or more second augmentations that have been included into the second augmented view based, at least in part, on the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations comprises:

an augmented view presenting module configured to present a second augmented view that includes one or more second augmentations having one or more formats that were selected based, at least in part, on one or more results of the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations.

200. The system of claim 199, wherein said augmented view presenting module configured to present a second augmented view that includes one or more second augmentations having one or more formats that were selected based, at least in part, on one or more results of the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations comprises:

an augmented view presenting module configured to present a second augmented view that includes one or more second augmentations that are selectively placed at one or more particular locations in the in the second augmented view, the placements of the one or more second augmentations at the one or more particular locations in the second augmented view being based, at least in part, on the associating.

201. The system of claim 199, wherein said augmented view presenting module configured to present a second augmented view that includes one or more second augmentations having one or more formats that were selected based, at least in part, on one or more results of the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations comprises:

an augmented view presenting module configured to present a second augmented view that includes one or more second augmentations that have one or more particular dimensions or shapes, the one or more particular dimensions or shapes of the one or more second augmentations being based, at least in part, on the associating.

202. The system of claim 142, wherein said augmented view presenting module configured to present a second augmented view of the actual scene or of a second actual scene from the real environment, the second augmented view having one or more second augmentations that have been included into the second augmented view based, at least in part, on the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations comprises:

an augmented view presenting module configured to present a second augmented view of the actual scene that includes one or more second augmentations that replaces the one or more augmentations in the second augmented view.

203. The system of claim 142, further comprising:

a correlation result relaying module configured to relay one or more results of the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations; and

an augmentation acquiring module configured to acquire, in response to said relaying, one or more second augmentations for presenting in a second augmented view of the actual scene or of a second actual scene from the real environment.

155-159. -159. (canceled)

161-163. -163. (canceled)

165-173. -173. (canceled)

183. (canceled)

186-189. -189. (canceled)

204-210. -210. (canceled)

211. An article of manufacture, comprising:

a non-transitory storage medium bearing;

one or more instructions for ascertaining one or more user reactions of a user reacting to a display of an augmented view of an actual scene from a real environment, the augmented view that was displayed having included one or more augmentations;

one or more instructions for associating the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view; and

one or more instructions for presenting a second augmented view of the actual scene or of a second actual scene from the real environment, the second augmented view having one or more second augmentations that have been included into the second view based, at least in part, on the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations.

212. A wearable augmented reality (AR) device, comprising:

a coupling device for coupling the wearable AR device to a user'"'"'s head;

one or more display monitors;

a user reaction ascertaining module configured to ascertain one or more user reactions of a user reacting to a display of an augmented view of an actual scene from a real environment, the augmented view that was displayed having included one or more augmentations;

a user reaction associating module configured to associate the one or more user reactions with at least one or more aspects associated with the one or more augmentations that were displayed with the augmented view; and

an augmented view presenting module configured to present a second augmented view of the actual scene or of a second actual scene from the real environment, the second augmented view having one or more second augmentations that have been included into the second view based, at least in part, on the associating of the one or more user reactions with the at least one or more aspects associated with the one or more augmentations.