PRESENTING AN AUGMENTED VIEW IN RESPONSE TO ACQUISITION OF DATA INFERRING USER ACTIVITY

• US 20140098128A1
• Filed: 11/09/2012
• Published: 04/10/2014
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, John D. Jr., Tegreene, Clarence T.
• Granted Patent: US 9,448,623 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computationally implemented methods and systems include obtaining visual data of an actual view of a scene from a real environment, determining whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have at least been acquired, and presenting, in response at least in part to determining that the activity-inferring data have at least been acquired, an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1-141. -141. (canceled)

142. A system, comprising:

one or more cameras;

a visual data obtaining module configured to obtain from the one or more cameras visual data of an actual view of a scene from a real environment;

an activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired; and

an augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired.

149. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

an activity-inferring visual data acquisition determining module configured to determine whether visual data that was obtained includes activity-inferring visual data that infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

150. The system of claim 149, wherein said activity-inferring visual data acquisition determining module configured to determine whether visual data that was obtained includes activity-inferring visual data that infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment comprises:

a visual cue determining module configured to determine whether the activity-inferring visual data that was obtained includes the activity-inferring visual data that provides multiple visual cues that when detected in combination infers the at least initial occurrence of the one or more user activities.

151. The system of claim 150 wherein said visual cue determining module configured to determine whether the activity-inferring visual data that was obtained includes the activity-inferring visual data that provides multiple visual cues that when detected in combination infers the at least initial occurrence of the one or more user activities comprises:

a visual cue determining module configured to determine whether the activity-inferring visual data that was obtained includes the activity-inferring visual data that indicates presence of multiple visible items in the scene from the real environment that when detected in combination infers the at least initial occurrence of the one or more user activities.

152. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

an activity-inferring audio data acquisition determining module configured to determine whether activity-inferring audio data have been acquired that infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

153. The system of claim 152, wherein said activity-inferring audio data acquisition determining module configured to determine whether activity-inferring audio data have been acquired that infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment comprises:

an activity-inferring audio data acquisition determining module configured to determine whether the activity-inferring audio data have been acquired that includes data for one or more identifiable sounds that when detected infers the at least initial occurrence of the one or more user activities.

155. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

a movement data acquisition determining module configured to determine whether movement data have been acquired that indicates one or more movements that infer the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

156. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

a personal information manager (PIM) data acquisition determining module configured to determine whether personal information manager (PIM) data have been acquired that infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

157. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

a sensor-provided data acquisition determining module configured to determine whether sensor-provided data have been acquired that infers occurrence of the one or more user activities associated with the scene from the real environment.

158. The computationally-implemented system of claim 157, wherein said sensor-provided data acquisition determining module configured to determine whether sensor-provided data have been acquired that infers occurrence of the one or more user activities associated with the scene from the real environment comprises:

a sensor-provided data acquisition determining module configured to determine whether sensor-provided data indicating one or more user physiological characteristics have been acquired that when detected as occurring infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

159. The computationally-implemented system of claim 157, wherein said sensor-provided data acquisition determining module configured to determine whether sensor-provided data have been acquired that infers occurrence of the one or more user activities associated with the scene from the real environment comprises:

a sensor-provided data acquisition determining module configured to determine whether sensor-provided data that indicates one or more characteristics associated with one or more user-employed machines that when detected as occurring infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

160. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

a social networking data acquisition determining module configured to determine whether social networking data have been acquired that infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

161. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

an activity-inferring data acquisition determining module including an activity-inferring visual data acquisition determining module, activity-inferring audio data acquisition determining module, location/orientation data acquisition determining module, movement data acquisition determining module, personal information manager (PIM) data acquisition determining module, sensor-provided data acquisition determining module, and/or social networking data acquisition determining module configured to determine whether the activity-inferring data have been acquired by determining whether the visual data that was obtained includes activity-inferring visual data that infers the at least initial occurrence of the one or more user activities have been acquired and determining whether one or more of activity-inferring audio data, location and/or orientation data, movement data, personal information manager (PIM) data, social networking data, and/or sensor-provided data have been acquired that infers the at least initial occurrence of the one or more user activities associated with the scene from the real environment.

163. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

an activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user leisure activities have been acquired.

164. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

an activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user machine operating activities have been acquired.

166. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

an activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user work or professional activities have been acquired.

167. The system of claim 142, wherein said activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired comprises:

an activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user academic activities have been acquired.

178. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmented view presenting module configured to present an augmented view including one or more dynamic functional augmentations that provide one or more dynamically-changing functional information.

187. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more time information and/or user physiological information in response to determining that activity-inferring data that infers one or more exercise and/or athletic activities have been acquired.

188. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more retail information in response to determining that activity-inferring data that infers one or more shopping activities have been acquired.

189. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more dietary information in response to determining that activity-inferring data that infers one or more dining activities have been acquired.

190. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more business, financial, and/or market information in response to determining that activity-inferring data that infers one or more employment or professional activities have been acquired.

191. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more environmental and/or weather information in response to determining that activity-inferring data that infers one or more leisure activities have been acquired.

192. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more reference information in response to determining that activity-inferring data that infers one or more educational and/or academic activities have been acquired.

193. The system of claim 142, wherein said augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired comprises:

an augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view in response, at least in part, to determining that the activity-inferred data have at least been acquired.

194. The system of claim 193, wherein said augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view in response, at least in part, to determining that the activity-inferred data have at least been acquired comprises:

an augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view via one or more wireless and/or wired networks.

195. The computationally-implemented system of claim 194, wherein said augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view via one or more wireless and/or wired networks comprises:

an augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with the scene from the real environment.

196. The computationally-implemented system of claim 195, wherein said augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with the scene from the real environment comprises:

an augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with one or more locations associated with the scene from the real environment.

197. The computationally-implemented system of claim 195, wherein said augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with the scene from the real environment comprises:

an augmentation soliciting and receiving module configured to solicit for and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with one or more items detected in the scene from the real environment.

198. The system of claim 142, further comprising:

a visual data obtaining module configured to obtain visual data from the one or more cameras of a second scene from the real environment, wherein the scene from the real environment is a first scene from the real environment;

an activity-inferring data acquisition determining module configured to determine whether a second activity-inferring data that infers the at least initial occurrence of one or more second user activities associated with the second scene from the real environment have been acquired, wherein the activity-inferring data that infers the at least initial occurrence of the one or more user activities of the first scene from the real environment is a first activity-inferring data; and

an augmented view presenting module configured to present, in response at least in part to determining by the activity-inferring data acquisition determining module that the second activity-inferring data have at least been acquired, a second augmented view of the second scene from the real environment, the second augmented view including one or more second augmentations that have been included into the second augmented view based, at least in part, on the second activity-inferring data, and wherein the augmented view including the one or more augmentations of the scene from the real environment is a first augmented view including one or more first augmentations of the first scene from the real environment;

202. The system of claim 198, wherein said augmented view presenting module configured to present, in response at least in part to determining by the activity-inferring data acquisition determining module that the second activity-inferring data have at least been acquired, a second augmented view of the second scene from the real environment, the second augmented view including one or more second augmentations that have been included into the second augmented view based, at least in part, on the second activity-inferring data, and wherein the augmented view including the one or more augmentations of the scene from the real environment is a first augmented view including one or more first augmentations of the first scene from the real environment comprises:

an augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are modified version or versions of the one or more first augmentations.

203. The system of claim 202, wherein said augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are modified version or versions of the one or more first augmentations comprises:

an augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are same or similar as the one or more first augmentations except for location or locations of the one or more second augmentations in the second augmented view being different from location or locations of the one or more first augmentations in the first augmented view.

204. The system of claim 202, wherein said augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are modified version or versions of the one or more first augmentations comprises:

an augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are same or similar as the one or more first augmentations except for the one or more second augmentations in the second augmented view being more transparent than the one or more first augmentations in the first augmented view.

205. The system of claim 202, wherein said augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are modified version or versions of the one or more first augmentations comprises:

an augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are same or similar as the one or more first augmentations except for the one or more second augmentations in the second augmented view being less transparent than the one or more first augmentations in the first augmented view.

206. The system of claim 202, wherein said augmented view presenting module configured to present a second augmented view including one or more second augmentations that is or are modified version or versions of the one or more first augmentations comprises:

an augmented view presenting module configured to present a second augmented view including the one or more second augmentations that is or are one or more disabled functional augmentations of one or more functional augmentations included in the first augmented view.

207. The system of claim 206, wherein said augmented view presenting module configured to present a second augmented view including the one or more second augmentations that is or are one or more disabled functional augmentations of one or more functional augmentations included in the first augmented view comprises:

an augmented view presenting module configured to present the second augmented view including the one or more second augmentations that is or are the one or more disabled functional augmentations of the one or more functional augmentations included in the first augmented view in response to a request made by a party associated with the second scene to disable the one or more functional augmentations.

208. The system of claim 198, wherein said augmented view presenting module configured to present, in response at least in part to determining by the activity-inferring data acquisition determining module that the second activity-inferring data have at least been acquired, a second augmented view of the second scene from the real environment, the second augmented view including one or more second augmentations that have been included into the second augmented view based, at least in part, on the second activity-inferring data, and wherein the augmented view including the one or more augmentations of the scene from the real environment is a first augmented view including one or more first augmentations of the first scene from the real environment comprises:

an augmented view presenting module configured to present one or more second augmentations that is or are one or more indicators that indicate that the one or more first augmentations have been removed.

209. The computationally-implemented system of claim 208, wherein said augmented view presenting module configured to present one or more second augmentations that is or are one or more indicators that indicate that the one or more first augmentations have been removed comprises:

an augmented view presenting module configured to present one or more second augmentations that is or are one or more indicators that can be selected or activated in order to reinstate the one or more first augmentations into the second augmented view.

210. The system of claim 142, further comprising:

a visual data obtaining module configured to obtain visual data from the one or more cameras of a second scene from the real environment, wherein the scene from the real environment is a first scene from the real environment;

an activity-inferring data acquisition determining module configured to determine whether a second activity-inferring data that infers at least initial occurrence of one or more second user activities associated with the second scene from the real environment have been acquired, wherein the activity-inferring data that infers at least initial occurrence of one or more user activities of the first scene from the real environment is a first activity-inferring data; and

a non-augmented view presenting module configured to present, in response at least in part to determining by the activity-inferring data acquisition determining module that the second activity-inferring data have at least been acquired, a view of the second scene that does not include the one or more augmentations.

211. The system of claim 210, further comprising:

a notification transmitting module configured to transmit a notification that notifies a party that the one or more augmentations have been removed.

143-148. -148. (canceled)

154. (canceled)

162. (canceled)

165. (canceled)

168-177. -177. (canceled)

179-186. -186. (canceled)

199-201. -201. (canceled)

212. An article of manufacture, comprising:

a non-transitory storage medium bearing;

one or more instructions for obtaining visual data of an actual view of a scene from a real environment;

one or more instructions for determining whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired; and

one or more instructions for presenting an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining that the activity-inferring data have at least been acquired.

213. A wearable augmented reality (AR) device, comprising:

a coupling device for coupling the wearable AR device to a user'"'"'s head;

one or more cameras;

a visual data obtaining module configured to obtain visual data from the one or more cameras of an actual view of a scene from a real environment;

an activity-inferring data acquisition determining module configured to determine whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have been acquired; and

an augmented view presenting module configured to present an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data, and the presenting being in response at least in part to determining by the activity-inferring data acquisition determining module that the activity-inferring data have at least been acquired.