Presenting an augmented view in response to acquisition of data inferring user activity

• US 9,448,623 B2
• Filed: 11/09/2012
• Issued: 09/20/2016
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, John D. Jr., Tegreene, Clarence T.
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US13/673,070
• Publication Number:
  US 20140098128A1
• Time in Patent Office: 1,411 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computationally implemented methods and systems include obtaining visual data of an actual view of a scene from a real environment, determining whether activity-inferring data that infers at least initial occurrence of one or more user activities associated with the scene from the real environment have at least been acquired, and presenting, in response at least in part to determining that the activity-inferring data have at least been acquired, an augmented view of the scene from the real environment, the augmented view including one or more augmentations that have been included into the augmented view based, at least in part, on the activity-inferring data. In addition to the foregoing, other aspects are described in the claims, drawings, and text.

Claims

1. A system, comprising:

one or more cameras;

a visual data obtaining module configured to obtain from the one or more cameras visual data of a real environment;

an activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user activities; and

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities; and

a non-augmented view presenting module configured to present one or more non-augmented portions of the view of the real environment that do not include the one or more augmentations based, least in part, on a determination that second visual data indicates one or more second user activities.

2. The system of claim 1, wherein said activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user activities comprises:

a visual cue determining module configured to determine whether the visual data includes multiple visual cues indicating one or more user activities.

3. The system of claim 2 wherein said visual cue determining module configured to determine whether the visual data includes multiple visual cues indicating one or more user activities comprises:

a visual cue determining module configured to determine whether the visual data includes a combination of multiple visible items indicating one or more user activities.

4. The system of claim 1, further comprising:

an activity-inferring audio data acquisition determining module configured to determine whether audio data indicates one or more user activities.

5. The system of claim 4, wherein said activity-inferring audio data acquisition determining module configured to determine whether audio data indicates one or more user activities comprises:

an activity-inferring audio data acquisition determining module configured to determine whether audio data indicates one or more identifiable sounds indicating at least one occurrence of the one or more user activities.

6. The system of claim 1, further comprising:

a movement data acquisition determining module configured to determine whether movement data indicates one or more user activities.

7. The system of claim 1, further comprising:

a personal information manager (PIM) data acquisition determining module configured to determine whether personal information manager (PIM) data indicates one or more user activities.

8. The system of claim 1, further comprising:

a sensor-provided data acquisition determining module configured to determine whether sensor-provided data indicates one or more user activities.

9. The system of claim 8, wherein said sensor-provided data acquisition determining module configured to determine whether sensor-provided data indicates one or more user activities comprises:

a sensor-provided data acquisition determining module configured to determine whether sensor-provided data includes one or more user physiological characteristics indicating one or more user activities.

10. The system of claim 1, further comprising:

a social networking data acquisition determining module configured to determine whether social networking data indicates one or more user activities.

11. The system of claim 1, further comprising:

an activity-inferring location/orientation data acquisition determining module configured to determine whether location and/or orientation data indicates one or more user activities.

12. The system of claim 1, wherein said activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user activities comprises:

an activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user leisure activities.

13. The system of claim 1, wherein said activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user activities comprises:

an activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user machine operating activities.

14. The system of claim 1, wherein said activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user activities comprises:

an activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user work or professional activities.

15. The system of claim 1, wherein said activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user activities comprises:

an activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user academic activities.

16. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view including one or more dynamic functional augmentations that provide dynamically-changing functional information.

17. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more of time information and/or user physiological information in response to determination that the visual data indicates one or more exercise and/or athletic activities.

18. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide retail information in response to determination that the visual data indicates one or more shopping activities.

19. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide dietary information in response to determination that the visual data indicates one or more dining activities.

20. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more of business, financial, and/or market information in response to determination that the visual data indicates one or more employment or professional activities.

21. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide one or more of environmental and/or weather information in response to determination that the visual data indicates one or more leisure activities.

22. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view including one or more augmentations that provide reference information in response to determination that the visual data indicates one or more educational and/or academic activities.

23. The system of claim 1, further comprising:

an augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view in response, at least in part, to determination that the visual data cates one or more user activities.

24. The system of claim 23, wherein said augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view in response, at least in part, to determination that the visual data cates one or more user activities comprises:

an augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view via one or more wireless and/or wired networks.

25. The system of claim 24, wherein said augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view via one or more wireless and/or wired networks comprises:

an augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with the real environment.

26. The system of claim 25, wherein said augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with the real environment comprises:

an augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with one or more locations associated with the real environment.

27. The system of claim 25, wherein said augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with the real environment comprises:

an augmentation soliciting and receiving module configured to solicit and receive the one or more augmentations to be included in the augmented view from one or more parties determined to be associated with one or more items detected in the real environment.

28. The system of claim 1, wherein the augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been modified in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities.

29. The system of claim 28, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been modified in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been moved in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities.

30. The system of claim 28, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been modified in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been made more transparent in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities.

31. The system of claim 28, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been modified in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been made less transparent in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities.

32. The system of claim 28, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been modified in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have been disabled in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities.

33. The system of claim 1, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have replaced one or more other augmentations in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities.

34. The system of claim 33, wherein said augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that have replaced one or more other augmentations in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities comprises:

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations that can be selected or activated to reinstate one or more other augmentations in the augmented view based, at least in part, on determination that the visual data indicates one or more user activities.

35. The system of claim 1, further comprising:

a notification transmitting module configured to transmit a notification that notifies a party that the one or more augmentations have been removed.

36. A wearable augmented reality (AR) device, comprising:

a coupling device for coupling the wearable AR device to a user'"'"'s head;

one or more cameras;

a visual data obtaining module configured to obtain from the one or more cameras visual data of a real environment;

an activity-inferring visual data acquisition determining module configured to determine whether the visual data indicates one or more user activities;

an augmented view presenting module configured to present an augmented view of the real environment including one or more augmentations based, at least in part, on a determination that the visual data indicates one or more user activities;

a non-augmented view presenting module configured to present one or more non-augmented portions of the view of the real environment that do not include the one or more augmentations based, least in part, on a determination that second visual data indicates one or more second user activities.

37. A system comprising:

circuitry configured for obtaining, from one or more cameras, visual data of a real environment;

circuitry configured for determining at least one user activity based at least partly on the obtained visual data of the real environment;

circuitry configured for presenting an augmented view of the real environment including one or more augmentations based at least partly on at least one user activity determined from the obtained visual data of the real environment; and

circuitry for presenting one or more non-augmented portions of the view of the real environment that do not include the one or more augmentations based at least in part a determination that second visual data indicates one or more second user activities.

38. The system of claim 37, wherein the configured for determining at least one user activity based at least partly on the obtained visual data of the real environment comprises:

circuitry configured for determining at least one user activity based at least partly on;

recognition of one or more visual items in obtained visual data of the real environment; and

geographic position information.

39. The system of claim 38, wherein the circuitry configured for presenting an augmented view of the real environment including one or more augmentations based at least partly on at least one user activity determined from the obtained visual data of the real environment comprises:

circuitry configured for selecting one or more augmentations to include into an augmented view the real environment based at least partly on;

at least one user activity determined from one or more visual items recognized using obtained visual data of the real environment; and

geographic position information.

40. The system of claim 37, wherein the presenting an augmented view of the real environment including one or more augmentations based at least partly on at least one user activity determined from the obtained visual data of the real environment includes:

circuitry configured for presenting one or more augmentations at one or more locations within an augmented view wherein the one or more locations are based at least partly on one or more locations of one or more recognized visual items within the real environment.