Formatting of one or more persistent augmentations in an augmented view in response to multiple input factors

• US 8,941,689 B2
• Filed: 10/09/2012
• Issued: 01/27/2015
• Est. Priority Date: 10/05/2012
• Status: Expired due to Fees

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, John D. Jr., Tegreene, Clarence T.
• Primary Examiner(s): Wu, Xiao M.
• Assistant Examiner(s): BUTTRAM, ALAN T
• Application Number:
  US13/648,012
• Publication Number:
  US 20140098126A1
• Time in Patent Office: 840 Days
• Field of Search: 345629-633, 715/757
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Computationally implemented methods and systems include presenting a first augmented view of a first scene from a real environment, the first augmented view to be presented including one or more persistent augmentations in a first one or more formats, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment, obtaining an actual view of a second scene from the real environment that is different from the actual view of the first scene, and presenting a second augmented view of the second scene from the real environment, the second augmented view to be presented including the one or more persistent augmentations in a second one or more formats that is based, at least in part, on multiple input factors.

Claims

1. A system, comprising:

one or more cameras;

an augmented view presenting module configured to present, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment, wherein said augmented view presenting module comprises;

an augmented view presenting module configured to present, via the augmented reality device, the first augmented view that includes the one or more persistent augmentations that are, for their inclusion in the first augmented view, independent of presence of the one or more visual cues in the actual view of the first scene and independent of location and orientation of the augmented reality device;

an actual view obtaining module configured to obtain, via the augmented reality device, using the one or more cameras an actual view of a second scene from the real environment, the actual view of the second scene being different from the actual view of the first scene; and

wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises;

a format soliciting module configured to solicit for the second one or more formats in response, at least in part, to said obtaining of the actual view of the second scene.

2. The system of claim 1, wherein said augmented view presenting module configured to present, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present a first augmented view that includes the one or more persistent augmentations and one or more non-persistent augmentations, the inclusion of the one or more non-persistent augmentations in the first augmented view being dependent on the presence of the one or more visual cues and/or presence of another one or more visual cues in the actual view of the first scene captured by the one or more cameras.

3. The system of claim 1, wherein said augmented view presenting module configured to present, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present a first augmented view that includes the one or more persistent augments disposed at one or more preselected locations of the first augmented view.

4. The system of claim 3, wherein said augmented view presenting module configured to present a first augmented view that includes the one or more persistent augments disposed at one or more preselected locations of the first augmented view comprises:

an augmented view presenting module configured to present a first augmented view that includes the one or more persistent augmentations disposed at the one or more preselected locations of the first augmented view, the one or more preselected locations of the first augmented view where the one or more persistent augmentations are to be disposed being independent of presence of one or more visual cues in the first augmented view.

5. The system of claim 3, wherein said augmented view presenting module configured to present a first augmented view that includes the one or more persistent augments disposed at one or more preselected locations of the first augmented view comprises:

an augmented view presenting module configured to present a first augmented view that includes the one or more persistent augmentations disposed at the one or more preselected locations of the first augmented view, the one or more preselected locations of the first augmented view where the one or more persistent augmentations are to be disposed being independent of absence of one or more visual cues in the first augmented view.

6. The system of claim 1, wherein said augmented view presenting module configured to present, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present the first augmented view that includes one or more persistent textual augmentations, the inclusion of the one or more persistent textual augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment.

7. The system of claim 6, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent textual augmentations, the inclusion of the one or more persistent textual augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic textual augmentations, the one or more persistent dynamic textual augmentations being one or more persistent alphabetical and/or numerical augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene.

8. The system of claim 7, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic textual augmentations, the one or more persistent dynamic textual augmentations being one or more persistent alphabetical and/or numerical augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent dynamic textual augmentations that content-wise at least dynamically changes over time.

9. The system of claim 7, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic textual augmentations, the one or more persistent dynamic textual augmentations being one or more persistent alphabetical and/or numerical augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent dynamic textual augmentations that content-wise at least dynamically changes in response to receiving an electronic message or electronic communication.

10. The system of claim 7, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic textual augmentations, the one or more persistent dynamic textual augmentations being one or more persistent alphabetical and/or numerical augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent dynamic textual augmentations that content-wise at least dynamically changes in response to sensor provided data.

11. The system of claim 1, wherein said augmented view presenting module configured to present, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present the first augmented view that includes one or more persistent graphical augmentations, the inclusion of the one or more persistent graphical augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment.

12. The system of claim 11, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent graphical augmentations, the inclusion of the one or more persistent graphical augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic graphical augmentations, the one or more persistent dynamic graphical augmentations being one or more persistent animated image augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene.

13. The system of claim 12, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic graphical augmentations, the one or more persistent dynamic graphical augmentations being one or more persistent animated image augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent dynamic graphical augmentations that dynamically changes over time.

14. The system of claim 12, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic graphical augmentations, the one or more persistent dynamic graphical augmentations being one or more persistent animated image augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent dynamic graphical augmentations that content-wise at least dynamically changes in response to receiving an electronic message or electronic communication.

15. The system of claim 12, wherein said augmented view presenting module configured to present the first augmented view that includes one or more persistent dynamic graphical augmentations, the one or more persistent dynamic graphical augmentations being one or more persistent animated image augmentations that dynamically changes with respect to one or more factors other than those related to the actual view of the first scene comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent dynamic graphical augmentations that content-wise at least dynamically changes in response to sensor provided data.

16. The system of claim 1, wherein said augmented view presenting module configured to present, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent augmentations that do not rely on the presence of one or more particular visual cues in the actual view of the first scene to exist in the first augmented view.

17. The system of claim 1, wherein said augmented view presenting module configured to present, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment comprises:

an augmented view presenting module configured to present the first augmented view that includes the one or more persistent augmentations that belong to a type of augmentation that was selected by a user for presentation in multiple augmented views of multiple scenes of the real environment.

18. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

an augmented view presenting module including a scene difference detecting module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, to detecting by the scene difference detecting module that the second scene from the real environment is different from the first scene from the real environment.

19. The system of claim 18, wherein said augmented view presenting module including a scene difference detecting module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, to detecting by the scene difference detecting module that the second scene from the real environment is different from the first scene from the real environment comprises:

an scene difference detecting module including an item finding module configured to detect that the second scene from the real environment is different from the first scene from the real environment when the item finding module finds at least one or more items in the second scene that were not found in the first scene from the real environment.

20. The system of claim 19, wherein said scene difference detecting module including an item finding module configured to detect that the second scene from the real environment is different from the first scene from the real environment when the item finding module finds at least one or more items in the second scene that were not found in the first scene from the real environment comprises:

an item finding module including an office item finding module configured to find the one or more items in the second scene that were not found in the first scene from the real environment when the office item finding module finds at least one or more office items in the second scene that were not found in the first scene from the real environment.

21. The system of claim 19, wherein said scene difference detecting module including an item finding module configured to detect that the second scene from the real environment is different from the first scene from the real environment when the item finding module finds at least one or more items in the second scene that were not found in the first scene from the real environment comprises:

an item finding module including a natural landscape item finding module configured to find the one or more items in the second scene that were not found in the first scene from the real environment when the natural landscape item finding module finds at least one or more natural landscape items in the second scene that were not present in the first scene from the real environment.

22. The system of claim 19, wherein said scene difference detecting module including an item finding module configured to detect that the second scene from the real environment is different from the first scene from the real environment when the item finding module finds at least one or more items in the second scene that were not found in the first scene from the real environment comprises:

an item finding module including a vehicle interior item finding module configured to find the one or more items in the second scene that were not found in the first scene from the real environment when the vehicle interior item finding module finds at least one or more interior items of a transportation vehicle in the second scene that were not present in the first scene from the real environment.

23. The system of claim 19, wherein said scene difference detecting module including an item finding module configured to detect that the second scene from the real environment is different from the first scene from the real environment when the item finding module finds at least one or more items in the second scene that were not found in the first scene from the real environment comprises:

an item finding module including an identifiable face finding module configured to find the one or more items in the second scene that were not found in the first scene from the real environment when the identifiable face finding module finds at least one or more identifiable faces of one or more persons in the second scene that were not present in the first scene from the real environment.

24. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

an augmented view presenting module including an audio input data responsive module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, by the audio input data responsive module to audio input data.

25. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

an augmented view presenting module including a location/orientation data responsive module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, by the location/orientation data responsive module to location and/or orientation data.

26. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

an augmented view presenting module including a movement data responsive module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, by the movement data responsive module to movement data.

27. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

an augmented view presenting module including a user physiological data responsive module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, by the user physiological data responsive module to user physiological data.

28. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

an augmented view presenting module including an eye tracking/gaze responsive module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, by the eye tracking/gaze responsive module to eye tracking and/or eye gaze data.

29. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

an augmented view presenting module including a personal information manager (PIM) data responsive module configured to present the second augmented view including the one or more persistent augmentations in the second one or more formats in response, at least in part, by the PIM data responsive module to PIM data.

30. The system of claim 1, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises:

a user activity determining module and a format selecting module configured to select the second one or more formats for application to the one or more persistent augmentations based on a determination by the user activity determining module of one or more user activities as inferred by the one or more input factors.

31. The system of claim 30, wherein said a user activity determining module and a format selecting module configured to select the second one or more formats for application to the one or more persistent augmentations based on a determination by the user activity determining module of one or more user activities as inferred by the one or more input factors comprises:

a user activity determining module and a format selecting module configured to select the second one or more formats based on the determination by the user activity determining module of the one or more user activities including at least one user recreational or leisure activity.

32. The system of claim 30, wherein said a user activity determining module and a format selecting module configured to select the second one or more formats for application to the one or more persistent augmentations based on a determination by the user activity determining module of one or more user activities as inferred by the one or more input factors comprises:

a user activity determining module and a format selecting module configured to select the second one or more formats based on the determination by the user activity determining module of the one or more user activities including at least one athletic activity.

33. The system of claim 30, wherein said a user activity determining module and a format selecting module configured to select the second one or more formats for application to the one or more persistent augmentations based on a determination by the user activity determining module of one or more user activities as inferred by the one or more input factors comprises:

a user activity determining module and a format selecting module configured to select the second one or more formats based on the determination by the user activity determining module of the one or more user activities including at least one vehicle operating activity.

34. The system of claim 30, wherein said a user activity determining module and a format selecting module configured to select the second one or more formats for application to the one or more persistent augmentations based on a determination by the user activity determining module of one or more user activities as inferred by the one or more input factors comprises:

a user activity determining module and a format selecting module configured to select the second one or more formats based on the determination by the user activity determining module of the one or more user activities including at least one academic activity.

35. The system of claim 1, wherein said format soliciting module configured to solicit for the second one or more formats in response, at least in part, to said obtaining of the actual view of the second scene comprises:

a format soliciting module configured to solicit for the second one or more formats via one or more wireless and/or wired networks.

36. The system of claim 35, wherein said format soliciting module configured to solicit for the second one or more formats via one or more wireless and/or wired networks comprises:

a format soliciting module configured to solicit for the second one or more formats from a party determined by an associated party determining module to be associated with one or more portions of the second scene.

37. The system of claim 36, wherein said format soliciting module configured to solicit for the second one or more formats from a party determined by an associated party determining module to be associated with one or more portions of the second scene comprises:

a format soliciting module configured to solicit for the second one or more formats from a party that was determined by the associated party determining module to be associated with a location associated with the second scene.

38. The system of claim 36, wherein said format soliciting module configured to solicit for the second one or more formats from a party determined by an associated party determining module to be associated with one or more portions of the second scene comprises:

a format soliciting module configured to solicit for the second one or more formats from a party that was determined by an item finding module to be associated with one or more visual identifiable items visually found in the second scene.

39. The system of claim 36, wherein said format soliciting module configured to solicit for the second one or more formats from a party determined by an associated party determining module to be associated with one or more portions of the second scene comprises:

a format soliciting module configured to solicit for the second one or more formats from a network server associated with the party.

40. An article of manufacture, comprising:

a non-transitory storage medium bearing;

one or more instructions for presenting, via an augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment, wherein said one or more instructions for presenting, via the augmented reality device, a first augmented view of a first scene from a real environment comprises;

one or more instructions for presenting, via the augmented reality device, a first augmented view that includes one or more persistent augmentations that do not depend on, for their existence in the first augmented view, the presence of the one or more visual cues in the actual view of the first scene and that do not depend on, for their existence in the first augmented view, location and orientation of the augmented reality device;

one or more instructions for obtaining, via the augmented reality device, an actual view of a second scene from the real environment, the actual view of the second scene being different from the actual view of the first scene; and

one or more instructions for presenting, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene, wherein said one or more instructions for presenting, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises;

one or more instructions for soliciting for the second one or more formats in response, at least in part, to said obtaining of the actual view of the second scene.

41. An augmented reality (AR) device, comprising:

a coupling device for coupling the AR device to a user'"'"'s head;

an augmented view presenting module configured to present, via the augmented reality device, a first augmented view of a first scene from a real environment, the first augmented view to be presented having one or more persistent augmentations in a first one or more formats, and the first augmented view to be presented being generated, at least in part, by augmenting an actual view that was captured by the one or more cameras of the first scene to include at least the one or more persistent augmentations, the inclusion of the one or more persistent augmentations in the first augmented view being independent of presence of one or more visual cues in the actual view of the first scene from the real environment, wherein said augmented view presenting module comprises;

an augmented view presenting module configured to present, via the augmented reality device, the first augmented view that includes the one or more persistent augmentations that are, for their inclusion in the first augmented view, independent of presence of the one or more visual cues in the actual view of the first scene and independent of location and orientation of the augmented reality device;

an actual view obtaining module configured to obtain, via the augmented reality device, using the one or more cameras an actual view of a second scene from the real environment, the actual view of the second scene being different from the actual view of the first scene; and

wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene, wherein said augmented view presenting module further configured to present, via the augmented reality device, a second augmented view of the second scene from the real environment, the second augmented view including the one or more persistent augmentations in a second one or more formats, the second one or more formats being based, at least in part, on the actual view of the second scene and one or more input factors unrelated to the actual view of the second scene comprises;

a format soliciting module configured to solicit for the second one or more formats in response, at least in part, to said obtaining of the actual view of the second scene.