Augmented reality system with activity templates

• US 9,406,170 B1
• Filed: 07/16/2012
• Issued: 08/02/2016
• Est. Priority Date: 07/16/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Grampurohit, Shweta D., Consolvo, Sunny A., Hazlewood, William R., Harrison, Beverly L.
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US13/550,446
• Time in Patent Office: 1,478 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 3/08 Projecting images onto non-planar surfaces, e.g. geodetic screens

Abstract

A projection and camera system creates an augmented reality environment in which images are projected onto a scene and user movement within the scene is captured. The system assists the user in performing various activities, such as productivity tasks, routine household tasks, and creative tasks. Predefined activity templates contain projectable patterns that, when projected into the environment in a certain sequence, guide the user through the activities.

Claims

1. A system comprising:

a memory;

a processor;

a projector to project two or more projection images into an environment;

a camera to capture one or more environment images of the environment, the one or more environment images used at least in part for determining user interaction with the two or more projection images projected into the environment;

the memory, configured to store one or more activity templates, each activity template including two or more patterns to be projected by the projector into the environment to assist a user in performing an activity; and

instructions stored in the memory and executed by the processor, to cause the processor to access the memory, facilitate use of at least one of the one or more activity templates, and perform operations comprising;

facilitating at least one of (1) user selection of the activity template from the memory, or (2) suggestion of the activity template to the user; and

providing patterns corresponding to the selected or suggested activity template for projection by the projector, the patterns provided in a sequence based on the user progressing through each pattern of the activity template as determined using the environment images captured by the camera.

2. The system of claim 1, wherein each of the two or more patterns comprises one or more of demarcations, symbols or word descriptions that, when projected by the projector, guide the user through corresponding steps of the activity.

3. The system of claim 1, wherein one of the activity templates comprises a folding template including a sequence of patterns to be projected by the projector onto a surface of a material to be folded, wherein the projected patterns depict fold lines on the material to guide the user in folding the material.

4. The system of claim 3, wherein the material comprises clothing fabric, and the projected sequence of patterns depict a series of fold lines on the clothing fabric.

5. The system of claim 3, wherein the material comprises paper and the projected sequence of patterns depict a series of fold lines on the paper.

6. The system of claim 1, wherein one of the activity templates comprises a sewing template including a sequence of patterns to be projected by the projector onto a surface of a material to guide the user in cutting the material into pieces and sewing the pieces together.

7. The system of claim 1, wherein one of the activity templates comprises a painting template including at least one pattern to be projected by the projector onto a surface to guide the user in painting on the surface.

8. The system of claim 1, wherein the activity templates comprises a board game template including one or more patterns to be projected by the projector onto a surface to form a board game image and thereby facilitate game play by the user.

9. The system of claim 1, wherein the operations further comprise monitoring user behavior over time and based at least partly on the user behavior, to determine a next activity to be performed by the user;

and

selecting an activity template that, when projected, is to assist the user with the determined next activity.

10. The system of claim 1, wherein the operations further comprise using the environment images captured by the camera to determine user progress through the activity.

11. A system comprising:

a memory;

a processor;

the memory configured to store one or more activity templates, each activity template including two or more patterns to be projected onto a surface to assist a user in performing an activity; and

instructions stored in the memory and executed by the processor, to cause the processor to select a given activity template from the memory and to provide the two or more patterns from the given activity template for projection onto the surface in a sequential order to guide the user through the activity.

12. The system of claim 11, further comprising a projector to project the patterns onto the surface.

13. The system of claim 12, further comprising a camera to capture one or more environment images of an environment encompassing the surface, the one or more environment images used at least in part for determining user interaction with the two or more patterns projected onto the surface.

14. The system of claim 11, wherein the two or more patterns are projected onto a surface of an object to assist the user in performing an activity associated with the object.

15. The system of claim 11, wherein the operations further comprise facilitating at least one of (1) user selection of the activity template from the memory, or (2) suggestion of the activity template to the user.

16. The system of claim 11, wherein the operations further comprise:

monitoring user behavior over time and based at least partly on the user behavior, determine a next activity to be performed by the user; and

recommending, to the user, an activity template suitable to assist with the next activity.

17. The system of claim 11, wherein the activity templates comprise household chores templates including sequences of patterns to be projected onto various surfaces to assist the user in doing household chores.

18. The system of claim 11, wherein the sequence is based at least in part on user progress associated with the activity.

19. One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

providing an activity template, the activity template including two or more patterns to be projected into an environment to guide a user through corresponding one or more steps in an activity;

choosing a first pattern from the activity template for projection into the environment to provide guidance to the user for performing a corresponding step in the activity;

receiving an indication of the user'"'"'s performance of the corresponding step; and

repeating the choosing and the receiving for a second pattern in the activity template.

20. The one or more non-transitory computer-readable storage media of claim 19, wherein the choosing and the receiving are repeated for one or more subsequent patterns after the second pattern until the user has performed the activity.

21. The one or more non-transitory computer-readable storage media of claim 19, wherein choosing a first pattern comprises enabling user selection of the activity template and selecting the first pattern in the activity template.

22. The one or more non-transitory computer-readable storage media of claim 19, wherein choosing the second pattern comprises:

capturing an image of the environment with a camera to determine when the user has performed the corresponding step; and

selecting the second pattern in the activity template that guides the user through a next step of the activity.

23. The one or more non-transitory computer-readable storage media of claim 19, wherein receiving the indication comprises detecting user interaction with the first pattern.

24. The one or more non-transitory computer-readable storage media of claim 19, wherein receiving the indication comprises capturing an image of the environment with a camera to monitor user interaction with the first pattern.

25. A method comprising:

providing an activity template including two or more patterns to be projected into an environment to guide a user through corresponding one or more steps in an activity;

projecting, in a sequence, the two or more patterns in the activity template to guide the user in performing the corresponding steps for the activity; and

determining, by one or more processors executing non-transitory computer-readable storage media storing computer-executable instructions, user progress through the activity as the patterns are projected in the sequence in order to advance from pattern to pattern.

26. The method of claim 25, wherein the determining comprises capturing images of the environment with a camera.

27. The method of claim 25, wherein the determining comprises receiving a user command.

28. The method of claim 27, wherein the user command comprises at least one of a voice command or a gesture.

29. The method of claim 25, wherein the activity templates comprise household chores templates including patterns to be projected onto various surfaces to guide the user through household chores.

30. The method of claim 25, wherein the activity templates comprises a game template including two or more patterns to be projected by the projector onto a surface to facilitate game play by the user.

31. The method of claim 25, further comprising:

monitoring user behavior over time;

based at least partly on the user behavior, determining a next activity to be performed by the user; and

selecting an activity template that includes two or more patterns that, when the two or more patterns are projected, are to assist the user with the determined next activity.