Providing features of an electronic product in an augmented reality environment

• US 10,891,800 B1
• Filed: 06/10/2020
• Issued: 01/12/2021
• Est. Priority Date: 09/29/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Stoyles, Justin D., Kuhn, Michael
• Primary Examiner(s): Amin, Jwalant
• Application Number:
  US16/897,551
• Time in Patent Office: 216 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection

Abstract

The present disclosure relates to providing a software feature of an electronic product in an augmented reality (AR) environment. In some embodiments, images are obtained using one or more image sensors, a determination is made whether the obtained images include printed media depicting the electronic product, when the obtained images include the printed media depicting the electronic product, a virtual object corresponding to the electronic product is displayed in the AR environment, and the software feature of the electronic product is provided with the virtual object.

Claims

1. A device for providing a software feature of an electronic product in an augmented reality (AR) environment, comprising:

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

obtaining images using one or more image sensors;

determining whether the obtained images include printed media depicting the electronic product;

in accordance with a first set of one or more conditions being satisfied, the first set of one or more conditions including a first condition that is satisfied when the obtained images include the printed media depicting a screen of the electronic product;

displaying a virtual object corresponding to the electronic product in the AR environment, wherein the virtual object includes a virtual screen overlaying the screen of the electronic product depicted in the printed media; and

in accordance with a second set of one or more conditions being satisfied, the second set of one or more conditions including a second condition that is satisfied when the virtual object corresponding to the electronic product is displayed;

displaying, with the virtual screen of the virtual object, the software feature of the electronic product.

2. The device of claim 1, wherein the one or more programs further include instructions for:

detecting a user interaction with the software feature; and

modifying the virtual object corresponding to the electronic product based on the user interaction.

3. The device of claim 2, wherein detecting the user interaction comprises:

detecting a touch contact on a touch-screen display at a location corresponding to the provided software feature.

4. The device of claim 2, wherein detecting the user interaction comprises:

detecting that a physical position of the user'"'"'s hand or finger corresponds to a position in the AR environment of the software feature.

5. The device of claim 1, wherein the first set of one or more conditions include a third condition that is satisfied when the printed media corresponds to predefined printed media.

6. The device of claim 5, wherein the predefined printed media is a predefined book, a predefined page of a predefined book, or a predefined depiction of the electronic product.

7. The device of claim 1, wherein the first set of one or more conditions include a sixth condition that is satisfied when a user input is detected.

8. The device of claim 1, wherein the second set of one or more conditions include a seventh condition that is satisfied when a user interaction with the virtual object is detected.

9. The device of claim 1, wherein providing the software feature includes displaying, with the virtual screen of the virtual object corresponding to the electronic product, an artificial boot-up sequence that depicts a boot-up sequence of the electronic product.

10. The device of claim 1, wherein providing the software feature includes displaying, with the virtual screen of the virtual object corresponding to the electronic product, a video depicting one or more aspects of the software feature.

11. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for:

obtaining images using one or more image sensors;

determining whether the obtained images include printed media depicting the electronic product;

in accordance with a first set of one or more conditions being satisfied, the first set of one or more conditions including a first condition that is satisfied when the obtained images include the printed media depicting a screen of the electronic product;

displaying a virtual object corresponding to the electronic product in the AR environment, wherein the virtual object includes a virtual screen overlaying the screen of the electronic product depicted in the printed media; and

in accordance with a second set of one or more conditions being satisfied, the second set of one or more conditions including a second condition that is satisfied when the virtual object corresponding to the electronic product is displayed;

displaying, with the virtual screen of the virtual object, the software feature of the electronic product.

12. The non-transitory computer-readable storage medium of claim 11, wherein the one or more programs further include instructions for:

detecting a user interaction with the software feature; and

modifying the virtual object corresponding to the electronic product based on the user interaction.

13. The non-transitory computer-readable storage medium of claim 12, wherein detecting the user interaction comprises:

detecting a touch contact on a touch-screen display at a location corresponding to the provided software feature.

14. The non-transitory computer-readable storage medium of claim 12, wherein detecting the user interaction comprises:

detecting that a physical position of the user'"'"'s hand or finger corresponds to a position in the AR environment of the software feature.

15. The non-transitory computer-readable storage medium of claim 11, wherein the first set of one or more conditions include a third condition that is satisfied when the printed media corresponds to predefined printed media.

16. The non-transitory computer-readable storage medium of claim 15, wherein the predefined printed media is a predefined book, a predefined page of a predefined book, or a predefined depiction of the electronic product.

17. The non-transitory computer-readable storage medium of claim 11, wherein the first set of one or more conditions include a sixth condition that is satisfied when a user input is detected.

18. The non-transitory computer-readable storage medium of claim 11, wherein the second set of one or more conditions include a seventh condition that is satisfied when a user interaction with the virtual object is detected.

19. The non-transitory computer-readable storage medium of claim 11, wherein providing the software feature includes displaying, with the virtual screen of the virtual object corresponding to the electronic product, an artificial boot-up sequence that depicts a boot-up sequence of the electronic product.

20. The non-transitory computer-readable storage medium of claim 11, wherein providing the software feature includes displaying, with the virtual screen of the virtual object corresponding to the electronic product, a video depicting one or more aspects of the software feature.

21. A method for providing a software feature of an electronic product in an augmented reality (AR) environment, the method comprising:

obtaining images using one or more image sensors;

determining whether the obtained images include printed media depicting the electronic product;

in accordance with a first set of one or more conditions being satisfied, the first set of one or more conditions including a first condition that is satisfied when the obtained images include the printed media depicting a screen of the electronic product;

displaying a virtual object corresponding to the electronic product in the AR environment, wherein the virtual object includes a virtual screen overlaying the screen of the electronic product depicted in the printed media; and

in accordance with a second set of one or more conditions being satisfied, the second set of one or more conditions including a second condition that is satisfied when the virtual object corresponding to the electronic product is displayed;

displaying, with the virtual screen of the virtual object, the software feature of the electronic product.

22. The method of claim 21, further comprising:

detecting a user interaction with the software feature; and

modifying the virtual object corresponding to the electronic product based on the user interaction.

23. The method of claim 22, wherein detecting the user interaction comprises:

detecting a touch contact on a touch-screen display at a location corresponding to the provided software feature.

24. The method of claim 22, wherein detecting the user interaction comprises:

detecting that a physical position of the user'"'"'s hand or finger corresponds to a position in the AR environment of the software feature.

25. The method of claim 21, wherein the first set of one or more conditions include a third condition that is satisfied when the printed media corresponds to predefined printed media.

26. The method of claim 25, wherein the predefined printed media is a predefined book, a predefined page of a predefined book, or a predefined depiction of the electronic product.

27. The method of claim 21, wherein the first set of one or more conditions include a sixth condition that is satisfied when a user input is detected.

28. The method of claim 21, wherein the second set of one or more conditions include a seventh condition that is satisfied when a user interaction with the virtual object is detected.

29. The method of claim 21, wherein providing the software feature includes displaying, with the virtual screen of the virtual object corresponding to the electronic product, an artificial boot-up sequence that depicts a boot-up sequence of the electronic product.

30. The method of claim 21, wherein providing the software feature includes displaying, with the virtual screen of the virtual object corresponding to the electronic product, a video depicting one or more aspects of the software feature.