Augmented reality quick-start and user guide

• US 10,509,619 B2
• Filed: 11/30/2015
• Issued: 12/17/2019
• Est. Priority Date: 12/15/2014
• Status: Active Grant

• Current Assignee: Hand Held Products Incorporated (Honeywell International Inc.)
• Original Assignee: Hand Held Products Incorporated (Honeywell International Inc.)
• Inventors: Todeschini, Erik, Sauerwein, Jr., James Timothy, Anderson, Donald
• Primary Examiner(s): Martello, Edward
• Application Number:
  US14/954,056
• Publication Number:
  US 20160171775A1
• Time in Patent Office: 1,478 Days
• Field of Search: None
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

A method includes receiving an image of a product, obtaining content relevant to using the product, and displaying the content in an augmented reality view of the product by overlaying the content on the image of the product on a display device.

Claims

1. A method of providing a quick-start or user guide for a product, the method comprising:

capturing, with a device comprising a display, an image of a product when the product appears within a field of view of the device;

obtaining, with the device, content comprising information associated with a first feature relevant to the product, in response to capturing the image of the product;

determining, with the device, a perspective of the display with respect to the product based on a skew of the image of the product for which the content to be displayed is obtained;

mapping, with the device, a 3D model of the product on the image of the product to identify components on the product within the field of view of the device based on the determined perspective;

displaying, with the device, the content in an augmented reality view of the product by overlaying the content on the image of the product on the display, wherein the image of the product itself defines context for obtaining the content and/or for displaying the content based on the mapping, such that the content aligns with the components identified in the field of view of the device;

highlighting, with the device, in the displayed augmented reality view one or more of the identified components in a sequence, wherein the sequence corresponds to an order in which the one or more identified components on the product can be manipulated to perform an operation with the product;

determining, with the device, if the product is no longer present within the field of view of the device; and

with the device, in response to determining that the product is no longer present in the field of view of the device, continuing the display of the content in the augmented reality view for a particular period of time, before disabling the display of the content.

2. The method of claim 1, further comprising:

detecting a user selection of a function on the product, at least in part, by detecting the user pressing a user selectable button on the product;

using the detected selection to select further content comprising information associated with a second feature relevant to the function on the product; and

presenting the further content in the augmented reality view of the product by overlaying the further content on the image of the product on the display.

3. The method of claim 1, wherein the step of capturing an image of a product comprises bringing the product into the field of view of the device.

4. The method of claim 1, wherein disabling the display of the content further comprises removing at least a portion of the displayed content, in response to determining that a corresponding portion of the product that defined context comprising a feature relevant for using the portion of the displayed content no longer appears within the field of view of the device.

5. The method of claim 1, wherein the content comprises a quick start guide to guide a user to use the product.

6. The method of claim 5, further comprising sequentially highlighting each of multiple pieces of presented content in an order corresponding to a selected operation on the product.

7. The method of claim 5, further comprising:

detecting a user selection of a function on the product; and

using the detected selection to highlight a sequential piece of the content on the display.

8. The method of claim 1, wherein the image of the product is a three-dimensional model that rotates based on a rotation of the product.

9. The method of claim 1, wherein the image of the product is augmented with content descriptive of the first feature of the product.

10. The method of claim 9, wherein the first feature includes user selectable buttons.

11. The method of claim 9, wherein the content includes selectable links to further detailed information regarding the first feature.

12. The method of claim 1, wherein the content comprises a view of inside the product during the operation.

13. The method of claim 12, wherein the product comprises a printer, the operation comprises loading media into the printer, and the content comprises a video of inner spindles and media being routed inside the printer corresponding to the operation.

14. The method of claim 1, wherein the product comprises at least one augmented reality marker, and wherein the augmented reality marker, when appearing within the field of view of the device, defines context for obtaining at least a portion of the content and/or for displaying at least a portion of the content.

15. The method of claim 1, wherein a first portion of the content comprises the 3D model of the product configured as an invisible framework, and a second portion of the content comprises a plurality of textured layers each configured to highlight a respective component of the product, and wherein the method further comprises selecting a first textured layer from among the plurality of textured layers when a user presses a user selectable button on the product, the first texture comprising content corresponding to a function of the product associated with the user selectable button.

16. The method of claim 1, wherein the content comprises a virtual model corresponding to the product, the virtual model having a region corresponding to a user selectable button on the product, and wherein the method further comprises registering a click event when an object crosses into the region and obtaining additional content, the additional content corresponding to a function of the product associated with the user selectable button.

17. A machine-readable storage device having instructions for execution by a processor to perform a method of providing a quick-start or user guide for a product, the method comprising:

capturing, with a device comprising a display, an image of a product when the product appears within a field of view of the device;

obtaining, with the device, content comprising information associated with a first feature relevant to the product, when the product appears within a field of view of the device;

determining, with the device, a perspective of the display with respect to the product based on a skew of the image of the product for which the content to be displayed is obtained;

mapping, with the device, a 3D model of the product on the image of the product to identify a component on the product within the field of view of the device based on the determined perspective;

displaying, with the device, the content in an augmented reality view of the product by overlaying the content on the image of the product on the display, wherein the image of the product itself defines context for obtaining the content and/or for displaying the content based on the mapping, such that the content aligns with the components identified in the field of view of the device;

highlighting, with the device in the displayed augmented reality view one or more of the identified components in a sequence, wherein the sequence corresponds to an order in which the one or more identified components on the product can be manipulated to perform an operation with the product;

determining, with the device, if the product is no longer present within the field of view of the device; and

with the device, in response to determining that the product is no longer present in the field of view of the device, continuing the display of the content in the augmented reality view for a particular period of time, before disabling the display of the content.

18. The machine-readable storage device of claim 17, wherein the method further comprises:

detecting a user selection of a function on the product, at least in part, by detecting a user input associated with a user selectable button on the product;

using the detected user selection to select further content comprising information associated with a second feature relevant to the function on the product; and

presenting the further content in the augmented reality view of the product by overlaying the further content on the image of the product on the display.

19. The machine-readable storage device of claim 17, wherein the content comprises a quick start guide to guide a user through steps to use the product.

20. The machine-readable storage device of claim 19, the method further comprising sequentially highlighting each of multiple pieces of presented content in an order corresponding to a selected operation on the product.

21. The machine-readable storage device of claim 17, the method further comprising:

detecting a user selection of a function on the product, at least in part, by detecting a user input associated with a user selectable button on the product; and

using the detected user selection to highlight a sequential piece of content on the display.

22. The machine-readable storage device of claim 17, wherein the displayed image of the product is a three-dimensional model that rotates based on a rotation the product by a user and is augmented with content descriptive of features of the product, including user selectable buttons and selectable links to further detailed content regarding the features.

23. A device comprising:

a display;

a processor; and

a memory device coupled to the processor and having a program stored thereon for execution by the processor to provide a quick-start or user guide for a product, which program, when executed by the processor, causes the processor to;

capture, with the device, an image of a product when the product appears within a field of view of the device;

obtain, with the device, content comprising information associated with a first feature relevant to the product when the product appears within a field of view of the device;

determine, with the device, a perspective of the display with respect to the product based on a skew of the image of the product for which the content to be displayed is obtained;

map, with the device, a 3D model of the product on the image of the product to identify a component on the product within the field of view of the device based on the determined perspective;

display, with the device, the content in an augmented reality view of the product by overlaying the content on the image of the product on the display, wherein the image of the product itself defines context for obtaining the content and/or for displaying the content based on the mapping, such that the content aligns with the components identified in the field of view of the display device;

highlight, with the device, in the displayed augmented reality view one or more of the identified components in a sequence, wherein the sequence corresponds to an order in which the one or more identified components on the product can be manipulated to perform an operation with the product;

determine, with the device, if the product is no longer present within the field of view of the device; and

with the device, in response to determining that the product is no longer present in the field of view of the device, continuing the display of the content in the augmented reality view for a particular period of time, before disabling the display of the content.

24. The device of claim 23, wherein the processor further executes the program to:

detect a user selection of a function on the product, at least in part, by detecting a user input associated with a user selectable button on the product;

using the detected user selection to select further content comprising information associated with a second feature relevant to the function on the product; and

present the further content in the augmented reality view of the product by overlaying the further content on the image of the product on the display.

25. The device of claim 23, wherein the content comprises a quick start guide to guide a user to use the product and wherein the program is further executed by the processor to sequentially highlight each of multiple pieces of presented content in an order corresponding to a selected operation on the product.