Systems and methods for providing task-based instructions

• US 9,460,314 B2
• Filed: 12/01/2014
• Issued: 10/04/2016
• Est. Priority Date: 12/01/2013
• Status: Active Grant

• Current Assignee: Upskill, Inc. (TeamViewer AG)
• Original Assignee: APX Labs, Inc.
• Inventors: Ballard, Brian Adams, Athey, James Leighton, Kim, Wonjae, Jenkins, Jeffrey Edward, English, Edward Robert, Reily, Todd Richard, Sciscenti, Michael Anthony
• Primary Examiner(s): NGUYEN, JENNIFER T
• Application Number:
  US14/556,607
• Publication Number:
  US 20150153571A1
• Time in Patent Office: 673 Days
• Field of Search: 345/7, 345/8, 345/156, 349/11
• US Class Current: 1/1
• CPC Class Codes: A45C 11/00 Receptacles for purposes not provided for in groups A45C1/00-A45C9/00 specially adapted for toiletry or cosmetic equipment A45D29/20, A45D44/18; travelling sewing kits A45F3/48
• CPC Class Codes: A45C 2011/002 for portable handheld communication devices, e.g. mobile phone, pager, beeper, PDA, smart phone
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 21/71 to assure secure computing or processing of information
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 : Scrolling or panning
• CPC Class Codes: G06F 3/0486 : Drag-and-drop
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 12/1822 : Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 67/52 : specially adapted for the location of the user terminal
• CPC Class Codes: H04M 1/72454 : according to context-related or environment-related conditions
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/631 : Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/635 : Region indicators; Field of view indicators
• CPC Class Codes: H04W 12/084 : using delegated authorisation, e.g. open authorisation [OAuth] protocol
• CPC Class Codes: H04W 4/08 : User group management
• CPC Class Codes: H04W 76/10 : Connection setup

Abstract

Systems and methods allow a user to interact with an augmented reality device. In one implementation, a wearable device for providing task-based instructions to a user includes a display; a network interface; a data input device configured to capture information relating to progress of a task; and at least one processing device. The at least one processing device is configured to provide to a server system, via the network interface, the captured information relating to progress of the task; receive from the server system, via the network interface, information relating to a next step in the task; and cause the information relating to the next step in the task to be shown on the display.

Claims

1. A wearable device for providing a user of the wearable device with task-based instructions, the wearable device comprising:

a display;

a network interface;

a data input device configured to capture information relating to progress of a task; and

at least one processing device configured to;

provide to a server system, via the network interface, the captured information relating to progress of the task, wherein the captured information includes an identifier of the wearable device, and the identifier is associated with at least one property of the wearable device;

receive from the server system, via the network interface, information relating to a next step in the task; and

cause the information relating to the next step in the task to be shown on the display based on the at least one property of the wearable device.

2. The wearable device of claim 1, wherein the data input device includes an image capture device configured to acquire image data during performance of steps in the task.

3. The wearable device of claim 1, wherein the data input device includes a voice recognition module configured to determine when the user audibly confirms completion of a task step.

4. The wearable device of claim 1, wherein the data input device includes a button for the user to press to confirm completion of a task step.

5. The wearable device of claim 1, wherein the network interface includes one or more of an Internet connection, a cellular connection, a WAN connection, or a LAN connection.

6. The wearable device of claim 1, wherein the information received from the server includes a video instruction for performing the next step in the task.

7. The wearable device of claim 1, wherein the information received from the server includes a text-based instruction for performing the next step in the task.

8. The wearable device of claim 1, wherein the information received from the server includes a graphical-based instruction for performing the next step in the task.

9. The wearable device of claim 1, wherein the information received from the server includes an audible instruction for performing the next step in the task.

10. The wearable device of claim 1, wherein the server system includes an enterprise resource planning system.

11. The wearable device of claim 1, wherein the at least one property of the wearable device is used to identify at least one of hardware and software configurations of the wearable device for the next step in the task.

12. The wearable device of claim 1, the wearable device further comprises receiving, from a user interface, a location on the display at which to display information relating to one or more steps in the task.

13. The wearable device of claim 12, wherein causing the information relating to the next step in the task to be shown on the display comprises causing the information relating to the next step in the task to be shown on the display at the location.

14. A method for providing a user of a wearable device with task-based instructions, the method comprising:

capturing, via a data input device of the wearable device, information relating to progress of a task;

providing to a server system, via a network interface, the captured information relating to progress of the task, wherein the captured information includes an identifier of the wearable device, and the identifier is associated with at least one property of the wearable device;

receiving from the server system, via the network interface, information relating to a next step in the task; and

causing the information relating to the next step in the task to be shown on a display of the wearable device based on the at least one property of the wearable device.

15. The method of claim 14, wherein the data input device includes an image capture device configured to acquire image data during performance of steps in the task.

16. The method of claim 14, wherein the data input device includes a voice recognition module configured to determine when the user audibly confirms completion of a task step.

17. The method of claim 14, wherein the data input device includes a button for the user to press to confirm completion of a task step.

18. The method of claim 14, further comprising using a voice recognition module to determine when the user audibly confirms completion of a task step.

19. The method of claim 14, wherein the captured information relating to progress of the task is received via a button for the user to press to confirm completion of a task step.

20. The method of claim 14, wherein the network interface includes one or more of an Internet connection, a cellular connection, a WAN connection, or a LAN connection.

21. The method of claim 14, wherein the information received from the server includes a video instruction for performing the next step in the task.

22. The method of claim 14, wherein the information received from the server includes a text-based instruction for performing the next step in the task.

23. The method of claim 14, wherein the information received from the server includes a graphical-based instruction for performing the next step in the task.

24. The method of claim 14, wherein the information received from the server includes an audible instruction for performing the next step in the task.

25. A non-transitory computer-readable storage medium storing program instructions executable by at least one processor to perform a method for providing a user of a wearable device with task-based instructions, the method comprising:

capturing, via a data input device of the wearable device, information relating to progress of a task;

providing to a server system, via a network interface, the captured information relating to progress of the task, wherein the captured information includes an identifier of the wearable device, and the identifier is associated with at least one property of the wearable device;

receiving from the server system, via the network interface, information relating to a next step in the task; and

causing the information relating to the next step in the task to be shown on a display of the wearable device based on the at least one property of the wearable device.

26. A wearable device for providing a user of the wearable device with task-based instructions, the wearable device comprising:

a display;

a network interface;

a data input device configured to capture information relating to progress of a task; and

at least one processing device configured to;

provide to a server system, via the network interface, the captured information relating to progress of the task, wherein the captured information includes an identifier of the wearable device, and the identifier is associated with at least one property of the wearable device;

receive from the server system, via the network interface, annotation information and information relating to a next step in the task, wherein the annotation information is associated with at least one object in an environment of the user; and

cause the information relating to the next step in the task and the annotation information to be shown on the display based on the at least one property of the wearable device.