Feedback based remote maintenance operations

• US 10,339,382 B2
• Filed: 09/13/2015
• Issued: 07/02/2019
• Est. Priority Date: 05/31/2015
• Status: Active Grant

• Current Assignee: Fieldbit Ltd. (Help Lightning, Inc.)
• Original Assignee: Fieldbit Ltd. (Help Lightning, Inc.)
• Inventors: Solomin, Shay, Meiron, Evyatar, Rapoport, Alex
• Primary Examiner(s): Liang, Dong Hui
• Application Number:
  US14/852,567
• Publication Number:
  US 20160350595A1
• Time in Patent Office: 1,388 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A method for feedback based maintenance wherein the method may include transmitting to an intermediate computerized system a first image that was acquired, at an acquisition point of time, by a head mounted display (HMD) mounted on a head of a first user; transmitting the first image from the intermediate computerized system and towards a computerized device of a second user; displaying on the computerized device of the second user the first image; receiving feedback, in response to the displaying and by the intermediate computerized system; generating, by the intermediate computerized system, an augmented image that comprises the first image and the feedback; transmitting to the HMD the augmented image; adjusting, by the HMD the augmented image to provide an amended augmented image that corresponds to visual information as viewed by the HMD at a display point in time; and displaying, by the HMD, the amended augmented image.

Claims

1. A method for feedback based maintenance, the method comprises:

transmitting to an intermediate computerized system a first image that was acquired, at an acquisition point of time, by a head mounted display (HMD) mounted on a head of a first user;

transmitting the first image from the intermediate computerized system and towards a computerized device of a second user;

wherein the intermediate computerized system differs from the computerized device of the second user and differs from the HMD;

displaying, on the computerized device of the second user, the first image;

following the displaying, receiving feedback, related to the first image;

wherein the feedback is provided by a user that differs from the first user;

generating, by the intermediate computerized system, an augmented image that comprises the first image and the feedback;

transmitting, to the HMD, the augmented image;

adjusting, by the HMD, the augmented image to provide an amended augmented image that corresponds to visual information as viewed by the HMD at a display point in time;

wherein the adjusting comprises overlaying at least a part of the augmented image over the visual information viewed by the HMD at the display point of time and matching a target object of the augmented image to the target object as appearing in the visual information viewed by the HMD at the display point of time;

wherein the overlaying of the at least part of the augmented image over the visual information viewed by the HMD at the display point of time comprises projecting staggered parts of the augmented image over the visual information viewed by the HMD at the display point of time;

and

displaying, by the HMD, the amended augmented image.

2. The method according to claim 1 comprising broadcasting the first image to multiple computerized devices of multiple users.

3. The method according to claim 2 further comprising receiving feedback from users of the multiple users and responding to feedback from only a predefined number of users of the multiple users.

4. The method according to claim 2 further comprising receiving feedback from more than one users of the multiple users and responding to feedback only from a single user of the multiple users.

5. The method according to claim 1 comprising receiving by the intermediate computerized system a feedback request from the first user to provide feedback related to the first image and transmitting the feedback request to the computerized device of the second user.

6. The method according to claim 5, comprising transmitting the feedback request and the first image to the intermediate computerized system at different times and through different channels.

7. The method according to claim 1 wherein the HMD is a see-through binocular head mounted display.

8. The method according to claim 7 wherein the adjusting, by the HMD the augmented image to provide an amended augmented image is preceded by performing a calibration process based upon a certain image of an object and feedback from the first user.

9. The method according to claim 1 comprising receiving, by the intermediate computerized system, the feedback at an absence of a feedback request from the first user to provide feedback related to the first image.

10. The method according to claim 1 wherein the user that differs from the first user is the second user.

11. The method according to claim 1 comprising relaying the first image and the feedback from the computerized device of the second user to a computerized device of a third user;

receiving additional feedback from the third user; and

sending the additional feedback from the third user.

12. The method according to claim 1 comprising relaying the first image from the computerized device of the second user to a computerized device of a third user;

and receiving the feedback from the third user and not from the second user.

13. The method according to claim 1 comprising projecting by the HMD a given user interface element and a selectable user interface element;

sensing, by a sensor, a head movement of the first user;

unchanging a position of the projection of the selectable user interface element when a speed of the head movement is below a first speed threshold;

changing the position of the projection of the selectable user interface element when the speed of the head movement exceeds the first speed threshold; and

returning the given user interface element and the selectable user interface element to an original position when sensing that the head movement is paused for predefined reset period and there is no overlap between the given user interface element and the selectable user interface element;

wherein when in the original position the given user interface element is projected on a center of the display of the HMD device.

14. The method according to claim 1 comprising transmitting to the intermediate computerized system a list of expert recipients;

sending requests to receive feedback to the expert recipients that are included in the list;

receiving, by the intermediate computerized system, deny responses or acceptance responses from the expert recipients that are included in the list;

selecting by the intermediate computerized system the second user out of one or more selected expert recipients that sent an acceptance response.

15. The method according to claim 1 wherein the first image comprises an image of the target object;

wherein the displaying, by the HMD, the amended augmented image, comprises displaying the feedback when the visual information as viewed by the HMD at the display point of time includes the image of the target object;

wherein the method comprises not displaying the feedback when the visual information as viewed by the HMD does not include the image of the target object.

16. The method according to claim 1 wherein the amended augmented image comprises augmented reality annotations;

wherein the method comprises removing the augmented reality annotations from the augmented image when detecting that the first user moved his focus away from the target object.

17. The method according to claim 1 wherein the amended augmented image comprises augmented reality annotations related to the target object;

wherein the method comprises displaying the augmented reality annotations related to the target object even when the target object is not viewed by the first user.

18. The method according to claim 1 wherein the overlaying of the at least part of the augmented image over the visual information viewed by the HMD at the display point of time comprises projecting a range of preset scaling values to the first user.

19. A non-transitory computer readable medium that once executed by a computer causes the computer to execute the steps of:

transmitting to an intermediate computerized system a first image that was acquired, at an acquisition point of time, by a head mounted display (HMD) mounted on a head of a first user;

transmitting the first image from the intermediate computerized system and towards a computerized device of a second user;

displaying on the computerized device of the second user the first image;

following the displaying, receiving feedback related to the first image by the intermediate computerized system;

wherein the feedback is provided by a user that differs from the first user;

generating, by the intermediate computerized system, an augmented image that comprises the first image and the feedback;

transmitting to the HMD the augmented image;

adjusting, by the HMD the augmented image to provide an amended augmented image that corresponds to visual information as viewed by the HMD at a display point in time;

wherein the adjusting comprises overlaying at least a part of the augmented image over the visual information viewed by the HMD at the display point of time and matching a target object of the augmented image to the target object as appearing in the visual information viewed by the HMD at the display point of time;

wherein the overlaying of the at least part of the augmented image over the visual information viewed by the HMD at the display point of time comprises projecting staggered parts of the augmented image over the visual information viewed by the HMD at the display point of time.

20. The non-transitory computer readable medium according to claim 19 that once executed by the computer causes the computer to execute the steps of relaying the first image and the feedback from the computerized device of the second user to a computerized device of a third user;

displaying on the computerized device of the third user the first image and receiving additional feedback from the third user.