Recording and providing for display images of events associated with power equipment

• US 10,074,402 B2
• Filed: 05/15/2013
• Issued: 09/11/2018
• Est. Priority Date: 05/15/2013
• Status: Active Grant

• Current Assignee: Hitachi Energy Limited
• Original Assignee: ABB Research Ltd. (ABB Ltd.)
• Inventors: Mahate, Shakeel M., Smiley, Karen J.
• Primary Examiner(s): Awad, Amr
• Assistant Examiner(s): Chow, Wing
• Application Number:
  US13/894,691
• Publication Number:
  US 20140341528A1
• Time in Patent Office: 1,945 Days
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/32 on separate auxiliary tracks of the same or an auxiliary record carrier
• CPC Class Codes: H04N 5/91 Television signal processing therefor

Abstract

Among other things, one or more techniques and/or systems are provided for identifying event images captured during an event associated with power equipment, such as a maintenance action. An entity, such as a maintenance technician, is equipped with an image recorder and an audio recorder. In some embodiments, the image recorder and/or audio recorder are wearable, such as mounted to eyewear and/or headwear worn by the entity. Audio that is recorded by the audio recorder is processed to identify events (e.g., key actions in a maintenance procedure). Based upon the identification of an event from the audio, event images (e.g., images captured temporally proximate to when the event was identified in the audio) associated with the event are identified and stored in a library. In some embodiments, one or more of such event images are later displayed to entities via a heads-up display, for example.

Claims

1. A method, comprising:

recording, via a wearable image recorder, images associated with a maintenance procedure;

recording, via a wearable audio recorder, audio associated with the maintenance procedure;

filtering images associated with events in the maintenance procedure from images not associated with events in the maintenance procedure based upon whether each image is associated with events in the maintenance procedure or is not associated with events in the maintenance procedure, the filtering comprising;

analyzing the audio and the images to identify a plurality of events, comprising a first event, a second event, a third event and a fourth event, of the maintenance procedure, the analyzing comprising;

identifying the first event based upon a non-verbal cue, in the audio, indicative of a vibration determined to indicate a failure of a first part in power equipment based upon a match between the non-verbal cue and a first audio indication paired with the first event in an audio library;

identifying the second event based upon a vocal change determined to indicate an increase in stress associated with a technician based upon a match between the vocal change and a second audio indication paired with the second event in the audio library;

identifying the third event based upon a keyword recognized via voice recognition in the audio, the identifying the third event based upon a match between the keyword and a third audio indication paired with the third event in the audio library; and

identifying the fourth event based upon a trigger image, recognized via image analysis in the images, indicative of a start of the fourth event;

determining that a first portion of the images and a first portion of the audio are associated with the first event based upon an analysis of timestamps in metadata associated with the first portion of the images and timestamps in metadata associated with the first portion of the audio;

determining that a second portion of the images and a second portion of the audio are associated with the second event based upon an analysis of timestamps in metadata associated with the second portion of the images and timestamps in metadata associated with the second portion of the audio;

determining that a third portion of the images and a third portion of the audio are associated with the third event based upon an analysis of timestamps in metadata associated with the third portion of the images and timestamps in metadata associated with the third portion of the audio;

determining that a fourth portion of the images and a fourth portion of the audio are associated with the fourth event based upon an analysis of timestamps in metadata associated with the fourth portion of the images and timestamps in metadata associated with the fourth portion of the audio; and

determining that a fifth portion of the images and a fifth portion of the audio are associated with routine actions in the maintenance procedure based upon an analysis of timestamps in metadata associated with the fifth portion of the images and timestamps in metadata associated with the fifth portion of the audio;

generating a library of events associated with the power equipment, the generating comprising;

responsive to determining that the first portion of the images and the first portion of the audio are associated with the first event, storing the first portion of the images and the first portion of the audio in a database of the library in association with the first event;

responsive to determining that the second portion of the images and the second portion of the audio are associated with the second event, storing the second portion of the images and the second portion of the audio in the database of the library in association with the second event;

responsive to determining that the third portion of the images and the third portion of the audio are associated with the third event, storing the third portion of the images and the third portion of the audio in the database of the library in association with the third event;

responsive to determining that the fourth portion of the images and the fourth portion of the audio are associated with the fourth event, storing the fourth portion of the images and the fourth portion of the audio in the database of the library in association with the fourth event; and

responsive to determining that the fifth portion of the images and the fifth portion of the audio are associated with the routine actions in the maintenance procedure, discarding the fifth portion of the images and the fifth portion of the audio to mitigate storage space to be consumed by a presentation associated with the maintenance procedure and to mitigate a temporal length of the presentation;

responsive to receiving a request from a user for images corresponding to performance of an event, identifying, in the library, a portion of the images and a portion of the audio indicative of the event;

analyzing the event to identify one or more keywords associated with the event specified by the request from the user;

responsive to identifying the one or more keywords, searching one or more databases, based upon the one or more keywords, to identify one or more documents associated with the event;

responsive to identifying the one or more documents, programmatically annotating textual information associated with the one or more documents to an event image in the portion of the images indicative of the event;

generating an audio-visual depiction of performance of the event based upon a combination of the portion of the images indicative of the event, the portion of the audio indicative of the event and the annotated textual information; and

controlling a graphical user interface to cause a screen to display an image stream of the audio-visual depiction of performance of the event while controlling a speaker to play an audio stream of the audio-visual depiction of performance of the event.

2. The method of claim 1, the wearable image recorder comprising an image recorder coupled to eyewear and the wearable audio recorder comprising an audio recorder coupled to at least one of a helmet, a wristwatch or a pennant.

3. The method of claim 1, the generating comprising:

merging the image stream with the audio stream for storage as a single file in a multimedia format.

4. The method of claim 1, comprising determining a time window for images and audio associated with each of the events using a start time identified based upon the identification of the corresponding event and an end time that is a defined amount of time after the start time.

5. The method of claim 1, the displaying performed via a heads-up display.

6. The method of claim 1, comprising determining a time window for images and audio associated with each of the events using a start time identified based upon the identification of the corresponding event and an end time identified based upon an identification of a spoken keyword, in the audio, determined to match a defined indicator of an end of an action.

7. The method of claim 1, the maintenance procedure comprising replacing a component of the power equipment, the first event comprising removing clips to access the component, the second event comprising uncoupling a wire harness to access the component, and the third event of the maintenance procedure comprising removing a screw to decouple the component from the power equipment.

8. The method of claim 1, the annotating the textual information comprising annotating the portion of the images associated with the event with a hyperlink to the one or more documents.

9. The method of claim 8, the hyperlink selectable via the graphical user interface.

10. A system, comprising:

an image recorder configured to capture images of power equipment associated with a maintenance procedure;

an audio recorder configured to capture audio associated with the maintenance procedure;

a filter configured to filter images associated with events in the maintenance procedure from images not associated with events in the maintenance procedure based upon whether each image is associated with events in the maintenance procedure or is not associated with events in the maintenance procedure, the filtering comprising;

analyzing the audio and the images to identify a plurality of events, comprising a first event, a second event, a third event, and a fourth event, of the maintenance procedure, the analyzing comprising;

identifying the first event based upon a non-verbal cue, in the audio, indicative of a vibration determined to indicate a failure of a first part in the power equipment based upon a match between the non-verbal cue and a first audio indication paired with the first event in an audio library;

identifying the second event based upon a vocal change determined to indicate an increase in stress associated with a technician based upon a match between the vocal change and a second audio indication paired with the second event in the audio library;

identifying the third event based upon a keyword recognized via voice recognition in the audio, the identifying the third event based upon a match between the keyword and a third audio indication paired with the third event in the audio library; and

identifying the fourth event based upon a trigger image, recognized via image analysis in the images, indicative of a start of the fourth event;

determining that a first portion of the images and a first portion of the audio are associated with the first event based upon an analysis of timestamps in metadata associated with the first portion of the images and timestamps in metadata associated with the first portion of the audio;

determining that a second portion of the images and a second portion of the audio are associated with the second event based upon an analysis of timestamps in metadata associated with the second portion of the images and timestamps in metadata associated with the second portion of the audio;

determining that a third portion of the images and a third portion of the audio are associated with the third event based upon an analysis of timestamps in metadata associated with the third portion of the images and timestamps in metadata associated with the third portion of the audio;

determining that a fourth portion of the images and a fourth portion of the audio are associated with the fourth event based upon an analysis of timestamps in metadata associated with the fourth portion of the images and timestamps in metadata associated with the fourth portion of the audio; and

determining that a fifth portion of the images and a fifth portion of the audio are associated with routine actions in the maintenance procedure based upon an analysis of timestamps in metadata associated with the fifth portion of the images and timestamps in metadata associated with the fifth portion of the audio;

a generator configured to generate a library of events associated with the power equipment, the generating comprising;

responsive to determining that the first portion of the images and the first portion of the audio are associated with the first event, storing the first portion of the images and the first portion of the audio in a database of the library in association with the first event;

responsive to determining that the second portion of the images and the second portion of the audio are associated with the second event, storing the second portion of the images and the second portion of the audio in the database of the library in association with the second event;

responsive to determining that the third portion of the images and the third portion of the audio are associated with the third event, storing the third portion of the images and the third portion of the audio in the database of the library in association with the third event;

responsive to determining that the fourth portion of the images and the fourth portion of the audio are associated with the fourth event, storing the fourth portion of the images and the fourth portion of the audio in the database of the library in association with the fourth event; and

responsive to determining that the fifth portion of the images and the fifth portion of the audio are associated with the routine actions in the maintenance procedure, discarding the fifth portion of the images and the fifth portion of the audio to mitigate storage space to be consumed by a presentation associated with the maintenance procedure and to mitigate a temporal length of the presentation; and

an image retrieval component configured to;

responsive to receiving a request from a user for images corresponding to performance of an event, identify, in the library, a portion of the images and a portion of the audio indicative of the event;

analyze the event to identify one or more keywords associated with the event specified by the request from the user;

responsive to identifying the one or more keywords, search one or more databases, based upon the one or more keywords, to identify one or more documents associated with the event;

responsive to identifying the one or more documents, programmatically annotate textual information associated with the one or more documents to an event image in the portion of the images indicative of the event;

generate an audio-visual depiction of performance of the event based upon a combination of the portion of the images indicative of the event, the portion of the audio indicative of the event and the annotated textual information; and

control a graphical user interface to cause a screen to display an image stream of the audio-visual depiction of performance of the event while controlling a speaker to play an audio stream of the audio-visual depiction of performance of the event.

11. The system of claim 10, the image recorder coupled to at least one of a helmet or eyewear.

12. The system of claim 11, at least one of the helmet or the eyewear comprising a heads-up display.

13. The system of claim 10, the analyzing comprising detecting a signature, of a keyword associated with the first event, in an intensity graph representative of an intensity of at least some of the audio.

14. The system of claim 10, the textual information comprising a hyperlink to the one or more documents, the hyperlink selectable via the graphical user interface.

15. The system of claim 10, comprising a display coupled to a helmet and configured to display at least some of the images.

16. The system of claim 10, the maintenance procedure comprising replacing a component of the power equipment, the first event comprising removing clips to access the component, the second event comprising uncoupling a wire harness to access the component, and the third event of the maintenance procedure comprising removing a screw to decouple the component from the power equipment.

17. The system of claim 10, comprising a display coupled to eyewear and configured to display at least some of the images.

18. The system of claim 10, comprising:

a heads-up display configured to display the portion of the images.

19. The system of claim 10, the image retrieval component configured to retrieve at least one of:

textual information associated with the portion of the images;

or

the portion of the audio.

20. A non-transitory computer-readable medium comprising instructions that when executed perform a method, comprising:

recording, via an image recorder, images associated with a maintenance procedure;

recording, via an audio recorder, audio associated with the maintenance procedure;

filtering images associated with events in the maintenance procedure from images not associated with events in the maintenance procedure based upon whether each image is associated with events in the maintenance procedure or is not associated with events in the maintenance procedure, the filtering comprising;

analyzing the audio and the images to identify a plurality of events, comprising a first event, a second event, a third event, and a fourth event, of the maintenance procedure, the analyzing comprising;

identifying the first event based upon a non-verbal cue, in the audio, indicative of a vibration determined to indicate a failure of a first part in power equipment based upon a match between the non-verbal cue and a first audio indication paired with the first event in an audio library;

identifying the second event based upon a vocal change determined to indicate an increase in stress associated with a technician based upon a match between the vocal change and a second audio indication paired with the second event in the audio library;

identifying the third event based upon a keyword recognized via voice recognition in the audio, the identifying the third event based upon a match between the keyword and a third audio indication paired with the third event in the audio library; and

identifying the fourth event based upon a trigger image, recognized via image analysis in the images, indicative of a start of the fourth event;

determining that a first portion of the images and a first portion of the audio are associated with the first event based upon an analysis of timestamps in metadata associated with the first portion of the images and timestamps in metadata associated with the first portion of the audio;

determining that a second portion of the images and a second portion of the audio are associated with the second event based upon an analysis of timestamps in metadata associated with the second portion of the images and timestamps in metadata associated with the second portion of the audio;

determining that a third portion of the images and a third portion of the audio are associated with the third event based upon an analysis of timestamps in metadata associated with the third portion of the images and timestamps in metadata associated with the third portion of the audio;

determining that a fourth portion of the images and a fourth portion of the audio are associated with the fourth event based upon an analysis of timestamps in metadata associated with the fourth portion of the images and timestamps in metadata associated with the fourth portion of the audio; and

determining that a fifth portion of the images and a fifth portion of the audio are associated with routine actions in the maintenance procedure based upon an analysis of timestamps in metadata associated with the fifth portion of the images and timestamps in metadata associated with the fifth portion of the audio;

generating a library of events associated with the power equipment, the generating comprising;

responsive to determining that the first portion of the images and the first portion of the audio are associated with the first event, storing the first portion of the images and the first portion of the audio in a database of the library in association with the first event;

responsive to determining that the second portion of the images and the second portion of the audio are associated with the second event, storing the second portion of the images and the second portion of the audio in the database of the library in association with the second event;

responsive to determining that the third portion of the images and the third portion of the audio are associated with the third event, storing the third portion of the images and the third portion of the audio in the database of the library in association with the third event;

responsive to determining that the fourth portion of the images and the fourth portion of the audio are associated with the fourth event, storing the fourth portion of the images and the fourth portion of the audio in the database of the library in association with the fourth event; and

responsive to determining that the fifth portion of the images and the fifth portion of the audio are associated with the routine actions in the maintenance procedure, discarding the fifth portion of the images and the fifth portion of the audio to mitigate storage space to be consumed by a presentation associated with the maintenance procedure and to mitigate a temporal length of the presentation;

responsive to receiving a request from a user for images corresponding to performance of an event, identifying, in the library, a portion of the images and a portion of the audio indicative of the event;

analyzing the event to identify one or more keywords associated with the event specified by the request from the user;

responsive to identifying the one or more keywords, searching one or more databases, based upon the one or more keywords, to identify one or more documents associated with the event;

responsive to identifying the one or more documents, programmatically annotating textual information associated with the one or more documents to an event image in the first portion of the images indicative of the event;

generating an audio-visual depiction of performance of the event based upon a combination of the portion of the images indicative of the event, the portion of the audio indicative of the event, and the annotated textual information; and

controlling a graphical user interface to cause a screen to display an image stream of the audio-visual depiction of performance of the event while controlling a speaker to play an audio stream of the audio-visual depiction of performance of the event.