Systems and methods for management and processing of electronic documents using video annotations

• US 9,792,024 B2
• Filed: 08/17/2015
• Issued: 10/17/2017
• Est. Priority Date: 08/17/2015
• Status: Active Grant

• Current Assignee: E-Plan, Inc.
• Original Assignee: E-Plan, Inc.
• Inventors: Chegini, Michael M., Reno, Stacy Ann, Kotovich, Leon
• Primary Examiner(s): Baderman, Scott
• Assistant Examiner(s): McVicker, Matthew G
• Application Number:
  US14/828,361
• Publication Number:
  US 20170052680A1
• Time in Patent Office: 792 Days
• CPC Class Codes: G06F 16/7328 Query by example, e.g. a complete video frame or video sequence graphical querying G06F16/7335
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 30/00 Computer-aided design [CAD]
• CPC Class Codes: G06F 30/13 Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 50/08 Construction
• CPC Class Codes: G11B 27/00 Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Systems and methods for managing and processing building plan documents including plan sheets. A toolbox of building project-related annotation tools comprising geometrical shapes and corresponding metadata indicating a criticality of a building project related defect is provided. An interface enables a selection of one or more video frames from a video file. A video frame editing area enables a user to drag and drop a shape over a portion of a frame displayed in the editing area to highlight a building defect. An interface enables the user to define a relationship between a video file and a building-related task. A user annotation, comprising a geometrical shape and corresponding metadata indicating a criticality of a building project related defect, of a video frame is received. An association of the video file with a first building project-related task is generated. A search interface enables the user to search for annotated video files associated with building projects.

Claims

1. A system for managing building plan documents, comprising:

one or more computing devices; and

non-transitory computer readable memory storing program code that when executed by the one or more computing devices is configured to cause the system to perform operations comprising;

receiving an electronic building plan document including a plurality of plan sheets;

providing a first of the plurality of plan sheets for display;

providing a user interface via which a user can select or enter a first comment associated with the first plan sheet;

providing a user interface via which the user can associate metadata, including at least a project type or discipline, with the first comment;

storing a first plurality of comments, including at least the first comment, in association with respective metadata;

providing a user interface enabling a user to upload a video file, image file, audio file;

providing an asset menu including a plurality of different physical structure types, including at least one of a high rise building, a low rise building, a private home, a hospital, a school, a bridge, a tunnel, or a water containment structure, via which the user can select a physical structure type;

receiving a user selection of a physical structure type via the asset menu;

selecting a toolbox of building project-related annotation tools, comprising a first plurality of different geometrical shapes configured to be selectively placed on video frames of structures corresponding to the selected physical structure type, wherein each geometrical shape in the first plurality of different geometrical shapes indicates a different physical structure defect type and is associated with respective metadata indicating the physical structure defect type, wherein the toolbox of building project-related annotation tools is selected based at least in part on the received user selection of the physical structure type;

providing a user interface comprising;

the toolbox of building project-related annotation tools selected based at least in part on the received user selection of the physical structure type;

an interface enabling a user to select one or more video frames from an uploaded video file;

an editing area configured to display at least one user selected frame and enabling the user to drag and drop a geometric shape that indicates a respective physical structure defect type over a portion of a frame displayed in the editing area and to size the geographic shape;

a user interface configured to enable a user to associate a criticality level from a plurality of criticality levels, including at least a critical event level and an informational event level, with the geometric shape dropped over the over the portion of a frame displayed in the editing area;

an association interface enabling the user to define a relationship between a video file, a building project, and a building-related task;

receiving a user annotation of at least one video frame, the user annotation comprising a geometrical shape selected by the user from the toolbox and corresponding metadata indicating a criticality of a building project related defect;

receiving an association of the video file with a first building project-related task comprising remediation or inspection;

storing the user annotation of the at least one video frame, the user annotation comprising the geometrical shape selected by the user from the toolbox and corresponding metadata indicating the criticality of the building project related defect and physical structure defect type;

storing the association of the video file with the first building project and the first building project-related task;

providing a search limiter user interface enabling the user to identify a building project for which a search is to be limited to;

providing a search interface enabling the user to search for building project-related annotated video files;

receiving a first user search query via the search interface enabling the user to search for building project-related annotated video files;

identifying, based at least in part on a comparison of the first search query with video file metadata, at least one building project-related annotated video file; and

providing search results comprising the at least one identified building project-related annotated video file, wherein the search results including the at least one identified building project-related annotated video file are limited to the building project identified via the search limiter user interface.

2. The system as defined in claim 1, the operations further comprising:

enabling the user to annotate an audio file; and

receiving an association of the audio file with at least one building project-related task.

3. The system as defined in claim 1, the operations further comprising:

identifying relations between two or more media files based at least in part on;

commonality of assets;

or

commonality of human resources involved in creating the two or more media files;

or

commonality of viewers, or

any combination of commonality of assets, commonality of human resources involved in creating the two or more media files, or commonality of viewers;

receiving a search query from the user requesting media files having one or more specified commonalities;

identifying two or more media files having the one or more specified commonalities;

providing search results indicating the identified two or more media files having the one or more specified commonalities.

4. The system as defined in claim 1, the operations further comprising:

providing a user interface configured to enable the user to select at least a first video frame from the video file;

receiving an annotation from the user for the first frame;

displaying, using a player, the first frame with the annotation;

accessing location information indicating where the first frame was recorded; and

indicating the location via a plan document or a map while the first frame is displayed.

5. The system as defined in claim 1, the operations further comprising:

providing a user interface configured to enable the user to select a plurality of video frames from the video file as a group;

receiving a geometric annotation from the user for a first frame in the group of selected video frames, wherein the geometric annotation is associated with a first X-Y location in the first frame in the group of selected video frames; and

associating the geometric annotation with each of the group of selected video frames;

playing, using a video player, the group of selected video frames, wherein the video player displays the geometric annotation at substantially the same X-Y position in each video frame in the group of selected video frames.

6. The system as defined in claim 1, wherein storing the user annotation of the at least one video frame comprises storing an identification of the geometrical shape selected by the user and a positing of the geometrical shape selected by the user within the at least one frame without modifying the visual content of the at least one frame.

7. The system as defined in claim 1, the operations further comprising tracking at least one annotated object through two or more frames utilizing one or more of kernel-based tracking, visual feature matching, blob tracking, or contour tracking.

8. The system as defined in claim 1, wherein the toolbox comprises a first geometric shape associated with metadata indicating an urgent issue, a second geometric shape associated with metadata indicating a major issue, and a third geometric shape associated with metadata indicating a minor issue.

9. The system as defined in claim 1, the operations further comprising providing a search filter control enabling the user to filter a search by building project related task.

10. The system as defined in claim 1, the operations further comprising enabling the user to include a link to the annotated video file in at least one plan sheet of at least one electronic building plan document.

11. The system as defined in claim 1, wherein the building-related task comprises an inspection task.

12. The system as defined in claim 1, wherein the building-related task comprises a remediation task.

13. The system as defined in claim 1, the operations further comprising providing a user interface configured to enable the user to associate an audio recording with at least one plan sheet of at least one electronic building plan document.

14. A system for managing building plan documents, comprising:

one or more computing devices; and

non-transitory computer readable memory storing program code that when executed by the one or more computing devices is configured to cause the system to perform operations comprising;

receiving an electronic building plan document including a plurality of plan sheets;

providing a user interface enabling a user to upload a video file;

providing an asset menu including a plurality of different physical structure types, including at least one of a high rise building, a low rise building, a private home, a hospital, a school, a bridge, a tunnel, or a water containment structure, via which the user can select a physical structure type;

receiving a user selection of a physical structure type via the asset menu;

selecting a toolbox of building project-related annotation tools, comprising a first plurality of different geometrical shapes configured to be selectively placed on video frames of structures corresponding to the selected physical structure type, wherein each geometrical shape in the first plurality of different geometrical shapes indicates a different physical structure defect type and is associated with respective metadata indicating the physical structure defect type, wherein the toolbox of building project-related annotation tools is selected based at least in part on the received user selection of the physical structure type;

providing a user interface comprising;

the toolbox of building project-related annotation tools selected based at least in part on the received user selection of the physical structure type;

an interface enabling a user to select one or more video frames from an uploaded video file;

an editing area configured to display at least one user selected frame and enabling the user to drag and drop a geometric shape that indicates a respective physical structure defect type over a portion of a frame displayed in the editing area and to size the geographic shape;

a user interface configured to enable a user to associate a criticality level from a plurality of criticality levels, including at least a critical event level and an informational event level, with the geometric shape dropped over the over the portion of a frame displayed in the editing area;

an association interface enabling the user to define a relationship between a video file and the building plan document;

receiving a user annotation of at least one video frame, the user annotation comprising a geometrical shape selected by the user from the toolbox and corresponding metadata indicating a criticality of a building project related defect;

receiving an association of the video file with a first building project-related task comprising remediation or inspection;

storing the user annotation of the at least one video frame, the user annotation comprising the geometrical shape selected by the user from the toolbox and corresponding metadata indicating the criticality of the building project related defect and physical structure defect type;

storing the association of the video file with the first building project and the first building project-related task;

providing a search limiter user interface enabling the user to identify a building project for which a search is to be limited to;

providing a search interface enabling the user to search for building project-related annotated video files;

receiving a project identification via the search limiter interface;

receiving a first user search query via the search interface enabling the user to search for building project-related annotated video files;

identifying, based at least in part on a comparison of the first search query with video file metadata, at least one building project-related annotated video file; and

providing search results comprising the at least one identified building project-related annotated video file, wherein the search results including the at least one identified building project-related annotated video file are limited to the project identified via the search limiter user interface.

15. The system as defined in claim 14, the operations further comprising:

enabling the user to annotate an audio file; and

receiving an association of the audio file with at least one building project-related task.

16. The system as defined in claim 14, the operations further comprising:

identifying relations between two or more media files based at least in part on;

commonality of assets;

commonality of human resources involved in creating the two or more media files; and

/or

commonality of viewers

receiving a search query from the user requesting media files having one or more specified commonalities;

identifying two or more media files having the one or more specified commonalities;

providing search results indicating the identified two or more media files having the one or more specified commonalities.

17. The system as defined in claim 14, the operations further comprising:

providing a user interface configured to enable the user to select at least a first video frame from the video file;

receiving an annotation from the user for the first frame;

displaying, using a player, the first frame with the annotation;

accessing location information indicating where the first frame was recorded; and

indicating the location via a plan document or a map while the first frame is displayed.

18. The system as defined in claim 14, the operations further comprising:

providing a user interface configured to enable the user to select a plurality of video frames from the video file as a group;

receiving a geometric annotation from the user for a first frame in the group of selected video frames, wherein the geometric annotation is associated with a first X-Y location in the first frame in the group of selected video frames;

associating the geometric annotation with each of the group of selected video frames; and

playing, using a video player, the group of selected video frames, wherein the video player displays the geometric annotation at substantially the same X-Y position in each video frame in the group of selected video frames.

19. The system as defined in claim 14, wherein storing the user annotation of the at least one video frame comprises storing an identification of the geometrical shape selected by the user and a positing of the geometrical shape selected by the user within the at least one frame without modifying the visual content of the at least one frame.

20. The system as defined in claim 14, the operations further comprising tracking at least one annotated object through two or more frames utilizing one or more of kernel-based tracking, visual feature matching, blob tracking, or contour tracking.

21. The system as defined in claim 14, wherein the toolbox comprises a first geometric shape associated with metadata indicating an urgent issue, a second geometric shape associated with metadata indicating a major issue, and a third geometric shape associated with metadata indicating a minor issue.

22. The system as defined in claim 14, the operations further comprising providing search a search filter control enabling the user to filter a search by building project related task.

23. The system as defined in claim 14, the operations further comprising enabling the user to include a link to the annotated video file in at least one plan sheet of at least one electronic building plan document.

24. The system as defined in claim 14, wherein the building project-related task comprises an inspection task.

25. The system as defined in claim 14, wherein the building project-related task comprises a remediation task.

26. The system as defined in claim 14, the operations further comprising providing a user interface configured to enable the user to associate an audio recording with at least one plan sheet of at least one electronic building plan document.

27. Non-transitory computer readable memory storing program code that when executed by one or more computing devices is configured to cause the one or more computing devices to perform operations comprising:

receiving an electronic building plan document including a plurality of plan sheets;

providing a user interface enabling a user to upload a video file;

providing an asset menu including a plurality of different physical structure types, including at least one of a high rise building, a low rise building, a private home, a hospital, a school, a bridge, a tunnel, or a water containment structure, via which the user can select a physical structure type;

receiving a user selection of a physical structure type via the asset menu;

selecting building project-related annotation tools, comprising a first plurality of different geometrical shapes configured to be selectively placed on video frames of structures corresponding to the selected physical structure type, wherein a given geometrical shape in the first plurality of different geometrical shapes indicates a respective physical structure defect type and is associated with respective metadata indicating the physical structure defect type, wherein the toolbox of building project-related annotation tools is selected based at least in part on the received user selection of the physical structure type;

providing a user interface comprising;

the building project-related annotation tools selected based at least in part on the received user selection of the physical structure type;

an interface enabling a user to select one or more video frames from an uploaded video file;

an editing area configured to display at least one user selected frame and enabling the user to drag and drop a geometric shape that indicates a respective physical structure defect type over a portion of a frame displayed in the editing area and to size the geographic shape;

a user interface configured to enable a user to associate a criticality level, indicating how critical a respective defect is, from a plurality of criticality levels with the geometric shape dropped over the over the portion of a frame displayed in the editing area;

an association interface enabling the user to define a relationship between a video file and the building plan document;

receiving a user annotation of at least one video frame, the user annotation comprising a geometrical shape selected by the user from the building project-related annotation tools and corresponding metadata indicating a criticality of a building project related defect;

receiving an association of the video file with a first building project-related task comprising remediation or inspection;

storing the user annotation of the at least one video frame, the user annotation comprising the geometrical shape selected by the user from the building project-related annotation tools and corresponding metadata indicating the criticality of the building project related defect and physical structure defect type;

storing the association of the video file with the first building project and the first building project-related task;

providing a search interface enabling the user to identify a building project for which a search is to be limited to and to search for building project-related annotated video files;

receiving a first user search query via the search interface enabling the user to search for building project-related annotated video files;

identifying, based at least in part on a comparison of the first search query with video file metadata, at least one building project-related annotated video file; and

providing search results comprising the at least one identified building project-related annotated video file, wherein the search results including the at least one identified building project-related annotated video file are limited to the project identified via the search limiter user interface.