System and method for recognition of handwritten diagram connectors

• US 10,417,491 B2
• Filed: 12/01/2015
• Issued: 09/17/2019
• Est. Priority Date: 10/19/2015
• Status: Active Grant

• Current Assignee: Myscript
• Original Assignee: Myscript
• Inventors: Melinand, Robin, Bednarowicz, Romain, Sidoli, Claire, Ric, Fabien, Bonnaud, Anne
• Primary Examiner(s): Rahmjoo, Manuchehr
• Application Number:
  US14/955,174
• Publication Number:
  US 20170109579A1
• Time in Patent Office: 1,386 Days
• Field of Search: 382198
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps

Abstract

A system, method and computer program product for hand-drawing diagrams including text and non-text elements on a computing device are provided. The computing device has a processor and a non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor. Display of a plurality of input diagram elements in interactive ink is performed on a display device associated with the computing device. At least one diagram element is identified as a connector which connects a plurality of diagram elements. Geometrical characteristics of the at least identified connector are determined and the diagram elements are re-displayed based on one or more interactions received with the interactive ink of the at least identified connector or one or more of the plurality of diagram elements connected thereby and in accordance with the determined geometrical characteristics.

Claims

1. A system for hand-drawing connections of diagrams including text and non-text elements on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the at least one non-transitory computer readable medium configured to:

cause display of a plurality of input diagram elements in interactive ink on a display device associated with the computing device;

identify at least one diagram element as a non-centered or a crossed connector which connects a plurality of diagram elements, each diagram element having a center of geometry, the non-centered or crossed connector defining a connection path which is offset from the centers of geometry of the connected diagram elements with a non-zero offset;

determine at least one of a projection line, point, and direction associated with the at least one identified connector; and

cause re-display of the diagram elements based on one or more interactions received with the interactive ink of the at least one identified non-centered or crossed connector or one or more of the plurality of diagram elements connected thereby and in accordance with the non-zero offset and the position of the determined at least one projection line, point, and direction, the position of the determined at least one projection line, point, and direction being updated when the diagram elements are re-displayed;

wherein an orientation of the at least one of a projection line, point, and direction associated with the at least one identified connector is determined based on an angle formed by a reference connector, and wherein the reference connector is connected nearest to a center of geometry of the at least one diagram element of the plurality of diagram elements.

2. A system as claimed in claim 1, wherein the at least one connector is identified based on characteristics of the connector and positional relationships between the diagram elements.

3. A system as claimed in claim 1, wherein the at least one projection line, point, and direction associated with the at least one identified connector is related to a connection path between geometrical features of the connected diagram elements.

4. A system as claimed in claim 3, wherein the center of geometry is one of the geometrical features of the connected diagram elements.

5. A system as claimed in claim 4, the at least one non-transitory computer readable medium being configured to determine the at least one projection line, point, and direction associated with the at least one identified connector to include a relationship between points of connection of the at least one connector with the connected diagram elements.

6. A system as claimed in claim 5, wherein the connection path is offset from the centers of geometry of the connected diagram elements based on the points of connection.

7. A system as claimed claim 1, wherein the interactive ink is digital ink.

8. A method for hand-drawing diagrams including text and non-text elements on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the method comprising:

displaying a plurality of input diagram elements in interactive ink on a display device associated with the computing device;

identifying at least one diagram element as a non-centered or a crossed connector which connects a plurality of diagram elements, each diagram element having a center of geometry, the non-centered or crossed connector defining a connection path which is offset from the centers of geometry of the connected diagram elements with a non-zero offset;

determining at least one of a projection line, point, and direction associated with the at least one identified; and

re-displaying the diagram elements based on one or more interactions received with the interactive ink of the at least one identified non-centered or crossed connector or one or more of the plurality of diagram elements connected thereby and in accordance with the non-zero offset and the position of the determined at least one projection line, point, and direction, the position of the determined at least one projection line, point, and direction being updated when the diagram elements are re-displayed;

wherein an orientation of the at least one of a projection line, point, and direction associated with the at least one identified connector is determined based on an angle formed by a reference connector, and wherein the reference connector is connected nearest to a center of geometry of the at least one diagram element of the plurality of diagram elements.

9. A method as claimed in claim 8, comprising identifying the at least one connector based on characteristics of the connector and positional relationships between the diagram elements.

10. A method as claimed in claim 8, wherein the at least one projection line, point, and direction associated with the at least one identified connector is related to a connection path between geometrical features of the connected diagram elements.

11. A method as claimed in claim 10, wherein the center of geometry is one of the geometrical features of the connected diagram elements.

12. A method as claimed in claim 11, wherein the at least one projection line, point, and direction associated with the at least one identified connector are determined to include a relationship between points of connection of the at least one connector with the connected diagram elements.

13. A method as claimed in claim 12, wherein the connection path is offset from the centers of geometry of the connected diagram elements based on the points of connection.

14. A method as claimed claim 8, wherein the interactive ink is digital ink.

15. A non-transitory computer readable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for hand-drawing diagrams including text and non-text elements on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the method comprising:

displaying a plurality of input diagram elements in interactive ink on a display device associated with the computing device;

identifying at least one diagram element as a non-centered or a crossed connector which connects a plurality of diagram elements, each diagram element having a center of geometry, the non-centered or crossed connector defining a connection path which is offset from the centers of geometry of the connected diagram elements with a non-zero offset;

determining at least one of a projection line, point, and direction associated with the at least one identified connector; and

re-displaying the diagram elements based on one or more interactions received with the interactive ink of the at least one identified non-centered or crossed connector or one or more of the plurality of diagram elements connected thereby and in accordance with the non-zero offset and the position of the determined at least one projection line, point, and direction, the position of the determined at least one projection line, point, and direction being updated when the diagram elements are re-displayed;

wherein an orientation of the at least one of a projection line, point, and direction associated with the at least one identified connector is determined based on an angle formed by a reference connector, and wherein the reference connector is connected nearest to a center of geometry of the at least one diagram element of the plurality of diagram elements.

16. A non-transitory computer readable medium as claimed in claim 15, comprising identifying the at least one connector based on characteristics of the connector and positional relationships between the diagram elements.

17. A non-transitory computer readable medium as claimed in claim 15, wherein at least one projection line, point, and direction associated with the at least one identified connector include a connection path between the geometrical features of the connected diagram elements.

18. A non-transitory computer readable medium as claimed in claim 17, wherein the center of geometry is one of the geometrical features of the connected diagram elements.

19. A non-transitory computer readable medium as claimed in claim 18, wherein the at least one projection line, point, and direction associated with the at least one identified connector are determined to include a relationship between points of connection of the at least one connector with the connected diagram elements.

20. A non-transitory computer readable medium as claimed in claim 19, wherein the connection path is offset from the centers of geometry of the connected diagram elements based on the points of connection.

21. A non-transitory computer readable medium as claimed claim 15, wherein the interactive ink is digital ink.