System and method of reflowing ink objects

• US 5,596,350 A
• Filed: 03/01/1995
• Issued: 01/21/1997
• Est. Priority Date: 08/02/1993
• Status: Expired due to Term

• Current Assignee: Apple Computer Incorporated (Apple Inc.)
• Original Assignee: Apple Computer Incorporated (Apple Inc.)
• Inventors: Pagallo, Giulia, Capps, Stephen P.
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): Liang, Regina
• Application Number:
  US08/397,549
• Time in Patent Office: 692 Days
• Field of Search: 345/172, 345/179, 345/180, 345/181, 345/182, 345/183, 345/141, 345/117, 345/901, 382/187, 382/188, 382/189, 382/181, 382/309, 178/18, 178/19
• US Class Current: 345/173
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text

Abstract

A pen-based system for reflowing handwritten ink objects is disclosed. The system divides the ink objects into paragraphs by identifying groups of ink objects separated from other groups by predefined distances. Each such paragraph is classified as either a graphics paragraph or a text paragraph; graphics paragraphs are formatted according to a graphics formatting protocol and text paragraphs are formatted according to a text formatting protocol. In addition, the system writes formatted paragraphs to destinations where they are displayed. The destination may be a screen on which the ink objects were originally written by a user and/or a printed page, a computer monitor, etc.

Claims

1. A method of formatting ink objects written with a pointer on an input tablet of a computer system, an ink object being handwritten information displayed as handwriting, the method comprising the following computer implemented steps:

(a) partitioning the ink objects entered on the input tablet coupled to the computer system into paragraphs by identifying groups of ink objects separated from other groups by at least a predefined vertical distance, each such group constituting a paragraph;

(b) automatically determining whether each identified paragraph is a text paragraph or a non-text paragraph, a text paragraph being a group of words separated from other paragraphs on the input tablet by at least said predefined vertical distance;

(c) formatting text paragraphs according to a text formatting protocol and formatting non-text paragraphs according to a non-text formatting protocol, said text and non-text formatting protocols employing different rules; and

(d) outputting the formatted non-text paragraphs and text paragraphs over an output device coupled to said computer system wherein the outputted paragraphs include said ink objects displayed as handwriting.

2. The method of claim 1 wherein the predefined vertical distance is between about 0 and 10 mm.

3. The method of claim 1 wherein the ink objects are partitioned into paragraphs by the following steps:

(i) defining bounding boxes around the individual ink objects which are not already in paragraphs, each bounding box having a top and a bottom;

(ii) from among the ink objects specified in step (1), identifying the top most located ink object on the input tablet designating that ink object'"'"'s bounding box as a seed, and marking that ink object as removed;

(iii) for each additional ink object not already in a paragraph, increasing its bounding box height by a defined amount, determining whether the top of its bounding box is higher than the bottom of the seed, and when the top of its bounding box is higher than the bottom of the seed, increasing the seed to include its bounding box, and marking the ink object as removed; and

(v) repeating steps (ii) and (iii) until all ink objects are marked.

4. The method of claim 1 wherein paragraphs containing more than a predefined percentage of ink objects of size greater than a predetermined magnitude are designated as graphics paragraphs.

5. The method of claim 1 the non-text paragraphs comprise at least one graphics paragraph and wherein paragraphs containing an ink object of size greater than about 10 mm are designated as graphics paragraphs.

6. The method of claim 1 wherein the non-text formatting protocol includes a protocol for formatting graphics paragraphs and the graphics formatting protocol includes a step of centering the graphics paragraph.

7. The method of claim 1 wherein the text formatting protocol includes the following steps:

(i) grouping the ink objects of the text paragraph into lines, each separated from other lines by a predetermined line separation distance; and

(ii) writing the lines to a destination such that the objects in the line are aligned on a predefined baseline on the destination.

8. The method of claim 7 further comprising a step of grouping the ink objects of a line into words, each word separated from the other words on its line by a predefined word separation distance.

9. The method of claim 1 further comprising the steps of:

(e) partitioning the ink objects within a paragraph into one or more lines by identifying groups of ink objects separated from any other groups in one direction by at least a predefined line distance, each such group constituting a line;

(f) partitioning the ink objects in the one or more lines into words displayed in handwritten form;

(g) separating the words in the one or more lines by defined word separation distances;

(h) separating the one or more lines in the paragraph by defined line separation distances when there is more than one line; and

(i) writing the words in the paragraph to a destination such that the words are aligned in one or more destination lines, wherein the one or more destination lines are separated from one another by the defined line separation distances when there is more than one destination line, and wherein the words in each destination line are separated from one another by the defined word separation distances.

10. The method of claim 9 wherein the words have bounding boxes and are partitioned into one or more lines by the following steps:

(i) identifying the topmost located ink object displayed on the tablet within said paragraph, designating that ink object'"'"'s bounding box as a seed, and marking that ink object as removed, provided the seed is not already in a line;

(ii) for each additional unmarked ink object, determining whether its midpoint is located between the top and the bottom of the seed, and when the midpoint is so located, increasing the seed to include the ink object'"'"'s bounding box and marking the ink object as removed;

(iii) repeating step (ii) until no new ink objects are added to the seed during step (ii), the ink objects within the seed being designated as a line; and

(iv) repeating steps (ii) through (iv) until all ink objects are marked.

11. The method of claim 10 further comprising the following steps:

(v) identifying thin lines having a height less than a preset line height; and

(vi) joining thin lines with the closest adjacent line.

12. The method of claim 10 wherein the seed is rectangular.

13. The method of claim 9 wherein the ink objects are partitioned into words by the following steps:

(i) determining the distance between two sequential ink objects in a line; and

(ii) joining two sequential ink objects when the distance between them is less than a predetermined value of about 5 mm.

14. The method of claim 9 further comprising the following steps:

(i) determining an average word distance between the words in each line of the paragraph;

(ii) setting the defined word separation distances equal to the average word distance;

(iii) determining an average line distance between each line in the paragraph; and

(iv) setting the defined line separation distances equal to the average line distance.

15. The method of claim 9 wherein the words have bounding boxes of defined lengths and the step of writing the words in the paragraph to a destination comprises the following steps:

(i) defining a destination width for each destination line;

(ii) writing words to the destination in the order in which they appear in the writing medium such that the words written to a destination line are separated from adjacent words by the defined word separation distances, wherein the words are written to the destination line until adding an additional word would cause the sum of the word bounding box lengths and the word separation distances to exceed the destination width for the destination line.

16. The method of claim 15 wherein the destination lines are left justified.

17. The method of claim 16 wherein the topmost destination line is indented.

18. The method of claim 15 wherein the words are written to more than one destination line, each of which has a baseline separated from the baselines of other destination lines by the defined line separation distances.

19. The method of claim 18 wherein the defined line separation distances are identical for each line baseline.

20. The method of claim 18 further comprising the following steps:

(i) defining baselines of the words; and

(ii) writing words to the destination such that the words'"'"' baselines are aligned with the destination line baselines.

21. The method of claim 20 wherein the step of defining the baseline of a word includes the following steps:

(iii) dividing the word bounding box into a plurality of vertical sections along a horizontal axis;

(iv) identifying the lowest vertical position of any ink object in each vertical section;

(v) dividing the word bounding box into a plurality of horizontal sections along a vertical axis;

(vi) identifying the horizontal section containing the greatest number of lowest ink object vertical positions; and

(vii) defining the word baseline in the horizontal section identified in step (vi).

22. The method of claim 9 wherein the defined word separation distances are equal to a preset value.

23. The method of claim 9 wherein the defined line separation distances are equal to a preset value.

24. The method of claim 1 wherein the output device is a screen on which the user writes ink information.

25. A system for reformatting ink information written by a user using a pointer on a tablet of a computer system, the system comprising:

(a) means for converting the ink information to discrete ink objects, an ink object being handwritten information displayed as handwriting;

(b) means for grouping the ink objects into paragraphs by identifying groups of ink objects separated from other groups by a predefined vertical distance, each such group constituting a paragraph;

(c) means for automatically determining whether each identified paragraph is either a text paragraph or a non-text paragraph;

a text paragraph being a group of words separated from other paragraphs on the input tablet by at least said predefined vertical distance;

(d) means for formatting text paragraphs according to a text formatting protocol to produce formatted text paragraphs including one or more ink objects displayed as handwriting; and

(e) means for formatting non-text paragraphs according to a non-text formatting protocol to produce formatted non-text paragraphs including one or more ink objects displayed as handwriting, said text and non-text formatting protocols employing different rules.

26. The system of claim 25 further comprising a writing medium on which the user writes ink information.

27. The system of claim 25 further comprising

(i) a destination on which the formatting text and non-text paragraphs are displayed;

and

(ii) means for displaying the formatted text and non-text paragraphs to the destination.

28. The system of claim 27 wherein the destination is a screen on which the user writes ink information.

29. The system of claim 27 wherein the destination is a printed page.

30. The system of claim 28 further comprising means for allowing the user to disable and enable the system for reformatting the ink information.

31. The system of claim 25 further comprising:

(f) means for partitioning the ink objects within a paragraph into one or more lines;

(g) means for partitioning the ink objects in the one or more lines into words displayed in handwritten form;

(h) means for separating the words in the one or more lines by defined word separation distances;

(i) means for separating the one or more lines in the paragraph by defined line separation distances when there is more than one line;

(j) a destination in which words are written in one or more destination lines; and

(k) means for writing the words in the paragraph to the destination wherein the words are aligned in the one or more destination lines and wherein the one or more destination lines are separated from one another by the defined line separation distances when there is more than one destination line, and wherein the words in each destination line are separated from one another by the defined word separation distances.

32. The system of claim 31 wherein the destination is the writing medium on which the ink objects were handwritten.

33. The system of claim 32 further comprising means for controlling the time when the means for writing words to the destination writes words to the destination.

34. The system of claim 32 wherein the means for controlling instructs the means for writing words to the destination to writes words to the destination at a preset period after the user has stopped using the pointer on the writing medium.

35. The system of claim 31 wherein the destination is a printed page.