Apparatus and method for supporting the implicit structure of freeform lists, outlines, text, tables and diagrams in a gesture-based input system and editing system

• US 6,525,749 B1
• Filed: 10/25/1996
• Issued: 02/25/2003
• Est. Priority Date: 12/30/1993
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Van Melle, William, Chiu, Patrick, Moran, Thomas P.
• Primary Examiner(s): Alam, Hosain T.
• Application Number:
  US08/736,883
• Time in Patent Office: 2,314 Days
• Field of Search: 395/326, 395/358, 395/761, 395/804, 345/326-358, 345/863, 345/173, 345/156, 345/179
• US Class Current: 715/863
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting

Abstract

A system features a data input scheme in which the nature of the entered data is irrelevant until the requirement made to perform a function with respect to that data. It is only when it becomes necessary to perform a function with respect to the data that the determination of the nature of the data become important. The desired function, which may include editing, rearranging, moving, etc., then results in a determination of the nature of the data in order to enable implementation of that specific function. The nature of the data can be determined to be in the form of a list, table, or diagram. To this end, a screen oriented display processing system is provided for the entry of informational data including a display means, a data entry means for entering informational data on the display means, a gesture entering means for entering handwritten gestural data on the display, the gestural data representing a gesture associated with at least a portion of the informational data on the display, an assignment means responsive to the entry of the gesture for assigning a function representative of the gesture to the portion of the informational data, and an implementing means responsive to the assignment means for implementing the function on the said portion of the informational data.

Claims

1. A display-oriented graphical editing system which allows for inputting and editing of data in the form of informal scribbling in handwriting and sketches, and inputting and editing of ASCII characters, the system comprising:

a substantially uniformed unlined display;

means for entering the data, which is unstructured freeform data, anywhere on said unlined display;

wherein the structure of said displayed data is undetermined at the time of their entry;

means for selecting an editing operation from among a plurality of editing operations to be performed on at least a portion of said displayed data, the editing operations capable of being performed including selectively editing of script, ASCII and diagrams;

means for designating said portion of said displayed data as at least one particular data structure before performing said editing operation;

wherein the at least one particular data structure of said selected portion of said displayed data may be changed after performing said editing operation; and

means for performing said editing operation on said selected portion of said displayed data;

wherein the operability of said editing operation is independent of the location of said selected portion of said displayed data, and said editing operations include line-wrapping of the script and ASCII characters entered on the unlined display when the one particular data structure is a text data structure.

44. The system of claim 1 wherein the operability of said editing operation is determined solely by said at least one particular data structure of said selected portion of said displayed data.

47. The method according to claim 1, wherein the editing operation is appropriately performed anywhere on the display for anyone of the list, text, outline, table and diagram.

2. A graphical editing method for a system including an unlined data interface surface configured without pre-set boundaries which define predetermined entry areas for selected data types, and user controllable means for generating unstructured freeform information anywhere on said unlined data interface surface which allows for informal scribbling in handwriting and sketches, and inputting of ASCII characters, wherein the structure of said information is undetermined at the time of its generation, the method comprising the steps of:

drawing a gesture on the unlined data interface surface for performing an operation on at least a portion of said information,

selectively determining a data structure, in response to said drawing of said gesture, for at least a subportion of said information on said unlined interface surface in accordance with said operation;

wherein said data structure may vary in accordance with said operation and said data structure is selectively each of a list, text, outline, table and diagram located at any location on the unlined data interface surface, and wherein the operation associated with the gesture is the same, irrespective of a location on the display where the gesture is located;

identifying said information affected by said operation with respect to said data structure;

computing structural parameters for said operation as a function of information identified; and

implementing said operation on said information.

3. A method for editing data on a display of a graphical editing system comprising the steps of:

entering unstructured freeform data anywhere on said display;

wherein said display has a substantially uniform blank unlined presentation, prior to inputting of the data;

designating at least a portion of said displayed data as at least one particular data structure;

wherein said at least one particular data structure may be changed; and

performing an editing operation on at least a subportion of said designated portion of said displayed data after said designation;

wherein the operability of said editing operation is independent of the location of said designated portion of said displayed data, and wherein when the data is designated as text data entered on the unlined display a line-wrapping procedure is made available to a user.

4. The method of claim 3 further comprising interpreting said editing operation as appropriate to said at least one data structure before performing said editing operation.

5. The method of claim 4 further comprising computing structural parameters for said editing operation as a function of said at least one data structure identified.

6. The method of claim 3 wherein said portion of said displayed data is designated as said at least one particular data structure by entering a structured selection gesture which implicitly identifies which at least one data structure to assign.

7. The method of claim 3 further comprising selecting said editing operation to be performed on said portion of said displayed data before performing said editing operation wherein said editing operation designates which at least one data structure to assign.

8. The method of claim 3 further comprising selecting said portion of said displayed data to which said data structure is assigned and selecting said editing operation to be performed on said portion of said displayed data before performing said editing operation wherein both said selection of said portion of said displayed data and said editing operation designate which at least one data structure to assign.

9. The method of claim 3 further comprising selecting, with a freeform gesture, at least a portion of said displayed data to which said data structure is assigned and selecting said editing operation to be performed on at least a portion of said displayed data.

10. The method of claim 3 further comprising selecting said editing operation to be performed by entering at least one gesture on said display.

11. The method of claim 3 wherein said at least one data structure is selected from the group of data structures consisting of a list, text sequence, table, diagram, node-link diagram and outline.

12. The method of claim 11 wherein said list comprises at least one vertical column of line items separated by spaces in accordance with said editing operation.

13. The method of claim 12 wherein said editing operation comprises aligning said at least one column and said line items therein.

14. The method of claim 11 wherein said at least one data structure comprises an outline which comprises a plurality of lines including a first line having a smaller indentation than a second line and wherein said editing operation comprises shifting said indentations to discrete indentation levels.

15. The method of claim 14 wherein said outline comprises a subtree and said editing operation comprises collapsing said subtree so that the lines within the subtree are not displayed.

16. The method of claim 15 wherein said editing operation comprises expanding said subtree so that the lines within the subtree are displayed.

17. The method of claim 3 further comprising designating a second portion of said displayed data other than said first portion of said displayed data and placing a border between said first and second portions.

18. The method of claim 17 further comprising inhibiting said editing operation from being performed on said second portion.

19. The method of claim 17 wherein said border inhibits a selection of at least a subportion of said first portion from extending into said second portion.

20. The method of claim 17 further comprising selecting a subportion of said first portion of displayed data and extending said selection into said second portion.

21. The method of claim 11 wherein said displayed data comprises first and second nodes comprising information, said data structure is a node-link diagram linking said first and second nodes with a link, and said editing operation comprises moving said first node and, in response to said move, adjusting said link so as to continue to link said first and second nodes without moving said second node.

22. The method of claim 21 wherein said link is distortable.

23. The method of claim 21 wherein said link is stretchable.

24. The method of claim 3 wherein said data is entered with a pointing device and said display is a touch panel.

25. The method of claim 3 wherein said data is entered with a keyboard.

26. The method of claim 3 wherein said displayed data comprises graphical objects.

27. The method of claim 3 further comprising undesignating said displayed data as said at least one data structure after said editing operation is completed or after a command by the user to terminate the designation.

28. The method of claim 27 further comprising deleting any record of said data structure previously designated from said system before or during said step of undesignating said displayed data.

29. The method of claim 3 wherein said step of performing said editing operation on said portion of said displayed data comprises leaving undisturbed by said editing operation any displayed data which does not conform to the designated data structure.

30. The method of claim 3 wherein said portion comprises all of said displayed data.

31. The method of claim 3 wherein said subportion comprises all of said portion of said displayed data.

45. The method of claim 3 wherein the operability of said editing operation is determined solely by said at least one particular data structure of said designated portion of said displayed data.

32. A method for editing data on a substantially blank unlined input area of a display of a graphical editing system which allows for informal scribbling in handwriting, sketches, and inputting ASCII characters with an input device, the method comprising the steps of:

entering the data, which is unstructured freeform data, anywhere on said unlined input area of the display;

wherein the structure of said displayed data is undetermined at the time of their entry;

designating at least a portion of said displayed data, which has been entered as freeform data, as at least one particular data structure and said data structure is selectively each of a list, text, outline, table and diagram, wherein said at least one particular data structure may be changed from time to time depending upon said designation, thereby creating a structured selection so that, if desired, an editing operation appropriate to said at least one data structure may be performed on at least a subportion of said designated portion of said displayed data after said designation, wherein an editing operation which may be performed includes a line wrapping editing procedure for the data designated as text, in of the unlined input area which is also configured to receive data designated as a diagram.

33. The method of claim 32 wherein said system comprises a memory and said method further comprises

generating a line image on said display;

searching through said memory in response to said line image generation to determine if said line image is a command gesture; and

if said line image is a command gesture, performing said editing operation in accordance with of a command selected by said command gesture and displaying said displayed data after said editing operation has been performed.

34. The method of claim 32 further comprising selecting said portion of said displayed data to which said data structure is assigned with a first gesture and selecting said editing operation to be performed on said subportion of said displayed data with a second gesture before performing said editing operation and wherein said designation of at least a portion of said displayed data as at least one data structure is implicitly identified by at least one of the gestures from the group consisting of a said first and second gestures.

35. A method for editing data on a substantially uniform unlined input area of a display of a graphical editing system which allows for informal scribbling in handwriting and sketches, and inputting ASCII characters with an input device, the method comprising the steps of:

entering unstructured freeform data anywhere on said substantially uniform unlined input area of the display;

wherein the structure of said displayed data is undetermined at the time of their entry;

selecting at least a portion of said displayed data by entering a gesture on said display;

selecting one of a plurality of editing operations configured to be performed on at least a subportion of said selected portion of said displayed data, at least one of the editing operations including text editing operations;

designating said selected portion of said displayed data as at least one particular data structure, thereby creating a structured selection and said data structure is selectively each of a list, text, outline, table and diagram;

wherein said at least one particular data structure may vary depending upon each said designation; and

performing said editing operation on at least a subportion of said structured selection;

wherein the operability of said editing operation is independent of the location of said selected portion of said displayed data, such that data entered at a drawing/text area of the uniform unlined input area can be edited in accordance with both text editing procedures and graphical editing procedures.

36. The method of claim 35 wherein said selection gesture which selects at least a portion of said displayed data either alone or in combination with said editing operation implicitly identifies to the graphical editing system which at least one data structure should be designated.

46. The method of claim 35 wherein the operability of said editing operation depends solely on said at least one particular data structure.

37. In a graphical editing system having a substantially uniform unlined display and a plurality of available data structures, a method for editing data which allows for informal scribbling in handwriting and sketches as well as ASCII characters at any location on the display and for selection of appropriate text based editing for editing of the handwriting and ASCII characters and appropriate graphics based editing for the editing of the sketches independent of the location of the handwriting ASCII characters, and sketches on the display, the method comprising:

entry of data in an unstructured freeform manner anywhere on said display, wherein said data is not assigned any of the plurality of available data structures at the time of their entry;

selection of at least a portion of said data with a selection command to indicate on which data to perform an editing operation; and

interpretation by said system of said selected data to associate with said selected data one of said plurality of available data structures which is selective each of a list, text, outline, table and diagram, whereby the system implicitly determines which of said plurality of available data structures is appropriate based upon said selected data and either said selection command or on said editing operation to be performed;

wherein said selected data may be associated with different said available data structure depending upon said editing operation to be performed and wherein, when the selected editing operation is a text editing operation a line wrapping operation is available for text located on the unlined display.

38. The method of claim 37 wherein said available data structures include a list, text, a table, a diagram and an outline.

39. The method of claim 37 further comprising selection of said editing operation.

40. The method of claim 39 wherein said selection command or selection of said editing operation is entered as a gesture.

41. The method of claim 37 wherein said system disregards the selected data structure for said selected data and treats said data as unstructured freeform after said selected data has been interpreted by said system as having at least one of said plurality of available data structures.

42. The method of claim 41 wherein after said system disregards the selected data structure, no record of the selected data structure is retained.

43. In a graphical editing system having a substantially uniform unlined display and a plurality of available data structures, a method for editing data which allows for informal scribbling in handwriting and sketches as well as entry of ASCII characters at any location on the display and for selection of appropriate text based editing for editing of the handwriting and ASCII characters, and appropriate graphics based editing for the editing of the sketches independent of the location of the handwriting and sketches on the display, the method comprising:

entry of data in an unstructured freeform manner anywhere on said display, wherein the structure of said data is undetermined at the time of their entry;

entry of an editing command, and

interpretation by said system of said data to associate with said data one of said plurality of available data structures which is selective each of a list, text, outline, table and diagram, whereby the system implicitly determines which of said plurality of data structures is appropriate based upon said data and said editing command;

wherein said data may be associated with different said plurality of available data structures depending upon said editing command to be performed and said editing command remains the same irrespective of where on the unlined display the editing command is located.