Method and system for enhanced inferred mode user interface operations

• US 9,659,279 B2
• Filed: 10/25/2013
• Issued: 05/23/2017
• Est. Priority Date: 10/25/2013
• Status: Expired due to Fees

• Current Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Saund, Eric, Janssen, Jr., William C., Mahoney, James V.
• Primary Examiner(s): Leggett, Andrea
• Application Number:
  US14/063,002
• Publication Number:
  US 20150121305A1
• Time in Patent Office: 1,306 Days
• Field of Search: 715710
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]

Abstract

A method and system is provided for the input of user interface commands. Particularly, command initiation events including at least one of: (i) a mouse press event on mouse pointer hardware, and (ii) at least one of a pen touch event, a stylus touch event, and a finger touch event on touch pointer hardware, are accepted. Then gesture stroke input events including at least one of: (i) a mouse drag event on the mouse pointer hardware, and (ii) at least one of a pen drag event, a stylus drag event and a finger drag event on the touch pointer hardware, are accepted. Additionally, command termination events including at least one of (i) a mouse release event on the mouse pointer hardware, and (ii) at least one of a pen lift event, a stylus lift event and a finger lift event on the touch pointer hardware, are accepted. The events are then interpreted as at least one of object selection or digital ink input operations without prior selection of a user input mode.

Claims

1. A method for input of user interface commands used to compose an electronic message on a display screen, the method comprising:

accepting command initiation events including at least one of;

(i) a mouse press event on mouse pointer hardware, and

(ii) at least one of a pen touch event, a stylus touch event, and a finger touch event on touch pointer hardware;

accepting gesture stroke input events including at least one of;

(i) a mouse drag event on the mouse pointer hardware, and

(ii) at least one of a pen drag event, a stylus drag event and a finger drag event on the touch pointer hardware;

accepting command termination events including at least one of;

(i) a mouse release event on the mouse pointer hardware, and

(ii) at least one of a pen lift event, a stylus lift event and a finger lift event on the touch pointer hardware;

interpreting the events as at least one of object selection or digital ink input operations without prior selection of user input mode;

composing the electronic message in a message composition region by selecting and transferring content from a referenced document external of the message composition region to the message composition region for use in composing the electronic message, wherein the selecting and transferring are performed where a single view is presented to the user; and

wherein the method is performed by use of an electronic computing device.

2. The method of claim 1 wherein ambiguity in said interpreting of said events is resolved by presenting a choice menu after completion of at least one of:

(i) the mouse release on the mouse pointer hardware and (ii) the pen lift event, the stylus lift event, and the finger lift event on the touch pointer hardware.

3. The method of claim 1 wherein the object selection operations include either of a lasso enclosure selection and a rectangle enclosure selection.

4. The method according to claim 1 wherein in a pointer-up state, the method further comprising:

receiving the press or touch event;

presenting a choice option to a user;

determining the presented choice option is at a choice option location on a canvas of the electronic display;

performing a choice action indicted by a selection of the choice option; and

entering a state of operation as determined by the choice action.

5. The method according to claim 1 wherein in a pointer-up state, the method further comprising:

receiving the press or touch event;

determining a choice option has been presented to a user;

determining the presented choice is at a location other than a choice option location on the canvas of the electronic display;

considering the press or touch event in relation to objects on the canvas of the electronic display;

determining the press or touch event is located over at least one selectable object of the objects on the canvas of the electronic display; and

modifying the at least one selectable object.

6. The method according to claim 5 wherein when it is determined the press or touch event is not located over at least one selectable object, further comprising,

entering a draw-gesture state.

7. The method according to claim 1 wherein in a pointer down state, the method further comprising:

entering the drag event;

determining the system is in an objects-selected state, with at least some objects selected; and

translating the selected objects across the canvas of the electronic display.

8. The method according to claim 1 wherein in a pointer down state, the method further comprising:

entering the drag event;

determining the system is in a draw-gesture state;

scoring gesture interpretations, from among a digital ink gesture, a selection lasso gesture, and a selection rectangle gesture to make an initial determination of a most likely gesture;

determining which of the gestures meet a predetermined threshold;

presenting, on the canvas of the electronic display, a visual indicator of the gesture that meets the predetermined threshold.

9. The method according to claim 1 further including:

providing the release event;

evaluating a gesture stroke in relation to objects on the canvas of the electronic display; and

determining the process is in a state of the selected objects.

10. The method according to claim 1 further including:

providing the release event;

determining the process is not in the state of the selected objects;

scoring gesture interpretation to identify a likelihood a user intent is one of candidate gestures, including a digital ink, a selection lasso, and a selection lasso;

undertaking an ambiguity test for the candidate gestures;

determining one of the candidate gestures is unambiguous;

determining the candidate gesture that is unambiguous is a selection operation; and

selecting objects enclosed by the unambiguous candidate gesture.

11. The method according to claim 10 further including:

determining the candidate gesture that is unambiguous is not a selection operation; and

accepting the candidate gesture as a digital ink object.

12. The method according to claim 10 further including:

determining the process is not in the state of the selected objects;

scoring gesture interpretations to identify a likelihood a user intent is one of candidate gestures, including a digital ink, a selection lasso, and a selection lasso;

undertaking an ambiguity test for the candidate gestures;

determining the ambiguity test is non-conclusive; and

presenting a post-gesture choice option to the user wherein the user unambiguously indicates the user intent.

13. The method according to claim 1 further including:

accepting the gesture is digital ink stroke;

determining the digital ink stroke matches a shape model for one of a predetermined class of graphic objects;

determining a clear match is not found; and

leaving the gesture stroke as digital ink.

14. The method according to claim 1 further including:

accepting the gesture as digital ink;

determining if the digital ink stroke matches a shape model for one of a predetermined class of graphic objects;

determining a clear match is found;

presenting an overlay image of the structured graphic object is presented on the canvas of the electronic display;

presenting a choice option to the user; and

replacing the digital ink stroke by the structured graphic object presented in the overlay, when the user confirms the selection by use of the choice option.

15. The method according to claim 1 further including composing an electronic message in a document composition region by use of at least one of the digital ink, the selection rectangle, and the selection lasso, without prior selection of mode.

16. The method according to claim 1 wherein the interpreting of the events further includes displaying a visual indicator of the inferred intent of the user input gesture when the user input is unambiguous before the release event, whereby the user is alerted of the gesture type before the release event.

17. A graphical input and display system having a user interface for creating and editing an electronic document comprising:

a user input interface configured to select and manipulate image objects;

an electronic display configured to display image objects and provide a system response to user selection actions;

an electronic processor, connected to receive requests for image edit operations and to access a memory structure, and to operate a user interaction arrangement configured to infer an intent of user input gestures as at least one of entry of digital ink, a selection rectangle, and a selection lasso without prior selection of mode, for the user input interface, wherein the user input interface includes either one or both of (i) a mouse pointer input device and (ii) a touch pointer input device, and wherein the user interaction arrangement is configured for entry of new image material of arbitrary graphical content and/or to select and to modify existing image material without pre-selecting input mode, the electronic processor further configured to display a visual indicator of the inferred intent of the user input gesture when the user input is unambiguous before the release event, whereby the user is alerted of the gesture type before the release event;

wherein the user interaction arrangement includes three entry points including, press or touch events, drag events and release events.

18. The system according to claim 17 wherein the user interface commands are used to compose an electronic message on a display screen the composition further including the electronic processor configured to:

select and transfer content from a referenced document external of a message composition region to the message composition region for use in composition of the electronic message,

wherein the select and transfer occur when a single view is presented to a user.

19. The system according to claim 17 wherein the electronic processor is further configured to display a visual indicator of the inferred intent of the user input gesture when the user input is unambiguous before the release event, whereby the user is alerted of the gesture type before the release event.

20. A method for utilizing a user interface on a display of an electronic computing device to compose an electronic message, the method comprising:

receiving a user input gesture;

inferring an intent of the user input gesture as at least one of;

digital ink, a selection rectangle, and a selection lasso, without prior selection of mode, for at least one of (i) a mouse pointer input device and (ii) a touch pointer input device, for entry of new image material of arbitrary graphical content and/or to select and to modify existing image material without pre-selecting an input mode,

wherein the receiving of the user input gesture includes receiving the input through at least one of three entry point events including, a press or touch event, a drag event and a release event of at least one of the mouse pointer input device and the touch pointer input device;

displaying a visual indicator of the inferred intent of the user input gesture when the user input is unambiguous before the release event, whereby the user is alerted of the gesture type before the release event;

composing the electronic message in a message composition region by selecting and transferring content from a referenced document external of the message composition region to the message composition region for use in composing the electronic message, wherein the selecting and transferring are performed where a single view is presented to the user; and

wherein the method is performed by use of an electronic computing device.