Multi-point interface for a graphical modeling environment

• US 8,525,813 B1
• Filed: 08/10/2007
• Issued: 09/03/2013
• Est. Priority Date: 12/29/2006
• Status: Active Grant

• Current Assignee: The MathWorks, Inc.
• Original Assignee: The MathWorks, Inc.
• Inventors: Smith, Paul F., Hosagrahara, Arvind S.
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US11/889,270
• Time in Patent Office: 2,216 Days
• Field of Search: 178 1801- 1811, 345/1.1, 345 21- 23, 345156-157, 345173-178, 700/17, 700/83, 715700-866
• US Class Current: 345/175
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A graphical model is displayed. A location of a portion of a user in relation to the displayed graphical model is sensed to obtain a sensed location of the portion of the user. A movement of the portion of the user in relation to the displayed graphical model and the sensed location of the portion of the user is sensed to obtain a sensed movement. The displayed graphical model is adjusted in response to the sensed movement of the portion of the user to obtain a displayed adjusted graphical model.

Claims

1. A computer-implemented method for displaying a graphical model on a display, the method comprising:

displaying the graphical model on the display, the display including a substantially planar surface for receiving tactile input from a user and for allowing the user to view the graphical model; and

manipulating the displayed graphical model based on an interaction between the user and the display, where the interaction includes;

receiving a first tactile input from the user on a first connection port, where the first connection port is associated with a first displayed element of the graphical model;

receiving a second tactile input from the user on a second connection port, where the second connection port is associated with a second displayed element of the graphical model; and

connecting or disconnecting the first displayed element and the second displayed element using the selected first connection port and the second connection port.

2. The method of claim 1, wherein the first tactile input from the user and the second tactile input from the user is selected from the group consisting of:

one finger on one hand;

one finger on one hand and one finger on another hand;

a plurality of fingers on one hand;

one finger on one hand and a plurality of fingers on another hand; and

a plurality of fingers on one hand and a plurality of fingers on another hand.

3. The method of claim 1, wherein a first location of the first tactile input from the user or the second tactile input by the user is selected from the group consisting of:

within an element of the displayed graphical model;

outside of an element of the displayed graphical model;

within a menu displayed on the display;

within a portion of the display not including the displayed graphical model;

around the perimeter of an element of the displayed graphical model;

on the perimeter of an element of the displayed graphical model.

4. The method of claim 1, further comprising:

sensing a movement of the first tactile input from the user, wherein the movement is selected from the group consisting of;

a motion of a finger;

motions of a plurality of fingers;

a series of motions of a finger; and

a series of motions of a plurality of fingers.

5. The method of claim 1, further comprising:

adjusting the displayed graphical model in response to the sensed movement of the first tactile input from the user, wherein adjusting the displayed graphical model is selected from the group consisting of;

dragging and dropping an element in the displayed graphical model;

resizing an element in the displayed graphical model;

displaying a hierarchy of sub-elements for an element displayed graphical model;

displaying a hierarchy of super-elements for an element displayed graphical model;

panning across a view of the displayed graphical model;

rotating an element in the displayed graphical model;

connecting or disconnecting elements in the displayed graphical model;

protecting and partitioning an area of the displayed graphical model;

simultaneously selecting a plurality of elements in the displayed graphical model;

copying and pasting an element in the displayed graphical model;

modifying a parameter of an element in the displayed graphical model; and

granting a user access to the displayed graphical model.

6. The method of claim 5, further comprising:

prior to the adjusting step;

sensing a fingerprint of the user; and

authorizing the user to adjust the displayed graphical model based on the sensed movement of the first tactile input from the user when the fingerprint is recognized; and

adjusting the displayed graphical model in response to the sensed movement of the first tactile input by the authorized user to obtain an adjusted displayed graphical model.

7. The method of claim 1, further comprising:

sensing a plurality of combinations of locations and movements of the first tactile input by the user, wherein the displayed graphical model is adjusted based on the plurality of combinations.

8. The method of claim 1, wherein the first tactile input from the user is sensed by a frustrated total internal reflection (FTIR) system comprising the display.

9. A computer-implemented method for displaying the dependencies of model elements, comprising:

downloading software to a computer system and storing the software in a non-transitory computer-readable storage medium, which software when executed by the computer system causes the computer system to perform operations comprising the method of claim 1.

10. A computer-implemented method for displaying the dependencies of model elements, comprising:

providing downloadable software to download to a computer system and storing the software in a non-transitory computer-readable storage medium, which software when executed by the computer system causes the computer system to perform operations comprising the method of claim 1.

11. A non-transitory computer-readable medium comprising software, which when executed by a computer, causes the computer to perform operations for displaying a graphical model on a display, the medium comprising:

instructions for displaying the graphical model on the display;

instructions for receiving tactile input from a user and manipulating the displayed graphical model based on an interaction between the user and the displayed graphical model, where the interaction includes one of;

part of the user interacting with the display or

the part and another part of the user simultaneously interacting with the display; and

instructions for manipulating the displayed graphical model, where the manipulating includes;

partitioning an area in the displayed graphical model with respect to non-partitioned areas of the displayed graphical model, and

protecting the partitioned area, where the protecting prevents editing the partitioned area in response to the received tactile input, and

where the partitioning excludes interfering with editable aspects of the non-partitioned areas of the displayed graphical model in response to the received tactile input.

12. The non-transitory computer-readable medium of claim 11, wherein the part of the user is selected from the group consisting of:

one finger on one hand;

one finger on one hand and one finger on another hand;

a plurality of fingers on one hand;

one finger on one hand and a plurality of fingers on another hand; and

a plurality of fingers on one hand and a plurality of fingers on another hand.

13. The non-transitory computer-readable medium of claim 11, wherein a location of the tactile input from the user is selected from the group consisting of:

within an element of the displayed graphical model;

outside of an element of the displayed graphical model;

within a menu displayed on the display;

within a portion of the display not including the displayed graphical model;

around the perimeter of an element of the displayed graphical model;

on the perimeter of an element of the displayed graphical model;

and on a connection port of an element of the displayed graphical model.

14. The non-transitory computer-readable medium of claim 11, further comprising:

instructions for recognizing a motion of the part of the user, wherein the motion of the part of the user is selected from the group consisting of;

an action of a finger;

actions of a plurality of fingers;

a series of actions of a finger; and

a series of actions of a plurality of fingers.

15. The non-transitory computer-readable medium of claim 14, further comprising:

instructions for modifying the displayed graphical model in response to the recognized motion of the part of the user, wherein instructions for modifying the displayed graphical model is selected from the group consisting of;

instructions for dragging and dropping an element in the displayed graphical model;

instructions for resizing an element in the displayed graphical model;

instructions for displaying a hierarchy of sub-elements for an element displayed graphical model;

instructions for displaying a hierarchy of super-elements for an element displayed graphical model;

instructions for panning across a view of the displayed graphical model;

instructions for zooming in or out of a view of the displayed graphical model;

instructions for rotating an element in the displayed graphical model;

instructions for connecting or disconnecting elements in the displayed graphical model;

instructions for protecting and partitioning an area of the displayed graphical model;

instructions for simultaneously selecting a plurality of elements in the displayed graphical model;

instructions for copying and pasting an element in the displayed graphical model;

instructions for modifying a parameter of an element in the displayed graphical model; and

instructions for granting a user access to the displayed graphical model.

16. The non-transitory computer-readable medium of claim 11, further comprising:

instructions for sensing a plurality of combinations of positions and motions of the user, wherein the displayed graphical model is adjusted based on the plurality of combinations.

17. The non-transitory computer-readable medium of claim 11, wherein the tactile input from the user is sensed by

a FTIR system comprising the display.

18. A computer-implemented graphical modeling system comprising:

a display to display a graphical model on the display, the displayed graphical model having at least a first element having a first connection port and a second element having a second connection port;

an interface to;

receive tactile input from a user, the tactile input from the user manipulating the displayed graphical model by;

selecting the first connection port and the second connection port, and connecting or disconnecting the first displayed element and the second displayed element using the selected first connection port and the second connection port,

the interface comprising a substantially planar surface for receiving the tactile input from the user and for allowing the user to view the graphical model,

where the interface manipulates the displayed graphical model based on an interaction between the user and the display, where the interaction is one of;

a portion of the user interacting with the display,

the portion and another portion of the user simultaneously interacting with the display, or

a plurality of portions of the user simultaneously interacting with the display.

19. The system of claim 18, wherein the portion of the user is selected from the group consisting of:

one finger on one hand;

one finger on one hand and one finger on another hand;

a plurality of fingers on one hand;

one finger on one hand and a plurality of fingers on another hand; and

a plurality of fingers on one hand and a plurality of fingers on another hand.

20. The system of claim 18, wherein a location of the tactile input from of the user is selected from the group consisting of:

within an element of the displayed graphical model;

outside of an element of the displayed graphical model;

within a menu displayed on the display;

within a portion of the display not including the displayed graphical model;

around the perimeter of an element of the displayed graphical model;

on the perimeter of an element of the displayed graphical model;

and on a connection port of an element of the displayed graphical model.

21. The system of claim 18, wherein the interface is further configured to:

sense a movement of the at least first portion of the user in relation to the displayed graphical model to obtain a sensed movement, wherein the movement of the user is selected from the group consisting of;

a motion of a finger;

motions of a plurality of fingers;

a series of motions of a finger; and

a series of motions of a plurality of fingers.

22. The system of claim 18, wherein adjusting the displayed graphical model is selected from the group consisting of

dragging and dropping an element in the displayed graphical model;

resizing an element in the displayed graphical model;

displaying a hierarchy of sub-elements for an element displayed graphical model;

displaying a hierarchy of super-elements for an element displayed graphical model;

panning across a view of the displayed graphical model;

zooming in or out of a view of the displayed graphical model;

rotating an element in the displayed graphical model;

connecting or disconnecting elements in the displayed graphical model;

protecting and partitioning an area of the displayed graphical model;

copying and pasting an element in the displayed graphical model;

modifying a parameter of an element in the displayed graphical model; and

granting a user access to the displayed graphical model.

23. The system of claim 18, wherein the interface is an frustrated total internal reflection system (FTIR) comprising the display.

24. A computer-implemented graphical modeling system comprising:

a display for displaying a graphical model;

an interface coupled to the display for receiving tactile input from a user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the interaction includes;

identifying a specific function to be assigned to a unique motion, recording the unique motion, and assigning the unique motion to the specific function, the unique motion having been previously unassigned to the specific function, and

modifying a parameter of an element in the displayed graphical model due to the tactile input from the user, where the interaction is a first part of the user continuously interacting with the display at a first location and a second part of the user continuously interacting with the display at a second location, a third location and an area between the second location and the third location.