Multi-point interface for a graphical modeling environment

• US 8,519,979 B1
• Filed: 12/29/2006
• Issued: 08/27/2013
• Est. Priority Date: 12/29/2006
• Status: Active Grant

• Current Assignee: The MathWorks, Inc.
• Original Assignee: The MathWorks, Inc.
• Inventors: Smith, Paul F., Hosagrahara, Arvind S.
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US11/647,458
• Time in Patent Office: 2,433 Days
• Field of Search: 178 1801- 1811, 345/1.1, 345 21- 23, 345156-157, 345173-178, 700/17, 700/83, 715700-866
• US Class Current: 345/175
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A computer-implemented graphical modeling system comprising: a graphical modeling environment to generate a graphical model; a display to display the graphical model; an interface to receive an input from a user for manipulating the displayed graphical model based on an interaction between the user and the display; wherein the interaction is one of one finger on one hand of the user interacting with the display; one finger on one hand of the user interacting with the display and one finger on another hand of the user of the user interacting with the display; a plurality of fingers on one hand of the user interacting with the display; one finger on one hand of the user interacting with the display and a plurality of fingers on another hand of the user interacting with the display; or a plurality of fingers on one hand of the user interacting with the display and a plurality of fingers on another hand of the user interacting with the display.

Claims

1. A computer-implemented graphical modeling system comprising:

a display to display a graphical model, where the graphical model comprises;

a first element having a first connection port, and

a second element having a second connection port; and

an interface to receive tactile input from a user, the interface comprising;

a substantially planar surface for receiving the tactile input from the user and for allowing the user to view the graphical model, and

at least one edge for receiving optical energy from a source proximate to the at least one edge, the optical energy internally reflected with respect to the substantially planar surface and another surface when the tactile input is absent with respect to the substantially planar surface,

where the interface manipulates the displayed graphical model based on an interaction between the user and the display, where the interaction is one of;

one part of the user interacting with the display;

one part and another part of the user simultaneously interacting with the display;

or

a plurality of parts of the user simultaneously interacting with the display, and the tactile input from the user manipulating the displayed graphical model by;

connecting the first and the second displayed element via a connection when the tactile input from the user selects the first connection port of the first displayed element and the second connection port of the second displayed element, and

at least one of;

adjusting the connection between the first displayed element and the second displayed element when the first displayed element is manipulated by additional tactile input from the user, or

removing the connection between the first displayed element and the second displayed element by additional tactile input from the user selecting the second connection port of the second displayed element and selecting the first connection port of the first displayed element.

2. The system of claim 1, wherein the one part of the user interacting with the display comprises:

one finger on one hand of the user interacting with the display.

3. The system of claim 1, wherein the one part and another part of the user simultaneously interacting with the display comprises:

one finger on one hand of the user interacting with the display and one finger on another hand of the user of the user simultaneously interacting with the display.

4. The system of claim 1, wherein the plurality of parts of the user simultaneously interacting with the display comprises:

a plurality of fingers on one hand of the user simultaneously interacting with the display;

one finger on one hand of the user interacting with the display and a plurality of fingers on another hand of the user simultaneously interacting with the display;

or

a plurality of fingers on one hand of the user interacting with the display and a plurality of fingers on another hand of the user simultaneously interacting with the display.

5. The system of claim 1, wherein the interaction comprises a multi-dimensional movement of a hand of the user.

6. The system of claim 1, wherein the interface receives simultaneous tactile input from the user and one or more additional users to manipulate the displayed graphical model based on an interaction between the user, at least one of the additional users, and the display.

7. The system of claim 1, wherein a multi-point interface comprises the display and the interface.

8. The system of claim 1, wherein a frustrated total internal reflection (FTIR) system comprises the display and the interface.

9. The system of claim 8, wherein the FTIR system further comprises:

a plurality of light emitting diodes.

10. The system of claim 1, wherein the interface comprises a camera.

11. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user manipulates the displayed graphical model to drag and drop an element in the displayed graphical model.

12. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactle input from the user manipulates the displayed graphical model to resize an element in the displayed graphical model.

13. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user displays a hierarchy of sub-elements for an element in the displayed graphical model.

14. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user displays a hierarchy of super-elements for an element in the displayed graphical model.

15. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user pans across a view of the displayed graphical model.

16. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user zooms in or zooms out of a view of the displayed graphical model.

17. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user manipulates the displayed graphical model to rotate an element in the displayed graphical model.

18. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user manipulates the displayed graphical model to protect and partition an area in the displayed graphical model.

19. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user manipulates the displayed graphical model to simultaneously select a plurality of elements in the displayed graphical model.

20. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user manipulates the displayed graphical model to copy and paste an element in the displayed graphical model.

21. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user manipulates the displayed graphical model to modify a parameter of an element in the displayed graphical model.

22. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user allows the user to operate on an element in the displayed graphical model, wherein the tactile input from the user comprises a fingerprint of the user.

23. The system of claim 1, wherein the display is at a first location, the system further comprising:

a second display to display the graphical model, the second display at a second location different from the first location; and

a second interface to receive tactile input from a user, the second interface comprising;

a substantially planar surface for receiving the tactile input from the user and for allowing the user to view the graphical model, and

at least one edge for receiving optical energy from a source proximate to the at least one edge, the optical energy internally reflected with respect to the substantially planar surface and another surface when the tactile input is absent with respect to the substantially planar surface; and

the second interface for manipulating the displayed graphical model based on an interaction between the user and the display.

24. The system of claim 1, wherein the interface receives a further tactile input from the user for manipulating the displayed graphical model based on an interaction between the user and the display, wherein the received further tactile input from the user records a unique motion and assigns the unique motion to a specific function.

25. The system of claim 1, wherein the displayed graphical model is processed by a distributed backend computer system comprised of a plurality of computer systems connected via a network.

26. The system of claim 25, wherein the displayed graphical model comprises a plurality of areas and each area is supported by a separate computer system from the distributed backend computer system.

27. The system of claim 1, wherein the interface comprises a camera and at least one glove.

28. The system of claim 27, wherein the glove comprises at least one light source attached to the glove.

29. A computer-implemented method for graphical modeling comprising:

displaying a graphical model;

receiving an input from a user for manipulating the displayed graphical model based on an interaction between the user and the displayed graphical model, where the interaction is indicated using frustrated total internal reflection (FTIR);

modifying a parameter of an element in the displayed graphical model due to the received input, where the interaction is a first part of the user continuously interacting with the display at a first location and a second part of the user continuously interacting with the display at a second location, a third location and an area between the second location and the third location;

receiving a second input from a user for manipulating the displayed graphical model based on a second interaction between the user and the displayed graphical model, where the second interaction is indicated using FTIR; and

manipulating the graphical model to partition and protect an area with respect to non-partitioned areas in the displayed graphical model due to the received second input, where the protection prevents editing the partitioned area in response to additional received input.

30. The method of claim 29, further comprising:

receiving a further input from a user for manipulating the displayed graphical model based on a third interaction between the user and the displayed graphical model, where the third interaction is indicated using FTIR; and

manipulating the graphical model to connect or disconnect elements in the displayed graphical model due to the received further input.

31. The method of claim 29, further comprising:

receiving a further input from a user for manipulating the displayed graphical model based on a third interaction between the user and the displayed graphical model, where the third interaction is indicated using FTIR; and

manipulating the graphical model to simultaneously select a plurality of elements in the displayed graphical model due to the received further input.

32. The method of claim 29, further comprising:

receiving a further input from a user for manipulating the displayed graphical model based on a third interaction between the user and the displayed graphical model, where the third interaction is indicated using FTIR; and

manipulating the graphical model to copy and paste an element in the displayed graphical model due to the received further input.

33. A non-transitory computer-readable medium for graphical modeling, the medium comprising:

instructions for displaying a graphical model;

instructions for receiving tactile input from a user and manipulating the displayed graphical model based on an interaction between the user and the displayed graphical model, where the tactile input and the interaction is indicated using frustrated total internal reflection (FTIR); and

instructions for manipulating the displayed graphical model, where the manipulating comprises;

partitioning an area in the displayed graphical model with respect to non-partitioned areas of the displayed graphical model based on a detected user defined outline of a perimeter of the partitioned area in response to the received input,

protecting the partitioned area, where the protecting prevents editing the partitioned area in response to the received input, and

distinguishing the partitioned area from the non-partitioned areas by displaying a graphical indication of the partitioned area on the display, and

where the manipulating excludes;

interfering with editable aspects of the non-partitioned areas of the displayed graphical model in response to the received input,

the interaction being one of;

one part of the user interacting with the display,

one part and another part of the user simultaneously interacting with the display, or

a plurality of parts of the user simultaneously interacting with the display.

34. The non-transitory computer-readable medium of claim 33, wherein the one part of the user interacting with the display comprises:

one finger on one hand of the user interacting with the display.

35. The non-transitory computer-readable medium of claim 33, wherein the one part and another part of the user simultaneously interacting with the display comprises:

one finger on one hand of the user interacting with the display and one finger on another hand of the user of the user simultaneously interacting with the display.

36. The non-transitory computer-readable medium of claim 33, wherein the plurality of parts of the user simultaneously interacting with the display comprises:

a plurality of fingers on one hand of the user simultaneously interacting with the display;

one finger on one hand of the user interacting with the display and a plurality of fingers on another hand of the user simultaneously interacting with the display;

or

a plurality of fingers on one hand of the user interacting with the display and a plurality of fingers on another hand of the user simultaneously interacting with the display.

37. The non-transitory computer-readable medium of claim 33, further comprising:

instructions for receiving a second tactile input from a user for manipulating the displayed graphical model based on a second interaction between the user and the displayed graphical model, where the second tactile input and the second interaction is indicated using FTIR; and

instructions for displaying a hierarchy of a sub-elements for an element in the displayed graphical model due to the received second tactile input.

38. The non-transitory computer-readable medium of claim 33, further comprising:

instructions for receiving a second tactile input from a user for manipulating the displayed graphical model based on a second interaction between the user and the displayed graphical model, where the second tactile input and the second interaction is indicated using FTIR; and

instructions for displaying a hierarchy of a super-elements for an element in the displayed graphical model due to the received second tactile input.

39. The non-transitory computer-readable medium of claim 33, further comprising:

instructions for receiving a second tactile input from a user for manipulating the displayed graphical model based on a second interaction between the user and the displayed graphical model, where the second tactile input and the second interaction is indicated using FTIR; and

instructions for manipulating the graphical model to protect and partition an area in the displayed graphical model due to the received second tactile input.

40. A computer-implemented graphical modeling system comprising:

a display to display a graphical model; and

an interface to receive tactile input from a user for manipulating the displayed graphical model based on an interaction between the user and a surface for registering internally reflected signals,

the interaction being one of;

one part of the user interacting with the display via the surface;

one part and another part of the user simultaneously interacting with the display via the surface;

or

a plurality of parts of the user simultaneously interacting with the display via the surface, and

the tactile input from the user defining a user unique motion to a user specified function, the defining including;

recording the user unique motion, the unique motion having been previously unassigned to the user specified function; and

assigning the user unique motion to the user specified function.

41. The computer-implemented graphical modeling system of claim 40, wherein the specific function is inserting a series of objects in the graphical model.