Apparatus for manipulating an object displayed on a display device by using a touch screen

• US 5,844,547 A
• Filed: 05/09/1995
• Issued: 12/01/1998
• Est. Priority Date: 10/07/1991
• Status: Expired due to Term

• Current Assignee: Apple Inc.
• Original Assignee: Fujitsu Limited
• Inventors: Kamata, Hajime, Fukue, Akiko, Okuyama, Satoshi, Minakuchi, Yu
• Primary Examiner(s): WU, XIAO MIN
• Application Number:
  US08/437,563
• Time in Patent Office: 1,302 Days
• Field of Search: 345/173, 345/174, 345/175, 345/176, 345/179, 345/180, 345/182, 345/123, 345/126, 345/156, 345/157, 345/145, 345/146, 345/358, 178/18, 178/19
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Apparatus and a method of operation manipulates the display of an object on the display surface of a display device in accordance with sensing touching contact on a touch panel, superimposed on the display surface and simulating movement of the displayed object in accordance with the characteristics of the touching contact on the touch panel adjacent to the displayed object image. Further, object data defining the display of the object image for each of a plurality of different states of the object, display information specifying the shape and physical properties of the object and the current display position thereof and file information relating to the stored data are stored and accessed for interpreting the simulated manipulation of the object by the characteristics of the touching contact therewith, for correspondingly manipulating and displaying the thus manipulated object image.

Claims

1. An apparatus affording simulated manipulation of an object in accordance with an image of the object displayed on a display surface of a display device, comprising:

a touch panel, mounted in superimposed relationship on the display surface of the display device and made of a transparent material through which the object image, displayed on the display surface, is transmitted, the touch panel being sensitive to characteristics of a touching contact on the touch panel adjacent the displayed object image, the characteristics including at least coordinate positions of, and time-based changes in, the adjacent touching contact, and outputting corresponding touch panel information representing the characteristics of the adjacent touching contact on the touch panel;

a plurality of data files which respectively store object data defining the display of the object image on the display surface of the display device in each of a plurality of different states of the object;

display information storage means for storing a plurality of different types of object information, including at least;

object information specifying the shape and physical properties of the object, and

display position information specifying the position on the display surface of the display device at which the object image currently is displayed; and

display control means, responsive to the touch panel information output by the touch panel, for recognizing the simulated manipulation of the object image and, further, responsive to the object data and the object information for the displayed object image and the recognized, simulated manipulation of the object, for correspondingly manipulating and displaying the manipulated object image on the display surface of the display device.

2. An apparatus as recited in claim 1, wherein the display control means is responsive to, and recognizes as a pick manipulation, touch panel information representing touching contact on the touch panel at first and second positions respectively adjacent first and second, opposite sides of the object at an initial display position and further to touch panel information representing continuing, coordinated movement of the first and second touching contact positions from the initial and to a further position, displaced by a distance from the initial position, for correspondingly moving the object image display by a corresponding said distance, from the initial position and to a displaced display positions, on the display surface.

3. An apparatus as recited in claim 1, wherein;

the display control means is responsive to, and recognizes as a push manipulation, touch panel information indicating both touching contact on the touch panel at a central position of the displayed object and also continuing movement of the touching contact by a distance from an initial position to a displaced position, simulating movement of the displayed object and the stopping of such movement while maintaining touching contact with the displayed object and, in response to the recognized push manipulation, correspondingly moves the displayed object on the display surface by a corresponding distance to a displaced display position and then fixes the display of the displayed object at the displaced display position.

4. An apparatus as recited in claim 1, wherein the display control means is responsive to, and recognizes as a push-while-rotate manipulation, touch panel information representing both touching contact on the touch panel at a position displaced from the center of the displayed object and also movement of the touching contact by a distance from an initial position to a displaced position and the ending of the movement while maintaining touching contact with the object and, in response to the recognition of the push-while-rotate manipulation, correspondingly both moves and rotates the object image display from the initial display position and through a corresponding distance to a displaced display position and, then, fixes the display of the object image at the displaced display position.

5. An apparatus as recited in claim 1, wherein the display control means is responsive to, and recognizes as a flip manipulation, touch panel information representing movement of the touching contact on the touch panel from an initial position displaced from the object image display at a speed higher than a predetermined speed and into simulated impacting contact with the displayed object image and, in response to the recognized flip manipulation, correspondingly both moves the object image display by a distance, which is proportional to the speed of movement of the touching contact on the touch panel when simulating contact with the object display image at the initial display position, and in a direction, which corresponds to the direction of the touching contact movement at the simulated contact with the object image at the initial display position.

6. An apparatus as recited in claim 1, wherein the display control means is responsive to, and recognizes as a flip-under-gravity manipulation, touch panel information representing movement of the touching contact on the touch panel at a speed greater than a predetermined speed, from an initial position displaced from the initial display position of the object image and into simulated impacting contact with the displayed object image at an initial object image display position and, further, is responsive to object information of the displayed object image specifying the object type as being subject to the effects of gravity, for correspondingly moving the object image display from the initial object image display position by a distance proportional to the speed with which the touching contact simulates contact with the object image at the initial object image display position and along a parabolic path of movement corresponding to the travel of the object when impacted into movement and subjected to the effects of gravity.

7. An apparatus as recited in claim 1, wherein:

the touch panel is sensitive to the level of pressure of the touching contact thereon; and

the display control means is responsive to, and recognizes, as a distort-restore manipulation, touch panel information representing touching contact on the touch panel simulating a pressure-applying contact on the object and the pressure level of the pressure-applying touching contact and the release of the pressure-applying contact and, based on object information specifying the object type as being elastic, for correspondingly altering the configuration of the object image from that as initially displayed, in accordance with, respectively, distorting the displayed object image by an amount proportional to the pressure level of the simulated pressure-applying contact and the elasticity of the object and subsequently restoring the initial configuration of the object image released by the pressure of the touching contact.

8. An apparatus as recited in claim 1, wherein the display information storage means further comprises means for storing file information which specifies a position of the object image portion being displayed on the display device, relative to the whole object image whose data is stored in one of said plurality of data files.

9. An apparatus as recited in claim 8, the display surface having a fixed perimeter and corresponding display dimensions, wherein:

the display control means is responsive to, and recognizes as a scroll manipulation, touch panel information indicating touching contact on the touch panel and associated, simulated movement of the displayed object by the movement of the touching contact while maintaining the adjacent touching contact relationship with the displayed object image, and further is responsive to object information, specifying that the displayed object is an "out-screen" object type and therefore has dimensions exceeding the dimensions of the display surface of the display device, for controlling, and thereby producing, a scrolling display of the object image on the display surface of the display device.

10. An apparatus as recited in claim 9, wherein the display control means is responsive to, and recognizes as the end of the scroll manipulation, touch panel information from the touch panel indicating the discontinuation of the continuing, coordinated movement of the touching contact and correspondingly the discontinuation of the simulated, continuing movement of the displayed object image and, in response thereto fixes the object image display at the current display position thereof on the display surface of the display device.

11. A method for simulating manipulation of an object in accordance with a touching contact adjacent an image of the object displayed on a display surface of a display device employing a touch panel, mounted in superimposed relationship on the display surface of the display device and made of a transparent material through which the object image, displayed on the display surface, is transmitted, the method comprising:

sensing characteristics of a touching contact on the touch panel adjacent the displayed object image, the characteristics including at least coordinate positions of, and time-based changes in, the adjacent touching contact, and producing corresponding touch panel output information representing the characteristics of the adjacent touching contact on the touch panel;

storing object data defining the display of the object image on the display surface of the display device in each of a plurality of different states of the object in a plurality of data files;

storing display information comprising a plurality of different types of object information, including at least;

object information specifying the shape and physical properties of the object, and

display position information specifying the position on the display surface of the display device at which the object image currently is displayed; and

responding to the touch panel information output by the touch panel for recognizing the simulated manipulation of the object and, further, to the object data and the object information for the displayed object image for correspondingly manipulating and displaying the manipulated object image on the display surface of the display device.

12. A method as recited in claim 11, further comprising:

responding to, and recognizing as a pick manipulation, touch panel information representing touching contact on the touch panel at first and second positions respectively adjacent first and second, opposite sides of the object at an initial display position and further to touch panel information representing continuing, coordinated movement of the first and second touching contact positions from the initial and to a further position, displaced by a distance from the initial position, for correspondingly moving the object image display by a corresponding said distance, from the initial position and to a displaced display position, on the display surface.

13. A method as recited in claim 11, further comprising:

responding to, and recognizing as a push manipulation, touch panel information indicating both touching contact on the touch panel at a central position of the displayed object and also continuing movement of the touching contact by a distance from an initial position to a displaced position, simulating movement of the displayed object and the stopping of such movement while maintaining touching contact with the displayed object and, in response to the recognized push manipulation, correspondingly moving the displayed object on the display surface by a corresponding distance to a displaced display position and then fixing the display of the displayed object at the displaced display position.

14. A method as recited in claim 11, further comprising:

responding to, and recognizing as a push-while-rotate manipulation, touch panel information representing both touching contact on the touch panel at a position displaced from the center of the displayed object and also movement of the touching contact by a distance from an initial position to a displaced position and the ending of the movement while maintaining touching contact with the object and, in response to the recognition of the push-while-rotate manipulation, correspondingly moving and rotating the object image display from the initial display position and through a corresponding distance to a displaced display position and then fixing the display of the object image at the displaced display position.

15. A method as recited in claim 11, further comprising:

responding to, and recognizing as a flip manipulation, touch panel information representing movement of the touching contact on the touch panel from an initial position displaced from the object image display at a speed higher than a predetermined speed and into simulated impacting contact with the displayed object image and, in response to the recognized flip manipulation, correspondingly both moving and rotating the object image display by a distance, which is proportional to the speed of movement of the touching contact on the touch panel when simulating contact with the object display image at the initial display position, and in a direction, which corresponds to the direction of the touching contact movement at the simulated contact with the object image at the initial display position.

16. A method as recited in claim 11, further comprising:

responding to, and recognizing as a flip-under-gravity manipulation, touch panel information representing movement of the touching contact on the touch panel at a speed greater than a predetermined speed, from an initial position displaced from the initial display position of the object image and into simulated impacting contact with the displayed object image at an initial object image display position and, further, responding to object information of the displayed object image specifying the object type as being subject to the effects of gravity, and correspondingly moving the object image display from the initial object image display position by a distance proportional to the speed with which the touching contact simulates contact with the object image at the initial object image display position and along a parabolic path of movement corresponding to the travel of the object when impacted into movement and subjected to the effects of gravity.

17. A method as recited in claim 11, further comprising:

responding to, and recognizing, as a distort-restore manipulation, touch panel information representing touching contact on the touch panel simulating a pressure-applying contact on the object and the pressure level of the pressure-applying touching contact and the release of the pressure-applying contact and, based on object information specifying the object type as being elastic, correspondingly altering the configuration of the object image from that as initially displayed, in accordance with, respectively, distorting the displayed object image by an amount proportional to the pressure level of the simulated pressure-applying contact and the elasticity of the object and subsequently restoring the initial configuration of the object image released by the pressure of the touching contact.

18. A method as recited in claim 11 wherein said responding further comprises:

providing, as said data files, a normal state display file which stores object data for displaying the normal state of the object, absent any simulated manipulation thereof, and plural sets of special state files respectively corresponding to the plural types of simulated manipulations of the object, each set storing initial and successive object data for displaying initial and successive special states of the object corresponding to successive conditions of the object resulting from the respective, simulated manipulation;

generating a touch report, in accordance with the touch panel information output by the touch panel in response to the touching contact on the touch panel adjacent a displayed object, the touch report identifying the object type of the displayed object and the recognized, simulated manipulation of the object;

in response to the touch report and the recognized, simulated manipulation provided thereby, accessing the stored display information including the special state file corresponding to both the object type and also the recognized, simulated manipulation of the object, and updating the stored display information with the initial object data of the special state file,

displaying the object in accordance with the updated display information at the display position defined by the stored display position information;

generating a further touch report and determining whether the simulated manipulation has continued or has ended, and;

(a) if the simulated manipulation has ended, accessing the normal display file and updating the stored display information in accordance with the object data of the normal display file and displaying the object at the display position specified by, and in accordance with the thus updated object information, and

(b) if the simulated manipulation has continued, accessing the special state file for obtaining therefrom the next successive object data corresponding to the next successive special state of the object, updating the stored display information with the thus obtained next successive object data, and displaying the object in accordance with the thus updated and stored display information and repeating the present step (b) for each successive, further touch report from which it is determined that the simulated manipulation has continued.

19. A method as recited in claim 18 wherein the simulated manipulation of the object is a pick manipulation.

20. A method as recited in claim 11, further comprising storing file information which specifies a position of the object image portion being displayed on the display device, relative to the whole object image of the stored object data.

21. A method as recited in claim 20, the display surface having a fixed perimeter and corresponding display dimensions, further comprising:

responding to, and recognizing as a scroll manipulation, touch panel information indicating touching contact on the touch panel and associated, simulated movement of the displayed object by the movement of the touching contact while maintaining the adjacent touching contact relationship with the displayed object image, and further is responsive to object information, specifying that the displayed object is an "out-screen" object type and therefore has dimensions exceeding the dimensions of the display surface of the display device, for controlling, and thereby producing, a scrolling display of the object image on the display surface of the display device.

22. A method as recited in claim 21, further comprising:

responding to, and recognizing as the end of the scroll manipulation, touch panel information from the touch panel indicating the discontinuation of the continuing, coordinated movement of the touching contact and correspondingly the discontinuation of the simulated, continuing movement of the displayed object image and, in response thereto, fixing the object image display at the current display position thereof on the display surface of the display device.

23. Apparatus for simulating manipulation of an object in accordance with touching contact adjacent an image of the object displayed on a display surface of a display devices comprising:

a memory storing, for each object image selectable for display on the display surface of the display device, object information specifying a type of the object and the shape and physical properties of the object;

a touch panel, mounted in superimposed relationship on the display surface of the display device and made of a transparent material through which an object image, selected and displayed on the display surface, is transmitted, coordinate positions being defined on the touch panel corresponding to coordinate positions of the display surface of the display device and the touch panel being sensitive and responsive to a touching contact thereon, adjacent a displayed object image, for outputting corresponding touch panel information, the touch panel output information comprising coordinate positions of, and time-based changes in, the adjacent touching contact; and

display control means, responsive to the touch panel output information, for controlling the display of an object image, including changing the display of the object image in response to a change in the coordinate positions of the adjacent touching contact on the touch panel adjacent to the displayed object image and in accordance with the stored object information.

24. An apparatus as recited in claims 23, wherein the display control means further is responsive to touch panel output information indicating time-based changes in the coordinate positions of the touching contact for controlling the displayed object image to simulate corresponding movement thereof.

25. A method for manipulating an image of an object displayed on a display surface of a display device, comprising:

storing, for each object image selectable for display on the display surface of the display device object information specifying a type of the object;

defining a coordinate system on the display surface of the display device;

displaying an image object image on the display surface of the display device, including controlling the coordinate position of the displayed object image;

sensing touching contact in relation to the display surface of the display device adjacent to the displayed object image and outputting corresponding adjacent touching contact information including the effective coordinate positions of the adjacent touching contact and time-based changes therein; and

controlling the display of the object image responsive to the touch panel output information, including changing the display of the object image, in response to a change in the coordinate positions of the touching contact on the touch panel adjacent to the displayed object image and in accordance with the stored object information.

26. A method as recited in claim 25, further comprising responding to touch panel output information indicating time-based changes in the coordinate positions of the touching contact for controlling the displayed object image to simulate corresponding movement thereof.

27. An apparatus affording simulated manipulation of an object in accordance with an image of the object displayed on a display surface of a display device, comprising:

a display device displaying an image of an object;

an input device responsive to a touching contact adjacent a position of the displayed object image as displayed by the display device, the adjacent touching contact simulating a manipulation of the object, and outputting corresponding touch information;

a characteristics detector receiving and detecting, from the received touch information, characteristics of the adjacent touching contact, the characteristics including the position of the adjacent touching contact and time-based changes of the position of the adjacent touching contact, and recognizing therefrom the corresponding object manipulation simulated thereby;

a storage unit storing object information specifying a type of the displayed object image; and

a display controller changing the display of the object image in accordance with the stored object information for the displayed object image and the recognized, simulated object manipulation.

28. An apparatus according to claim 27, wherein:

in response to detected characteristics indicating that two touching contacts are located at opposite sides of the object image and move and stop with a distance therebetween, said display controller controls the display of the object image on the display device so that the object moves on a display surface of the display device from where the two touching contacts are located at opposite sides of the object image to where the two touching contacts stop with the distance therebetween.

29. An apparatus according to claim 27, wherein:

in response to the detected characteristics indicating that the touching contact position is located at the object and moves while keeping in contact with the object and the object is sufficiently large relative to, and extends beyond, the display or only a portion of the object image is displayed on the display device, the display controller displays the object by scrolling the object image on the display device.

30. An apparatus according to claim 29, wherein:

in response to the detected characteristics indicating that a moving, touching contact stops moving, the display controller displays the object image on the display device so that the scrolling of the object image stops on the display.

31. An apparatus as recited in claim 27, further comprising an information storage unit storing information specifying a position of a portion of the object image being displayed on the display device, relative to the whole object image.

32. An apparatus according to claim 27, wherein:

in response to the detected characteristics indicating that the moving, touching contact is located at a geometric center or at a center of gravity of the object and moves and stops while maintaining contact with the object, said display controller displays the object so that the object moves on a display surface thereof from where the touching contact is located at the geometric center or the center of gravity to where the moving, touching contact stops moving.

33. An apparatus according to claim 27, wherein:

in response to the detected characteristics indicating that touching contact is located at a position off the geometric center or off the center of gravity thereof and moves and stops while maintaining contact with the object, said display controller displays the object so that the object moves while rotating on the display, from where the touching contact is located at a position to where the touching contact stops.

34. An apparatus according to claim 27, wherein:

in response to the detected characteristics indicating that touching contact touches the object, coming in a direction from a position apart therefrom and at a speed higher than a predetermined speed, said display controller displays the object so that the object moves a distance in the direction and at a speed proportional to the speed with which the touching contact touches the object.

35. An apparatus according to claim 27, wherein:

said apparatus further comprises a display information storage unit storing object information specifying a type of the object as "rollable"; and

in response to the detected characteristics indicating that the touching contact touches the object image, moving and stopping while maintaining contact with the object, the display controller displays the object image in a rolling condition.

36. An apparatus according to claim 27, wherein:

the input device is sensitive to an amount of pressure applied thereon and, when the detected characteristics indicate the touching contact touches the object with an amount of pressure, the display controller displays the object image so that the object image varies in a degree of distortion and/or restoration according to the amount of pressure of the touching contact.

37. An apparatus according to claim 27, wherein the input device is a panel, mounted with a display device, which is sensitive to the characteristics of touching contact on the panel.

38. An image display and manipulating apparatus, comprising:

a display device displaying an image of the object;

an input device inputting a touching contact adjacent a position of the object image as displayed by the display device;

a characteristics detector receiving and detecting, from the received touch information, characteristics of the adjacent touching contact, the characteristics including the position on the object image of the adjacent touching contact and time based changes of the position of the adjacent touching contact, and recognizing therefrom the corresponding object manipulation simulated thereby;

a display information storage unit storing object information including at least an object type, which specifies physical properties of the object; and

a display controller changing the display of the object image in accordance with the stored object information specifying physical properties of the object and the recognized simulated object manipulation.

39. An apparatus manipulating a displayed image of an object, comprising:

a display device displaying an image of the object;

an input device inputting a touching contact adjacent a position of the object image displayed by the display device;

a characteristics detector receiving and detecting, from the received touch information, characteristics of the adjacent touching contact, the characteristics including a position of the adjacent touching contact on the object image and time based changes of the adjacent touching contact input by said input device;

a display information storage unit storing object information including at least an object type, which specifies physical properties of the object, and position information, which specifies a position at which the object image is displayed on the display device; and

a display controller changing the display of the object image in accordance with the stored object information and the detected characteristics.

40. An apparatus simulating manipulation of an object in accordance with touching contact adjacent art image of the object displayed on a display surface of a display device, the apparatus comprising:

a touch panel mounted in relation to a display surface of a display device and sensitive to characteristics of a touching contact adjacent the object image, the characteristics including at least a position of, and time-based changes in, the adjacent touching contact, for outputting corresponding touch panel information representing a simulated movement of the object responsive to the adjacent touching contact;

a first display information storage means for storing object information including at least;

an object type which specifies physical properties of the object, and

position information which specifies the position where the object image is displayed on the display device; and

a display controller recognizing a type of manipulation of the object image in accordance with the information representing a simulated movement of the object, output by the touch panel, and the object type stored in the display information storage unit and correspondingly manipulating and displaying the object image on the display device.

41. An apparatus as recited in claim 40, further comprising a second display information storage unit storing information specifying a position of a portion of the object being displayed on the display device, relative to the whole object.

42. A display apparatus simulating manipulation of an object, comprising:

a display device producing a display of an image of an object;

a display information storage unit storing object information, including at least an object type which specifies physical properties of the object;

a device responsive to touching contact thereon, adjacent to the displayed object image, simulating an actual touching contact with an actual object corresponding to the displayed object image;

a detecting unit detecting characteristics of the adjacent touching contact including a position of the adjacent touching contact relatively to the displayed object image and time-based changes of the position of the adjacent touching contact and recognizing therefrom the corresponding object manipulation simulated thereby; and

a display controller changing the display of the image of the object in accordance with the stored object information specifying Physical properties of the object and the recognized simulated object manipulation.