Three-dimensional sliding object arrangement method and system

• US 9,513,765 B2
• Filed: 10/24/2008
• Issued: 12/06/2016
• Est. Priority Date: 12/07/2007
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Miyazaki, Reiko, Miyashita, Ken
• Primary Examiner(s): Levy, Amy M
• Application Number:
  US12/257,663
• Publication Number:
  US 20090150775A1
• Time in Patent Office: 2,965 Days
• Field of Search: 715/723, 715/712, 715/716, 715/719, 715/782, 715/848, 715/852, 345/173
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

An information display terminal includes a first operation information acquiring unit that acquires first operation information including first location information on a display screen via a touch panel including a contact detection surface and the display screen. A three-dimensional coordinate space is projected onto the display screen. An object is arranged within the three-dimensional coordinate space on the basis of three-dimensional coordinate information of the object. The information display terminal also includes a second operation information acquiring unit that acquires second operation information including second location information on the display screen via a contact detection surface provided on a surface on a side opposite to the display screen. In addition, the information display terminal includes a display controlling unit that controls a display of the object in the three-dimensional coordinate space on the display screen on the basis of the acquired first operation information and second operation information.

Claims

1. An information display method for displaying an object of a graphical user interface on a display screen, comprising:

projecting a three-dimensional coordinate space onto the display screen, the object arranged within the three-dimensional coordinate space as part of a group of related objects on the basis of three-dimensional coordinate information of the object;

acquiring first operation information from a touch panel having a contact detection surface overlaid on the display screen, the first operation information including first location information on the display screen corresponding to a contact location on the touch panel and including first contact pressure information, a size of a region of the contact detection surface of the touch panel through which input is received being adjusted based on a user stroke by detecting a start point and an end point of the user stroke and setting the region of the contact detection surface as a quadrilateral having a diagonal that connects the start point and the end point of the user stroke;

acquiring second operation information from a touch pad that includes a contact detection surface and that is provided on a surface opposite the display screen, the second operation information including second location information on the display screen and second contact pressure information;

controlling display of a first pointer corresponding to the first location information and a second pointer corresponding the second location information in the three-dimensional coordinate space on the display screen; and

controlling a display of the object in the three-dimensional coordinate space on the display screen on the basis of the acquired first operation information and second operation information,

wherein the group of related objects form an object column in which the objects are aligned in a direction of a first coordinate axis of the three-dimensional coordinate space to overlap,

the object column is specified on the basis of the first location information included in the first operation information or second location information included in the second operation information,

upon indication from the first operation information of a sliding operation on the contact detection surface overlaid on the display screen, the objects in the specified object column are caused to transition from alignment with the first coordinate axis of the three-dimensional coordinate space to a first state in which the objects in the specified column are aligned with a second coordinate axis of the three-dimensional coordinate space while the objects maintain a same relative position with respect to the first coordinate axis, and to transition from the first state to a second state in which the objects of the specified column maintain a same relative alignment with respect to the second coordinate axis while transitioning to a same position with respect to the first coordinate axis, the second coordinate axis being orthogonal to the first coordinate axis, and the objects in the specified object column are displayed without overlap after transitioning to the first state and in the second state, the sliding operation being performed after receipt of the second operation information, and

a layout display region is displayed in which a layout of a plurality of object columns is displayed using a plurality of icons, which correspond to objects in each of the plurality of object columns.

2. A computer-readable storage medium encoded with a program that, when executed, causes a computer to execute an information display method for displaying an object of a graphical user interface on a display screen, wherein the information display method comprises:

projecting a three-dimensional coordinate space onto the display screen, the object arranged within the three-dimensional coordinate space as part of a group of related objects on the basis of three-dimensional coordinate information of the object;

acquiring first operation information from a touch panel having a contact detection surface overlaid on the display screen, the first operation information including first location information on the display screen corresponding to a contact location on the touch panel and including first contact pressure information, a size of a region of the contact detection surface of the touch panel through which input is received being adjusted based on a user stroke by detecting a start point and an end point of the user stroke and setting the region of the contact detection surface as a quadrilateral having a diagonal that connects the start point and the end point of the user stroke;

acquiring second operation information from a touch pad that includes a contact detection surface and that is provided on a surface opposite the display screen, the second operation information including second location information on the display screen and second contact pressure information;

controlling display of a first pointer corresponding to the first location information and a second pointer corresponding the second location information in the three-dimensional coordinate space on the display screen; and

controlling a display of the object in the three-dimensional coordinate space on the display screen on the basis of the acquired first operation information and second operation information,

wherein the group of related objects form an object column in which the objects are aligned in a direction of a first coordinate axis of the three-dimensional coordinate space to overlap,

the object column is specified on the basis of the first location information included in the first operation information or second location information included in the second operation information,

upon indication from the first operation information of a sliding operation on the contact detection surface overlaid on the display screen, the objects in the specified object column are caused to transition from alignment with the first coordinate axis of the three-dimensional coordinate space to a first state in which the objects in the specified column are aligned with a second coordinate axis of the three-dimensional coordinate space while the objects maintain a same relative position with respect to the first coordinate axis, and to transition from the first state to a second state in which the objects of the specified column maintain a same relative alignment with respect to the second coordinate axis while transitioning to a same position with respect to the first coordinate axis, the second coordinate axis being orthogonal to the first coordinate axis, and the objects in the specified object column are displayed without overlap after transitioning to the first state and in the second state, the sliding operation being performed after receipt of the second operation information, and

a layout display region is displayed in which a layout of a plurality of object columns is displayed using a plurality of icons, which correspond to objects in each of the plurality of object columns.

3. An information display terminal for displaying an object of a graphical user interface on a display screen, comprising:

circuitry configured to

acquire first operation information from a touch panel having a contact detection surface overlaid on the display screen, the first operation information including first location information corresponding to a contact location on the touch panel and including first contact pressure information, wherein a three-dimensional coordinate space is projected onto the display screen, and the object is arranged within the three-dimensional coordinate space as part of a group of related objects on the basis of three-dimensional coordinate information of the object, the circuitry adjusting a size of a region of the contact detection surface of the touch panel through which input is received based on a user stroke by detecting a start point and an end point of the user stroke and setting the region of the contact detection surface as a quadrilateral having a diagonal that connects the start point and the end point of the user stroke,

acquire second operation information from a touch pad that includes a contact detection surface and that is provided on a surface of the display terminal that is opposite the display screen, the second operation information including second location information on the display screen and second contact pressure information,

control display of a first pointer corresponding to the first location information and a second pointer corresponding the second location information in the three-dimensional coordinate space on the display screen, and

control a display of the object in the three-dimensional coordinate space on the display screen on the basis of the acquired first operation information and second operation information,

wherein

the group of related objects form an object column in which the objects are aligned in a direction of a first coordinate axis of the three-dimensional coordinate space to overlap,

the circuitry specifies the object column on the basis of the first location information included in the first operation information or second location information included in the second operation information,

upon indication from the first operation information of a sliding operation on the contact detection surface overlaid on the display screen, the circuitry causes objects in the specified object column to transition from alignment with the first coordinate axis of the three-dimensional coordinate space to a first state in which the objects in the specified column are aligned with a second coordinate axis of the three-dimensional coordinate space while the objects maintain a same relative position with respect to the first coordinate axis, and to transition from the first state to a second state in which the objects of the specified column maintain a same relative alignment with respect to the second coordinate axis while transitioning to a same position with respect to the first coordinate axis, the second coordinate axis being orthogonal to the first coordinate axis, and the objects in the specified object column are displayed without overlap after transitioning to the first state and in the second state, the sliding operation being performed after receipt of the second operation information, and

the circuitry further controls display of a layout display region in which a layout of a plurality of object columns is displayed using a plurality of icons, which correspond to objects in each of the plurality of object columns.

4. The information display terminal according to claim 3, wherein the other of the first operation information and the second operation information includes second scroll information obtained in accordance with the corresponding first or second contact pressure information, and the circuitry controls the display so that the group of related objects included in the specified object column scroll in a direction on the first coordinate axis on the basis of the acquired second scroll information.

5. The information display terminal according to claim 3, wherein the circuitry controls the display so that the group of related objects scroll when the degree of the first or second contact exceeds a predetermined threshold value.

6. The information display terminal according to claim 3, wherein the first and second operation information respectively include first and second contact duration information, and

the circuitry controls an acceleration of a display on which the group of related objects scroll on the basis of at least one of the first and second contact duration information.

7. The information display terminal according to claim 3, wherein the circuitry controls an acceleration of display on which the group of related objects scroll on the basis of a detection area of the contact on at least one of the touch panel and the touch pad.

8. The information display terminal according to claim 3, wherein the first operation information includes first start point information and first end point information as the first location information obtained from a moving state of a contact point detected on the contact detection surface provided on the display screen, the second operation information includes second start point information acquired at the same time as the first start point information and second end point information acquired at the same time as the first end point information as the second location information obtained from a moving state of a contact point detected on the contact detection surface provided on the surface on the side opposite to the display screen, and

the circuitry specifies a frontmost object included in a specific object column on the display screen on the basis of the acquired first start point information, specifies a rearmost object included in the object column on the basis of the acquired second start point information, links the acquired first end point information to a location to which the frontmost object moves, and links the acquired second end point information to a location to which the rearmost object moves, and thus controls the display so that the plurality of objects included in the object column move in a plane perpendicular to the first coordinate axis.

9. The information display terminal according to claim 3, wherein the first operation information includes first start point information and first end point information as the first location information obtained from a moving state of a contact point detected on the contact detection surface provided on the display screen, the second operation information includes second start point information acquired at the same time as the first start point information and second end point information acquired at the same time as the first end point information as the second location information obtained from a moving state of a contact point detected on the contact detection surface provided on the surface on the side opposite to the display screen, and

the circuitry calculates an amount of change in an angle between a reference axis obtained from the acquired first start point information and the second start point information and a movement axis obtained from the acquired first end point information and the second end point information, and changes an angle of an axis connecting a viewpoint indicating a frontmost apex of a pyramid which is a visual hull onto which the three-dimensional coordinate space is projected, and a reference point indicating a center of a projection surface, and thus, controls the display of the object on the display screen which is the projection surface in such a state that the angle of the axis is changed relative to the three-dimensional coordinate space.

10. The information display terminal according to claim 3, wherein the object is provided in a two-dimensional coordinate plane on the basis of two-dimensional coordinate information specific to the object,

the first operation information includes first start point information and first end point information as the first location information obtained from a moving state of a contact point detected on the contact detection surface provided on the display screen,

the second operation information includes second start point information acquired at the same time as the first start point information and second end point information acquired at the same time as the first end point information as the second location information obtained from a moving state of a contact point detected on the contact detection surface provided on the surface on the side opposite to the display screen, and

the circuitry specifies a frontmost object included in a specific object column on the display screen on the basis of the acquired first start point information, specifies a rearmost object included in the object column on the basis of the acquired second start point information, links the acquired first end point information to a location to which the frontmost object moves, and links the acquired second end point information to a location to which the rearmost object moves, and thus controls the display so that the group of related objects included in the object column move to change a degree of overlap between the objects.

11. The information display terminal according to claim 3, wherein a second operation region which is an object of an operation on the contact detection surface provided on the surface on the side opposite to the display screen is smaller than a first operation region which is an object of an operation on the contact detection surface provided on the display screen, and the circuitry makes such an adjustment that the second location information on the display screen obtained from the second operation region corresponds to the first location information on the display screen obtained in the first operation region.

12. The information display terminal according to claim 11, wherein the circuitry is further configured to set operation region information for specifying a range of the second operation region on the basis of the second operation information acquired for each user.

13. The information display terminal according to claim 12, wherein the circuitry is further configured to store the operation region information set, and to read out the operation region information for a specific user in response to a request for a change to the specific user and sets the range of the second operation region.

14. The information display terminal according to claim 12, wherein the circuitry is further configured to store a number of operation region information instances in accordance with a direction in which the display screen is used, and to determine the direction in which the display screen is used, read out an operation region information instance in accordance with the direction in which the display screen is used, and set the range of the second operation region.

15. The information display terminal according to claim 14, wherein the circuitry determines the direction in which the display screen is used on the basis of the second operation information acquired from a user.

16. The information display terminal according to claim 14, wherein the circuitry is further configured to determine the direction in which the display screen is used on the basis of results of detection by an acceleration sensor.

17. The information display terminal according to claim 3, wherein the group of related objects form an object column, and the circuitry controls the display for operating the object column and the display for operating the objects which form the object column, in accordance with an acquisition state of the first location information included in the first operation information and/or the second location information included in the second operation information and a display state of the object column.

18. The information display terminal according to claim 17, wherein the circuitry controls the display of the object column so that an amount of an overlap between the objects changes and the group of related objects move in response to a continuous change in the first location information continuously acquired within a predetermined time after an acquisition of the second location information when the second location information is acquired in such a state that the group of related which form the object column are displayed in an initial location or in the slide location after the object column is specified by the first location information.

19. The information display terminal according to claim 18, wherein an object placed on a front surface side on the display screen from among the group of related objects which are placed in the object column to at least partially overlap is displayed larger than an object placed on a rear surface side, the circuitry controls the display of the object column so that the group of related objects move while changing the amount of the overlap between the objects in response to the continuous change in the first location information which is continuously acquired, and the smaller the amount of the overlap between the objects is, the further the objects move to the front surface side and the larger the objects are displayed, while the greater the amount of the overlap between the objects is, the further the objects move to the rear surface side and the smaller the objects are displayed.

20. The information display terminal according to claim 18, wherein an object placed on a front surface side on the display screen from among the group of related which are placed in the object column to at least partially overlap is displayed larger than an object placed on a rear surface side, and

the circuitry controls the display of the object column so that the plurality of objects move to a frontmost surface side to be displayed in a list with the same size when the group of related objects forming the object column are arranged so as not to overlap for a display.

21. The information display terminal according to claim 20, wherein the circuitry controls the display of the object column so that the group of related objects are displayed larger as the objects move to the front surface side.

22. The information display terminal according to claim 20, wherein the circuitry controls the display of the objects so that contents of the objects are displayed when the objects are specified by the first location information in such a state that the group of related objects are displayed in a list.

23. The information display terminal according to claim 17, wherein the circuitry controls the display of the object column so that the group of related objects forming the object column move to the initial location and are aligned to overlap when the first and second location information are not acquired within a predetermined period in such a state that the group of related objects forming the object column are displayed in the slide location after the object column is specified by the first location information.

24. The information display terminal according to claim 17, wherein the circuitry controls the display of the object column so that the object column moves in response to a continuous change in the first location information which is continuously acquired in a state where the group of related objects forming the object column are displayed in the initial location after the object column is specified by the first location information.

25. The information display terminal according to claim 17, wherein the circuitry controls the display of the objects so that an object of focus that is one of the group of related objects forming the object column shifts to another object placed on any one of a front surface side or a rear surface side of an object on the display screen which is the object of focus when the first location information is acquired, and shifts to another object placed on the other of the front surface side or the rear surface side of the object on the display screen which is the object of focus when the second location information is acquired when the first location information or the second location information which does not specify the object column is acquired in such a state that the group of related objects forming the object column are displayed in the initial location after the object column is specified by the first location information.

26. The information display terminal according to claim 3, wherein in the layout display region, icons corresponding to the object column specified are displayed with a different visual characteristic from other icons in the layout display region in order to identify the object column specified as being displayed.

27. The information display terminal according to claim 26, wherein the icons corresponding to the object column specified are black, and the other icons are white.