User interface system

• US 7,810,050 B2
• Filed: 03/28/2006
• Issued: 10/05/2010
• Est. Priority Date: 03/28/2005
• Status: Expired due to Fees

• Current Assignee: Godo Kaisha IP Bridge 1 (IP Bridge, Inc.)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Yamashita, Atsushi, Iisaka, Atsushi, Hirai, Takuya, Terada, Tomohiro
• Primary Examiner(s): Bashore; William L
• Assistant Examiner(s): LONG, ANDREA NATAE
• Application Number:
  US11/884,955
• Publication Number:
  US 20080163131A1
• Time in Patent Office: 1,652 Days
• Field of Search: 715/863, 715/866, 345/173
• US Class Current: 715/863
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A user interface system having an improved usability is provided. The user interface system comprises: an operation section (21) an operation section for receiving an instruction inputted by a user; a finger position detecting section (32) for detecting a group of first coordinate values, each indicating a current position of a fingertip, of a user, which is placed on an operation surface of the operation section; a component storing section (4) for storing component data representing a GUI (Graphical User Interface) component; an assignment section (34) for uniquely assigning the component data stored in the component storing section to each of the first coordinate values detected by the finger position detecting section (32); an image generating section (35) for generating a GUI image representing a user interface in which the GUI component represented by the component data stored in the component storing section (4) is uniquely associated with each of the first coordinate values detected by the finger position detecting section (32), in accordance with an assignment performed by the assignment section; and a display controlling section (36) for displaying the GUI image generated by the image generating section (35) on a display screen.

Claims

1. A user interface system comprising:

an operation section for receiving an instruction inputted by a user;

an image pickup device set so as to be able to image an operation performed with a hand of the user;

a finger position detecting section for detecting, in an image captured by the image pickup device, a group of first coordinate values respectively indicating current positions of fingertips of the user, which are placed on an operation surface of the operation section;

a finger identifying section for identifying, in the image captured by the image pickup device, a plurality of finger types to which the first coordinate values, detected by the finger position detecting section, respectively correspond;

a component storing section for storing component data representing GUI (Graphical User Interface) components;

an assignment section for uniquely assigning the component data stored in the component storing section to the first coordinate values corresponding to the plurality of finger types identified by the finger identifying section;

an image generating section for generating a GUI image representing a user interface in which the GUI components represented by the component data stored in the component storing section are uniquely associated with the first coordinate values corresponding to the plurality of finger types identified by the finger identifying section, in accordance with an assignment performed by the assignment section; and

a display controlling section for displaying the GUI image generated by the image generating section on a display screen.

2. The user interface system according to claim 1, further comprising a function executing section for erasing one of the GUI components associated with a finger bent by the user among fingers included in a hand image captured by the image pickup device, and

for executing a function of one of the GUI components which is assigned to one of the coordinate values indicating one of the current positions of the fingertips corresponding to a last remaining finger.

3. The user interface system according to claim 1, wherein

the image generating section generates the GUI image in which the positions indicated by the first coordinate values corresponding to the plurality of finger types identified by the finger identifying section are respectively connected, by lines, to the GUI components represented by the component data assigned by the assignment section.

4. The user interface system according to claim 1, wherein

the image generating section respectively superimposes the GUI components represented by the component data assigned by the assignment section on a display image at the positions indicated by the first coordinate values corresponding to the plurality of finger types identified by the finger identifying section or in a vicinity thereof.

5. The user interface system according to claim 1, wherein

the finger identifying section further identifies whether the fingertips corresponding to the identified plurality of finger types belong to a right hand or a left hand, and

the assignment section respectively assigns the component data, which are different from each other and which are stored in the component storing section, to the first coordinate values corresponding to the plurality of finger types identified by the finger identifying section in accordance with a hand type identified by the finger identifying section.

6. The user interface system according to claim 1, wherein

the image generating section generates the GUI image in which a size of each of the GUI components stored in the component storing section is changed in accordance with the first coordinate values detected by the finger position detecting section.

7. The user interface system according to claim 1, further comprising:

a contact position detecting section for detecting a second coordinate value indicating a position, on the operation surface of the operation section, which is in contact with the user;

a contact finger determining section for specifying one of the GUI components uniquely assigned to one of the first coordinate values which substantially coincides with the second coordinate value; and

a function executing section for executing a function assigned to one of the GUI components which is specified by the contact finger determining section.

8. The user interface system according to claim 1, wherein

in response to an operation performed by the user, the assignment section reassigns the component data, once uniquely assigned to the first coordinate values corresponding to the plurality of finger types identified by the finger identifying section, to other component data.

9. The user interface system according to claim 8, wherein

the operation surface of the operation section is divided into a plurality of partial areas, and a plurality of types of appliances to be controlled by the user interface system are uniquely assigned to the plurality of partial areas, and

the assignment section respectively assigns, to the first coordinate values, the component data representing one of the plurality of types of appliances uniquely assigned to one of the plurality of partial areas which corresponds to a position of the hand of the user.

10. The user interface system according to claim 1, wherein

the assignment section assigns no component data to one of the first coordinate values indicating one of the positions of the fingertips corresponding to a specific finger type, and uniquely assigns the component data, which are different from each other between a plurality of types of appliances to be controlled by the user interface system, to the other first coordinate values indicating the positions of the fingertips corresponding to remaining finger types, and

the image generating section displays, for one of the plurality of types of appliances which is currently being selected in response to an operation performed with one of the fingertips corresponding to the specific finger type, the GUI image in which the other first coordinate values, indicating the positions of the fingertips corresponding to the remaining finger types which are identified by the finger identifying section, are uniquely associated with the component data which are stored in the component storing section and which are assigned to the fingertips corresponding to the remaining finger types.

11. The user interface system according to claim 7, wherein

the assignment section assigns a plurality of pieces of the component data, which are previously determined, to a predetermined finger,

the image generating section respectively disposes the GUI components represented by the plurality of pieces of the component data at positions which are in the vicinity of the first coordinate values indicating the current positions of the fingertips of the user and which do not overlap with each other,

when the user contacts at least two points on the operation surface of the operation section, the contact position detecting section detects the second coordinate value indicating an initial contact position and further detects the second coordinate value indicating a second contact position, in response to the respective contacts performed by the user,

the contact finger determining section specifies the finger type the user uses to contact the operation surface, based on the second coordinate value which indicates the initial contact position and which is detected by the contact position detecting section and the first coordinate values corresponding to the plurality of finger types identified by the finger identifying section, and further specifies one of the GUI components currently designated by the user based on the second coordinate value which indicates the second contact position and which is detected by the contact position detecting section and the GUI components represented by the plurality of pieces of the component data assigned to the specified finger type.

12. A display method used in a user interface system, the display method comprising:

an operation step of receiving, with an operation section of the user interface system, an instruction inputted by a user;

an image pickup step of imaging an operation performed with a hand of the user;

a finger position detecting step of detecting, in an image captured in the image pickup step, a group of first coordinate values respectively indicating current positions of fingertips of the user which are placed on an operation surface of the operation section;

a finger identifying step of identifying, in the image captured in the image pickup step, a plurality of finger types to which the first coordinate values, detected in the finger position detecting, step, respectively correspond;

an assignment step of uniquely assigning component data, which are stored in a memory device included in the user interface system and which represent GUI (Graphical User Interface) components, to the first coordinate values corresponding to the plurality of finger types identified by the finger identifying step;

an image generating step of generating a GUI image representing a user interface in which the GUI components stored in the memory device are uniquely associated with the first coordinate values corresponding to the plurality of finger types identified by the finger identifying step, in accordance with an assignment performed by the assignment step; and

a display controlling step of displaying the GUI image generated by the image generating step on a display screen included in the user interface system.