Light-based finger gesture user interface

• US 8,416,217 B1
• Filed: 03/20/2012
• Issued: 04/09/2013
• Est. Priority Date: 11/04/2002
• Status: Expired due to Term

• Current Assignee: Neonode, Inc.
• Original Assignee: Neonode, Inc.
• Inventors: Eriksson, Thomas, Pettersson, Robert, Jansson, Anders, Goertz, Magnus, Laveno Mangelsdorff, Jochen, Leine, Per
• Primary Examiner(s): Cerullo, Liliana
• Application Number:
  US13/424,592
• Publication Number:
  US 20130093727A1
• Time in Patent Office: 385 Days
• Field of Search: 345173-184, 178 1801- 1902, 3404071-4072, 2503361-395, 353/42, 341 22- 31, 382/124
• US Class Current: 345/175
• CPC Class Codes: G06F 1/1616 with folding flat displays, e.g. laptop computers or notebooks having a clamshell configuration, with body parts pivoting to an open position around an axis parallel to the plane they define in closed position
• CPC Class Codes: G06F 1/1643 the display being associated to a digitizer, e.g. laptops that can be used as penpads details related to the relative motion of the display enclosure with respect to the body enclosure, e.g. to move between laptop and tablet PC configuration G06F1/1615
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 2203/0339 Touch strips, e.g. orthogonal touch strips to control cursor movement or scrolling; single touch strip to adjust parameter or to implement a row of soft keys
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A light-based finger gesture user interface for an electronic device including a housing for an electronic device, a display mounted in the housing, a cavity, separated from the display, penetrating two opposite sides of the housing, a detector mounted in the housing operative to detect an object inserted in the cavity, and a processor connected to the detector and to the display for causing the display to render a visual representation in response to output from the detector.

Claims

1. A light-based finger gesture user interface for an electronic device comprising:

a housing for an electronic device comprising inner sidewalls extending from a bottom opening to a top opening to form a cavity;

a touch screen display mounted in said housing such that the whole cavity is disposed along a side of the display, separate from the display;

a detector mounted in said inner sidewalls, comprising a plurality of light emitters that project light beams into the cavity and a plurality of light receivers that measure intensities of blocked and unblocked ones of the projected light beams; and

a processor connected to said detector and to said display for generating touch position data in response to a first object touching said display based on output received from said display, for generating cavity insertion data in response to a second object, different than the first object, being inside the cavity based on output received from said detector, and for combining the touch position data and the cavity insertion data into input for the electronic device, wherein the cavity insertion data includes location and orientation of the second object inside the cavity.

2. The user interface of claim 1, wherein said housing comprises first and second sections connected by a hinge that enables folding the two sections onto each other, and wherein said display is mounted in said first section and said cavity is located in said second section within a frame that fits around the edges of said first section when the sections are folded.

3. The user interface of claim 1, wherein said cavity is located within a handle for carrying the device.

4. The user interface of claim 1 wherein said cavity is formed by extending a retractable frame that surrounds the electronic device on three sides thereof.

5. The user interface of claim 1, wherein said processor determines at least five location and orientation attributes in response to the second object being inserted inside the cavity;

namely,

(i) location along an x-axis;

(ii) location along a y-axis;

(iii) location along a z-axis;

(iv) polar angle in the xz-plane; and

(v) polar angle in the yz-plane,

wherein the locations and polar angles are with reference to x and y axes along the length and width of the top opening of the cavity and to a z axis along the height of the cavity from top to bottom.

6. The user interface of claim 5, wherein the second object comprises two fingers, and wherein said processor further distinguishes between four multi-finger gestures performed in said cavity;

namely,

(i) clockwise rotation in the xy-plane;

(ii) counter-clockwise rotation in the xy-plane;

(iii) finger pinch in the xy-plane; and

(iv) finger spread in the xy-plane.

7. The user interface of claim 5, wherein said processor renders on said display a graphic being rotated in response to changes in polar angle of the second object inside the cavity.

8. A light-based finger gesture user interface for an electronic device comprising:

a housing for an electronic device comprising inner sidewalls extending from a bottom opening to a top opening to form a cavity;

a touch screen display mounted in said housing for rendering output generated by an application program running on the electronic device, wherein the whole cavity is disposed along a side of the display, separate from the display;

a sensor mounted in said inner sidewalls, comprising a plurality of light emitters that project light beams into the cavity and a plurality of light receivers that measure intensities of blocked and unblocked ones of the projected light beams; and

a processor in said housing connected to said display and to said sensor, for executing the application program, for generating touch position data in response to a first object touching said display based on output received from said display, for generating cavity insertion data in response to a second object, different than the first object, being inside the cavity based on output received from said sensor, and for combining the touch position data and the cavity insertion data into input for the application program wherein the cavity insertion data includes location and orientation of the second object inside the cavity.

9. The user interface of claim 8, wherein the second object comprises two fingers, and wherein said processor distinguishes via the cavity insertion data between four multi-finger gestures performed in the cavity;

namely,

(i) clockwise rotation in the xy-plane;

(ii) counter-clockwise rotation in the xy-plane;

(iii) finger pinch in the xy-plane; and

(iv) finger spread in the xy-plane.

10. The user interface of claim 8, wherein the second object comprises a finger, and wherein:

said processor determines at least five location and orientation attributes in response to the finger being inside the cavity;

namely,

(i) location along an x-axis;

(ii) location along a y-axis;

(iii) location along a z-axis;

(iv) polar angle in the xz-plane; and

(v) polar angle in the yz-plane,

wherein the locations and polar angles are with reference to x and y axes along the length and width of the top opening of the cavity and to a z axis along the height of the cavity from top to bottom;

said processor causes said display to pan a displayed image in response to a translation gesture of the finger in the cavity; and

said processor causes said display to rotate a displayed image in response to a change in one or both of the polar angles of the finger in the cavity.

11. The user interface of claim 8, wherein the second object comprises one or more fingers, and wherein said processor causes said display to render animated navigation of snapshots of widgets, documents, web page bookmarks, album artwork or photographs in response to detecting via the cavity insertion data a one finger or a multi-finger gesture in the cavity, wherein a speed of the animated navigation is determined by the number of fingers performing the gesture.

12. The user interface of claim 8, wherein the second object comprises two fingers, and wherein said processor causes said display to render opening of a widget, document, web page, album, or photograph in response to detecting via the cavity insertion data a multi-finger gesture.

13. The user interface of claim 11, wherein said processor causes said display to render:

fast navigation of the snapshots in response to detecting the one-finger gesture; and

slow navigation of the snapshots in response to detecting the multi-finger gesture.

14. The user interface of claim 9, wherein said processor causes said display to render:

opening of a widget, document, web page, album or photograph in response to detecting the finger spread in the xy-plane; and

closing of an open widget, document, web page, album or photograph in response to detecting the finger pinch in the xy-plane.

15. The user interface of claim 8, wherein the object comprises a finger, wherein the application program defines a context, and wherein said processor activates a function based on the context in response to detecting via the cavity insertion data a gesture of the finger performed inside the cavity.

16. The user interface of claim 15, wherein the gesture comprises tilting the finger inside the cavity at an acute angle once, and wherein the function causes said display to render a pop-up menu of context-relevant options.

17. The user interface of claim 8 wherein the second object comprises a finger, and wherein said processor causes said display to render one or more graphics, identifies via the touch position data one of the rendered graphics as being a selected graphic, and instructs the application program to process the selected graphic in response to detecting via the cavity insertion data a gesture of the finger inside the cavity.

18. The user interface of claim 17, wherein the selected graphic is selected text within a document, and wherein the application program causes said display to maintain the selected text at its location within the document and to move unselected text surrounding the selected text within the document, thereby positioning the selected text at a different location within the document.

19. The user interface of claim 18, wherein the application program causes said display to fix the selected text at the different location in response to detecting via the cavity insertion data a second finger gesture in the cavity.

20. The user interface of claim 17, wherein the selected graphic is a selected icon, and wherein the application program causes said display to maintain the selected icon at its location on said display and to move unselected icons surrounding the selected icon on said display, thereby positioning the selected icon at a different location.

21. The user interface of claim 20, wherein the application program causes said display to fix the selected icon at the different location in response to detecting via the cavity insertion data a second finger gesture in the cavity.

22. The user interface of claim 17, wherein the application program is a communication program, wherein the selected graphic is a selected list entry in a contact list, and wherein the application program

initiates a phone call to the selected entry in response to detecting via the cavity insertion data a first gesture, and

opens an SMS dialog addressed to the selected entry in response to detecting via the cavity insertion data a second gesture.

23. The user interface of claim 17, wherein the application program is a calendar program that presents a time interval divided into units of a given resolution, wherein the selected graphic is a selected time interval, and wherein the application program causes said display to increase or decrease the resolution to present a respectively greater or lesser time interval centered at the selected time interval.

24. The user interface of claim 17, wherein the application program is a file manager that presents files, wherein the selected graphic is a selected file, wherein the gesture is extendable, and wherein the file manager

presents properties of the selected file in response to detecting via the cavity insertion data an initial gesture, and

opens the selected file in response to detecting via the cavity insertion data an extension of the initial gesture.

25. The user interface of claim 17, wherein the application program is a file manager that presents files within a file system that has multiple levels of folders, wherein the selected graphic is a selected file, wherein the gesture is extendable, and wherein the file manager

presents properties of the selected file in response to detecting via the cavity insertion data an initial gesture,

displays the selected file as an icon in its folder in response to detecting via the cavity insertion data an extension of the initial gesture, and

displays its folder as an icon in its next level folder in response to detecting via the cavity insertion data a further extension of the initial gesture.

26. The user interface of claim 17, wherein the application program is a social network application, wherein the selected graphic is a person in the social network, wherein the gesture is extendable, and wherein the application program

presents a first plurality of persons connected to the selected person in response to detecting via the cavity insertion data an initial gesture, and

presents a second plurality of persons connected to the first plurality of persons in response to detecting via the cavity insertion data an extension of the initial gesture.

27. The user interface of claim 17, wherein the application program is a music library manager, wherein the selected graphic is a song or artist in the music library, wherein the gesture is extendable, and wherein the music library manager

presents a first plurality of songs related to the selected song or artist in response to detecting via the cavity insertion data an initial gesture, and

presents a second plurality of songs connected to the first plurality of songs in response to detecting via the cavity insertion data an extension of the initial gesture.

28. The user interface of claim 17, wherein the application program is a reservation program for restaurants or hotels, wherein the selected graphic is a selected restaurant or hotel, wherein the gesture is extendable, and wherein the reservation program

presents a plurality of restaurants or hotels similar to the selected restaurant or hotel in terms of style, cost, location or other parameter, in response to detecting via the cavity insertion data an initial gesture, and

presents additional similar restaurants or hotels in response to detecting via the cavity insertion data an extension of the initial gesture.

29. The user interface of claim 8 wherein the top and bottom openings of the cavity are uncovered, wherein the object comprises a finger, and wherein said processor causes said display to render an image, identifies via the touch position data a location within the image as being a selected location, and instructs the application program to process the selected location in response to detecting via the cavity insertion data a gesture of the finger in the cavity.

30. The user interface of claim 8, wherein said processor receives the output from said display and the output from said sensor substantially simultaneously.