Interaction techniques for flexible displays

• US 20070247422A1
• Filed: 03/30/2007
• Published: 10/25/2007
• Est. Priority Date: 03/30/2006
• Status: Abandoned Application

• Current Assignee: xuuk, Inc.
• Original Assignee: xuuk, Inc.
• Inventors: Vertegaal, Roel, Holman, David
• Application Number:
  US11/731,447
• Publication Number:
  US 20070247422A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1652 the display being flexible, e.g. mimicking a sheet of paper, or rollable
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

The invention relates to a set of interaction techniques for obtaining input to a computer system based on methods and apparatus for detecting properties of the shape, location and orientation of flexible display surfaces, as determined through manual or gestural interactions of a user with said display surfaces. Such input may be used to alter graphical content and functionality displayed on said surfaces or some other display or computing system.

Claims

1. A method for capturing location, orientation and shape of one or more flexible display surface(s) comprising the steps of:

a) Determining the location in three dimensions of one or more Points within said flexible display surface(s);

b) Calculating a three dimensional model of the shape, orientation and location of said flexible display surface(s);

c) Clustering locations of Points and fitting curves through said measured locations of Points to determine the three dimensional model; and

d) Optionally determining the relative locations of Points such that the state of the shape or deformation of said flexible display surface(s) can be recognized.

2. The method of claim 1 wherein the flexible display surface is a three dimensional surface made of a material, selected from a group consisting of:

paper, cardboard, paper-like materials, electronic paper, thin substrate displays, thin-film substrate displays, flexible substrate displays, liquid crystal devices, liquid crystal diodes, light emitting devices, light emitting diodes, organic light emitting devices, stacked organic light emitting devices, transparent organic light emitting devices, polymer light emitting devices, organic light emitting diodes, stacked organic light emitting diodes, transparent organic light emitting diodes, polymer light emitting diodes, optical fibres, styrofoam, plastics, epoxy resin, textiles, e-textiles, clothing, skin of a living or dead human or other organism, body of a living or dead human or other organism, carbon-based materials and any three-dimensional object or model.

3. The method of claim 1 wherein a Point is a light reflective marker, embedded or otherwise affixed to said flexible display surface, and where the device for capturing three-dimensional location is an active or passive computer vision system that comprises one or more cameras.

4. The method of claim 1 wherein a Point is an accelerometer embedded or otherwise affixed to said flexible display surface, and where acceleration of said accelerometer is used to calculate the three dimensional position or velocity of said Point.

5. The method of claim 1 wherein Points are inferred from properties of said flexible display surface, as extracted from a background by a computer vision algorithm using properties of said flexible display surface that include shape, color, image or brightness.

6. The method of claim 1 wherein deformation of said flexible display surface is determined by measuring the intensity of light passing through one or more optical fiber mounted along said flexible display surface.

12. A method for projecting Image(s) onto a surface corrected for shape, orientation and location of said surface through a model obtained by methods of claim 1 and 2, onto said surface(s), using a series of projector(s) mounted such that they project upon the flexible display surface(s), and cover the space through which said flexible display surface(s) may move.

13. The method of claim 12 wherein the Image is a three dimensional model consisting of the shape and/or location and/or orientation of said surface, and wherein said three dimensional model is texture-mapped with a second Image selected from a group consisting of:

the contents of a computer window, the contents of a computer file or document, any other static electronic image(s)., and any moving electronic images.

14. The method of claim 12 wherein the surface is a three dimensional surface made of a material selected from a group consisting of:

paper, cardboard, paper-like materials, electronic paper, thin substrate displays, thin-film substrate displays, flexible substrate displays, liquid crystal devices, liquid crystal diodes, light emitting devices, light emitting diodes, organic light emitting devices, stacked organic light emitting devices, transparent organic light emitting devices, polymer light emitting devices, organic light emitting diodes, stacked organic light emitting diodes, transparent organic light emitting diodes, polymer light emitting diodes, optical fibres, styrofoam, plastics, epoxy resin, textiles, e-textiles, clothing, skin of a living or dead human or other organism, body of a living or dead human or other organism, carbon-based materials and any three-dimensional object or model.

7. A method for capturing the location in three dimensions of the finger(s) of one or multiple hands or some tool held by one or multiple hand(s) for the purpose of determining location of said finger(s) or said tool or said hand(s) within a flexible display surface, comprising the steps of:

a) Measuring the location in three dimensions of one or more Point(s) located on said fingers or stylus; and

b) Relating said location of said Points to a coordinate system defined by said flexible display surface so as to obtain a position relative to said coordinate system.

8. The method of claim 7 wherein a Point is a light reflective marker, embedded or otherwise affixed to said flexible display surface, and where the device for capturing three-dimensional location is an active or passive computer vision system that comprises one or more cameras.

9. The method of claim 8 where the marker is selected from a group consisting of:

infrared reflective semisphere or sphere, infrared reflective pattern or object, sphere, semisphere or pattern reflecting specific color(s) in the visible light spectrum, and infrared reflective ink pattern.

10. The method of claim 7 wherein a Point is an accelerometer embedded or otherwise affixed to said flexible display surface, and where acceleration of said accelerometer is used to calculate the three dimensional position or velocity of said Point.

11. The method of claim 7 wherein the location of fingers or tools are sensed through other means known in the art, including but not limited to touch screen, capacitive sensors, electromagnetic field tracking or other forms of computer vision.

15. A method for providing input to a computer system that uses properties of shape, orientation and/or location of one or more flexible surface(s) associated with said computer system, or deformation of said properties, wherein said properties are selected from a group consisting of:

a) Hold, wherein a single flexible surface is activated as a destination of computer commands, or activates associated computing commands, by holding it with one or two hands, and where said surface remains the active surface until another such surface is activated.

b) Collocate, wherein collocating multiple flexible surfaces is used to create a larger flexible surface, which act serves as input to a computer system.

c) Collate, wherein multiple flexible surfaces are organized by stacking them on top of one another, and where such organization is used as input to a computer system.

d) Flip or Turn, wherein rotating a flexible surface around its horizontal or vertical axes such that one of the extremities of the surfaces is lifted up, then folded over, is used as input to a computer system.

e) Fold, wherein folding a flexible surface around any of its axes serves as a means of input to a computer system.

f) Part-fold, wherein partly folding a flexible surface on serves as input to a computer system.

g) Semi-permanent fold, wherein the act or shape resulting from folding a flexible surface around any of its axes in such way that it remains in a folded state after release, serves as input to a computing system.

h) Roll, wherein the act of changing the shape of a flexible surface such that said shape transitions from planar to cylindrical or vice versa serves as input to a computing system.

i) Bend, wherein bending a flexible surface around any of its axes serves as input to a computing system.

j) Rub, wherein providing a rubbing gesture in which the hand or finger or some tool is moved back and forth over a flexible surface is used as input to a computing system.

k) Staple, wherein the act of impacting a first flexible surface with a second flexible surface serves as input to a computing system; and

l) Pointing, wherein the location of such hand(s), tool or finger(s) serve as input to a computing system.

16. The method of claim 15 wherein property j is applied to a first flexible surface collated on top of a second flexible surface, and wherein said property causes the content of said first flexible surface to be copied or otherwise moved onto said second flexible display surface.

17. The method of claim 16 wherein said second flexible surface is a traditional or rigid computer display terminal, and wherein said content is moved from said second flexible surface to said first flexible surface if said first flexible surface does not display an image, or vice versa when said first flexible surface does display an image.

18. The method of claim 16 wherein said second flexible surface is any computing peripheral that has a processing action and corresponding software associated, and wherein content is moved from said first flexible surface to said computing peripheral for processing.

19. The method of claim 18 wherein said computing peripheral is a printer or network peripheral, and wherein said content is moved to said printer or network peripheral for printing, or to a remote location for printing or viewing on a computing system.

20. The method of claim 15 wherein said input to a computing system causes a command to execute on said computing system and wherein said command is selected from a group consisting of:

a) Activate, wherein a file or computer content, image, selection, or window associated with or displaying on said flexible surface is selected for other commands, such as but not limited to editing commands.

b) Zoom in or Enlarge, wherein an image or content of a file associated with said flexible surface is enlarged or zoomed in on.

c) Zoom out or Reduce, wherein an image associated with said flexible surface is reduced or zoomed out of.

d) Organize, wherein some property of file(s), digital information, text, images, or other computer content associated with or displaying on said flexible surface(s) is organized or sorted digitally in a way that matches properties of the physical organization of said flexible surface(s), such as, but not limited to, their physical order.

e) Scroll, wherein an image or content of a file associated with said flexible surface is scrolled, such that a portion of said image, or content of said file is revealed that is currently not rendered, or that is contiguous to what is currently rendered on said flexible surface, or some other display.

f) Page Down, wherein a section of the content of a file that is subsequent to the section of said content that is currently displayed on or associated with said flexible surface, or some other display, is navigated to such that it causes said subsequent section to render on said flexible surface or display.

g) Page Up, wherein a section of the content of a file that precedes the section of said content that is currently displayed on or associated with said flexible surface, or some other display, is navigated to such that it causes said subsequent section to render on said flexible surface or display.

h) Navigate, wherein an arbitrary section of the content of a file associated with the flexible surface, or some online content or hyperlink associated with said surface is navigated to such that it causes said content to render on said flexible surface or some other display.

i) Page Back or Forward, wherein a section of the content of a file, or some online content, webpage or hyperlink that precedes or follows the content currently displayed or associated with the flexible surface, is navigated to such that it causes said content to render on said flexible surface or some other display.

j) Open or Close, wherein some file or digital information associated with said flexible display is opened or closed, read into memory, or out to a permanent medium.

k) Sleep or Wake, wherein said flexible display surface is de-activated, or activated from a state of reduced energy use.

l) Deformation, wherein the shape of a three dimensional model associated with said flexible surface is deformed in such way that it follows the deformation of said flexible surface, in any dimension.

m) Save, wherein the file associated with the flexible display is saved to a hard drive or other permanent medium.

n) Move or Copy, wherein a section of the content of a file, or other digital information, or some selection thereof, currently associated with said flexible surface is transferred to another flexible surface or computing device.

o) Duplicate, wherein the information or file associated with one flexible surface is made identical to that of a second flexible surface so as to clone or duplicate said information on said second flexible surface.

p) Select, where items or images, or both, displayed within a flexible display surface are selected; and

q) Stationary, wherein said flexible surface or some other display shows a set of icons indicating a set of computing applications, or potential functionality of said flexible surface or display and wherein said computing system refrains to context of interaction of said computing system to said application functionality, after selecting said icon of said application.

21. The method of claim 15 wherein the flexible surface is a flexible display or flexible display surface.