User Interfacing

• US 20080018591A1
• Filed: 07/20/2006
• Published: 01/24/2008
• Est. Priority Date: 07/20/2006
• Status: Abandoned Application

• Current Assignee: Candledragon, Inc.
• Original Assignee: Candledragon, Inc.
• Inventors: Goldman, Andrew M., Pittel, Ilya, Liberman, Sergey, Pittel, Arkady, Elektrov, Stanislav V.
• Application Number:
  US11/490,736
• Publication Number:
  US 20080018591A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1616 with folding flat displays, e.g. laptop computers or notebooks having a clamshell configuration, with body parts pivoting to an open position around an axis parallel to the plane they define in closed position
• CPC Class Codes: G06F 1/1639 the display being based on projection
• CPC Class Codes: G06F 1/1649 the additional display being independently orientable, e.g. for presenting information to a second user
• CPC Class Codes: G06F 1/1673 Arrangements for projecting a virtual keyboard
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: H04M 1/021 using combined folding and rotation motions combined with translation movement H04M1/0235
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04M 1/0272 for a projector or beamer module assembly constructional features of projectors per se G03B21/00; constructional details of projectors using electronic spatial light modulators per se H04N9/31
• CPC Class Codes: H04M 1/72445 for supporting Internet browser applications
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability

Abstract

A display is projected, information representing an image of the projected display and at least a portion of a pointing device in a vicinity of the projected display is optically captured, and the display is updated based on the captured image information.

Claims

1. A method comprising

projecting a display,

optically capturing information representing an image of the projected display and at least a portion of a pointing device in a vicinity of the projected display, and

updating the display based on the captured image information.

2. The method of claim 1 in which the pointing device comprises a finger.

3. The method of claim 1 in which the pointing device comprises a stylus.

4. The method of claim 1 in which the image of the pointing device includes information about whether the pointing device is activated.

5. The method of claim 1 in which the image of the portion of the pointing device comprises light emitted by the pointing device.

6. The method of claim 5 also comprising emitting light from the pointing device based on light from the projector.

7. The method of claim 6 in which the light is emitted from the pointing device asynchronously with the light emitted by the projector.

8. The method of claim 7 in which the image of the pointing device is captured when the pointing device is emitting light and the image of the display is captured when the projector is emitting light.

9. The method of claim 1 also comprising blocking visible light and transmitting infrared light.

10. The method of claim 1 in which the image of the portion of the pointing device comprises light reflected by the pointing device.

11. The method of claim 10 also comprising illuminating the pointing device.

12. The method of claim 11 also comprising projecting the display and illuminating the pointing device in alternating frames.

13. The method of claim 11 also comprising directing light into an ellipse around a previous location of the pointing device, and enlarging the ellipse until the captured image includes light reflected by the pointing device.

14. The method of claim 11 in which illuminating the pointing device comprises energizing a light source when a signal indicates that the pointing device is in use.

15. The method of claim 1 in which projecting the display comprises reflecting light with a micromirror device.

16. The method of claim 1 in which projecting the display comprises reflecting infrared light.

17. The method of claim 16 in which

projecting the display comprises projecting an image with a first subset of micromirrors of the micromirror device,

the method also comprising directing light in a common direction with a second subset of micromirrors of the micromirror device.

18. The method of claim 7 in which the first subset of micromirrors reflect visible light, and the second subset reflect infrared light.

19. The method of claim 1 in which capturing information representing an image of at least a portion the pointing device comprises capturing movement of the pointing device.

20. The method of claim 19 in which the movement of the pointing device comprises handwriting.

21. The method of claim 19 in which updating the display comprises one or more of

creating, modifying, moving, or deleting a user interface element based on movement of the pointing device,

editing text in an interface element based on movement of the pointing device, and

drawing lines based on movement of the pointing device.

22. The method of claim 19 in which the display is projected within a field of view, and updating the display comprises changing the field of view based on movement of the pointing device.

23. The method of claim 19 also comprising

interpreting the movement of the pointing device as selection of a hyperlink in the display, and

updating the display to display information corresponding to the hyperlink.

24. The method of claim 19 also comprising

interpreting the movement of the pointing device as an identification of another device, and

initiating a communication with the other device based on the identification.

25. The method of claim 24 in which initiating the communication comprises placing a telephone call.

26. The method of claim 24 in which initiating the communication comprises

assembling handwriting into a text message, and

transmitting the text message.

27. The method of claim 24 in which initiating the communication comprises

assembling handwriting into an email message, and

transmitting the email message.

28. The method of claim 1 in which

projecting a display comprises projecting an output image and projecting an image of a set of user interface elements, and

capturing the image information includes identifying which projected user interface elements the pointing device is in the vicinity of.

29. The method of claim 28 in which the image of a set of user interface element s comprises an image of a keyboard.

30. The method of claim 1 in which updating the display comprises adjusting the shape of the display to compensate for distortion found in the captured image of the display.

31. The method of claim 1 in which updating the display comprises repeatedly determining an angle to a surface based on the captured information representing an image of the display, and adjusting the shape of the display based on the angle.

32. The method of claim 31 in which projecting the display includes projecting reference marks and determining an angle includes determining distortion of the reference marks.

33. The method of claim 1 in which updating the display comprises adjusting the display to appear undistorted when projected at a known angle.

34. The method of claim 33 in which the known angle is based on an angle between a projecting element and a base surface of a device housing the projecting element.

35. The method of claim 1 in which projecting the display comprises altering a shape of the projected display based on calibration parameters stored in a memory.

36. The method of claim 1 also comprising capturing an image of a surface.

37. The method of claim 36 also comprising creating a file system object representing the image of the surface.

38. The method of claim 37 also comprising recognizing the image of the surface as a photograph, and in which the file system object is an image file representing the photograph.

39. The method of claim 37 also comprising recognizing the image of the surface as an image of a writing, and in which the file system object is a text file representing the writing.

40. The method of claim 1 also comprising

capturing information representing movement of the pointing device, and

editing a file system object based on movement of the pointing device.

41. The method of claim 40 in which editing comprises adding, deleting, moving, or modifying text.

42. The method of claim 40 in which editing comprises adding, deleting, moving, or modifying graphical elements.

43. The method of claim 40 in which editing comprises adding a signature.

44. The method of claim 1 in which the display comprises a computer screen bitmap image.

45. The method of claim 1 in which the display comprises a vector-graphical image.

46. The method of claim 45 in which the vector-graphical image is monochrome.

47. The method of claim 45 in which the vector-graphical image comprises multiple colors.

48. The method of claim 45 in which projecting the display comprises reflecting light along a sequence of line segments using at least a subset of micromirrors of a micromirror device.

49. The method of claim 1 also comprising

generating the display by removing content from an image, and

in which projecting the display comprises projecting the remaining content.

50. The method of claim 48 in which removing content from an image comprises removing image elements composed of bitmaps.

51. The method of claim 1 in which projecting the display comprises projecting a representation of items each having unique coordinates,

the method also comprising

detecting a location touched by the pointing device, and

correlating the location to at least one of the projected items.

52. The method of claim 1 also comprising

transmitting the captured information representing images to a server,

receiving a portion of an updated display from the server, and

in which updating the display comprises adding the received portions of an updated display to the projected display.

53. An apparatus comprising

a projector,

a camera, and

a processor programmed to

receive input from the camera including an image of a projected interface and a pointing device,

generate an interface based on the input, and

use the projector to project the interface.

54. The apparatus of claim 53 in which the projector has a first field of view, the camera has a second field of view, and the first and second fields of view at least partially overlap.

55. The apparatus of claim 53 in which the projector has a first field of view, the camera has a second field of view, and the first and second fields of view do not overlap.

56. The apparatus of claim 53 in which the projector has a first field of view, the camera has a second field of view, and at least one of the first and second fields of view can be repositioned.

57. The apparatus of claim 53 in which the projector and the camera can be repositioned relative to the rest of the apparatus.

58. The apparatus of claim 53 in which the camera comprises a filter that blocks visible light and admits infrared light.

59. The apparatus of claim 53 also comprising a source of light positioned to illuminate the pointing device.

60. The apparatus of claim 53 also comprising a sensor positioned to receive light from the pointing device.

61. The apparatus of claim 53 in which the projector comprises a micromirror device.

62. The apparatus of claim 60 in which a subset of micromirrors of the micromirror device are adapted to reflect infrared light.

63. The apparatus of claim 53 also comprising wireless communications circuitry.

64. The apparatus of claim 53 also comprising a memory storing a set of instructions for the processor.

65. An apparatus comprising

a projector having a first field of view,

a camera having a second field of view, the first and second fields of view not overlapping,

wireless communications circuitry, and

a processor programmed to

receive input from the camera including an image of a projected interface and a pointing device,

generate an interface based on the input, and

use the projector to project the interface.

66. An apparatus comprising

a light source, and

a cone-shaped reflector positioned within a path of light from the light source.