METHODS AND SYSTEMS FOR INTERACTING WITH PROJECTED USER INTERFACE

• US 20120017147A1
• Filed: 07/16/2010
• Published: 01/19/2012
• Est. Priority Date: 07/16/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Mark, John Liam
• Granted Patent: US 9,134,799 B2
• Time in Patent Office: Days
• US Class Current: 715/702
• CPC Class Codes: G06F 1/1639 the display being based on projection
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04M 1/0272 for a projector or beamer module assembly constructional features of projectors per se G03B21/00; constructional details of projectors using electronic spatial light modulators per se H04N9/31
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector
• CPC Class Codes: H04M 2250/54 including functional features of a projector or beamer module assembly
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability
• CPC Class Codes: H04N 9/3179 Video signal processing therefor

Abstract

Aspect methods and systems enable interaction with projected user interface through user input gestures performed on a mobile device. The mobile device may project the user interface image on an external surface, such as from a pico projector, and detect user input gestures performed by users. The mobile device may interpret the detected user input gestures, and change the projected user interface image in response to the interpreted user input gestures. User input gestures may be in the form of touch gestures performed on a touchscreen of the mobile device and/or in the form of hand gestures performed by turning, rotating and/or hoping the mobile device.

Claims

1. A method for providing a user interface for a mobile device, comprising:

projecting a user interface image;

detecting a user input gesture;

identifying functionality associated with the detected user input gesture; and

changing the projected user interface image based on the interpreted gesture.

2. The method of claim 1, wherein detecting a user input gesture comprises:

receiving a touch event from a touch screen display; and

correlating the received touch event to a particular touch gesture.

3. The method of claim 1, wherein projecting a user interface image comprises projecting an image that appears to be three dimensional when viewed with 3-D glasses.

4. The method of claim 1, wherein the projected image is configured so that image objects appear positioned about a surface of a sphere, and wherein changing the projected user interface image based on the interpreted gesture comprises changing the projected user interface image so that the sphere appears to rotate about one or two axes to position different image objects in a foreground position.

5. The method of claim 1, further comprising:

highlighting an image object located in a center of the projected user interface image; and

selecting the highlighted image object in response to a user input.

6. The method of claim 1, further comprising:

establishing a communication link with a second mobile device;

receiving a message from the second mobile device including information regarding changes to the user interface image based on the interpreted gesture; and

changing the user interface image based on the information received from the second mobile device regarding changes to the user interface image.

7. The method of claim 1, wherein detecting a user input gesture comprises:

receiving signals from orientation sensors within the mobile device; and

correlating the received sensor signals to a hand gesture.

8. The method of claim 7, further comprising:

identifying a functionality corresponding to the correlated hand gesture; and

implementing the identified functionality,

wherein changing the projected user interface image based on the interpreted gesture comprises changing the user interface image based upon the implemented functionality.

9. The method of claim 7, further comprising adjusting an angle of the projected user interface image to maintain the projected image approximately steady within a limited angle of tilt or turn.

10. The method of claim 7, further comprising changing the projected user interface image to maintain image objects in an approximately constant position with respect to an external surface onto which the image is projected.

11. The method of claim 7, wherein the orientation sensors comprise one or more accelerometers.

12. The method of claim 11, wherein correlating the received sensor signals to a hand gesture comprises recognizing the hand gesture as being one of a tilt of the mobile device, a turn of the mobile device, a rotation of the mobile device, and a rapid movement of the mobile device.

13. The method of claim 11, wherein implementing the identified functionality comprises accomplishing one of scrolling a list, moving an image object, changing a zoom factor applied to an image object displayed within the projected user interface image, selecting an object within the projected user interface image, copying an object displayed within the projected user interface image, deleting an object displayed within the projected user interface image, executing an application associated with an object displayed within the projected user interface image, and movement of a pointing icon displayed within the user interface image.

14. A mobile device, comprising:

a projector; and

a processor coupled to the processor, wherein the processor is configured with processor-executable instructions to perform operations comprising;

projecting a user interface image;

detecting a user input gesture;

identifying functionality associated with the detected user input gesture; and

changing the projected user interface image based on the interpreted gesture.

15. The mobile device of claim 14, wherein the processor is configured with processor-executable instructions to perform operations such that detecting a user input gesture comprises:

receiving a touch event from a touch screen display; and

correlating the received touch event to a particular touch gesture.

16. The mobile device of claim 14, wherein the processor is configured with processor-executable instructions to perform operations such that projecting a user interface image comprises projecting an image that appears to be three dimensional when viewed with 3-D glasses.

17. The mobile device of claim 14, wherein the processor is configured with processor-executable instructions to perform operations such that:

the projected image is configured so that image objects appear positioned about a surface of a sphere; and

changing the projected user interface image based on the interpreted gesture comprises changing the projected user interface image so that the sphere appears to rotate about one or two axes to position different image objects in a foreground position.

18. The mobile device of claim 14, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

highlighting an image object located in a center of the projected user interface image; and

selecting the highlighted image object in response to a user input.

19. The mobile device of claim 14, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

establishing a communication link with a second mobile device;

receiving a message from the second mobile device including information regarding changes to the user interface image based on the interpreted gesture; and

changing the user interface image based on the information received from the second mobile device regarding changes to the user interface image.

20. The mobile device of claim 14, further comprising orientation sensors, wherein the processor is configured with processor-executable instructions to perform operations such that detecting a user input gesture comprises:

receiving signals from the orientation sensors; and

correlating the received sensor signals to a hand gesture.

21. The mobile device of claim 20, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

identifying a functionality corresponding to the correlated hand gesture; and

implementing the identified functionality,

wherein changing the projected user interface image based on the interpreted gesture comprises changing the user interface image based upon the implemented functionality.

22. The mobile device of claim 20, wherein the processor is configured with processor-executable instructions to perform operations further comprising adjusting an angle of the projected user interface image to maintain the projected image approximately steady within a limited angle of tilt or turn.

23. The mobile device of claim 20, wherein the processor is configured with processor-executable instructions to perform operations further comprising changing the projected user interface image to maintain image objects in an approximately constant position with respect to an external surface onto which the image is projected.

24. The mobile device of claim 20, wherein the orientation sensors comprise one or more accelerometers.

25. The mobile device of claim 24, wherein the processor is configured with processor-executable instructions to perform operations such that correlating the received sensor signals to a hand gesture comprises recognizing the hand gesture as being one of a tilt of the mobile device, a turn of the mobile device, a rotation of the mobile device, and a rapid movement of the mobile device.

26. The mobile device of claim 24, wherein the processor is configured with processor-executable instructions to perform operations such that implementing the identified functionality comprises accomplishing one of scrolling a list, moving an image object, changing a zoom factor applied to an image object displayed within the projected user interface image, selecting an object within the projected user interface image, copying an object displayed within the projected user interface image, deleting an object displayed within the projected user interface image, executing an application associated with an object displayed within the projected user interface image, and movement of a pointing icon displayed within the user interface image.

27. A mobile device, comprising:

means for projecting a user interface image;

means for detecting a user input gesture;

means for identifying functionality associated with the detected user input gesture; and

means for changing the projected user interface image based on the interpreted gesture.

28. The mobile device of claim 27, wherein means for detecting a user input gesture comprises:

means for receiving a touch event from a touch screen display; and

means for correlating the received touch event to a particular touch gesture.

29. The mobile device of claim 27, wherein means for projecting a user interface image comprises means for projecting an image that appears to be three dimensional when viewed with 3-D glasses.

30. The mobile device of claim 27, wherein:

means for projecting a user interface image comprises means for projecting an image so that image objects appear positioned about a surface of a sphere; and

means for changing the projected user interface image based on the interpreted gesture comprises means for changing the projected user interface image so that the sphere appears to rotate about one or two axes to position different image objects in a foreground position.

31. The mobile device of claim 27, further comprising:

means for highlighting an image object located in a center of the projected user interface image; and

means for selecting the highlighted image object in response to a user input.

32. The mobile device of claim 27, further comprising:

means for establishing a communication link with a second mobile device;

means for receiving a message from the second mobile device including information regarding changes to the user interface image based on the interpreted gesture; and

means for changing the user interface image based on the information received from the second mobile device regarding changes to the user interface image.

33. The mobile device of claim 27, wherein means for detecting a user input gesture comprises:

means for sensing orientation of the mobile device; and

means for correlating the sensed orientation of the mobile device to a hand gesture.

34. The mobile device of claim 33, further comprising:

means for identifying a functionality corresponding to the correlated hand gesture; and

means for implementing the identified functionality,

wherein means for changing the projected user interface image based on the interpreted gesture comprises means for changing the user interface image based upon the implemented functionality.

35. The mobile device of claim 33, further comprising means for adjusting an angle of the projected user interface image to maintain the projected image approximately steady within a limited angle of tilt or turn.

36. The mobile device of claim 33, further comprising means for changing the projected user interface image to maintain image objects in an approximately constant position with respect to an external surface onto which the image is projected.

37. The mobile device of claim 33, wherein means for sensing orientation of the mobile device comprises one or more accelerometers.

38. The mobile device of claim 37, wherein means for correlating the received sensor signals to a hand gesture comprises means for recognizing the hand gesture as being one of a tilt of the mobile device, a turn of the mobile device, a rotation of the mobile device, and a rapid movement of the mobile device.

39. The mobile device of claim 37, wherein means for implementing the identified functionality comprises means for accomplishing one of scrolling a list, moving an image object, changing a zoom factor applied to an image object displayed within the projected user interface image, selecting an object within the projected user interface image, copying an object displayed within the projected user interface image, deleting an object displayed within the projected user interface image, executing an application associated with an object displayed within the projected user interface image, and movement of a pointing icon displayed within the user interface image.

40. A non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor of a mobile device to perform operations comprising:

a projector; and

a processor coupled to the processor, wherein the processor is configured with processor-executable instructions to perform operations comprising;

projecting a user interface image;

detecting a user input gesture;

identifying functionality associated with the detected user input gesture; and

changing the projected user interface image based on the interpreted gesture.

41. The mobile device of claim 40, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations such that detecting a user input gesture comprises:

receiving a touch event from a touch screen display; and

correlating the received touch event to a particular touch gesture.

42. The mobile device of claim 40, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations such that projecting a user interface image comprises projecting an image that appears to be three dimensional when viewed with special 3-D glasses.

43. The mobile device of claim 40, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations such that:

the projected image is configured so that image objects appear positioned about a surface of a sphere; and

changing the projected user interface image based on the interpreted gesture comprises changing the projected user interface image so that the sphere appears to rotate about one or two axes to position different image objects in a foreground position.

44. The mobile device of claim 40, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations further comprising:

highlighting an image object located in a center of the projected user interface image; and

selecting the highlighted image object in response to a user input.

45. The mobile device of claim 40, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations further comprising:

establishing a communication link with a second mobile device;

receiving a message from the second mobile device including information regarding changes to the user interface image based on the interpreted gesture; and

changing the user interface image based on the information received from the second mobile device regarding changes to the user interface image.

46. The mobile device of claim 40, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations such that detecting a user input gesture comprises:

receiving signals from orientation sensors within the mobile device; and

correlating the received sensor signals to a hand gesture.

47. The mobile device of claim 46, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations further comprising:

identifying a functionality corresponding to the correlated hand gesture; and

implementing the identified functionality,

wherein changing the projected user interface image based on the interpreted gesture comprises changing the user interface image based upon the implemented functionality.

48. The mobile device of claim 46, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations further comprising adjusting an angle of the projected user interface image to maintain the projected image approximately steady within a limited angle of tilt or turn.

49. The mobile device of claim 46, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations further comprising changing the projected user interface image to maintain image objects in an approximately constant position with respect to an external surface onto which the image is projected.

50. The mobile device of claim 46, wherein the orientation sensors comprise one or more accelerometers.

51. The mobile device of claim 50, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations such that correlating the received sensor signals to a hand gesture comprises recognizing the hand gesture as being one of a tilt of the mobile device, a turn of the mobile device, a rotation of the mobile device, and a rapid movement of the mobile device.

52. The mobile device of claim 50, wherein the stored processor-executable instructions are configured to cause a processor of a mobile device to perform operations such that implementing the identified functionality comprises accomplishing one of scrolling a list, moving an image object, changing a zoom factor applied to an image object displayed within the projected user interface image, selecting an object within the projected user interface image, copying an object displayed within the projected user interface image, deleting an object displayed within the projected user interface image, executing an application associated with an object displayed within the projected user interface image, and movement of a pointing icon displayed within the user interface image.