MOBILE DEVICE AND METHOD FOR CONTROLLING GRAPHICAL USER INTERFACE THEREOF

• US 20150002475A1
• Filed: 03/20/2014
• Published: 01/01/2015
• Est. Priority Date: 06/27/2013
• Status: Abandoned Application

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Tiao, Kuo-Tung, Jeng, Tzuan-Ren, Lin, Hsien-Chang, Chen, Chia-Chen, Zeng, Wen-Jun, Huang, Kuo-Chung, Yang-Mao, Shys-Fan, Wang, Yao-Lien
• Application Number:
  US14/221,269
• Publication Number:
  US 20150002475A1
• Time in Patent Office: Days
• US Class Current: 345/175
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A mobile device is provided, which includes a camera unit, a sensor unit, a see-through display, and a processor. The camera unit takes an image of a finger and a surface. The sensor unit generates a sensor signal in response to a motion of the finger. The taking of the image and the generation of the sensor signal are synchronous. The see-through display displays a GUI on the surface. The processor is coupled to the camera unit, the sensor unit, and the see-through display. The processor uses both of the image and the sensor signal to detect a touch of the finger on the surface. The processor adjusts the GUI in response to the touch.

Claims

1. A mobile device, comprising:

a camera unit, taking an image of a finger and a surface;

a sensor unit, generating a sensor signal in response to a motion of the finger, wherein the taking of the image and the generation of the sensor signal are synchronous;

a see-through display, displaying a graphical user interface (GUI) on the surface; and

a processor, coupled to the camera unit, the sensor unit, and the see-through display, using both of the image and the sensor signal to detect a touch of the finger on the surface, and adjusting the GUI in response to the touch.

2. The mobile device of claim 1, wherein the camera unit comprises two cameras configured to take stereo images of the finger and the surface, wherein the processor uses the stereo images to estimate a distance between the finger and the surface, and wherein the processor uses the distance as an auxiliary to detect the touch.

3. The mobile device of claim 1, wherein the see-through display displays the GUI on the surface when the processor identifies the surface in the image.

4. The mobile device of claim 1, wherein the processor analyzes the sensor signal to detect the touch when the processor identifies both of the finger and the surface in the image, wherein the processor ignores the sensor signal and does not attempt to detect the touch when processor fails to identify at least one of the finger and the surface in the image.

5. The mobile device of claim 1, wherein the processor starts identifying and tracking a fingertip of the finger after the processor identifies the finger in the image, wherein the processor analyzes the sensor signal to detect the touch when the fingertip is in a touch-operable area of the GUI, wherein the processor ignores the sensor signal and does not attempt to detect the touch when processor fails to identify the fingertip or when the fingertip is in a non-touch-operable area of the GUI.

6. The mobile device of claim 1, wherein the processor starts identifying and tracking a fingertip of the finger after the processor identifies the finger in the image, wherein the processor confirms the touch when the fingertip is in a touch-operable area of the GUI and either a magnitude of the sensor signal is larger than a preset threshold or a waveform of the sensor signal matches a preset waveform.

7. The mobile device of claim 6, wherein the sensor unit comprises a temperature sensor configure to measure an ambient temperature of the finger, wherein the sensor unit further comprises a humidity sensor configure to measure an ambient humidity of the finger, wherein the processor adjusts the preset threshold according to the ambient temperature and the ambient humidity.

8. The mobile device of claim 1, wherein the processor detects a multiple click of the finger on the surface by analyzing the image to confirm a position of the multiple click and analyzing the sensor signal to confirm a number of clicks of the finger on the surface, wherein the processor performs a preset function in response to the multiple click.

9. A method for controlling a graphical user interface (GUI) of a mobile device, comprising:

taking an image of a finger and a surface;

generating a sensor signal in response to a motion of the finger, wherein the taking of the image and the generation of the sensor signal are synchronous;

displaying the GUI on the surface;

using both of the image and the sensor signal to detect a touch of the finger on the surface; and

adjusting the GUI of the mobile device in response to the touch.

10. The method of claim 9, further comprising:

taking stereo images of the finger and the surface;

using the stereo images to estimate a distance between the finger and the surface; and

using the distance as an auxiliary to detect the touch.

11. The method of claim 9, wherein the step of displaying the GUI comprises:

displaying the GUI on the surface when the processor identifies the surface in the image.

12. The method of claim 9, further comprising:

analyzing the sensor signal to detect the touch when both of the finger and the surface are identified in the image; and

ignoring the sensor signal and not attempting to detect the touch when failing to identify at least one of the finger and the surface in the image.

13. The method of claim 9, further comprising:

starting identifying and tracking a fingertip of the finger after identifying the finger in the image;

analyzing the sensor signal to detect the touch when the fingertip is in a touch-operable area of the GUI; and

ignoring the sensor signal and not attempting to detect the touch when failing to identify the fingertip or when the fingertip is in a non-touch-operable area of the GUI.

14. The method of claim 9, further comprising:

starting identifying and tracking a fingertip of the finger after identifying the finger in the image; and

confirming the touch when the fingertip is in a touch-operable area of the GUI and either a magnitude of the sensor signal is larger than a preset threshold or a waveform of the sensor signal matches a preset waveform.

15. The method of claim 14, further comprising:

measuring an ambient temperature and an ambient humidity of the finger;

adjusting the preset threshold according to the ambient temperature and the ambient humidity.

16. The method of claim 9, further comprising:

detecting a multiple click of the finger on the surface by analyzing the image to confirm a position of the multiple click and analyzing the sensor signal to confirm a number of clicks of the finger on the surface; and

performing a preset function in response to the multiple click.

17. The method of claim 9, further comprising:

detecting a preset motion of the finger on the surface by tracking a motion of the finger in the image and analyzing the sensor signal to confirm continued contact of the finger on the surface; and

performing a preset function in response to the preset motion.