Touchless tablet method and system thereof

• US 8,614,669 B2
• Filed: 03/07/2007
• Issued: 12/24/2013
• Est. Priority Date: 03/13/2006
• Status: Expired due to Fees

• Current Assignee: Navisense
• Original Assignee: Navisense
• Inventors: Marc, Boillot Andre
• Primary Examiner(s): Bukowski, Kenneth
• Application Number:
  US11/683,416
• Publication Number:
  US 20070211031A1
• Time in Patent Office: 2,484 Days
• Field of Search: 178/18.04, 178/18.09, 178/19.02, 178/19.05, 345156-184
• US Class Current: 345/158
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/043 using propagating acoustic waves

Abstract

A system (100) and method for a touchless tablet that produces a touchless sensory field over a form (111). The touchless tablet includes a touchless sensing unit (110) for identifying a finger action above the form, and a controller (130) communicatively coupled to the sensing unit for associating the finger action with at least one form component on the form. The touchless tablet identifies a selection of a form component (122) based on a location and action of the finger above the form. A display (140) connected to the touchless tablet can expose a graphical application, wherein a touchless selection of a form component corresponds to a selection of a graphical component on the graphical application.

Claims

1. A touchless tablet comprising:

a camera that takes a picture of a form,

a touchless sensing unit operatively coupled to the camera for identifying a touchless finger action above the form within a three dimensional (3D) ultrasonic sensory space; and

a controller communicatively coupled to said touchless sensing unit, wherein said controller

determines a location of form components in the picture captured by said camera,

creates a virtual layout of virtual form components in a three-dimensional ultrasonic touchless sensing space based on the location of the form components in the picture;

presents on a display a visual layout of visual form components also based on the location of the form components in the picture,

applies a combinational weighting of a Time of Flight (TOF) ultrasonic distance measurement corresponding to a coarse estimated location of the finger and a differential Time of Flight (dTOF) ultrasonic measurement corresponding to a relative displacement as the finger accelerates or decelerates between a far distance and a close distance;

creates a vector from a first location at the far distance to a second location at the close distance for a forward finger movement, or, a vector from the first location at the close distance to the second location at the far distance for a retracting finger movement in the ultrasonic sensory space; and

predicts a destination of the finger on said form from said vector to provide zoom in and zoom out and associates the touchless finger action in the ultrasonic sensory space on at least one virtual form component in the virtual layout of the three-dimensional ultrasonic touchless sensing space with at least one visual form component of the visual layout on said form and presented on the display based on the predicted destination,

wherein said touchless tablet identifies a selection of said virtual form component based on said location and action of said finger in the three-dimensional ultrasonic touchless sensing space.

2. The touchless tablet of claim 1, further comprising an off-axis element placing the camera above the touchless finger actions to capture the picture of said form using optical camera elements, and said controller identifies form components in said picture and creates graphical components on a display corresponding to said form components of said picture.

3. The touchless tablet of claim 1, wherein said sensing unit further comprises:

a detector, wherein said detector identifies a position of said finger in a sensory field above the form by

emits a plurality of ultrasonic pulses from a first ultrasonic transducer configured to transmit the ultrasonic pulses;

estimates for a plurality of ultrasonic transducers a time of flight between transmitting one of the ultrasonic pulses and receiving a reflected ultrasonic signal corresponding to a reflection off the finger;

calculates for the plurality of ultrasonic transducers a differential receive time between the reflected ultrasonic signal and a previously received reflected ultrasonic signal, and

determines a location and relative displacement of the finger from a combination of said time of flight measurements and said differential receive time measurements for mapping the virtual components of the VUI to the user components of the UI; and

a timer cooperatively connected to said detector, wherein said timer determines a length of time said finger is at a position over a form component.

4. The touchless tablet of claim 3, wherein said plurality of ultrasonic transducers are microphones that alternate between capturing a sound of a finger tap on a surface presenting said form component and reflected ultrasonic signals off the finger, such that said detector affirms said position of said finger when said sound of said finger tap is detected.

5. The touchless tablet of claim 1, wherein said touchless tablet is:

a portion of a frame positioned on an adjacent second side of said form to allow a user to see an unobstructed view of said form,

wherein said frame contains at least one sensory element for detecting a finger movement within said frame.

6. The touchless tablet of claim 1, wherein said finger action includes at least one of a touchless depressing action, a touchless release action, a touchless hold action, and a touchless dragging action.

7. The touchless tablet of claim 1, where the display is communicatively coupled to said touchless tablet for exposing the graphical components, wherein said selection of a form component within a sensory field above the form corresponds to a selection of the graphical component on said display.

8. The touchless tablet of claim 1, wherein said selection performs an action on the form object that produces a response from a graphical component in the graphical application, wherein the graphical component corresponds to the form component.

9. The touchless tablet of claim 2, wherein said touchless sensing unit comprises:

a fixed array of ultrasonic transducers positioned on at least one side of a form for producing a sensing field over said form for determining a first and second coordinate of a finger within said sensing field;

the off-axis element comprises an ultrasonic transducer for providing a third dimension of ultrasonic sensory measurement for determining a third coordinate of said finger within said sensing field; and

a processor for determining a three-dimensional coordinate and action of said finger within said sensing field.

10. A virtual screen comprising:

a sensing unit containing an arrangement of sensing elements for generating an ultrasonic touchless sensory field representing the virtual screen;

a camera that takes a picture of a form, where the form is thereafter not needed and removed,

a processor communicatively coupled to said touchless sensing unit and the camera for

determining a location of form components in the picture captured by said camera,

creating a virtual layout of virtual form components in said ultrasonic touchless sensory field based on the location of the form components in the picture;

presenting on a display a visual layout of visual form components also based on the location of the form components in the picture

determining a touchless finger location and a touchless finger action within said touchless sensory field; and

associating the touchless finger location and touchless finger action on a virtual form component in the virtual layout with a visual form component in the visual layout, and

controlling a graphical application of the visual form component according to said touchless finger location and said touchless finger action la applying a combinational weighting of a Time of Flight (TOF) ultrasonic distance measurement corresponding to a coarse estimated location of the finger and a differential Time of Flight (dTOF) ultrasonic measurement corresponding to a relative displacement as the finger accelerates or decelerates between a far distance and a close distance to the sensing unit.

11. The virtual screen of claim 10, wherein said sensing unit comprises two or more acoustic transducer to produce the touchless sensory field.

12. The virtual screen of claim 10, wherein said finger action includes at least one of a finger push action, a finger hold action, a finger release action, and a finger slide action.

13. The virtual screen of claim 10, where the touchless sensory field is produced over the display that receives commands for controlling a graphical application on the display and provides a third dimension of depth interaction.

14. The virtual screen of claim 10, further comprising a communication unit for transmitting said finger action and location using at least one among a USB, a BlueTooth, and a ZigBee communication link.

15. The virtual screen of claim 10, further comprising a communication device that receives said finger location and action and adjusts at least one user interface control of the communication device.

16. A method of navigation in a virtual screen comprising:

capturing by way of a camera a picture of a user interface and determining a location of user interface components in the picture;

creating by way of an ultrasonic sensing unit a three dimensional (3D) sensory space with a virtual layout of virtual user interface components corresponding to the location of user interface components in the picture;

presenting on a display a visual layout of visual user interface components also based on the location of user interface components in the picture;

estimating a time of flight (TOF) between when an ultrasonic pulse is transmitted from a first ultrasonic transducer and when a reflection of said ultrasonic pulse off the finger in said 3D sensory space is received from a plurality of ultrasonic transducers;

estimating a differential time of flight (dTOF) between a first reflected ultrasonic signal and a second ultrasonic reflected received from the ultrasonic transducer for the plurality of ultrasonic transducers,

estimating a finger position within said 3D sensory space corresponding to one of the virtual user interface components in the virtual layout by applying a combinational weighting of the TOF and the dTOF as the finger accelerates and decelerates between a far distance and a close distance, where the TOF corresponds to an estimated location of the finger and the dTOF corresponds to a relative displacement of the finger;

determining one of a forward or retracting finger movement associated with said finger position; and

controlling the visual user interface component in the visual layout corresponding to the virtual user interface component according to said finger movement by adjusting a graphical portion of said user interface to where the finger is pointed with respect to said forward or retracting finger movement.

17. The method of claim 16, wherein said determining comprises:

identifying a first depth of a finger during said forward or retracting finger movement within said 3D ultrasonic sensory space;

identifying a second depth of said finger during said forward or retracting finger movement within said 3D ultrasonic sensory space;

creating a vector from said first and said second depth based on said TOF and said dTOF ultrasonic measurements;

predicting a destination of said finger on said graphical application from said vector; and

adjusting a region of focus of a zoom-in and zoom-out of the graphical portion of said visual user interface based on said destination with respect to said forward or retracting finger movement.

18. The method of claim 17, wherein a distant depth of said 3D sensory space corresponds to a broad region and a close depth of said 3D sensory space corresponds to a narrow region of focus, such that said zoom-out broadens when the user'"'"'s finger is farther from a sensing unit producing said 3D sensory space, and said zoom-in narrows when the user'"'"'s finger is closer to said sensing unit.