Visual toolkit for a virtual user interface

• US 8,578,282 B2
• Filed: 03/07/2007
• Issued: 11/05/2013
• Est. Priority Date: 03/15/2006
• Status: Expired due to Fees

• Current Assignee: Navisense
• Original Assignee: Navisense
• Inventors: Boillot, Marc Andre
• Primary Examiner(s): Pesin, Boris
• Assistant Examiner(s): Vuu, Henry
• Application Number:
  US11/683,413
• Publication Number:
  US 20070220437A1
• Time in Patent Office: 2,435 Days
• Field of Search: 715/776, 715/863, 715/864
• US Class Current: 715/762
• CPC Class Codes: G06F 8/34 Graphical or visual programming

Abstract

An Integrated Development Environment (IDE) (100) for creating a touchless Virtual User Interface (VUI) 120 is provided. The IDE can include a development window (152) for graphically presenting a visual layout of user interface (UI) components (161) that respond to touchless sensory events in a virtual layout of virtual components (261), and at least one descriptor (121) for modifying a touchless sensory attribute of a user component. The touchless sensory attribute describes how a user component responds to a touchless touchless sensory event on a virtual component.

Claims

1. An Integrated Development Environment (IDE), executing computer instructions in a computer-readable storage medium of a computer system, suitable for use in developing a Virtual User Interface (VUI) application, the IDE running on the computer system and comprising:

a development window for graphically constructing a visual layout of a User Inteface (UI) to correspond to a virtual layout of a Virtual User Interface (VUI) generated by an ultrasonic sensing unit externally coupled to the computer system and with its own targeted processor and memory for receiving executable program code compiled from the IDE specific to instructions of the targeted processor; and

at least one descriptor that identifies a response of a user interface component in the UI to touchless sensory events applied to a corresponding virtual component in the VUI generated by the ultrasonic sensing unit,

a compiler for compiling a source code of the VUI application into at least one code object using imported target processor and memory configurations from the ultrasonic sensing unit;

a linker for converting said code object into relocatable code based on the targeted processor and memory of the ultrasonic sensing unit;

a code builder for building said relocatable code into an executable code object; and

a flashing module for managing a down loading of the executable program code object over a connection into a flash memory of the externally coupled ultrasonic sensing unit associated with the target processor and memory,

where the computer system is communicatively coupled to the ultrasonic sensing unit by a wired or wireless connection for developing the VUI on the computer and down loading the executable code object into the ultrasonic sensing unit.

2. The IDE of claim 1, further comprising a component palette having user components that are dragged and dropped in the UI to create Virtual components in the VUI.

3. The IDE of claim 1, further comprising a dialog window that presents at least one modifiable touchless sensory attribute of a touchless sensory event associated with a virtual component.

4. The IDE of claim 1, wherein a touchless sensory attribute comprises at least one of an absolute location of a finger, a duration of time of a finger at a location, a relative displacement of a finger, a velocity of a finger, and an acceleration of a finger in the VUI.

5. The IDE of claim 1, wherein a touchless sensory event comprises at least one of a touchless finger positioning, a touchless finger push, a touchless finger release, a touchless finger pause, a touchless finger sign, a touchless scroll, and a touchless select.

6. The IDE of claim 1, wherein the computer system receives touchless sensory events from the ultrasonic sensing unit producing a touchless sensing field,

wherein the ultrasonic sensing unit generates the touchless sensory events in response to touchless finger actions in the VUI where there are no markers, fixtures, gloves, or objects in physical contact with the finger.

7. The IDE of claim 1, wherein a touchless sensory event on a virtual component produces one of audio or visual feedback on a corresponding UI component in the display window.

8. The IDE of claim 2, further comprising a form sheet, wherein said ultrasonic sensing unit visually projects the virtual layout containing the Virtual components on the form sheet.

9. The IDE of claim 1, where

the linker for converting said code object into relocatable code identifies regions of memory targeted by the processor of the ultrasonic sensing unit, and links the relocatable code to those regions of memory including the at least one descriptor;

and

the code builder for building said relocatable code into an executable code object that accesses the targeted memory of the processor in the external ultrasonic sensing unit and at least one descriptor on the ultrasonic sensing device.

10. The IDE of claim 9, where

the flashing module resides on the computer system and is temporarily coupled to the ultrasonic sensing unit for loading the executable program code object with the at least one descriptor into the flash memory of the ultrasonic sensing unit via the wired or wireless connection.

11. The IDE of claim 10, further comprising at least one of:

a cable for connecting said IDE to the ultrasonic sensing unit; and

a wireless device for temporarily connecting said IDE to a the ultrasonic sensing unit to perform wireless flashing,

wherein the flashing module manages the download of the executable program code object over a serial port, a USB, a Bluetooth, or a ZigBee connection.

12. A visual toolkit, executing computer instructions in a computer-readable storage medium of a computer system running on the computer system, the visual toolkit comprising:

a development window that presents a visual layout of re-locatable user components in a User Interface (UI);

at least one descriptor in the development window that describes how a user interface component of the UI responds to touchless sensory events received from an ultrasonic sensing unit unit externally coupled to the computer system and with its own targeted processor and memory for receiving executable program code compiled by the visual toolkit;

a compiler for compiling a source code of the User Interface into at least one code object using imported target processor and memory configurations from the ultrasonic sensing unit;

a linker for converting said code object into relocatable code based on the targeted processor and memory of the ultrasonic sensing unit;

a code builder for building said relocatable code into an executable code object on the computer system; and

a flashing module for managing a down loading of the executable code object over a connection into a flash memory of the externally coupled ultrasonic sensing unit associated with the target processor and memory,

wherein the ultrasonic sensing unit is externally coupled to the computer system via a wired or wireless connection and generates a touchless ultrasonic sensing field with a Virtual User Interface (VUI) from the at least one descriptor and the visual layout;

wherein an arranging of the re-locatable user components in the UI creates a virtual layout of virtual components in the touchless sensing field for producing the Virtual User Interface (VUI) generated by the ultrasonic sensing unit.

13. The visual toolkit of claim 12, wherein executable code object of the VUI application represents the visual layout in the UI.

14. The visual toolkit of claim 12, wherein the executable code object is a compiled:

Java program, C++program, or .NET program.

15. The visual toolkit of claim 12, wherein the visual toolkit provides programming through C, C++, Java, or .NET, and

wherein the flashing module manages the download of the executable program code object over a serial port, a USB, a Bluetooth, or a ZigBee connection.

16. The visual toolkit of claim 12, wherein the driver converts a touches sensory event on a virtual component in the VUI to an action on a user component in the UI.

17. The visual toolkit of claim 12, wherein the visual toolkit implements a sensory Application Programming Interface (API) for providing portability across computing platforms, wherein the sensory API exposes methods, fields, event listeners, and event handlers for processing touchless finger actions in the VUI directed to the executable code in the ultrasonic sensing unit.

18. A system comprising:

an ultrasonic sensing unit with its own targeted processor and memory for producing a sensing field and generating touchless sensory events response to touchless finger actions in the sensing field, where there are no markers, fixtures, gloves, or objects in physical contact with the finger;

an Integrated Development Environment (IDE) running on a computer, the IDE comprising;

a development window for graphically constructing a visual layout of a User Inteface (UI) to correspond to a virtual layout of a Virtual User Interface (VUI) generated by the ultrasonic sensing unit that is externally coupled to the computer system to receive executable program code compiled from the IDE specific to the targeted processor and memory;

a compiler for compiling a source code of the User Interface into at least one code object using imported target processor and memory configurations from the ultrasonic sensing unit;

a linker for converting said code object into relocatable code based on the targeted processor and memory of the ultrasonic sensing unit;

a code builder for building said relocatable code into an executable code object on the computer system;

a flashing module for managing a down loading of the executable code object over a connection into a flash memory of the externally coupled ultrasonic sensing unit associated with the target processor,

and, a wireless communication link between the IDE and the ultrasonic sensing unit for developing the VUI on the IDE and loading the executable code object onto the flash memory of the externally coupled ultrasonic sensing unit by way of the flashing module,

wherein an arranging of the re-locatable user components in the UI creates a virtual layout of virtual components in the ultrasonic sensing field for producing the Virtual User Interface (VUI) generated by the ultrasonic sensing unit.