Method and apparatus for developing and playing natural user interface applications

• US 9,501,140 B2
• Filed: 11/05/2012
• Issued: 11/22/2016
• Est. Priority Date: 11/05/2012
• Status: Active Grant

• Current Assignee: Harvest Moon Series 96 of Allied Security Trust I (Allied Security Trust)
• Original Assignee: Artissimo Designs, LLC
• Inventors: Ovadia Nahon, Dolly, Matza, Kfir, Matza, Sarel
• Primary Examiner(s): IPAKCHI, MARYAM M
• Application Number:
  US13/668,365
• Publication Number:
  US 20140129935A1
• Time in Patent Office: 1,478 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

An apparatus method and computer program product for developing and executing NUI applications. The apparatus for developing the NUI application comprising a development platform comprising a design manager for indicating a NUI object and a graphic object, wherein the graphic object is designed to track movements of the NUI object, based on movements of an object captured by a NUI device and associated with the NUI object when the application is executed; an interactive manager for managing interactive actions between the NUI object and the graphic object; and a NUI manager for defining a NUI device to be supported by a computing platform executing the application, wherein the application is developed graphically.

Claims

1. An apparatus having a processing unit and a storage device, the apparatus comprising:

a natural user interface (NUI) development platform for developing a graphic NUI application, the graphic NUI application to be used by an end-user while the graphic NUI application is being executed without using the NUI development platform, the NUI development platform comprising;

a design manager for indicating and linking during design time of the graphic NUI application, at least one graphic object and at least one NUI object, the at least one NUI object representing an object to be captured by a NUI device, such that during playing of the graphic NUI application, the at least one graphic object is intended to track the movements of the at least one NUI object, based on movements of the object;

an interactive manager for managing during design time interactive actions between the at least one NUI object and the at least one graphic object, such that during playing of the graphic NUI application actions of the at least one NUI object cause actions of the at least one graphic object; and

a NUI manager for defining a NUI device to be supported by a computing platform executing the NUI application,

wherein the NUI application is developed graphically, and wherein the NUI application employs the NUI device when executed and displays the object as captured.

2. The apparatus of claim 1, wherein the at least one graphic object is designed to track the at least one NUI object on at least one tracking point or on at least tracking area comprising at least three tracking points defined on the at least one NUI object.

3. The apparatus of claim 1, wherein the design manager provides for indicating a reference point on the at least one NUI object for determining relative locations between at least one tracking point on the at least one NUI object and locations on the at least one graphic object.

4. The apparatus of claim 1, wherein the graphic object is associated with a pinning point associated with a physical property, such that the at least one graphic object or part thereof tracks the at least one NUI object in accordance with the physical property, when the NUI application is executed.

5. The apparatus of claim 1, wherein the at least one graphic object is selected from the group consisting of:

a three dimensional representation of a physical object;

a two dimensional representation of a physical object;

an image;

a video signal;

an audio signal;

a drawing;

an illustration; and

a recorded object.

6. The apparatus of claim 1, wherein the NUI device is selected from the group consisting of:

a camera, a web camera, a depth camera, a computer gaming device, a virtual reality device, a leap, an EEG helmet, a smart phone, a tablet, and a computing platform equipped with a touch sensor.

7. The apparatus of claim 1, further comprising a gesture manager for defining gestures available in the system for the NUI device.

8. The apparatus of claim 1, further comprising a passive media manager for handling passive media, the passive media manager comprising a media encoder for associating an identifier with a video signal.

9. The apparatus of claim 1, wherein the interactive actions are provided by at least one interactive object.

10. The apparatus of claim 9, wherein the at least one interactive object is associated with programming commands when the NUI application is executed.

11. The apparatus of claim 1, wherein the interactive manager provides for at least one connector type, each of the at least one connector type associated with a conditional statement.

12. The apparatus of claim 1, wherein the at least one NUI object is associated with a predefined gesture and connected to at least one interactive object by a connector in a flowchart-like visual programming.

13. The apparatus of claim 1, wherein the NUI development platform provides a developer with visual representation of the human body or parts thereof, and tools to edit live video in a visual environment.

14. The apparatus of claim 1, wherein the NUI development platform provides for object recording during design time, such that the object is recognized once captured by the NUI device in playing time.

15. A computer-implemented method performed by a computerized device, comprising:

receiving a NUI graphic application developed using a graphical user interface of a NUI development platform, the user interface comprising a scene containing at least one layer representing at least one object, comprising;

receiving at least one graphic object, and at least one NUI object, the at least one NUI object representing an object to be captured by a NUI device, such that during playing of the graphic NUI application, the at least one graphic object and the NUI object are displayed, and the at least one graphic object is designed to track the movements of the at least one NUI object, based on movements of the object, and

receiving interactive actions between the at least one NUI object and the at least one graphic object, such that during playing of the graphic NUI application actions of the at least one NUI object cause actions of the at least one graphic object;

receiving an indication of a NUI device to be supported by a computing platform executing the NUI application; and

storing the NUI application on a storage device, wherein the NUI application is developed graphically, and wherein the graphic NUI application to be used by an end-user while the graphic NUI application is being executed without using the NUI development platform, and wherein the NUI application employs the NUI device when executed.

16. The computer-implemented method of claim 15, wherein the NUI application is selected from the group consisting of:

a user interacting with an object; and

a user interacting with a projected surface.