Method and Apparatus for Developing and Playing Natural User Interface Applications

• US 20140129935A1
• Filed: 11/05/2012
• Published: 05/08/2014
• Est. Priority Date: 11/05/2012
• Status: Active Grant

• Current Assignee: Harvest Moon Series 96 of Allied Security Trust I (Allied Security Trust)
• Original Assignee: Artissimo Designs, LLC
• Inventors: OVADIA NAHON, Dolly, MATZA, Kfir, MATZA, Sarel
• Granted Patent: US 9,501,140 B2
• Time in Patent Office: Days
• US Class Current: 715/716
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

An apparatus method and computer program product for developing and executing NUI applications. The apparatus for developing the NUI application comprising a development platform comprising a design manager for indicating a NUI object and a graphic object, wherein the graphic object is designed to track movements of the NUI object, based on movements of an object captured by a NUI device and associated with the NUI object when the application is executed; an interactive manager for managing interactive actions between the NUI object and the graphic object; and a NUI manager for defining a NUI device to be supported by a computing platform executing the application, wherein the application is developed graphically.

Claims

1. An apparatus having a processing unit and a storage device, the apparatus comprising:

a natural user interface (NUI) development platform for developing a NUI application, the development platform comprising;

a design manager for indicating at least one NUI object, and at least one graphic object, such that the at least one graphic object is designed to track the movements of the at least one NUI object, based on movements of an object captured by a NUI device and associated with the at least one NUI object when the NUI application is executed;

an interactive manager for managing interactive actions between the at least one NUI object and the at least one graphic object; and

a NUI manager for defining a NUI device to be supported by a computing platform executing the NUI application,

wherein the NUI application is developed graphically, and

wherein the NUI application employs the NUI device when executed and displays the NUI object as captured.

2. The apparatus of claim 1, wherein the at least one graphic object is designed to track the at least one NUI object on at least one tracking point or on at least tracking area comprising at least three tracking points defined on the at least one NUI object.

3. The apparatus of claim 1, wherein the design manager provides for indicating a reference point on the at least one NUI object for determining relative locations between at least one tracking point on the at least one NUI object and locations on the at least one graphic object.

4. The apparatus of claim 1, wherein the graphic object is associated with a pinning point associated with a physical property, such that the at least one graphic object or part thereof tracks the at least one NUI object in accordance with the physical property, when the NUI application is executed.

5. The apparatus of claim 1, wherein the at least one graphic object is selected from the group consisting of:

a three dimensional representation of a physical object;

a two dimensional representation of a physical object;

an image;

a video signal;

an audio signal;

a drawing;

an illustration; and

a recorded object.

6. The apparatus of claim 1, wherein the NUI device is selected from the group consisting of:

web camera, a depth camera, a Kinect™

device, a WII®

remote control, a leap, a ringbow, an EEG helmet, a PS3®

device, a smart phone a tablet, a XTION, a Microsoft'"'"'s Digits, a Microsoft PixelSense, a Smart Window, a Google glass, a Motorola HC1, a PMD[Vision]®

camera, and a computing platform equipped with a touch sensor.

7. The apparatus of claim 1, further comprising a gesture manager for defining gestures available in the system for the NUI device.

8. The apparatus of claim 1, further comprising a passive media manager for handling passive media, the passive media manager comprising a media encoder for associating an identifier with a video signal.

9. The apparatus of claim 1, wherein the interactive actions are provided by at least one interactive object.

10. The apparatus of claim 9, wherein the at least one interactive object is associated with programming commands when the NUI application is executed.

11. The apparatus of claim 1, wherein the interactive manager provides for at least one connector type, each of the at least one connector type associated with a conditional statement.

12. The apparatus of claim 1, wherein the at least one NUI object is associated with a predefined gesture and connected to at least one interactive object by a connector in a flowchart-like visual programming.

13. The apparatus of claim 1, wherein the development platform provides a developer with visual representation of the human body or parts thereof, and tools to edit live video in a visual environment.

14. The apparatus of claim 1, wherein the development platform provides for object recording, such that the object is recognized once captured by the NUI device.

15. An apparatus having a processing unit and a storage device, the apparatus comprising:

a player for executing a NUI application, the player comprising;

a NUI controller for receiving input from a NUI device;

a decoder for identifying a gesture in the input received from the NUI device;

a local interactive media storage for storing a graphic object; and

an interactive media executor for integrating the graphic object with the input such that the graphic object and an NUI object comprising an image of a human body or part thereof, wherein the image is captured by the NUI device, are displayed, and the graphic object tracks the NUI object identified in the input in accordance with a gesture of the NUI object.

16. The apparatus of claim 15, wherein the player further comprises a physicality engine for implementing physical properties and behaviors defined on the graphic object when reacting to the NUI object.

17. The apparatus of claim 15, wherein the NUI device is selected from the group consisting of:

web camera, a depth camera, a Kinect™

device, a WII®

remote control, a leap, a ringbow, an EEG helmet, a PS3®

device, a smart phone a tablet, a XTION, a Microsoft'"'"'s Digits, a Microsoft PixelSense, a Smart Window, a Google glass, a Motorola HC1, a PMD[Vision]®

camera, and a computing platform equipped with a touch sensor.

18. The apparatus of claim 15 wherein the NUI device is selected as a most suitable device connected to the apparatus.

28. The apparatus of claim 15, wherein the gesture of the NUI object tracked by the graphic object is not a predetermined gesture.

19. An apparatus having a processing unit and a storage device, the apparatus comprising:

a player for executing a NUI application, the player comprising;

an interactive media loader for loading a NUI application;

a NUI controller for receiving input from a NUI device;

a decoder for identifying a NUI object in the input received from the NUI device;

a passive media receiver for receiving passive media; and

an interactive media executor for integrating output from the NUI application comprising input from the NUI device, with the passive media such that the NUI object is displayed with the passive media,

wherein the NUI application is received separately from the passive media, and

wherein the NUI application employs the NUI device when executed.

20. The apparatus of claim 19, wherein the passive media is a TV broadcast.

21. The apparatus of claim 19 further comprising a signal decoder for identifying a signal in the passive media, and integrating a NUI object associated with the NUI device into the passive media

22. The apparatus of claim 19, wherein the NUI application is received from a cloud service to a television, a set up box, or a computing platform in accordance with at least one parameter associated with a user of the apparatus.

29. The apparatus of claim 19, wherein the gesture of the NUI object tracked by the graphic object is not a predetermined gesture.

23. A computer-implemented method performed by a computerized device, comprising:

receiving a NUI application developed using a graphical user interface, the user interface comprising a scene containing at least one layer representing at least one object, comprising;

receiving at least one NUI object, and at least one graphic object, such that the at least one graphic object and the NUI object are displayed when the NUI application is executed, and the at least one graphic object is designed to track the movements of the NUI object, based on movements of an object captured by a NUI device and associated with the NUI object when the NUI application is executed, and

receiving interactive actions between the at least one NUI object and the at least one graphic object;

receiving an indication to a NUI device to be supported by a computing platform executing the NUI application; and

storing the NUI application on a storage device,

wherein the NUI application is developed graphically,

wherein the NUI application employs the NUI device when executed.

24. The computer-implemented method of claim 23, wherein the NUI application is selected from the group consisting of:

a user interacting with an object; and

a user interacting with a projected surface.

25. A computer-implemented method performed by a computerized device, comprising:

receiving streamed data;

sniffing the streamed data for a signal;

responsive to identifying the signal in the streamed data, searching for corresponding downloaded media comprising a NUI application;

responsive to locating the downloaded media, integrating at a location within the streamed data associated with a location in which the signal was identified, output from the NUI application being executed,

wherein the NUI application employs the NUI device when executed.

26. A computer program product comprising:

a non-transitory computer readable medium;

a first program instruction for receiving a NUI application, comprising;

an instruction for receiving at least one NUI object, and at least one graphic object, such that the at least one graphic object and the NUI object are displayed when the NUI application is executed, and the at least one graphic object is designed to track the movements of the NUI object, based on movements of an object captured by a NUI device and associated with the NUI object when the NUI application is executed, and

an instruction for receiving interactive actions between the at least one NUI object and the at least one graphic object;

a second program instruction for receiving an indication to a NUI device to be supported by a computing platform when the NUI application is executed; and

a third program instruction for storing the NUI application on a storage device,

wherein the NUI application is developed graphically,

wherein said first, second, and third program instructions are stored on said non-transitory computer readable medium, and

wherein the NUI application employs the NUI device when executed.

27. A computer program product comprising:

a non-transitory computer readable medium;

a first program instruction for receiving streamed data;

a second program instruction for sniffing the streamed data for a signal;

a third program instruction for responsive to identifying the signal in the streamed data, searching for corresponding downloaded media comprising a NUI application; and

a fourth program instruction for responsive to locating the downloaded media, integrating the streamed data at a location within the streamed data associated with a location in which the signal was identified, the streamed data with input from the NUI application being executed,

wherein the NUI application employs the NUI device when executed, and

wherein said first, second, third, and fourth program instructions are stored on said non-transitory computer readable medium.