Systems and methods for a voice- and gesture-controlled mobile application development and deployment platform

• US 8,898,630 B2
• Filed: 03/14/2013
• Issued: 11/25/2014
• Est. Priority Date: 04/06/2011
• Status: Active Grant

• Current Assignee: Media Direct Incorporated
• Original Assignee: Media Direct Incorporated
• Inventors: Hirsch, Scott, Pereymer, Arsen, Rajpal, Sunny
• Primary Examiner(s): Zhen, Li B
• Assistant Examiner(s): CHEN, SEN THONG
• Application Number:
  US13/831,155
• Publication Number:
  US 20140109046A1
• Time in Patent Office: 621 Days
• Field of Search: 717/107
• US Class Current: 717/107
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 8/60 Software deployment
• CPC Class Codes: G06F 8/70 Software maintenance or management
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs

Abstract

Systems and methods for developing, customizing, and deploying mobile device applications through voice and/or gesture interactions are provided through a mobile application development and deployment platform. Preferably, these systems and methods are implemented in an Internet based environment that allows non-technical users to build sophisticated, highly-customizable cross-platform mobile applications. The platform allows users to use voice and/or gesture interactions to select, input, create, customize, and combine various content, design characteristics, and application components, such as modules, some of which utilize features and functionality associated with various mobile devices and mobile operating systems. In certain embodiments, the platform allows users to compile the mobile application for various mobile operating systems and generates a configuration file for the mobile application that can be distributed to end users for execution on various mobile devices. When the mobile application is installed on, or executed by the mobile device, the configuration file may enable the retrieval of various data associated with the mobile application.

Claims

1. A system for allowing users to use voice commands to develop mobile applications or portions thereof that are capable of being compiled to run on a plurality of mobile operating systems, the system comprising:

a computing device having physical memory storing instructions that cause the computing device to;

provide a mobile application development platform adapted to assist users, including visually-impaired users, with the development and customization of mobile applications, wherein the mobile application development platform represents application data associated with a mobile application in accordance with a data model that defines relationships among a hierarchy of tables including;

a head table that is used to represent an instance of the mobile application and that includes an identifier for the mobile application;

an app type table that includes a first set of attributes for one or more templates that can be applied to the mobile application; and

a module table that includes a second set of attributes that are used to represent one or more customizable components included in the mobile application;

receive a first set of voice commands from a user to develop the mobile application through the mobile application development platform, the first set of voice commands at least identifying an application type property that identifies a template included in the app type table;

query one or more of the tables in the data model to identify a plurality of customizable components that are available for use with the identified template;

send information associated with the plurality of customizable components to be presented to the user through a set of audio outputs, the plurality of customizable components pertaining to a functionality, design and content of a mobile application to be developed by the user;

receive from the user through a second set of voice commands a plurality of customizable component selections pertaining to the plurality of presented customizable components;

associate application data with the mobile application based, at least in part, on the plurality of customizable component selections received from the user through the second set of voice commands, the application data representing user-defined functionality, design and content for the mobile application;

store the application data associated with the mobile application in accordance with the data model;

generate compiled data for the mobile application based on the stored application data using build tools associated with at least one mobile operating system selected by the user through a third set of voice commands, wherein generating compiled data includes querying a database that stores the tables associated with the data model to retrieve information associated with the mobile application and including at least a portion of the retrieved information in a configuration file; and

generate the configuration file comprising the identifier for the mobile application, the configuration file being adapted for transmission to, and execution on, a mobile device running the at least one mobile operating system;

wherein the identifier enables the retrieval of at least a portion of the compiled application data by the mobile device in response to the mobile application being installed on or executed by the mobile device.

2. The system according to claim 1, wherein the mobile application development platform includes a web based service accessed by the user through the Internet.

3. The system according to claim 1, wherein the mobile application development platform is accessed through a software application installed on a computer associated with the user.

4. The system according to claim 1, the physical memory storing instructions that further cause the computing device to receive from the user through a fourth set of voice commands one or more identifying properties for the mobile application.

5. The system according to claim 4, wherein the one or more identifying properties are selected from the group consisting of a name, a type, a category, a description, a unique identifier, and a date.

6. The system according to claim 1, wherein the second set of voice commands comprises one or more voice commands from a pre-defined list of voice commands supported by the mobile application development platform.

7. The system according to claim 1, wherein the second set of voice commands comprises a one or more voice responses to an audio menu output presented by the mobile application development platform.

8. The system according to claim 1, wherein the second set of voice commands comprises voice specifications provided by the user during a natural language dialog with the mobile application development platform.

9. The system according to claim 1, wherein the set of audio outputs comprises a user-configurable subset of a detailed audio description of the plurality of customizable components pertaining to the functionality, design and content of the mobile application.

10. A method for developing and customizing mobile applications or portions thereof that are compiled to run on a plurality of mobile operating systems, the method being implemented through instructions stored in a physical memory associated with a computing device, which cause the computing device to perform the steps of:

providing a mobile application development platform adapted to assist users with the development and customization of mobile applications, wherein the mobile application development platform represents application data associated with a mobile application in accordance with a data model that defines relationships among a hierarchy of tables including;

a head table that is used to represent an instance of the mobile application and that includes an identifier for the mobile application;

an app type table that includes a first set of attributes for one or more templates that can be applied to the mobile application; and

a module table that includes a second set of attributes that are used to represent one or more customizable components included in the mobile application;

receiving at the computing device a first set of voice commands from a user to develop the mobile application, the first set of voice commands at least identifying an application type property that identifies a template included in the app type table;

querying one or more of the tables in the data model to identify a plurality of customizable components that are available for use with the identified template;

presenting the plurality of customizable components to the user through a set of audio outputs, the plurality of customizable components pertaining to functionality, design and content of the mobile application to be customized by the user;

receiving from the user through a second set of voice commands at the computing device a plurality of customizable component selections pertaining to the plurality of presented customizable components;

associating application data with the mobile application based, at least in part, on the plurality of customizable component selections, the application data being stored in the physical memory associated with the computing device;

storing the application data associated with the mobile application in accordance with the data model;

generating at the computing device compiled data for the mobile application based on the application data using build tools associated with at least one mobile operating system selected by the user through a third set of voice commands, wherein generating compiled data includes querying a database that stores the tables associated with the data model to retrieve information associated with the mobile application and including at least a portion of the retrieved information in a configuration file; and

generating at the computing device the configuration file comprising the identifier for the mobile application, the configuration file being adapted for transmission to, and execution on, a mobile device running the at least one mobile operating system;

wherein the identifier enables the retrieval of at least a portion of the compiled application data by the mobile device in response to the mobile application being installed on or executed by the mobile device.

11. The method according to claim 10, wherein the instructions stored in the physical memory associated with the computing device further cause the computing device to perform the steps:

receiving a fourth set of voice selections from the user of one or more identifying properties from the mobile application selected from the group consisting of a name, a type, a category, a description, a unique identifier, and a date.

12. A system for allowing users to use touch gestures to develop mobile applications or portions thereof that are capable of being compiled to run on a plurality of mobile operating systems, the system comprising:

a computing device comprising a touch screen display, the computing device further having physical memory storing instructions that cause the computing device to;

provide a mobile application development platform adapted to assist users with the development and customization of mobile applications, wherein the mobile application development platform represents application data associated with a mobile application in accordance with a data model that defines relationships among a hierarchy of tables including;

a head table that is used to represent an instance of the mobile application and that includes an identifier for the mobile application;

an app type table that includes a first set of attributes for one or more templates that can be applied to the mobile application; and

a module table that includes a second set of attributes that are used to represent one or more customizable components included in the mobile application;

receive a request from a user through one or more first gestures to develop the mobile application through the mobile application development platform, the one or more first gestures at least identifying an application type property that identifies a template included in the app type table;

query one or more of the tables in the data model to identify a plurality of customizable components that are available for use with the identified template;

send information associated with the plurality of customizable components to be presented to the user, the plurality of customizable components pertaining to a functionality, design and content of a mobile application to be developed by the user;

receive from the user through one or more second gestures a plurality of customizable component selections pertaining to the plurality of presented customizable components;

associate application data with the mobile application based, at least in part, on the plurality of customizable component selections received from the user through the one or more second gestures, the application data representing user-defined functionality, design and content for the mobile application;

store the application data associated with the mobile application in accordance with the data model;

generate compiled data for the mobile application based on the application data using build tools associated with at least one mobile operating system selected by the user through one or more third gestures, wherein generating compiled data includes querying a database that stores the tables associated with the data model to retrieve information associated with the mobile application and including at least a portion of the retrieved information in a configuration file; and

generate the configuration file comprising the identifier for the mobile application, the configuration file being adapted for transmission to, and execution on, a mobile device running the at least one mobile operating system;

wherein the identifier enables the retrieval of at least a portion of the compiled application data by the mobile device in response to the mobile application being installed on or executed by the mobile device.

13. The system according to claim 12, wherein the mobile application development platform includes a web based service accessed by the user through the Internet.

14. The system according to claim 12, wherein the mobile application development platform is accessed through a software application installed on a computer associated with the user.

15. The system according to claim 12, wherein the touch screen display of the computing device is configured to receive user input through at least one of simple gestures or multi-touch gestures.

16. The system according to claim 12, wherein the information associated with the plurality of customizable components is presented to the user as graphical elements on the touch screen display.

17. The system according to claim 16, the physical memory storing instructions that further cause the computing device to:

receive one or more fourth gestures performed by the user on the graphical elements of the touch screen display to modify the plurality of customizable components; and

modify the application data associated with the mobile application, based on the modifications to the plurality of customizable components.

18. The system according to claim 17 wherein the one or more fourth gestures performed by the user on the graphical elements of the touch screen display to modify the plurality of customizable components comprise one or more gestures performed by the user to resize the graphical elements.

19. The system according to claim 17 wherein the one or more fourth gestures performed by the user on the graphical elements of the touch screen display to modify the plurality of customizable components comprise one or more gestures performed by the user to arrange the graphical elements with respect to each other.

20. The system according to claim 17 wherein the one or more fourth gestures performed by the user on the graphical elements of the touch screen display to modify the plurality of customizable components comprise one or more gestures performed by the user to modify content associated with the customizable components.

21. The system according to claim 12, the physical memory storing instructions that further cause the computing device to receive a request to reconfigure a mapping between a set of gestures and a set of actions to a new mapping between the set of gestures and the set of actions.