PERSISTENT COMPANION DEVICE CONFIGURATION AND DEPLOYMENT PLATFORM

• US 20170206064A1
• Filed: 03/30/2017
• Published: 07/20/2017
• Est. Priority Date: 03/15/2013
• Status: Abandoned Application

• Current Assignee: Ntt Disruptions Co.
• Original Assignee: Jibo, Inc.
• Inventors: Michaud, Avida, Laberge, Francois, Ross, Jonathan Louis, Saund, Carolyn Marothy, Breazeal, Cynthia, Faridi, Fardad
• Application Number:
  US15/474,841
• Publication Number:
  US 20170206064A1
• Time in Patent Office: Days
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: G01S 3/8083 determining direction of source
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/36 Software reuse
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A development platform for developing a skill for a persistent companion device (PCD) includes an asset development library having an application programming interface (API) configured to enable a developer to at least one of find, create, edit and access one or more content assets utilizable for creating a skill, an expression tool suite having one or more APIs via which are received one or more expressions associated with the skill as specified by the developer wherein the skill is executable by the PCD in response to at least one defined input, a behavior editor for specifying one or more behavioral sequences of the PCD for the skill and a skill deployment facility having an API for deploying the skill to an execution engine of the PCD.

Claims

1. A system for developing a skill for a persistent companion device (PCD) comprising:

an asset development library, accessible via an application programming interface (API) executing on a processor, configured to enable a developer to at least one of find, create, edit and access one or more content assets utilizable for creating a skill that is executable by the PCD;

an animation tool suite executing on the processor having one or more APIs via which operation of one or more physical elements of the PCD for the skill including at least two of an electronic display, a plurality of movable body segments, a speech output system, and a multi-color light source are specified by the developer, wherein the skill is executable by the PCD in response to at least one input that is defined by the developer;

a behavior editor executing on the processor for specifying one or more behavioral sequences of the PCD for the skill; and

a skill deployment facility executing on the processor for deploying the skill to an execution engine for executing the skill.

2. The system of claim 1, wherein the skill deployment facility deploys the skill via an API.

3. The system of claim 1, wherein the behavior editor facilitates operation of a sensory input system and an expressive output system of the PCD.

4. A system for enabling development of a persistent companion device (PCD) skill using a software development kit (SDK) comprising:

a logic level mapping system operating on a processor configured to map received inputs to the PCD to coded responses; and

a PCD behavior tool suite operating on the processor adapted to configure a perceptual engine of the PCD comprising;

a vision function system configured via the behavior tool suite to detect one or more vision function events and to inform the logic level mapping system of the one or more detected vision function events; and

a speech/sound recognition and understanding system configurable by the behavior tool suite to detect defined sounds and to inform the logic level mapping system of the detected speech/sounds; and

a PCD animation tool suite operating on the processor adapted to configure an expression engine to generate one or more animations expressive of at least one defined state in response to at least one input and to transmit the one or more animations to the logic level mapping system for mapping of the animations to the inputs.

5. The system of claim 4, wherein the defined state is at least one of an emotional state, a persona state, a cognitive state, and a state expressing a defined level of energy.

6. A system for configuring a persistent companion device (PCD) to perform a skill, comprising:

a software development kit executing on a networked server, comprising

a plurality of animation user interface screens through which a user configures animation associated with the skill, the plurality of user interface screens facilitating specification of the operation of physical elements of the PCD including at least two of an electronic display, a plurality of movable body segments, a speech output system, and a multi-color light source; and

a plurality of behavior user interface screens through which the user configures behavior of the PCD for coordinating robot actions and decisions associated with the skill, the plurality of behavior user interface screens facilitating operation of an expressive output system of the PCD in response to a sensory input system of the PCD;

wherein a graphical representation of the PCD in at least one of the animation user interface screens and the behavior user interface screens represents the movement of the PCD in response to inputs based on the configuration by the user.

7. The system of claim 6, further comprising a gaze orientation user interface screen through which a user configures the PCD to expressively orient a display screen of the PCD toward a target located in proximity to the PCD as a point in a three-dimensional space relative to the PCD, the PCD responding to the target in at least one of a single-shot mode and a continuous target-tracking mode.

8. A system for animating a persistent companion device (PCD), comprising:

an animation editor executing on a networked server providing access to PCD animation configuration and control functions of the PCD via a software development kit;

an electronic interface to a PCD, the PCD configured with a plurality of interconnected moveable body segments, motors for rotation thereof, at least one light ring, an electronic display screen, and an audio system;

a PCD animation application programming interface via which the animation editor controls at least a portion of the features of the PCD; and

a plurality of animation builders configurable by a user of the animation editor, the animation builders spawning animation instances that specify active animation sessions.

9. The system of claim 8, further comprising a behavior transition system for specifying transition of the PCD from a first animation instance to a second animation instance in response to a signal.

10. A system for controlling behaviors of a persistent companion device (PCD), comprising:

a behavior editor executing on a networked server providing access to PCD behavior configuration and control functions of the PCD via a software development kit;

a plurality of behavior tree data structures accessible by the behavior editor that facilitate controlling behavior and control flow of autonomous robot operational functions, the operational functions including a plurality of sensor input functions and a plurality expressive output functions, wherein the plurality of behavior tree data structures organize control of robot operational functions hierarchically, wherein at least one behavior tree data structure is associated with at least one skill performed by the PCD;

wherein each behavior tree data structure comprises a plurality of behavior nodes, each of the plurality of behavior nodes associated with one of four behavior states consisting of an invalid state, an in-progress state, a successful state, and a failed state; and

wherein each behavior tree data structure comprises at least one parent behavior node, the at least one parent node referencing at least one child behavior node and adapted to initiate at least one of sequential child behavior node operation, parallel child behavior node operation, switching among child behavior nodes, and randomly activating a referenced child behavior node.

11. The system of claim 10, wherein at least a portion of the behavior nodes are each configured with a behavior node decorator that functions to modify a state of its behavior node by performing at least one of preventing a behavior node from starting, forcing an executing behavior node to succeed, forcing an executing behavior node to fail, re-executing a behavior node.

12. A system for recognizing speech with a persistent companion device (PCD), comprising:

a PCD speech recognition configuration system that facilitates natural language understanding by a PCD, the system comprising a plurality of user interface screens by which a user operates a speech rule editor executing on a networked computer to configure speech understanding rules comprising at least one of an embedded rule and a custom rule;

a software development kit comprising library of embedded speech understanding rules accessed by the user via the networked server; and

a robot behavior association function of the software development by which a user associates speech understanding rules with at least one of a listen-type PCD behavior and a listen success decorator that the user configures to cause the PCD to perform an operation based on a successful result of a condition tested by the listen success decorator.

13. A persistent companion device (PCD) control configuration system comprising:

a PCD animation configuration system that facilitates controlling expressive output of the PCD through playback of scripted animations, responsive operation of the PCD for events detected by event listeners that are configurable by a user, and a plurality of animation layers that facilitate specifying animation commands;

a PCD behavior configuration system that facilitates controlling mechanical and electronic operation of the PCD;

a PCD gaze orientation configuration system that facilitates determining directional activity of the gaze of the PCD by specifying a target and a gaze PCD functional mode of at least one of single-shot and target-tracking; and

a PCD speech recognition configuration system comprising a plurality of embedded rules for recognizing human speech, and a user interface for customizing rules for recognizing human speech, wherein the human speech is captured by an audio sensor input system of the PCD.

14. The system of claim 13, wherein controlling mechanical and electronic operation of the PCD through robot behavior comprises controlling transitions between animated behaviors, controlling a plurality of animated behaviors in at least one of parallel control and sequential control, and controlling a plurality of child behaviors based on a behavior tree of parent and child behaviors, wherein a child behavior is activated based on one of a switch condition for selecting among the child behaviors and randomly selecting among the child behaviors.