APPARATUS AND METHODS FOR PROVIDING A PERSISTENT COMPANION DEVICE

• US 20150314454A1
• Filed: 07/15/2015
• Published: 11/05/2015
• Est. Priority Date: 03/15/2013
• Status: Abandoned Application

• Current Assignee: Jibo, Inc., Ntt Disruptions Co.
• Original Assignee: Jibo, Inc.
• Inventors: Breazeal, Cynthia, Michaud, Avida, Laberge, Francois, Ross, Jonathan Louis, Saund, Carolyn Marothy, Querze, Elio Dante III, Faridi, Fardad
• Application Number:
  US14/799,704
• Publication Number:
  US 20150314454A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 11/001 with emotions simulating means
• CPC Class Codes: B25J 11/0015 Face robots, animated artificial faces for imitating human expressions
• CPC Class Codes: B25J 9/0003 Home robots, i.e. small robots for domestic use
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A development platform for developing a skill for a persistent companion device (PCD) includes an asset development library having an application programming interface (API) configured to enable a developer to at least one of find, create, edit and access one or more content assets utilizable for creating a skill, an expression tool suite having one or more APIs via which are received one or more expressions associated with the skill as specified by the developer wherein the skill is executable by the PCD in response to at least one defined input, a behavior editor for specifying one or more behavioral sequences of the PCD for the skill and a skill deployment facility having an API for deploying the skill to an execution engine of the PCD.

Claims

23. A platform for enabling development of a skill using a software development kit (SDK) comprising:

a logic level module configured to map received inputs to coded responses; and

a perceptual level module comprising;

a vision function module configured to detect one or more vision function events and to inform the logic level module of the one or more detected vision function events; and

a speech/sound recognition and understanding module configured to detect defined sounds and to inform the logic level module of the detected speech/sounds; and

an expression engine configured to generate one or more animations expressive of defined emotional/persona states and to transmit the one or more animations to the logic level module.

1. A development platform for developing a skill for a persistent companion device (PCD) comprising:

an asset development library having an application programming interface (API) configured to enable a developer to at least one of find, create, edit and access one or more content assets utilizable for creating a skill that is executable by the PCD;

an expression tool suite having one or more APIs via which receive one or more expressions associated with the skill as specified by the developer wherein the skill is executable by the PCD in response to at least one defined input;

a behavior editor for specifying one or more behavioral sequences of the PCD for the skill; and

a skill deployment facility having an API for deploying the skill to an execution engine for executing the skill.

2. The development platform of claim 1 wherein at least one of the APIs of the asset development library and the expression tool suite comprises a Javascript API.

3. The development platform of claim 1 wherein the behavioral editor utilizes a graphical user interface to display the one or more behavioral sequences.

4. The development platform of claim 3, wherein at least one behavioral sequence is displayed by graphically representing a plurality of sequential physical positions of the PCD for the sequence.

5. The development platform of claim 3, wherein at least one behavioral sequence is displayed by representing a multi-modal expression of the PCD from a plurality of modes selected from the group consisting of plurality of sequential physical positions of the PCD, a sequence of sounds for the PCD, a sequence of graphics displayed by the PCD, a sequence of lighting effects displayed by the PCD and a sequence of speech by the PCD.

6. The development platform of claim 3, wherein at least one behavioral sequence comprises at least one of a control signal to another device, a software application, and an input from at least one of another device, a software application for behaviors and an asset development.

7. The development platform of claim 3 wherein the one or more behavioral sequences are specified using a directed graph.

8. The development platform of claim 7 wherein the directed graph is a recursive directed graph.

9. The development platform of claim 1 wherein the expression tool suite enables specification of at least one of a mood, a mode and a persona configured to modulate the expression of a given skill by a PCD based on at least one of a mood, a mode, a user attribute model and a persona.

10. The development platform of claim 1 further comprising a sensory input handling module for handling one or more sensory inputs that can be captured by the PCD.

11. The development platform of claim 10, wherein the sensory input module of the PCD has a defined local perceptual space in which the PCD is adapted to detect and track sensory input associated with the environment in which the PCD is deployed.

12. The development platform of claim 1 further comprising a simulation module for at least one of executing code developed for the skill and simulating the local perceptual space of the PCD, wherein the simulation module and the PCD can execute the same code.

13. The development platform of claim 12, wherein the PCD and the simulation module can act as input sources or output targets for each other.

14. The development platform of claim 1, wherein the platform is adapted for delivery of a skill to a PCD that includes at least one interface for receiving and executing the skill.

15. The development platform of claim 1, wherein the at least one defined input comprises at least one of a digital input and a physical input.

16. The development platform of claim 1, wherein the execution engine is deployed on the PCD.

17. The development platform of claim 1, wherein the execution engine is deployed as part of the development platform.

18. The development platform of claim 1, wherein the execution engine is deployed as part of a simulator for the behavior of the PCD.

19. The development platform of claim 1 wherein the asset development library comprises one or more library assets.

20. The development platform of claim 1 wherein the expression tool suite comprises a timeline editor configured to temporally coordinate multi-model expressions of the PCD.

21. The development platform of claim 20 wherein the multi-model expressions are selected from the group consisting of speech, sound effects, sound filters, physical movements, screen graphics, and LED lighting effects.

22. The development platform of claim 1 comprising a real-time debugger/editor to support live coding for WYSIWYG viewing of at least one of resulting code and other asset changes on at least one of the simulator output and the PCD.

24. The platform of claim 23, further comprising a multi-modal data collection and machine-learning module for enabling automated or human-guided improvement of at least one of a skill and a component of a skill.

25. The platform of claim 23, further comprising a persona module inclusive of character-based interactivity with emotive, social and task-based attributes for creating one of a stored and learned digital representation for a persona for the PCD.

26. The platform of claim 23, further comprising a user attribute module for at least one of learning and storing attributes of a user of the PCD.

27. The platform of claim 23, wherein the animations consist of effects selected from the group consisting of sequences of physical positions of the PCD, sounds for the PCD, graphics displayed by the PCD, lighting effects displayed by the PCD and speech by the PCD.

28. The platform of claim 23 wherein the expression engine receives an input selected from the group consisting of pre-recorded audio, speech, text, touch, textually formatted output utterances, animations, textures, data from other device and applications.

29. The platform of claim 23 wherein the speech/sound recognition and understanding module receives an input generated via a process selected from the group consisting of phrase spotting, non-phrase spotting, non-speech sound recognition, automatic speech recognition, the application of natural language rules, tools for data collection and training of new recognition and understanding models or improvement of existing ones, the identification of the user, and the direction of the source of sound.

30. The platform of claim 23 wherein communication between the logic level module and the perceptual level module operates to translate perceived and internal events into expressed skills.

31. The platform of claim 23, wherein communication between the logic level module and the perceptual module operates to at least one of provide an input and accept an input from an interface with at least one of another device, a web service, a mobile application, a web application, a cloud computing resource, a cloud storage resource, a social media interface and a video conferencing interface.

32. The platform of claim 23, wherein communication between the logic level module and the perceptual level module operates to modulate at least one of the persona and the behavior of the PCD.

33. The platform of claim 32 wherein the communication operates with contextual information comprising a user attribute model.

34. The platform of claim 23 wherein pre-recorded audio and textually formatted output utterances are processed by a text-to-speech and audio splicing/filtering engine to create a speech component of an animation.

35. The platform of claim 34, wherein the expression engine generates a sound effect associated with the speech component of the animation.

36. The platform of claim 23 wherein the vision function module is configured to perform functions selected from the group consisting of picture taking, video recording, face and facial feature tracking, person tracking, object recognition, location of the PCD, emotion recognition, gesture recognition, gesture tracking and facial recognition.

37. The platform of claim 23 wherein the type of speech input to be accepted by the PCD is defined by a user via a user interface.

38. The platform of claim 23, further comprising a publication module for publishing a skill to at least one of an online store and a cloud-based skill repository where a skill can be at least one of discovered, purchased and downloaded to a PCD of a user.

39. The platform of claim 23 wherein the one or more vision function events comprise at least one of a social visual feature and an emotive visual feature.

40. The platform of claim 38 wherein the emotive visual feature comprises at least one of a depth feature, a color feature, an illumination feature, and an infra-red (IR) feature.

41. The platform of claim 23 wherein the defined sounds comprise at least one of a social feature and an emotive sound based feature.

42. The platform of claim 23, wherein communication between the logic level module and the perceptual level module operates to modulate a user attribute model.

43. The platform of claim 42 wherein communication operates with contextual information comprising the persona model.

44. The platform of claim 23 wherein inputs comprise reusable multi-modal input-output modules for behavior tree development, library asset development, and skill development.

43-1. The platform of claim 38, wherein the skill is downloaded from the cloud.