Apparatus and methods for providing a persistent companion device

• US 10,391,636 B2
• Filed: 03/13/2014
• Issued: 08/27/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Ntt Disruptions Co.
• Original Assignee: SQN Venture Income Fund LP (SQN Venture Partners LLC)
• Inventors: Breazeal, Cynthia
• Primary Examiner(s): Mott, Adam R
• Application Number:
  US14/210,037
• Publication Number:
  US 20140277735A1
• Time in Patent Office: 1,993 Days
• CPC Class Codes: B25J 11/0005 Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
• CPC Class Codes: B25J 9/0003 Home robots, i.e. small robots for domestic use
• CPC Class Codes: G03B 2217/005 Blur detection
• CPC Class Codes: G03B 2217/007 Details of energy supply or management
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet

Abstract

A method includes providing a telecommunications enabled robotic device adapted to persist in an environment of a user, receiving an instruction to photograph one or more persons in the environment according to a time parameter and photographing the one or more persons in accordance with the time parameter resulting in one or more photographs.

Claims

1. A method comprising:

recording, utilizing a telecommunications-enabled robotic device adapted to persist in an environment of a user, an environment using one or more sensors and a camera located in the robotic device, the robotic device including a first segment housing the sensors and camera, a second segment, and a third segment, the first, second and third segments independently rotatable with respect to each other, wherein an end of the first segment facing a first end of the second segment has a generally circular periphery;

identifying a location of one or more persons in the environment based on at least one sensory input recorded by the sensors or camera;

performing a first rotation of at least one of the second segment and the third segment to aim the camera and sensors at the one or more persons;

executing a social interaction operation performing a second rotation of at least one of the second segment and the third segment in combination with a natural language interaction and electronic display screen graphics, wherein the social interaction operation is generated based on the data recorded by the sensors and camera;

executing an instruction to photograph the one or more persons according to a first parameter and via the social interaction; and

changing the mode of operation from the first mode of operation to a second mode of operation based, at least in part, on the first parameter.

2. The method of claim 1 wherein the first parameter is a time parameter.

3. The method of claim 2 wherein the time parameter is selected from the group consisting of a predetermined amount of time, a rate at which photographs are to be taken, a start time and end time.

4. The method of claim 1 wherein the at least one sensory input is selected from the group consisting of an auditory input, a visual input, a touch input and a remote device input.

5. The method of claim 1 wherein executing an instruction to photograph the one or more persons comprises utilizing at least one of sound location and facial detection or facial recognition to identify each of the one or more persons to be photographed.

6. The method of claim 1 wherein the first parameter causes the robot device to search for a target person to photograph or video, select the target person to photo or video, and track the target person via movement while capturing photo or video.

7. The method of claim 1 further comprising:

observing an emotional state of a person to be photographed; and

emitting, in response to the observed emotional state, a cue to provoke a preferred emotional state of the person.

8. The method of claim 7 wherein said cue is derived, at least in part, from data received from the cloud.

9. The method of claim 7 further comprising repeating the steps of observing and emitting until an objective is completed.

10. The method of claim 9 wherein the objective is determined based, at least in part, on at least one of the first mode of operation and the second mode of operation.

11. The method of claim 1 further comprising:

receiving from a user a second parameter; and

photographing the one or more persons in accordance with the second parameter.

12. The method of claim 11 wherein the second parameter is a requirement to take at least one photograph of each of the one or more persons or a video.

13. The method of claim 11 further comprising broadcasting a request to the persons to be photographed, or receiving an input from the persons, to improve the image composition.

14. The method of claim 13 wherein the request directs the at least one of the one or more persons to remain stationary.

15. The method of claim 1 further comprising:

broadcasting a request for an identifier of at least one of the one or more persons;

receiving an identifier from the at least one of the one or more persons; and

associating the received identifier with at least one of a photograph and a video of the at least one of the one or more persons.

16. The method of claim 1 wherein the instruction to photograph is received from the user.

17. The method of claim 1, wherein rotation of at least one of the second and third segments is controlled by a vestibulo-ocular reflex algorithm.

18. The method of claim 1, wherein the first end of the second segment, a second end of the second segment, and an end of the third segment comprise the generally circular periphery.

19. The method of claim 18, wherein the generally circular periphery of at least two of the first end of the second segment, the second end of the second segment, and the end of the third segment substantially align during rotation.

20. The method of claim 1, the robotic device further comprising a plurality of operational modules selected from a list of modules consisting of multi-modal machine perception, speech recognition and expressive speech synthesis.

21. The method of claim 1 wherein at least one of the electronic display screen graphics comprises an eye graphic.

22. The method of claim 1 further comprising the robotic device asking at least one question to clarify the received instruction.

23. The method of claim 1 wherein the social interaction operation is generated based on the data recorded by the sensors and camera and data retrieved from a remote server.

24. The method of claim 1 wherein a motor of the robotic device is mounted in an elastomeric material for absorbing noise produced by armature bearings of the motor.

25. The method of claim 1 wherein the data recorded by the sensors is combined into a probabilistic model of physically measurable user state, using particle filtering techniques.

26. A computer readable medium forming a part of a telecommunications enabled robotic device adapted to persist in an environment of a user, the computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising:

recording, utilizing a telecommunications-enabled robotic device adapted to persist in an environment of a user, an environment using one or more sensors and a camera located in the robotic device, the robotic device including a first segment housing the sensors and camera, a second segment, and a third segment, the first, second and third segments independently rotatable with respect to each other, wherein an end of the first segment facing an end of the second segment has a generally circular periphery;

identifying a location of one or more persons in the environment based on at least one sensory input recorded by the sensors or camera;

performing a first rotation of at least one of the second segment and the third segment to aim the camera and sensors at the one or more persons;

executing a social interaction operation performing a second rotation of at least one of the second segment and the third segment in combination with a natural language interaction and electronic display screen graphics, wherein the social interaction operation is generated based on the data recorded by the sensors and camera;

executing an instruction to photograph the one or more persons according to a first parameter and via the social interaction; and

changing the mode of operation from a first mode of operation to a second mode of operation based, at least in part, on the first parameter.

27. The computer readable medium of claim 26 wherein the first parameter is a time parameter.

28. The computer readable medium of claim 27 wherein the time parameter is selected from the group consisting of a predetermined amount of time, a rate at which photographs are to be taken, a start time and end time.

29. The computer readable medium of claim 26 wherein the at least one sensory input is selected from the group consisting of an auditory input, a visual input, a touch input and a remote device input.

30. The computer readable medium of claim 26 wherein executing an instruction to photograph the one or more persons comprises utilizing at least one of sound location and facial recognition to identify each of the one or more persons to be photographed.

31. The computer readable medium of claim 26 further comprising uploading the at least one of a photograph and a video to the cloud.

32. The computer readable medium of claim 26 further comprising enabling a user to perform an action on the at least one of a photograph and a video the action selected from the group consisting of deleting, editing and cropping.

33. The computer readable medium of claim 26 further comprising uploading the at least one of a photograph and a video to a social media web site.

34. The computer readable medium of claim 26 further comprising emailing the at least one of a photograph and a video to a social media web site.

35. The computer readable medium of claim 26 further comprising:

observing an emotional state of a person to be photographed; and

emitting, in response to the observed emotional state, a cue to provoke a preferred emotional state of the person.

36. The computer readable medium of claim 35 wherein said cue is derived, at least in part, from data received from the cloud.

37. The computer readable medium of claim 35 further comprising repeating the steps of observing and emitting until an objective is completed.

38. The computer readable medium of claim 37 wherein the objective is determined based, at least in part, on at least one of the first mode of operation and the second mode of operation.

39. The computer readable medium of claim 26 further comprising:

receiving from a user a second parameter; and

photographing the one or more persons in accordance with the second parameter.

40. The computer readable medium of claim 39 wherein the second parameter is a requirement to take at least one photograph of each of the one or more persons.

41. The computer readable medium of claim 39 further comprising broadcasting a request to at least one of the one or more persons to remain stationary.

42. The computer readable medium of claim 26 further comprising:

broadcasting a request for an identifier of at least one of the one or more persons;

receiving an identifier from the at least one of the one or more persons; and

associating the received identifier with at least one of a photograph and a video of the at least one of the one or more persons.

43. The computer readable medium of claim 26 wherein the instruction to photograph is received from the user.

44. The computer readable medium of claim 26 wherein the social interaction operation is generated based on the data recorded by the sensors and camera and data retrieved from a remote server.

45. The computer readable medium of claim 26 wherein a motor of the robotic device is mounted in an elastomeric material for absorbing noise produced by armature bearings of the motor.

46. The computer readable medium of claim 26 wherein the data recorded by the sensors is combined into a probabilistic model of physically measurable user state, using particle filtering techniques.