Apparatus and methods for context determination using real time sensor data

• US 9,821,470 B2
• Filed: 09/17/2014
• Issued: 11/21/2017
• Est. Priority Date: 09/17/2014
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Laurent, Patryk, Petre, Csaba, Izhikevich, Eugene M.
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US14/489,368
• Publication Number:
  US 20160075016A1
• Time in Patent Office: 1,161 Days
• Field of Search: 700245
• CPC Class Codes: B25J 13/006 by means of a wireless system for controlling one or several manipulators
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: Y10S 901/03 Teaching system

Abstract

Computerized appliances may be operated by users remotely. In one exemplary implementation, a learning controller apparatus may be operated to determine association between a user indication and an action by the appliance. The user indications, e.g., gestures, posture changes, audio signals may trigger an event associated with the controller. The event may be linked to a plurality of instructions configured to communicate a command to the appliance. The learning apparatus may receive sensory input conveying information about robot'"'"'s state and environment (context). The sensory input may be used to determine the user indications. During operation, upon determine the indication using sensory input, the controller may cause execution of the respective instructions in order to trigger action by the appliance. Device animation methodology may enable users to operate computerized appliances using gestures, voice commands, posture changes, and/or other customized control elements.

Claims

1. A computerized method for providing a remote control command to a computerized device based on a sequence of digital images, the method comprising:

determining a discrepancy measure based on a comparison of pixels of a current image of the sequence of digital images to a reference image;

determining a salient feature based on an analysis of the discrepancy measure, the salient feature being associated with a portion of the pixels within the current image; and

based on an existence of a previously established association between an occurrence of a user indication associated with an action by the computerized device and the salient feature, automatically transmitting a command to the computerized device, the command configured to cause the computerized device to execute the action;

wherein;

the salient feature comprises a representation of a user body portion;

the reference image comprises an image acquired prior to the current image without the representation of the user body portion; and

the reference image is based on a low pass filter operation on a plurality of images from the sequence of digital images, where the individual ones of the plurality of images precede the current image.

2. The computerized method of claim 1, wherein:

the determining the discrepancy measure comprises determining a difference image based on a pixel wise difference operation between the current image and the reference image, and the difference image comprises one or more background pixel values and two or more areas of pixels having values different from the one or more background pixel values; and

the analysis of the discrepancy measure comprises selecting one area from the two or more areas with a winner takes all process.

3. The computerized method of claim 2, wherein the winner takes all process is configured based on determining, for a given area of the two or more areas, one or more of:

a total number of pixels within the given area, a sum of values of pixels within the given area, and a sum of deviations from a reference pixel value of a plurality of values of pixels within the given area.

4. The computerized method of claim 1, further comprising receiving the sequence of digital images from a camera that is remote from the computerized device.

5. The computerized method of claim 4, further comprising:

storing information related to one or more user indications, each of the one or more user indications corresponding to a state;

wherein

the command is associated with the one or more user indications; and

the automatically transmitting the command is based on an occurrence of the state.

6. The computerized method of claim 5, wherein the state comprises a position of the computerized device in an environment of the computerized device.

7. The computerized method of claim 1, further comprising:

characterizing an environment external to the computerized device; and

conveying one or more of a position, a gesture, or a movement of a person via a parameter.

8. The computerized method of claim 1, further comprising:

acquiring the reference image prior to the current image without the representation of the user body portion; and

wherein the salient feature comprises a representation of a user body portion.

9. The computerized method of claim 1, wherein the automatically transmitting the command comprises transmitting via an infrared signal.

10. The computerized method of claim 1, wherein the automatically transmitting the command comprises transmitting via a radio frequency signal.

11. The computerized method of claim 1, further comprising:

executing the action within an environment comprising one or more objects characterized by one or more corresponding object motion parameters;

the executed action is characterized by a motion parameter of a robotic device; and

the executed action is configured based on a modification of the motion parameter based on a value associated with a corresponding one of the one or more object motion parameters.

12. The computerized method of claim 1, wherein:

the computerized device comprises a household appliance configured to perform a cleaning task of a user premises; and

a state comprises information related to the user premises.

13. The computerized method of claim 12, further comprising:

loading another set of instructions which were previously configured to cause execution of another task by another computerized device;

wherein the loading the another set of instructions is triggered automatically by the computerized device based on a characteristic of the premises.

14. A computerized method for providing a remote control command to a computerized device based on a sequence of digital images, the method comprising:

determining a discrepancy measure based on a comparison of pixels of a current image of the sequence of digital images to a reference image;

determining a salient feature based on an analysis of the discrepancy measure, the salient feature being associated with a portion of the pixels within the current image;

based on an existence of a previously established association between an occurrence of a user indication associated with an action by the computerized device and the salient feature, automatically transmitting a command to the computerized device, the command configured to cause the computerized device to execute the action; and

loading another set of instructions which were previously configured to cause execution of another task by another computerized device;

wherein;

the computerized device comprises a household appliance configured to perform a cleaning task of a user premises;

a state comprises information related to the user premises; and

the loading the another set of instructions is triggered automatically by the computerized device based on a characteristic of the user premises.

15. The computerized method of claim 14, wherein the reference image is based on a low pass filter operation on a plurality of images from the sequence of digital images, where the individual ones of the plurality of images precede the current image.

16. The computerized method of claim 15, wherein:

the determining the discrepancy measure comprises determining a difference image based on a pixel wise difference operation between the current image and the reference image, and the difference image comprises one or more background pixel values and two or more areas of pixels having values different from the one or more background pixel values; and

the analysis of the discrepancy measure comprises selecting one area from the two or more areas with a winner takes all process configured to select one area from the two or more areas.

17. The computerized method of claim 16, wherein the winner takes all process is configured based on determining, for a given area of the two or more areas, one or more of:

a total number of pixels within the given area, a sum of values of pixels within the given area, and a sum of deviations from a reference pixel value of a plurality of values of pixels within the given area.

18. The computerized method of claim 14, further comprising receiving the sequence of digital images from a camera that is remote from the computerized device.

19. The computerized method of claim 14, further comprising:

characterizing an environment external to the computerized device; and

conveying one or more of a position, a gesture, or a movement of a person via a parameter.

20. The computerized method of claim 14, further comprising:

acquiring the reference image prior to the current image without the representation of the user body portion; and

the salient feature comprises a representation of a user body portion.

21. The computerized method of claim 14, wherein:

the computerized device comprises a household appliance configured to perform a cleaning task of a user premises; and

a state comprises information related to the user premises.

22. A non-transitory computer-readable storage medium having instructions embodied thereon to determine an association between a sensory context and an action indication for an appliance device, the instructions when executed by a processing apparatus cause the processing apparatus to:

determine a first sensory context version based on a first sensory modality and a high pass filter operation version using the first sensory modality, the high pass filter operation characterized by a decay time scale;

determine a second sensory context version based on a second sensory modality;

when the second sensory context version occurs within a first time window from occurrence of the first sensory context version, assign the first sensory context version and the second context version as the sensory context; and

associate the sensory context with the action indication based on occurrence of the action indication within a second time window from at least one of the first sensory context version or the second sensory context version;

wherein the association between the sensory context and the action indication for the appliance device is configured to enable automatic provision of a command to the appliance device based on an occurrence of the sensory context, the command configured to cause the appliance device to execute the action; and

wherein one or more data of the first sensory modality comprises a sequence of images, and the decay time scale is at least five times longer than the second time window.

23. The non-transitory computer-readable storage medium of claim 22, where the instructions further cause the processing apparatus to:

determine a first sensory context version comprising a transformation of individual ones of a sequence of input images to transformed images, the individual ones of the transformed images characterized by a first data rate that is at least ten times lower than a second data rate of respective input images; and

where one or more data of the first sensory modality comprises a sequence of input images provided by a video camera.

24. A non-transitory computer-readable storage medium having instructions embodied thereon to determine an association between a sensory context and an action indication for an appliance device, the instructions when executed by a processing apparatus cause the processing apparatus to:

determine a first sensory context version based on a first sensory modality, the first sensory context version comprising a transformation of individual ones of a sequence of input images to transformed images, the individual ones of the transformed images characterized by a first data rate that is at least ten times lower than a second data rate of respective input images;

determine a second sensory context version based on a second sensory modality;

when the second sensory context version occurs within a first time window from occurrence of the first sensory context version, assign the first sensory context version and the second context version as the sensory context; and

associate the sensory context with the action indication based on occurrence of the action indication within a second time window from at least one of the first sensory context version or the second sensory context version;

wherein the association between the sensory context and the action indication for the appliance device is configured to enable automatic provision of a command to the appliance device based on an occurrence of the sensory context, the command configured to cause the appliance device to execute the action;

wherein one or more data of the first sensory modality comprises a sequence of input images provided by a video camera.

25. The non-transitory computer-readable storage medium of claim 24, where the instructions further cause the processing apparatus to:

determine the first sensory context based on a high pass filter operation version using a first sensory modality, the high pass filter operation characterized by a decay time scale;

where one or more data of the first sensory modality comprises a sequence of images; and

the decay time scale is at least five times longer than the second time window.