APPARATUS AND METHODS FOR TRAINING ROBOTS UTILIZING GAZE-BASED SALIENCY MAPS

• US 20150339589A1
• Filed: 05/21/2014
• Published: 11/26/2015
• Est. Priority Date: 05/21/2014
• Status: Abandoned Application

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Fisher, Dimitry
• Application Number:
  US14/284,120
• Publication Number:
  US 20150339589A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/16 Programme controls programme controls in general G05B19/00, e.g. numerical pogramme controls G05B19/18; recording or playback systems G05B19/42
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: G05B 2219/33034 Online learning, training
• CPC Class Codes: G05B 2219/36039 Learning task dynamics, process
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 99/00 Subject matter not provided for in other groups of this subclass
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

Robotic devices may be trained using saliency maps derived from gaze of a trainer. In navigation applications, the saliency map may correspond to portions of the environment being observed by a driving instructor during training using a gaze detector. During an operation, a driver assist robot may utilize the saliency map in order to assess attention of the driver, detect potential hazards, and issue alerts. Responsive to a detection of a mismatch between the driver current attention and the target attention derived from the saliency map, the robot may issue a warning, and/or prompt the driver of an upcoming hazard. A data processing apparatus may employ gaze based saliency maps in order to analyze, e.g., surveillance camera feeds for intruders, open doors, hazards, policy violations (e.g., open doors).

Claims

1. A system configured for determining a saliency map, the system comprising:

a first sensing apparatus configured to provide sensory input associated with a task being executed by a robotic device operable by a trainer;

a second sensing apparatus configured to provide information related to a gaze parameter associated with a present gaze of the trainer;

one or more processors communicatively coupled with one or both of the first sensing apparatus or the second sensing apparatus, the one or more processors being configured to execute computer program instructions to cause the one or more processors to;

determine one or more features within the sensory input using an adaptive process;

determine a salient area within the sensory input based on the gaze parameter;

associate the salient area with at least one of the one or more features; and

update a learning parameter of the process based on an evaluation of the association;

wherein;

the learning process is characterized by a performance measure;

the update is configured to effectuate autonomous execution of the task by the robotic device in an absence of the trainer; and

the saliency map comprises the salient area.

2. The system of claim 1, wherein:

the present gaze is configured to convey information related to direction of eye sight of the trainer;

the sensory input comprises a first image and a second image both conveying information related to an environment surrounding the robotic device during execution of the task; and

the gaze parameter is determined based on an operation configured using to a first portion within the first image and a second portion of the second image being gazed at by the trainer.

3. The system of claim 2, wherein the operation comprises a weighted average of the first portion and the second portion.

4. The system of claim 1, wherein:

the sensory input comprises an image characterized by a spatial extent, the image conveying information related to an environment surrounding the robotic device during execution of the task;

the present gaze of the trainer is characterized by a plurality of areas within the spatial extent being observed by the trainer, a given area within the spatial extent being characterized by a duration of the present gaze directed to the given area, a location of the given area within the spatial extent, and a perimeter of the given area; and

the gaze parameter is determined based on a spatial average of the individual areas.

5. The system of claim 4, wherein:

the sensory input comprises another image conveying information related to the environment surrounding the robotic device during execution of the task; and

the gaze parameter is determined based on a temporal average of the individual areas associated with the image and the other image.

6. The system of claim 4, wherein:

the association of the salient area with the at least one of the one or more features comprises determining a first location within the image associated with the salient area and a second location within the image associated with the at least one of the one or more features; and

the evaluation comprises a determination of a similarity measure between the first location and the second location.

7. The system of claim 6, wherein:

the one or more processors are configured to operate a network of a plurality of computerized neurons configured to implement the learning process; and

the network comprises an input layer of neurons and an output layer of neurons.

8. The system of claim 7, wherein:

the similarity measure is configured to provide a discrepancy between the first location and the second location; and

the update is configured based on propagation of the discrepancy from the output layer back to the input layer.

9. The system of claim 1, further comprising:

a nonvolatile storage medium configured to store the updated learning parameter;

wherein the second sensing apparatus comprises;

an optical gaze tracker comprising a transmitter element configured to illuminate an eye of the trainer; and

a receiver element configured to detect a waveform reflected by the eye.

10. A non-transient computer-readable storage medium having instructions embodied thereon, the instructions being executable to cause one or more processors to:

determine of a gaze of a person executing a task;

determine one or more features in sensory input associated with the task;

select a salient feature from the one or more features, the selection being based on an operation of a predictor process characterized by a parameter;

associate an area of the gaze of the person with a portion of the sensory input; and

provide an indication to the person, the indication conveying information associated with the salient feature and the area;

wherein the parameter is based on an evaluation of gaze of another person during a prior execution of the task prior.

11. The apparatus of claim 10, wherein the indication comprises an alert for the person, the alert being responsive to a discrepancy between (i) an area of the sensory input associated with the salient feature and (ii) the area of the gaze, the alarm being configured to attract attention of the person to the discrepancy.

12. The apparatus of claim 11, wherein the alarm comprises one or more of an audible indication, a visible indication, or tactile indication.

13. The apparatus of claim 11, wherein:

the task comprises navigating a trajectory by a vehicle;

the alarm is configured to indicate to the person the area of the sensory input associated with the salient feature; and

the alarm is configured to cause generation of a graphical user interface element on a display component of the vehicle, the display component configured to present to the person at least a portion of the sensory input.

14. The apparatus of claim 13, wherein:

the silent feature comprises an object disposed proximate the trajectory; and

the graphical user interface element conveys one or more of a location of the object or a boundary of the object.

15. The apparatus of claim 10, wherein

the salient feature is determined based on determining a salient area within the sensory input;

and

the indication comprises an alert for the person, the alert being responsive to an absence of the gaze within the salient area for a period of time.

16. The apparatus of claim 15, wherein:

the task comprises navigating a trajectory by a vehicle;

the sensory input comprises a sequence of frames obtained at an inter frame duration; and

the interval comprises a period of multiple inter-frame duration.

17. The apparatus of claim 16, wherein:

for an inter frame duration of 40 milliseconds, the interval is selected to be greater than 400 milliseconds.

18. A method for operating a robotic apparatus to perform a task, the method comprising:

for a given visual scene;

determining a feature within a portion of a digital image of the visual scene, the determination being based on an analysis of a saliency map associated with the task, the saliency map being representative of one or more areas of preferential attention by a human trainer; and

executing the task based on an association between with the feature and the task;

wherein;

the saliency map is determined by a learning process of the robotic apparatus;

the association between with the feature and the task is determined by the learning process;

the learning process has been previously trained to execute the task using gaze of the human trainer.

19. The method of claim 18, further comprising:

using the saliency map, as determined from the human gaze, to specify the feature associated with the robotic apparatus so that the robotic apparatus learns the association between the feature and the task.