System, method and product for utilizing prediction models of an environment

• US 10,245,724 B2
• Filed: 06/09/2016
• Issued: 04/02/2019
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Shmuel Ur Innovations Ltd.
• Original Assignee: Shmuel Ur Innovations Ltd.
• Inventors: Ur, Shmuel, Dabija, Vlad, Hirshberg, David
• Primary Examiner(s): Do, Truc M
• Application Number:
  US15/177,411
• Publication Number:
  US 20170355078A1
• Time in Patent Office: 1,027 Days
• Field of Search: None
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/1669 characterised by special application, e.g. multi-arm co-operation, assembly, grasping
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39536 Planning of hand motion, grasping
• CPC Class Codes: G05B 2219/40425 Sensing, vision based motion planning
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A first method comprising: predicting a scene of an environment using a model of the environment and based on a first scene of the environment obtained from sensors observing scenes of the environment; comparing the predicted scene with an observed scene from the sensors; and performing an action based on differences determined between the predicted scene and the observed scene. A second method comprising applying a vibration stimuli on an object via a computer-controlled component; obtaining a plurality of images depicting the object from a same viewpoint, captured during the application of the vibration stimuli. The second method further comprising comparing the plurality of images to detect changes occurring in response to the application of the vibration stimuli, which changes are attributed to a change of a location of a boundary of the object; and determining the boundary of the object based on the comparison.

Claims

1. A method comprising:

obtaining a first scene of an environment from one or more sensors, wherein the one or more sensors observe scenes of the environment;

predicting, using a model of the environment and based on the first scene of the environment, a predicted second scene of the environment;

obtaining an observed second scene from the one or more sensors, wherein the observed second scene occurs later in time than the first scene;

wherein prior to said obtaining the observed second scene, performing computations based on the predicted second scene;

comparing the predicted second scene with the observed second scene; and

performing an action, using the computations and based on differences determined in said comparing between the predicted second scene and the observed second scene, whereby utilizing the computations based on the predicted second scene to reduce response time of performing the action in response to the observed second scene.

2. The method of claim 1,

wherein said performing the action comprises utilizing the computations, whereby processing time occurring after said obtaining the observed second scene is reduced by relying on the predicted second scene.

3. The method of claim 1, wherein the computations are computations of an initial movement plan of a robotic device, wherein the initial movement plan is based on the predicted second scene, wherein said performing comprises instructing the robotic device to move in accordance with the initial movement plan or a modification thereof, whereby the robotic device is moved.

4. The method of claim 2, wherein the computations are computations of a movement plan of a robotic device, wherein the movement plan is based on the predicted second scene, wherein said performing comprises:

in response to determining that the observed second scene is identical to the predicted second scene, performing the movement plan, whereby moving the robotic device.

5. The method of claim 1, further comprising:

wherein said performing the action comprises;

compressing the observed second scene based on the differences between the predicted second scene and the observed second scene, wherein said compressing creates a compressed representation of the observed second scene,

whereby the observed second scene is obtainable from the compressed representation of the observed second scene by a device capable of performing a prediction, using the model of the environment, based on the first scene;

whereby the first scene and the observed second scene are retainable in a computer-readable medium using a reduced amount of space in comparison to retaining the first scene and the observed second scene directly.

6. The method of claim 5, further comprises:

transmitting the first scene of the environment to a computing device;

wherein said performing the action further comprises;

transmitting the compressed representation to the computing device;

whereby the computing device is capable of obtaining the observed second scene by performing a prediction based on the first scene and using the model of the environment to obtain a predicted scene of the environment and modifying the predicted scene using the compressed representation to obtain the observed second scenes,

whereby reducing amount of bandwidth required for transmission the first scene and the second observed scene without losing information.

7. The method of claim 1, further comprises:

wherein the one or more sensors comprise a first sensor and a second sensor, wherein the first sensor is different than the second sensor;

wherein said obtaining the first scene comprises obtaining the first scene from a first sensor;

wherein said obtaining the observed second scene comprises obtaining the observed scene from a second sensor.

8. The method of claim 7, wherein the observed second scene occurs later in time than the first scene.

9. The method of claim 7, wherein the observed second scene occurs at the same time as the first scene.

10. The method of claim 7, wherein scenes of the environment observed by the first sensor and scenes of the environment observed by the second sensor demonstrate different views of a same location in the environment.

11. The method of claim 7, wherein the first sensor and the second sensor observe different locations of the environment.

12. The method of claim 1, wherein the one or more sensors comprise one or more cameras, wherein scenes obtained from the one or more sensors are images of the environment.

13. A method comprising:

obtaining a first scene of an environment from one or more sensors, wherein the one or more sensors observe scenes of the environment, wherein the environment comprises a pile of physical objects;

determining a stimuli force to be applied on a physical object of the pile of physical objects;

predicting, using a model of the pile of physical objects and based on the first scene of the environment, a predicted second scene of the environment

in response to application of the stimuli force on the physical object;

wherein after said predicting, applying the stimuli force on the physical object; and

obtaining an observed second scene from the one or more sensors after said applying;

comparing the predicted second scene with the observed second scene; and

performing an action based on differences determined in said comparing between the predicted second scene and the observed second scene, wherein said performing the action comprises updating, based on the differences between the predicted second scene and the observed second scene, the model of the pile.

14. The method of claim 13, wherein said performing the action further comprises:

based on the updated model, determining an order of picking up objects from the pile.

15. The method of claim 13 further comprises determining, based on said comparing, a mechanical coupling between two or more objects of the pile of physical objects that was not modeled in the model of the pile, wherein said updating the model comprises introducing to the model a modeling of the mechanical coupling.

16. The method of claim 15 further comprises selecting the object upon which the stimuli force is to be applied, wherein said selecting comprises selecting the object based on a determination that applying stimuli force on the object is expected to provide information useful for validating or refining the model of the pile.

17. The method of claim 13, wherein the stimuli force is applied by one or more robotic devices.

18. The method of claim 13, wherein the stimuli force is a vibration force.

19. The method of claim 18, wherein a frequency of the vibration force is below a sampling frequency of the one or more sensors.

20. The method of claim 18, wherein a frequency of the vibration force is above a sampling frequency of the one or more sensors, wherein the frequency is correlated with the sampling frequency.

21. The method of claim 13, wherein an intensity of the stimuli force is below a maximal threshold, wherein the maximal threshold is an intensity capable of permanently changing a structure of the pile.

22. The method of claim 13, wherein an intensity of the stimuli force is above a minimal threshold, wherein the minimal threshold is an intensity capable of moving the physical object in a manner detectable by the one or more sensors.