Selection of a device or object using a camera

• US 10,095,216 B2
• Filed: 05/23/2016
• Issued: 10/09/2018
• Est. Priority Date: 05/29/2015
• Status: Active Grant

• Current Assignee: KUKA Roboter GmbH (Midea Group Co., Ltd)
• Original Assignee: KUKA Roboter GmbH (Midea Group Co., Ltd)
• Inventors: Hietmann, Gerhard
• Primary Examiner(s): Tran, Khoi H
• Assistant Examiner(s): Peche, Jorge O
• Application Number:
  US15/162,056
• Publication Number:
  US 20160346936A1
• Time in Patent Office: 869 Days
• CPC Class Codes: G05B 19/409 characterised by using manual data input [MDI] or by using control panel, e.g. controlling functions with the panel; characterised by control panel details or by setting parameters G05B19/408, G05B19/4093 take precedence
• CPC Class Codes: G05B 2219/39444 Display of position, of shape of robot and tool
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/02 Arm motion controller
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A method for actuating a manipulator system by means of a portable end device, a corresponding portable end device, and a robot system. The portable end device includes a 3-D camera that is a plenoptic camera. A three-dimensional image is acquired by means of the 3-D camera of the portable end device of at least part the manipulator system to be actuated. The image identifies at least one actuatable object in the manipulator system. A task is selected for the identified object to actuate. By selecting the identified actuatable object, the object is actuated to perform the task.

Claims

1. A method for actuating objects in a manipulator system by means of a portable end device, wherein the portable end device includes a camera, a device control unit and a rendering device, wherein the device control unit performs the steps of:

acquiring an image, by means of the camera in the portable end device, imaging at least a portion of the manipulator system;

obtaining access to reference data sets for at least the manipulator system wherein reference data sets include camera image data of at least the portion of the manipulator system;

identifying at least one feature of the manipulator system by performing an image recognition process using the camera image and the reference data sets, wherein the at least one feature comprises a visible aspect of the manipulator system that is apparent in the acquired camera image and characteristic of the manipulator system or an actuatable object of the manipulator system;

classifying the imaged manipulator system based on the at least one characteristic feature;

displaying the acquired camera image of the manipulator system on the rendering device of the portable end device;

identifying the at least one actuatable object of the manipulator system based on the at least one characteristic feature and the reference data sets for the identified manipulator system;

rendering an optical marking the camera image diplayed on the rendering device of the portable end device, the optical marking indicating the at least ;

one actuatable object of the manipulator system; and

actuating the actuatable object of the manipulator system upon receiving an input via a user interface of the portable end device to select the optical marking.

2. The method in accordance with claim 1, wherein the camera is a 3-D camera and the acquired image is a three-dimensional image.

3. The method in accordance with claim 2, wherein the image includes 3-D image data and the feature comprises at least a portion of a physical component of the manipulator that is visible in the 3-D image.

4. The method in accordance with claim 3, wherein the physical component is a gripper.

5. The method in accordance with claim 1, wherein the reference data sets for the manipulator identify and include camera image data of the at least one actuatable object.

6. The method in accordance with claim 1, wherein the rendering device includes a touch panel.

7. The method in accordance with claim 6, further including the step of rendering of context-dependent data, taking into account the determined feature, based on operator input.

8. The method in accordance with claim 7, wherein the operator assistance comprises a selection of one of a start-up mode or a programming mode of the portable end device.

9. The method in accordance with claim 1, wherein:

at least two actuatable objects of the manipulator system are identified and at least two optical markings are rendered on the camera image displayed on rendering device of the portable end device,

each optical marking specifies one of the identified objects of the manipulator system, and

wherein the actuating step comprises the step of selecting at least one of the optical markings specifying one of the identified objects rendered on the rendering device.

10. The method in accordance with claim 1, wherein the step of actuating the actuatable object of the manipulator system includes a step of querying, providing, creating, or modifying data of the manipulator system, which data describe at least one of the following:

adjusting or orienting a manipulator;

parameterizing an end effector;

calibrating the manipulator and/or the end effector;

defining a work area;

parameterizing the object of the manipulator system;

defining a coordinate system;

or

a program sequence.

11. The method in accordance with claim 1, wherein the step of actuating the actuatable object of the manipulator system includes a step of creating, modifying, or and/or executing a function or a program directed to at least one of the following:

action of an end effector;

moving the object of the manipulator system;

controlling the object of the manipulator system;

moving a manipulator;

controlling the manipulators;

or

changing a program sequence.

12. The method in accordance with claim 1, wherein the camera is a plenoptic camera, and wherein the acquisition of the image includes acquisition of a four-dimensional light field, by means of the plenoptic camera, of the at least one object of the manipulator system to he actuated.

13. A portable end device including a camera, a rendering device and a device control unit, wherein the device control unit is configured to:

acquire an image, by means of the camera in the portable end device, of at least a portion of a manipulator system;

obtain access to reference data sets for the manipulator system, wherein the reference data sets include camera image data of at least the portion of the manipulator system;

determine a feature of the manipulator system by performing an image recognition process using the camera image and the reference data sets, wherein the at least one feature comprises a visible aspect of the manipulator system that is apparent in the acquired camera image and characteristic of the manipulator system or an actuatable object of the manipulator system;

classifying the imaged manipulator system based on the at least one characteristic feature;

display the acquired camera image of the manipulator system on the rendering device of the portable end device;

identify the at least one actuatable object of the manipulator system based on the at least one characteristic feature and the reference data sets for the manipulator system;

render an optical marking on the camera image diplayed on the rendering device of the portable end device, the optical marking indicating the at least one actuatable object of the manipulator system; and

actuate the actuatable object of the manipulator system upon receiving an input via a user interface of the portable end device to select the optical marking.

14. The portable end device in accordance with claim 13, wherein the camera is a 3-D camera.

15. The portable end device in accordance with claim 14, wherein the camera is a plenoptic camera.

16. The portable end device in accordance with claim 14, wherein the image includes 3-D image data and the feature comprises at least a portion of a physical component of the manipulator that is visible in the 3-D image.

17. A manipulator system, including a manipulator and a portable end device that includes a camera and a rendering device, wherein the manipulator system furthermore has a manipulator control unit that is configured to:

acquire an image, by means of the camera in the portable end device, of at least a portion of the manipulator system;

obtain access to reference data sets for the manipulator system, wherein the reference data sets include camera image data of at least the portion of the manipulator system;

identify at least one feature of the manipulator system by performing an image recognition process using the camera image and the reference data sets, wherein the at least one feature comprises a visible aspect of the manipulator system that is apparent in the acquired camera image and characteristic of the manipulator system or an actuatable object of the manipulator system;

classifying the imaged manipulator system based on the at least one characteristic feature;

display the acquired camera image of the manipulator system on the rendering device of the portable end device;

identify the at least one actuatable object of the manipulator system based on the at least one characteristic feature and the reference data sets for the manipulator system;

render an optical marking on the camera image diplayed on the rendering device of the portable end device, the optical marking indicating the at least one actuatable object of the manipulator system; and

actuate the actuatable object of the manipulator system upon receiving an input via a user interface of the portable end device to select the optical marking.

18. The manipulator system in accordance with claim 17, wherein the camera is a 3-D camera.

19. The manipulator system in accordance with claim 18, wherein the camera is a plenoptic camera.

20. The manipulator system in accordance with claim 18, wherein the image includes 3-D image data and the feature comprises at least a portion of a physical component of the manipulator that is visible in the 3-D image.