Robot apparatus, method for controlling the same, and computer program

• US 9,962,839 B2
• Filed: 11/23/2016
• Issued: 05/08/2018
• Est. Priority Date: 03/08/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Genta, Miyamoto, Atsushi, Shimizu, Satoru
• Primary Examiner(s): Tran, Dalena
• Application Number:
  US15/359,939
• Publication Number:
  US 20170072569A1
• Time in Patent Office: 531 Days
• Field of Search: 700245, 700257-259, 382103, 382153-155, 382159, 382173-174, 382190, 348169, 901 47, 901 1- 4, 901 14, 901 30, 901 31, 901 46
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/37555 Camera detects orientation, position workpiece, points of workpiece
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G05B 2219/40099 Graphical user interface for robotics, visual robot user interface
• CPC Class Codes: G05B 2219/40298 Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/45108 Aid, robot for aid to, assist human disabled

Abstract

A robot apparatus includes an output unit that displays an image including an object on a screen, an input unit that receives an operation performed by a user for specifying information relating to an approximate range including the object in the image, an object extraction unit that extracts information regarding a two-dimensional contour of the object on the basis of the specification received by the input unit, and a position and attitude estimation unit that estimates information regarding a three-dimensional position and attitude of the object on the basis of the information regarding the two-dimensional contour.

Claims

1. A control device of a movable apparatus comprising:

at least one processor; and

at least one computer-readable medium having stored therein instructions that, when executed by the at least one processor, causes the at least one processor to perform;

extracting two-dimensional feature information regarding an object in an image captured by a camera attached to the movable apparatus;

calculating information regarding a three-dimensional position of the object, the three-dimensional position of the object being a position of the object in a world coordinate system, wherein calculating the information regarding the three-dimensional position of the object is performed based on a position of the object within the image captured by the camera attached to the movable apparatus and position information of the camera or the movable apparatus in the world coordinate system; and

controlling the movable apparatus based on the calculated information regarding the three-dimensional position of the object.

2. The control device of the movable apparatus according to claim 1, wherein calculating the information regarding the three-dimensional position of the object comprises calculating the information regarding the three-dimensional position of the object based on a specification of the object in the image received by an input interface.

3. The control device of the movable apparatus according to claim 2, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform:

causing a display to display the image.

4. The control device of the movable apparatus according to claim 3, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform:

causing the display to display annotation information related to the two-dimensional feature information on the displayed image.

5. The control device of the movable apparatus according to claim 3, wherein the display is integrated with an input interface comprising a touch panel, and the specification of the object comprises information based on touch input on the touch panel.

6. The control device for a movable apparatus according to claim 5, wherein controlling the movable apparatus comprises controlling at least the movable apparatus and the camera based on the touch input of a position of the displayed annotation information on the touch panel.

7. The control device of the movable apparatus according to claim 2, wherein calculating the information regarding the three-dimensional position of the object comprises receiving a specification of an approximate range including the object.

8. The control device of the movable apparatus according to claim 2, wherein:

extracting the two-dimensional feature information regarding the object is based on the specification received by the input interface.

9. The control device of the movable apparatus according to claim 2, wherein:

the instructions, when executed by the at least one processor, further cause the at least one processor to receive an instruction of an action from the input interface to cause on the object, and

controlling the movable apparatus comprises controlling the movable apparatus based on the instruction to cause the action on the object.

10. The control device of the movable apparatus according to claim 9, wherein the action on the object is an action of holding the object.

11. The control device of the movable apparatus according to claim 1, wherein controlling the movable apparatus comprises planning a movement of a movable part of the movable apparatus based on the calculated information regarding the three-dimensional position of the object.

12. The control device of the movable apparatus according to claim 1, wherein calculating the information regarding the three-dimensional position of the object comprises calculating an attitude of the object, and controlling the movable apparatus comprises controlling the movable apparatus based on the calculated information regarding the three-dimensional position of the object and the calculated attitude of the object.

13. The control device of the movable apparatus according to claim 12, wherein controlling the movable apparatus comprises planning a movement of a movable part of the movable apparatus based on the calculated information regarding the three-dimensional position of the object and the calculated attitude of the object.

14. The control device of the movable apparatus according to claim 1, wherein the camera comprises a stereo camera configured to capture a plurality of images from a plurality of points, and the three-dimensional position in the world coordinate system of the camera is determined based on the plurality of images captured by the stereo camera.

15. A control system comprising:

the control device of the movable apparatus according to claim 1;

the movable apparatus; and

an input interface,

wherein the movable apparatus comprises a movable part, the camera, and a control circuit.

16. The control system of the movable apparatus according to claim 15, wherein the movable part comprises a rotating actuator and is configured to provide driving force to the movable apparatus.

17. The control system of the movable apparatus according to claim 16, wherein the movable part comprises a drive wheel.

18. The control system of the movable apparatus according to claim 15, wherein the control circuit is configured to control the movable part without receiving a stored model from a database.

19. The control system of the movable apparatus according to claim 1, the control device further comprising:

a first communication unit configured to receive the image from the camera wirelessly;

a second communication unit configured to transmit control communication to the movable apparatus wirelessly; and

a third communication unit configured to transmit display information to the display.

20. A robot control device comprising:

at least one processor; and

at least one non-transitory computer-readable medium having stored therein instructions that, when executed by the at least one processor, cause the at least one processor to;

extract two-dimensional feature information regarding an object in an image captured by a sensor attached to a movable robot;

estimate information regarding a three-dimensional position of the object, the three-dimensional position of the object being a position of the object in a world coordinate system, wherein estimating the information regarding the three-dimensional position of the object is performed based on position information in a coordinate system of the sensor and position information of the sensor in the world coordinate system; and

control the movable robot based on the estimated position of the object.