System and method for controlling master and slave manipulator

• US 6,470,236 B2
• Filed: 12/18/2001
• Issued: 10/22/2002
• Est. Priority Date: 12/19/2000
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ohtsuki, Tomoyuki
• Primary Examiner(s): Cuchlinski, Jr., William A.
• Assistant Examiner(s): Marc, McDieunel, Marc, McDieunel
• Application Number:
  US10/023,631
• Publication Number:
  US 20020123825A1
• Time in Patent Office: 308 Days
• Field of Search: 700/245, 700/247, 700/251, 700/259, 700/257, 700/264, 700/258, 318/568.11, 600/102, 600/101, 600/117, 600/126, 600/109, 600/118, 600/429, 600/595, 600/407, 600/427, 600/474, 600/476, 606/1, 606/16, 606/130, 606/139, 606/19, 606/46, 606/167, 606/564, 606/205, 345/157, 414/2, 414/1, 701/23, 128/898, 901/8
• US Class Current: 700/247
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/40144 Force sensation feedback from slave
• CPC Class Codes: G05B 2219/40169 Display of actual situation at the remote site
• CPC Class Codes: G05B 2219/40405 Master slave position control
• CPC Class Codes: G05B 2219/45123 Electrogoniometer, neuronavigator, medical robot used by surgeon to operate

Abstract

In a system and a method for remotely controlling a slave manipulator easily and highly accurately, operations of three master manipulators are consolidated, and one slave manipulator is remotely controlled in accordance with the consolidation result, thereby ensuring that the slave manipulator is moved along a target path.

Claims

1. A manipulator system comprising a first master manipulator including a first handling part handled by a first operator, a second master manipulator including a second handling part handled by a second operator, and a slave manipulator including a treating part for treating an object,

the first master manipulator comprising:

first detection means for detecting an absolute attitude, in a space within which the first handling part is allowed to move, of the first handling part; and

first transmission means for transmitting the absolute attitude of the first handling part;

the second master manipulator comprising;

second detection means for detecting an absolute attitude, in a space within which the second handling part is allowed to move, of the second handling part; and

second transmission means for transmitting the absolute attitude of the second handling part;

the slave manipulator comprising;

first acquisition means for acquiring the absolute attitude of the first handling part transmitted from the first transmission means of the first master manipulator, and the absolute attitude of the second handling part transmitted from the second transmission means of the first master manipulator;

first consolidation means for consolidating the absolute attitude of the first handling part and the absolute attitude of the second handling part; and

first control means for controlling the attitude of the treating part in accordance with the result of the consolidation performed by the first consolidation means.

2. A manipulator system according to claim 1, wherein the absolute attitude of the first handling part is the position and the state, of the first handling part, in a space within which the first handling part is allowed to move, and

the absolute attitude of the second handling part is the position and the state, of the second handling part, in a space within which the second handling part is allowed to move.

3. A manipulator system according to claim 1, wherein the first consolidation means of the slave manipulator calculates the weighted sum of the absolute attitude of the first handling part and the absolute attitude of the second handling part, using predetermined weighting factors for respective terms.

4. A manipulator system according to claim 1, wherein

the save manipulator further comprises:

third detection means for detecting a first force or a first torque applied to the treating part from the object being treated; and

third transmission means for transmitting the first force or the first torque,

the first master manipulator further comprises;

second acquisition means for acquiring the first force or the first torque transmitted from the third transmission means of the slave manipulator;

first determination means for determining a second force or a second torque to be perceived by the first operator, in accordance with the first force or the first torque acquired by the second acquisition means; and

second control means for controlling the first handling part so that the first operator perceives the second force or the second torque, and

the second master manipulator further comprises;

third acquisition means for acquiring the first force or the first torque transmitted from the third transmission means of the slave manipulator;

second determination means for determining a third force or a third torque to be perceived by the second operator, in accordance with the first force or the first torque acquired by the third acquisition means; and

third control means for controlling the second handling part so that the second operator perceives the third force or the third torque.

5. A manipulator system according to claim 1, wherein

the slave manipulator further comprises third transmission means for transmitting the result of the consolidation performed by the first consolidation means,

the first master manipulator further comprises:

second acquisition means for acquiring the result of the consolidation transmitted from the third transmission means of the slave manipulator;

first calculation means for calculating the difference between the absolute attitude of the first handling part and the result of the consolidation acquired by the second acquisition means;

first determination means for determining a force or a torque to be perceived by the first operator, in accordance with the result of the calculation performed by the first calculation means; and

second control means for controlling the first handling part so that the first operator perceives the force or the torque determined by the first determination means,

the second master manipulator further comprises;

third acquisition means for acquiring the result of the consolidation transmitted from the third transmission means of the slave manipulator;

second calculation means for calculating the difference between the absolute attitude of the second handling part and the result of the consolidation acquired by the third acquisition means;

second determination means for determining a force or a torque to be perceived by the second operator, in accordance with the result of the calculation performed by the second calculation means; and

third control means for controlling the second handling part so that the second operator perceives the force or the torque determined by the second determination means.

6. A manipulator system according to claim 1, wherein

the slave manipulator further comprises:

third detection means for detecting a first force or a first torque applied to the treating part from the object being treated; and

third transmission means for transmitting the first force or the first torque and the result of the consolidation performed by the first consolidation means,

the first master manipulator further comprises;

second acquisition means for acquiring the first force or the first torque and the result of the consolidation transmitted from the third transmission means of the slave manipulator;

first calculation means for calculating the difference between the absolute attitude of the first handling part and the result of the consolidation acquired by the second acquisition means;

first determination means for determining a second force or a second torque to be perceived by the first operator, in accordance with the first force or the first torque acquired by the second acquisition means and in accordance with the difference calculated by the first calculation means; and

second control means for controlling the first handling part so that the first operator perceives the second force or the second torque,

the second master manipulator further comprises;

third acquisition means for acquiring the first force or the first torque and the result of the consolidation transmitted from the third transmission means of the slave manipulator;

second calculation means for calculating the difference between the absolute attitude of the second handling part and the result of the consolidation acquired by the third acquisition means;

second determination means for determining a third force or a third torque to be perceived by the second operator, in accordance with the first force or the first torque acquired by the third acquisition means and in accordance with the difference calculated by the second calculation means; and

third control means for controlling the second handling part so that the second operator perceives the third force or the third torque.

7. A manipulator system according to claim 1, wherein

the first master manipulator further comprises first display control means for controlling a cue so that the first operator can operate the first handling part in synchronization with the operation of the second operator on the second handling part in accordance with the cue, and

the second master manipulator further comprises second display control means for controlling the cue so that the second operator can operate the second handling part in synchronization with the operation of the first operator on the first handling part in accordance with the cue.

8. A manipulator system according to claim 1, wherein

the first master manipulator further comprises first output control means for controlling a sound/voice cue so that the first operator can operate the first handling part in synchronization with the operation of the second operator on the second handling part in accordance with the sound/voice cue, and

the second master manipulator further comprises second output control means for controlling the sound/voice cue so that the second operator can operate the second handling part in synchronization with the operation of the first operator on the first handling part in accordance with the sound/voice cue.

9. A manipulator system according to claim 1, further comprising a third master manipulator including a third handling part handled by a third operator, wherein

the third master manipulator further includes:

third detection means for detecting a relative attitude of the third handling part with respect to a predetermined reference attitude; and

third transmission means for transmitting the relative attitude of the third handling part detected by the third detection means,

the first acquisition means of the slave manipulator further acquires the relative attitude of the third handling part transmitted from the third transmission means of the third master manipulator, and

the first consolidation means consolidates the absolute attitude of the first handling part, the absolute attitude of the second handling part, and the relative attitude of the third handling part.

10. A manipulator system according to claim 9, wherein the first consolidation means of the slave manipulator calculates the weighted sum of the absolute attitude of the first handling part, the absolute attitude of the second handling part, and the relative attitude of the third handling part, using predetermined weighting factors for respective terms.

11. A manipulator system according to claim 9, wherein

the slave manipulator further comprises:

fourth detection means for detecting a first force or a first torque applied to the treating part from the object being treated; and

fourth transmission means for transmitting the first force or the first torque,

the first manipulator further comprises;

second acquisition means for acquiring the first force or the first torque transmitted from the fourth transmission means of the slave manipulator;

third acquisition means for acquiring the absolute attitude of the second handling part transmitted of the second transmission means of the second master manipulator;

second consolidation means for consolidating the absolute attitude of the first handling part and the absolute attitude of the second handling part;

first calculation means for calculating the difference between the absolute attitude of the first handling part and the result of the consolidation performed by the second consolidation means;

first determination means for determining a second force or a second torque to be perceived by the first operator, in accordance with the first force or the first torque acquired by the second acquisition means and in accordance with the difference calculated by the first calculation means; and

second control means for controlling the first handling part so that the first operator perceives the second force or the second torque,

the second master manipulator further comprises;

fourth acquisition means for acquiring the first force or the first torque transmitted from the fourth transmission means of the slave manipulator;

fifth acquisition means for acquiring the absolute attitude of the first handling part transmitted of the first transmission means of the first master manipulator;

third consolidation means for consolidating the absolute attitude of the second handling part and the absolute attitude of the first handling part;

second calculation means for calculating the difference between the absolute attitude of the second handling part and the result of the consolidation performed by the third consolidation means;

second determination means for determining a third force or a third torque to be perceived by the second operator, in accordance with the first force or the first torque acquired by the fourth acquisition means and in accordance with the difference calculated by the second calculation means; and

third control means for controlling the second handling part so that the second operator perceives the third force or the third torque, and

the third master manipulator further comprises;

sixth acquisition means for acquiring the first force or the first torque transmitted from the fourth transmission means of the slave manipulator;

third determination means for determining a fourth force or a fourth torque to be perceived by the third operator, in accordance with the first force or the first torque acquired by the sixth acquisition means and in accordance with the relative attitude of the third handling part; and

fourth control means for controlling the third handling part so that the third operator perceives the fourth force or the fourth torque.

12. A manipulator system according to claim 11, wherein

the first master manipulator further comprises seventh acquisition means for acquiring the relative attitude of the third handling part transmitted from the third transmission means of the third master manipulator,

the first determination means of the first master manipulator determines the second force or the second torque such that if a value corresponding to the relative attitude of the third handling part is smaller than a predetermined threshold value, the second force or the second torque is determined in accordance with the first force or the first torque and the result of the consolidation performed by the second consolidation means, however if the value corresponding to the relative attitude of the third handling part is equal to or greater than the predetermined threshold value, the second force or the second torque is determined in accordance with only the result of the consolidation performed by the second consolidation means,

the second master manipulator further comprises eighth acquisition means for acquiring the relative attitude of the third handling part transmitted from the third transmission means of the third master manipulator,

the second determination means of the second master manipulator determines the third force or the third torque such that if the value corresponding to the relative attitude of the third handling part is smaller than the predetermined threshold value, the third force or the third torque is determined in accordance with the first force or the first torque and the result of the consolidation performed by the third consolidation means, however if the value corresponding to the relative attitude of the third handling part is equal to or greater than the predetermined threshold value, the third force or the third torque is determined in accordance with only the result of the consolidation performed by the third consolidation means, and

the third determination means of the third master manipulator determines the fourth force or the fourth torque such that if the value corresponding to the relative attitude of the third handling part is smaller than the predetermined threshold value, the fourth force or the fourth torque is determined in accordance with only the relative attitude of the third handling part, however if the value corresponding to the relative attitude of the third handling part is equal to or greater than the predetermined threshold value, the fourth force or the fourth torque is determined in accordance with the relative attitude of the third handling part and the first force or the first torque.

13. A manipulator control method in a manipulator system including a first master manipulator including a first handling part handled by a first operator, a second master manipulator including a second handling part handled by a second operator, and a slave manipulator including a treating part for treating an object, the method comprising:

a first detection step for detecting an absolute attitude, in a space within which the first handling part is allowed to move, of the first handling part of the first master manipulator;

a first transmission step for transmitting the absolute attitude of the first handling part;

a second detection step for detecting an absolute attitude, in a space within which the second handling part is allowed to move, of the second handling part of the second master manipulator;

a second transmission step for transmitting the absolute attitude of the second handling part;

a first acquisition step for acquiring the absolute attitude of the first handling part transmitted in the first transmission step and the absolute attitude of the second handling part transmitted in the second transmission step;

a first consolidation step for consolidating the absolute attitude of the first handling part and the absolute attitude of the second handling part; and

a first control step for controlling the attitude of the treating part in accordance with the result of the consolidation performed in the first consolidation step.

14. A manipulator control method according to claim 13, wherein

the absolute attitude of the first handling part is the position and the state, of the first handling part, in a space within which the first handling part is allowed to move, and

the absolute attitude of the second handling part is the position and the state, of the second handling part, in a space within which the second handling part is allowed to move.

15. A manipulator control method according to claim 13, wherein the first consolidation step calculates the weighted sum of the absolute attitude of the first handling part and the absolute attitude of the second handling part, using predetermined weighting factors for respective terms.

16. A manipulator control method according to claim 13, further comprising:

a third detection step for detecting a first force or a first torque applied to the treating part of the slave manipulator from the object being treated;

a third transmission step for transmitting the first force or the first torque,

a second acquisition step for acquiring the first force or the first torque transmitted in the third transmission step;

a first determination step for determining a second force or a second torque to be perceived by the first operator, in accordance with the first force or the first torque acquired in the second acquisition step;

a second control step for controlling the first handling part so that the first operator perceives the second force or the second torque,

a third acquisition step for acquiring the first force or the first torque transmitted in the third transmission step;

a second determination step for determining a third force or a third torque to be perceived by the second operator, in accordance with the first force or the first torque acquired in the third acquisition step; and

a third control step for controlling the second handling part so that the second operator perceives the third force or the third torque.

17. A manipulator control method according to claim 13, further comprising:

a third transmission step for transmitting the result of the consolidation performed in the first consolidation step;

a second acquisition step for acquiring the result of the consolidation transmitted in the third transmission step;

a first calculation step for calculating the difference between the absolute attitude of the first handling part and the result of the consolidation acquired in the second acquisition step;

a first determination step for determining a force or a torque to be perceived by the first operator, in accordance with the result of the calculation performed in the first calculation step;

a second control step for controlling the first handling part so that the first operator perceives the force or the torque determined in the first determination step;

a third acquisition step for acquiring the result of the consolidation transmitted in the third transmission step;

a second calculation step for calculating the difference between the absolute attitude of the second handling part and the result of the consolidation acquired in the third acquisition step;

a second determination step for determining a force or a torque to be perceived by the second operator, in accordance with the result of the calculation performed in the second calculation step; and

a third control step for controlling the second handling part so that the second operator perceives the force or the torque determined in the second determination step.

18. A manipulator control method according to claim 13, further comprising:

a third detection step for detecting a first force or a first torque applied to the treating part of the slave manipulator from the object being treated;

a third transmission step for transmitting the first force or the first torque and the result of the consolidation performed in the first consolidation step;

a second acquisition step for acquiring the first force or the first torque and the result of the consolidation transmitted in the third transmission step;

a first calculation step for calculating the difference between the absolute attitude of the first handling part and the result of the consolidation acquired in the second acquisition step;

a first determination step for determining a second force or a second torque to be perceived by the first operator, in accordance with the first force or the first torque acquired in the second acquisition step and in accordance with the difference calculated in the first calculation step;

a second control step for controlling the first handling part so that the first operator perceives the second force or the second torque;

a third acquisition step for acquiring the first force or the first torque and the result of the consolidation transmitted in the third transmission step;

a second calculation step for calculating the difference between the absolute attitude of the second handling part and the result of the consolidation acquired in the third acquisition step;

a second determination step for determining a third force or a third torque to be perceived by the second operator, in accordance with the first force or the first torque acquired in the third acquisition step and in accordance with the difference calculated in the second calculation step; and

a third control step for controlling the second handling part so that the second operator perceives the third force or the third torque.

19. A manipulator control method according to claim 13, further comprising:

a first display control step for controlling a cue so that the first operator can operate the first handling part in synchronization with the operation of the second operator on the second handling part in accordance with the cue, and

a second display control step for controlling the cue so that the second operator can operate the second handling part in synchronization with the operation of the first operator on the first handling part in accordance with the cue.

20. A manipulator control method according to claim 13, further comprising:

a first output control step for controlling a sound/voice cue so that the first operator can operate the first handling part in synchronization with the operation of the second operator on the second handling part in accordance with the sound/voice cue, and

a second display control step for controlling the cue so that the second operator can operate the second handling part in synchronization with the operation of the first operator on the first handling part in accordance with the cue.

21. A manipulator control method according to claim 13, wherein the manipulator system further includes a third master manipulator including a third handling part handled by a third operator, and the method further comprises:

a third detection step for detecting a relative attitude of the third handling part of the third master manipulator with respect to a predetermined reference attitude; and

a third transmission step for transmitting the relative attitude of the third handling part detected in the third detection step,

and wherein the first acquisition step further acquires the relative attitude of the third handling part of the third master manipulator transmitted in the third transmission step, and

the first consolidation step consolidates the absolute attitude of the first handling part, the absolute attitude of the second handling part, and the relative attitude of the third handling part.

22. A manipulator control method according to claim 21, wherein the first consolidation step calculates the weighted sum of the absolute attitude of the first handling part, the absolute attitude of the second handling part, and the relative attitude of the third handling part, using predetermined weighting factors for respective terms.

23. A manipulator control method according to claim 21, further comprising:

a fourth detection step for detecting a first force or a first torque applied to the treating part of the slave manipulator from the object being treated;

a fourth transmission step for transmitting the first force or the first torque,

a second acquisition step for acquiring the first force or the first torque transmitted in the fourth transmission step;

a third acquisition step for acquiring the absolute attitude of the second handling part transmitted in the second transmission step;

a second consolidation step for consolidating the absolute attitude of the first handling part and the absolute attitude of the second handling part;

a first calculation step for calculating the difference between the absolute attitude of the first handling part and the result of the consolidation performed in the second consolidation step;

a first determination step for determining a second force or a second torque to be perceived by the first operator, in accordance with the first force or the first torque acquired in the second acquisition step and in accordance with the difference calculated in the first calculation step;

a second control step for controlling the first handling part so that the first operator perceives the second force or the second torque;

a fourth acquisition step for acquiring the first force or the first torque transmitted in the fourth transmission step;

a fifth acquisition step for acquiring the absolute attitude of the first handling part transmitted in the first transmission step;

a third consolidation step for consolidating the absolute attitude of the first handling part and the absolute attitude of the second handling part;

a second calculation step for calculating the difference between the absolute attitude of the second handling part and the result of the consolidation performed in the third consolidation step;

a second determination step for determining a third force or a third torque to be perceived by the second operator, in accordance with the first force or the first torque acquired in the fourth acquisition step and in accordance with the difference calculated in the second calculation step;

a third control step for controlling the second handling part so that the second operator perceives the third force or the third torque;

a sixth acquisition step for acquiring the first force or the first torque transmitted in the fourth transmission step;

a third determination step for determining a fourth force or a fourth torque to be perceived by the third operator, in accordance with the first force or the first torque acquired in the sixth acquisition step and in accordance with the relative attitude of the third handling part; and

a fourth control step for controlling the third handling part so that the third operator perceives the fourth force or the fourth torque.

24. A manipulator control method according to claim 23, the method further comprising:

a seventh acquisition step for acquiring the relative attitude of the third handling part transmitted in the third transmission step,

wherein the first determination step determines the second force or the second torque such that if a value corresponding to the relative attitude of the third handling part is smaller than a predetermined threshold value, the second force or the second torque is determined in accordance with the first force or the first torque and the result of the consolidation performed in the second consolidation step, however if the value corresponding to the relative attitude of the third handling part is equal to or greater than the predetermined threshold value, the second force or the second torque is determined in accordance with only the result of the consolidation performed in the second consolidation step,

the method further comprising an eighth acquisition step for acquiring the relative attitude of the third handling part transmitted in the third transmission step,

wherein the second determination step determines the third force or the third torque such that if a value corresponding to the relative attitude of the third handling part is smaller than a predetermined threshold value, the third force or the third torque is determined in accordance with the first force or the first torque and the result of the consolidation performed in the third consolidation step, however if the value corresponding to the relative attitude of the third handling part is equal to or greater than the predetermined threshold value, the third force or the third torque is determined in accordance with only the result of the consolidation performed in the third consolidation step,

and wherein the third determination step determines the fourth force or the fourth torque such that if a value corresponding to the relative attitude of the third handling part is smaller than a predetermined threshold value, the fourth force or the fourth torque is determined in accordance with the value corresponding to the relative attitude of the third handling part, however if the value corresponding to the relative attitude of the third handling part is equal to or greater than the predetermined threshold value, the fourth force or the fourth torque is determined in accordance with the value corresponding to the relative attitude of the third handling part and the first force or the first torque.

25. A storage medium in which is stored a computer-readable program for controlling a manipulator system including a first master manipulator including a first handling part handled by a first operator, a second master manipulator including a second handling part handled by a second operator, and a slave manipulator including a treating part for treating an object, the program comprising:

a first detection step for detecting an absolute attitude, in a space within which the first handling part is allowed to move, of the first handling part of the first master manipulator;

a first transmission step for transmitting the absolute attitude of the first handling part;

a second detection step for detecting an absolute attitude, in a space within which the second handling part is allowed to move, of the second handling part of the second master manipulator;

a second transmission step for transmitting the absolute attitude of the second handling part;

a first acquisition step for acquiring the absolute attitude of the first handling part transmitted in the first transmission step and the absolute attitude of the second handling part transmitted in the second transmission step;

a first consolidation step for consolidating the absolute attitude of the first handling part and the absolute attitude of the second handling part; and

a first control step for controlling the attitude of the treating part in accordance with the result of the consolidation performed in the first consolidation step.

26. A master manipulator having a handling part handled by an operator, the master manipulator comprising:

detection means for detecting the attitude of the handling part; and

transmission means for transmitting the attitude of the handling part to a slave manipulator thereby allowing the slave manipulator to control a treating part of the slave manipulator so as to process an object in accordance with a result of consolidation of the attitude of the handling part and the attitude of a handling part of another master manipulator supplied from said another master manipulator.

27. A method for controlling a master manipulator having a handling part handled by an operator, the method comprising:

a detection step for detecting the attitude of the handling part; and

a transmission step for transmitting the attitude of the handling part to a slave manipulator thereby allowing the slave manipulator to control a treating part of the slave manipulator so as to process an object in accordance with a result of consolidation of the attitude of the handling part and the attitude of a handling part of another master manipulator supplied from said another master manipulator.

28. A storage medium in which is stored a computer-readable program for controlling a master manipulator having a handling part handled by an operator, the program comprising:

a detection step for detecting the attitude of the handling part; and

a transmission step for transmitting the attitude of the handling part to a slave manipulator thereby allowing the slave manipulator to control a treating part of the slave manipulator so as to process an object in accordance with a result of consolidation of the attitude of the handling part and the attitude of a handling part of another master manipulator supplied from said another master manipulator.

29. A slave manipulator having a treating part for treating an object, the salve manipulator comprising:

acquisition means for acquiring the attitude of a first handling part of a first master manipulator transmitted from the first master manipulator and the attitude of a second handling part of a second master manipulator transmitted from the second master manipulator;

consolidation means for consolidating the attitude of the first handling part and the attitude of the second handling part; and

control means for controlling the attitude of the treating part in accordance with the result of the consolidation performed by the consolidation means.

30. A method for controlling a slave manipulator having a treating part for treating an object, the method comprising:

an acquisition step for acquiring the attitude of a first handling part of a first master manipulator transmitted from the first master manipulator and the attitude of a second handling part of a second master manipulator transmitted from the second master manipulator;

a consolidation step for consolidating the attitude of the first handling part and the attitude of the second handling part; and

a control step for controlling the attitude of the treating part in accordance with the result of the consolidation performed in the consolidation step.

31. A storage medium in which is stored a computer-readable program for controlling a slave manipulator having a treating part for treating an object, the program comprising:

an acquisition step for acquiring the attitude of a first handling part of a first master manipulator transmitted from the first master manipulator and the attitude of a second handling part of a second master manipulator transmitted from the second master manipulator;

a consolidation step for consolidating the attitude of the first handling part and the attitude of the second handling part; and

a control step for controlling the attitude of the treating part in accordance with the result of the consolidation performed in the consolidation step.