Robot device, method of controlling robot device, computer program, and program storage medium

• US 10,421,190 B2
• Filed: 09/26/2017
• Issued: 09/24/2019
• Est. Priority Date: 05/25/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sawada, Tsutomu
• Primary Examiner(s): Grant, II, Jerome
• Application Number:
  US15/716,022
• Publication Number:
  US 20180015617A1
• Time in Patent Office: 728 Days
• Field of Search: 382100
• CPC Class Codes: B25J 11/0085 Cleaning
• CPC Class Codes: B25J 19/0058 Means for cleaning manipulators, e.g. dust removing means
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 9/1674 characterised by safety, monitoring, diagnostic
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39053 Probe, camera on hand scans many points on own robot body, no extra jig
• CPC Class Codes: G05D 1/00 Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots drive control systems specially adapted for autonomous road vehicles B60W60/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor

Abstract

Provided is an excellent robot device capable of preferably detecting difference between dirt and a scratch on a lens of a camera and difference between dirt and a scratch on a hand. A robot device detects a site in which there is the dirt or the scratch using an image of the hand taken by a camera as a reference image. Further, this determines whether the detected dirt or scratch is due to the lens of the camera or the hand by moving the hand. The robot device performs cleaning work assuming that the dirt is detected, and then this detects the difference between the dirt and the scratch depending on whether the dirt is removed.

Claims

1. An information processing device, comprising:

circuitry configured to

detect whether there is a foreign object on either an image capturing device or a subject by comparing a reference image with a first image,

determine which of the image capturing device and the subject has the foreign object thereon by comparing the first image with a second image on a condition that the foreign object is detected on the image capturing device or on the subject, and

control performance of clean work on the image capturing device or the subject after the determining which of the image capturing device and the subject has the foreign object thereon, wherein

the reference image is an image of a specific site of a subject captured by the image capturing device,

the first image is the image of the specific site of the subject, in a same posture as that taken when the reference image is obtained, captured by the image capturing device, and

the second image is the image of the specific site of the subject, in a different posture as that taken when the first image is obtained, captured by the image capturing device.

2. The device according to claim 1, wherein the circuitry is configured to

determine which of the image capturing device and the subject has the foreign object thereon by comparing the reference image with a third image after the performance of clean work,

wherein the third image is the image of the specific site of the subject, in a same posture as that taken when the reference image is obtained, captured by the image capturing device after the performance of clean work.

3. The device according to claim 1, wherein the circuitry is configured to

detect that there is no foreign object on the image capturing device and the subject when a correlation between the first image and the reference image is higher than a predetermined value for an entirety of the first image and the reference image, and

detect that there is the foreign object on either the image capturing device or the subject when it is determined that a site exists in which the correlation is lower than the predetermined value.

5. The device according to claim 1, wherein the circuitry is configured to

capture the specific site of the device as the subject.

4. An information processing device, comprising:

circuitry configured to

detect whether there is a foreign object on either an image capturing device or a subject by comparing a reference image with a first image, and

determine which of the image capturing device and the subject has the foreign object thereon by comparing the first image with a second image on a condition that the foreign object is detected on the image capturing device or on the subject, wherein

the reference image is an image of a specific site of a subject captured by the image capturing device,

the first image is the image of the specific site of the subject, in a same posture as that taken when the reference image is obtained, captured by the image capturing device,

the second image is the image of the specific site of the subject, in a different posture as that taken when the first image is obtained, captured by the image capturing device,

the circuitry is further configured to

detect that there is no foreign object on the image capturing device and the subject when a correlation between the first image and the reference image is higher than a predetermined value for an entirety of the first image and the reference image, and

detect that there is the foreign object on either the image capturing device or the subject when it is determined that a site exists in which the correlation is lower than the predetermined value, and

the circuitry is further configured to, when the circuitry detects that there is the foreign object on either of the image capturing device or the subject,

determine that there is the foreign object on the image capturing device when the correlation between the first image and the second image in the site, in which the correlation between the first image and the reference image is lower, is higher than the predetermined value, and

determine that there is the foreign object on the subject when the correlation between the first image and the second image in the site, in which the correlation between the first image and the reference image is lower, is lower than the predetermined value.

6. An information processing method, comprising:

detecting whether there is a foreign object on either an image capturing device or a subject by comparing a reference image with a first image;

determining, using circuitry, which of the image capturing device and the subject has the foreign object thereon by comparing the first image with a second image on a condition that the foreign object is detected on the image capturing device or on the subject; and

controlling performance of clean work on the image capturing device or the subject after the determining which of the image capturing device and the subject has the foreign object thereon, wherein

the reference image is an image of a specific site of a subject captured by the image capturing device,

the first image is the image of the specific site of the subject, in a same posture as that taken when the reference image is obtained, captured by the image capturing device, and

the second image is the image of the specific site of the subject, in a different posture as that taken when the first image is obtained, captured by the image capturing device.

7. The information processing method according to claim 6, further comprising:

determining which of the image capturing device and the subject has the foreign object thereon by comparing the reference image with a third image after the performance of clean work,

wherein the third image is the image of the specific site of the subject, in a same posture as that taken when the reference image is obtained, captured by the image capturing device after the performance of clean work.

8. The information processing method according to claim 6, wherein the detecting further comprises:

detecting that there is no foreign object on the image capturing device and the subject when a correlation between the first image and the reference image is higher than a predetermined value for an entirety of the first image and the reference image; and

detecting that there is the foreign object on either the image capturing device or the subject when it is determined that a site exists in which the correlation is lower than the predetermined value.

9. The information processing method according to claim 6, further comprising:

capturing the specific site of the device as the subject.

10. An information processing method, comprising:

detecting whether there is a foreign object on either an image capturing device or a subject by comparing a reference image with a first image; and

determining, using circuitry, which of the image capturing device and the subject has the foreign object thereon by comparing the first image with a second image on a condition that the foreign object is detected on the image capturing device or on the subject, wherein

the reference image is an image of a specific site of a subject captured by the image capturing device,

the first image is the image of the specific site of the subject, in a same posture as that taken when the reference image is obtained, captured by the image capturing device, and

the second image is the image of the specific site of the subject, in a different posture as that taken when the first image is obtained, captured by the image capturing device, wherein

the detecting further comprises;

detecting that there is no foreign object on the image capturing device and the subject when a correlation between the first image and the reference image is higher than a predetermined value for an entirety of the first image and the reference image; and

detecting that there is the foreign object on either the image capturing device or the subject when it is determined that a site exists in which the correlation is lower than the predetermined value, and

when the detecting detects that there is the foreign object on either of the image capturing device or the subject, the determining further comprises;

determining that there is the foreign object on the image capturing device when the correlation between the first image and the second image in the site, in which the correlation between the first image and the reference image is lower, is higher than the predetermined value; and

determining that there is the foreign object on the subject when the correlation between the first image and the second image in the site, in which the correlation between the first image and the reference image is lower, is lower than the predetermined value.

11. A non-transitory computer readable medium having stored thereon a program that when executed by processing circuitry causes the processing circuitry to implement an information processing method, comprising:

detecting whether there is a foreign object on either an image capturing device or a subject by comparing a reference image with a first image;

determining which of the image capturing device and the subject has the foreign object thereon by comparing the first image with a second image on a condition that the foreign object is detected on the image capturing device or on the subject; and

controlling performance of clean work on the image capturing device or the subject after the determining which of the image capturing device and the subject has the foreign object thereon, wherein

the reference image is an image of a specific site of a subject captured by the image capturing device,

the first image is the image of the specific site of the subject, in a same posture as that taken when the reference image is obtained, captured by the image capturing device, and

the second image is the image of the specific site of the subject, in a different posture as that taken when the first image is obtained, captured by the image capturing device.