Robot device, method of controlling robot device, computer program, and program storage medium

• US 9,802,320 B2
• Filed: 04/29/2016
• Issued: 10/31/2017
• Est. Priority Date: 05/25/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Sawada, Tsutomu
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US15/142,611
• Publication Number:
  US 20160236352A1
• Time in Patent Office: 550 Days
• CPC Class Codes: B25J 11/0085 Cleaning
• CPC Class Codes: B25J 19/0058 Means for cleaning manipulators, e.g. dust removing means
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 9/1674 characterised by safety, monitoring, diagnostic
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39053 Probe, camera on hand scans many points on own robot body, no extra jig
• CPC Class Codes: G05D 1/00 Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots drive control systems specially adapted for autonomous road vehicles B60W60/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor

Abstract

Provided is an excellent robot device capable of preferably detecting difference between dirt and a scratch on a lens of a camera and difference between dirt and a scratch on a hand.

A robot device 100 detects a site in which there is the dirt or the scratch using an image of the hand taken by a camera 305 as a reference image. Further, this determines whether the detected dirt or scratch is due to the lens of the camera 305 or the hand by moving the hand. The robot device 100 performs cleaning work assuming that the dirt is detected, and then this detects the difference between the dirt and the scratch depending on whether the dirt is removed.

Claims

1. An image capturing device, comprising:

circuitry configured to

control capturing of a first image,

control capturing of a second image, and

determine whether there is a foreign object on the image capturing device by comparing the first image with a reference image,

wherein, in a case that it is determined that there is the foreign object on the image capturing device, the circuitry instructs removing the foreign object by cleaning work of a cleaning robot to permit the image capturing device to capture unobstructed, and further compares a second image captured by the image capturing device after the cleaning work on the image capturing device with at least one of the reference image or the first image to further determine whether there is still the foreign object on the image capturing device.

2. The device according to claim 1, wherein the circuitry is configured to

calculate a correlation between the first image and the reference image;

determine whether a site exists in which the correlation between the first image and the reference image is below a predetermined value;

determine that there is no foreign object on the image capturing device when the correlation between the first image and the reference image is higher than the predetermined value for an entirety of the first image and the reference image; and

determine that there is the foreign object on the image capturing device when it is determined that the site exists in which the correlation is lower than the predetermined value.

3. The device according to claim 2, wherein the circuitry is configured to determine that there is no foreign object on the image capturing device by comparing a third image of a second item different from a first item represented in the first image, captured by the image capturing device, with the first image.

4. The device according to claim 3, wherein the circuitry is configured to calculate a correlation between the first image and the third image, and

wherein the circuitry is configured to determine that there is the foreign object on the image capturing device in a case that a correlation between the third image and the first image is higher than a predetermined value in a low-correlation site and determine that there is the foreign object on the first item in the case that the correlation between the third image and the first image is lower than a predetermined value in the low-correlation site determined by comparison between the second image with the first image, the low-correlation site is a site for which the correlation between the second image and the first image is lower than a predetermined value.

5. The device according to claim 4,

wherein in the case that there is the foreign object on the first item, the circuitry instructs cleaning work to clean the first item.

6. The device according to claim 1, wherein the cleaning work to be performed on the image capturing device includes using a moveable part to clean the image capturing device.

7. The device according to claim 1, wherein the circuitry is configured to

calculate a correlation between a plurality of sites in the first image and the plurality of sites in the reference image, and

determine whether any site of the plurality of sites exists in which the correlation between the first image and the reference image is below a predetermined value by determining that there is no foreign object on the image capturing device when the correlation between the first image and the reference image is higher than the predetermined value for each site of the plurality of sites, and determining that there is the foreign object on the image capturing device when it is determined that any site of the plurality of sites exists in which the correlation is lower than the predetermined value.

16. The device according to claim 1, wherein the circuitry is configured to

automatically determine whether there is the foreign object on the image capturing device, and

in response to the automatic determination, instruct removing the foreign object by cleaning work of the cleaning robot.

8. A method of controlling an image capturing device, comprising:

controlling capturing of a first image;

controlling capturing of a second image; and

determining whether there is a foreign object on a lens of the image capturing device by comparing the first image with a reference image,

wherein, in a case that it is determined that there is the foreign object on the lens of the image capturing device, the method further comprises instructing cleaning work of a cleaning robot to permit the image capturing device to capture unobstructed, and comparing a second image captured by the image capturing device after the cleaning work on the lens with at least one of the reference image or the first image to further determine whether there is still the foreign object on the image capturing device.

9. The method according to claim 8, further comprising:

calculating a correlation between the first image and the reference image;

determining whether a site exists in which the correlation between the first image and the reference image is below a predetermined value;

determining that there is no foreign object on the lens of the image capturing device when the correlation between the first image and the reference image is higher than the predetermined value for an entirety of the first image and the reference image; and

determining that there is the foreign object on the lens of the image capturing device when it is determined that the site exists in which the correlation is lower than the predetermined value.

10. The method according to claim 9, further comprising:

determining that there is no foreign object on the lens of the image capturing device by comparing a third image of a second item different from a first item represented in the first image, captured by the image capturing device, with the first image.

11. The method according to claim 10, further comprising:

calculating a correlation between the first image and the third image; and

determining that there is the foreign object on the lens of the image capturing device in a case that a correlation between the third image and the first image is higher than a predetermined value in a low-correlation site and determine that there is the foreign object on the first item in the case that the correlation between the third image and the first image is lower than a predetermined value in the low-correlation site determined by comparison between the second image with the first image, the low-correlation site is a site for which the correlation between the second image and the first image is lower than a predetermined value.

12. The method according to claim 11, further comprising:

in the case that there is the foreign object on the first item, instructing cleaning work to clean the first item.

13. The method according to claim 8, wherein the cleaning work to be performed on the image capturing device includes using a moveable part to clean the lens of the image capturing device.

14. The method according to claim 8, further comprising:

calculating a correlation between a plurality of sites in the first image and the plurality of sites in the reference image; and

determining whether any site of the plurality of sites exists in which the correlation between the first image and the reference image is below a predetermined value by determining that there is no foreign object on the lens of the image capturing device when the correlation between the first image and the reference image is higher than the predetermined value for each site of the plurality of sites, and determining that there is the foreign object on the lens of the image capturing device when it is determined that any site of the plurality of sites exists in which the correlation is lower than the predetermined value.

15. A non-transitory program storage medium storing thereon control program that when executed by a computer causes the computer to implement a method of controlling an image capturing device, comprising:

controlling capturing of a first image;

controlling capturing of a second image, and

determining whether there is a foreign object on a lens of the image capturing device by comparing the first image with a reference image,

wherein, in a case that it is determined that there is the foreign object on the lens of the image capturing device, the method further comprises instructing cleaning work of a cleaning robot to permit the image capturing device to capture unobstructed, and comparing a second image captured by the image capturing device after the cleaning work on the lens with at least one of the reference image or the first image to further determine whether there is still the foreign object on the image capturing device.