Pre-touch localization on a reflective surface

• US 9,823,782 B2
• Filed: 11/20/2015
• Issued: 11/21/2017
• Est. Priority Date: 11/20/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gu, Yu, Hwang, Inseok, Liu, Su, Rozner, Eric J., Wei, Yaoguang, Yoo, Chungkuk
• Primary Examiner(s): Snyder, Adam J
• Application Number:
  US14/946,994
• Publication Number:
  US 20170147151A1
• Time in Patent Office: 732 Days
• Field of Search: 345175, 178 1809
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Aspects include a method, system, and computer program product for determining the location of an object relative to a reflective surface. The method includes acquiring an image of an object and the reflective surface with a camera, the camera positioned adjacent the reflective surface. The object is identified in the image. A reflection of the object on the reflective surface is identified in the image. Three-dimensional coordinates of at least a portion of the object are determined based at least in part on the object and the reflection of the object in the image.

Claims

1. A method comprising:

acquiring a first image of an object and a reflective surface at a first time with a camera positioned adjacent the reflective surface and having a field of view perpendicular to the reflective surface, wherein the image is reflected into the camera by a mirror positioned adjacent the camera, the mirror being positioned to reflect at least a portion of the reflective surface and a space adjacent the reflective surface;

acquiring a second image of the object and the reflective surface with the camera at a second time;

comparing the first image and the second image and determining a differential between the first image and the second image to identify movement;

identifying in the first image and the second image the object based at least in part on the differential between the first image and the second image;

identifying a convex polygon representing the object based at least in part on the differential between the first image and the second image;

identifying in the first image and the second image a reflection of the object on the reflective surface, wherein the identifying the reflection of the object on the reflective surface comprises;

defining a region of interest;

subtracting pixel values of a portion of a video displayed on the reflective surface within the region of interest;

analyzing a shape within the region of interest; and

determining the shape to have a shape similar to the object in the first image and the second image;

determining three-dimensional coordinates of at least a portion of the object based at least in part on the convex polygon and the reflection of the object in the first image and the second image; and

transforming the three-dimensional coordinates into surface coordinates comprising a graphical marker displayed on the reflective surface.

2. The method of claim 1 further comprising performing a predetermined action in response to the surface coordinates.

3. The method of claim 2 wherein the predetermined action is performed in response to the object remaining in substantially the same position for a predetermined amount of time.

4. The method of claim 2 wherein:

the reflective surface is a graphical display; and

the predetermined action includes displaying a graphical image at the surface coordinates.

5. The method of claim 1 further comprising removably coupling a mobile computing device on an angle relative to the reflective surface, wherein the camera is coupled to the mobile computing device.

6. The method of claim 1 wherein acquiring of the image enables unilaterally provisioning computing capabilities identify the object, the reflection of the object in the reflective surface, and determine the three-dimensional coordinates of at least the portion of the object.

7. The method of claim 1 further comprising:

acquiring a second image of the object and the reflective surface, the second image being acquired after the first image;

identifying in the second image the object;

identifying in the second image the convex polygon representing the object;

identifying in the second image a reflection of the object on the reflective surface; and

determining a second three-dimensional coordinates of at least a portion of the object based at least in part on the convex polygon and the reflection of the object in the second image.

8. The method of claim 1 further comprising:

determining a region of interest from the first image, the object being positioned within the region of interest;

comparing at least one shape in the region of interest in the second image to the object in the first image; and

wherein the object is identified in the second image based at least in part on the region of interest and the comparison of the at least one shape to the object.

9. A system, the system comprising:

a reflective surface;

a camera positioned adjacent the reflective surface and having a field of view perpendicular to the reflective surface;

a mirror positioned adjacent the reflective surface and the camera, the mirror oriented to reflect at least a portion of the reflective surface and a space adjacent the reflective surface;

a memory having computer readable instructions; and

one or more processors for executing the computer readable instructions, the one or more processors being coupled to communicate with the camera, the computer readable instructions comprising;

acquiring a first image of an object and the reflective surface with the camera at a first time via the mirror;

acquiring a second image of the object and the reflective surface with the camera at a second time;

comparing the first image and the second image and determining a differential between the first image and the second image to identify movement;

identifying in the first image and the second image the object based at least in part on the differential between the first image and the second image;

identifying a convex polygon representing the object based at least in part on the differential between the first image and the second image;

identifying in the first image and the second image a reflection of the object on the reflective surface, wherein the identifying the reflection of the object on the reflective surface comprises;

defining a region of interest;

subtract pixel values of a portion of a video displayed on the reflective surface within the region of interest;

analyzing a shape within the region of interest; and

determining the shape to have a shape similar to the object in the first image and the second image;

determining three-dimensional coordinates of at least a portion of the convex polygon based at least in part on the object and the reflection of the object in the first image and the second image; and

transforming the three-dimensional coordinates into surface coordinates comprising a graphical marked displayed on the reflective surface.

10. The system of claim 9 further comprising performing a predetermined action in response to the surface coordinates.

11. The system of claim 10 wherein the predetermined action is performed in response to the object remaining in substantially the same position for a predetermined amount of time.

12. The system of claim 10 wherein the predetermined action includes displaying a graphical image at the surface coordinates.

13. The system of claim 9 further comprising removably coupling a mobile computing device on an angle relative to the reflective surface, wherein the camera is coupled to the mobile computing device.

14. A computer program product for determining a pre-touch location of an object relative to a reflective surface, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform:

acquiring a first image of the object and the reflective surface at a first time with a camera, the camera arranged adjacent the reflective surface and having a field of view perpendicular to the reflective surface, wherein the image is reflected into the camera by a mirror positioned adjacent the camera, the mirror being positioned to reflect at least a portion of the reflective surface and a space adjacent the reflective surface;

acquiring a second image of the object and the reflective surface with the camera at a second time;

comparing the first image and the second image and determining a differential between the first image and the second image to identify movement;

identifying in the first image and the second image the object based at least in part on a differential between the first image and the second image;

identifying a convex polygon representing the object based at least in part on the differential between the first image and the second image;

identifying in the first image and the second image a reflection of the object on the reflective surface, wherein the identifying the reflection of the object on the reflective surface comprises;

defining a region of interest;

subtracting pixel values of a portion of a video displayed on the reflective surface within the region of interest;

analyzing a shape within the region of interest; and

determining the shape to have a shape similar to the object in the first image and the second image;

determining three-dimensional coordinates of at least a portion of the object based at least in part on the convex polygon and the reflection of the object in the first image and second image; and

transforming the three-dimensional coordinates into surface coordinates comprising a graphical marker displayed on the reflective surface.

15. The computer program product of claim 14 further comprising performing a predetermined action in response to the surface coordinates.

16. The computer program product of claim 15 wherein the predetermined action is performed in response to the object in remaining substantially the same position for a predetermined amount of time.

17. The computer program product of claim 14 wherein:

the reflective surface is a graphical display; and

the predetermined action includes displaying a graphical image at the surface coordinates.