Movement prediction for image capture

• US 9,129,400 B1
• Filed: 09/23/2011
• Issued: 09/08/2015
• Est. Priority Date: 09/23/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ivanchenko, Volodymyr V., Hovden, Gunnar
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): BEASLEY, DEIRDRE L
• Application Number:
  US13/243,487
• Time in Patent Office: 1,446 Days
• Field of Search: 348/169
• US Class Current: 1/1
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

A camera of a computing device can capture two or more images of a region including an object of interest, in order to allow for separation of the object from a background of the images through a process such as image subtraction. In order to compensate for rotations of the device between image captures, an element such as an electronic gyroscope can be used to monitor changes in orientation and predict an amount of shift of objects between images. The predicted shift can be used to attempt to align images captured around the time of the rotation, in order to enable subtraction or similar processes by effectively removing the shifting effect of the rotation.

Claims

1. A computer-implemented method of tracking motion of a target object via a computing device, comprising:

tracking a respective orientation of the computing device over at least a period of time using at least one embedded sensor of the computing device;

capturing a first image using a camera of the computing device at a first time, the first image including a representation of the target object and a respective representation of one or more background elements;

capturing a second image using the camera of the computing device at a second time, the second image including the representation of the target object and the respective representation of the one or more background elements;

calculating, an angular change in orientation of the computing device between the first time and the second time using the respective orientation tracked by the at least one embedded sensor between the first time and the second time;

calculating, a first number of pixels corresponding to a first shift of the respective representation of the one or more background elements in the second image with respect to the respective representation of the one or more background elements in the first image based at least in part upon the angular change in orientation;

calculating a second number of pixels corresponding to a second shift of the respective representation of the one or more background elements in the first image with respect to the respective representation of the one or more background elements in the second image based at least in part upon the angular change in orientation;

generating a third image by a first shifting of second intensity and color values of the second image by the first number of pixels to align the respective representation of the one or more background elements in the first image and the third image;

generating a fourth image by a second shifting of first intensity and color values of the first image by the second number of pixels to align the respective representation of the one or more background elements in the second image and the fourth image;

substantially removing the respective representation of the one or more background elements from;

(a) the first image by subtracting third intensity and color values of the third image from the first intensity and color values of the first image and (b) the second image by subtracting fourth intensity and color values of the fourth image from the second intensity and color values of the second image; and

determining the motion of the target object based on the first image and the second image each with the respective representation of the one or more background elements substantially removed.

2. The method of claim 1, wherein calculating the angular change in orientation of the computing device between the first time and the second time includes:

receiving angular velocity and timestamp information from the at least one embedded sensor; and

calculating an angular change for at least one axis of rotation.

3. The method of claim 2, wherein the timestamp information does not match the first time and the second time, and the method further comprises:

interpolating respective orientation information for the first time and the second time using the respective orientation tracked for timestamps adjacent to the first time and the second time.

4. The method of claim 1, wherein an amount of the first shift is calculated in part using an angular field of view and a number of pixels of the camera.

5. The method of claim 1, further comprising:

illuminating the target object using at least one illumination element of the computing device while capturing one of the first image or the second image.

6. The method of claim 1, wherein the at least one embedded sensor includes at least one of an accelerometer, an electronic gyroscope, an electronic compass, or an inertial sensor.

7. A computer-implemented method of providing input to an electronic device, comprising:

obtaining a first image and a second image each including a representation of an object and a respective representation of one or more background elements using a camera of the electronic device, the first image being captured at a first time and the second image being captured at a second time;

determining, using information generated by at least one embedded sensor of the electronic device, at least a rotational change in orientation of the electronic device between the first time and the second time;

estimating a first amount of image shift of the second image relative to the first image and a second amount of image shift of the first image relative to the second image due at least in part to the rotational change;

substantially removing the respective representation of the one or more background elements from;

(a) the first image using a first mapping of respective image information of the first image and the second image and (b) the second image using a second mapping of the respective information of the first image and the second image, the first mapping being based at least in part upon the first amount of image shift, and the second mapping being based at least in part upon the second amount of image shift; and

determining motion of the object based on the first image and the second image each with the respective representation of the one or more background elements substantially removed.

8. The computer-implemented method of claim 7, wherein substantially removing the respective representation of the one or more background elements from the first image and the second image includes:

generating a third image by a first shifting of second intensity and color values of the second image by the first amount of image shift to align the respective representation of the one or more background elements in the first image and the third image;

generating a fourth image by a second shifting of first intensity and color values of the first image by the second amount of image shift to align the respective representation of the one or more background elements in the second image and the fourth image; and

subtracting;

(a) third intensity and color values of the third image from the first intensity and color values of the first image and (b) fourth intensity and color values of the fourth image from the second intensity and color values of the second image.

9. The computer-implemented method of claim 7, further comprising:

illuminating the object using at least one illumination element of the electronic device during a capture time of one of the first image or the second image such that at least a portion of the representation of the object has a change in brightness between the first image and the second image.

10. The computer-implemented method of claim 7, further comprising:

determining an amount of lateral translation of the electronic device between the first time and the second time using at least one translation sensor,

wherein at least one of the first mapping or the second mapping is further based at least in part upon the amount of lateral translation.

11. The computer-implemented method of claim 7, wherein the at least one embedded sensor includes at least one of an accelerometer, an electronic gyroscope, an electronic compass, or an inertial sensor.

12. The computer-implemented method of claim 7, wherein the camera is a still image camera and the first image and the second image are still images, or the camera is a video camera and the first image and the second image are frames of video information.

13. The computer-implemented method of claim 7, wherein the object includes at least one feature of a user, and the method further comprises:

determining at least one gesture performed by the user to provide input to the electronic device using the motion of the object.

14. The computer-implemented method of claim 7, wherein the first image and the second image include at least one of ambient light image information or reflected infrared image information.

15. The computer-implemented method of claim 7, further comprising:

deactivating image capture if no motion or gesture is detected within a specified period of inactivity.

16. A computing device, comprising:

a device processor;

a camera;

an embedded sensor; and

a memory device including instructions that, upon being executed by the processor, cause the computing device to;

obtain a first image and a second image using the camera, the first image being captured at a first time and the second image being captured at a second time, the first image and the second image each including a representation of an object and a respective representation of one or more background elements;

determine a rotational change in orientation of the computing device during a period including the first time and the second time using the embedded sensor;

estimate a first amount of image shift of the second image relative to the first image and a second amount of image shift of the first image relative to the second image based at least in part upon the rotational change;

minimize the respective representation of the one or more background elements in;

(a) the first image using a first map of respective image information of the first image and the second image and (b) the second image using a second map of the respective image information of the first image and the second image, the first map being based at least in part upon the first amount of image shift, and the second map being based at least in part upon the second amount of image shift; and

determine motion of the object based on the first image and the second image each with the respective representation of each of the one or more background elements minimized.

17. The computing device of claim 16, further comprising:

at least one illumination element operable to illuminate the object during a capture time of one of the first image or the second image such that at least a portion of the object has a change in brightness between the first image and the second image.

18. The computing device of claim 16, further comprising:

a lateral sensor element operable to determine an amount of lateral translation of the computing device between the first time and the second time,

wherein at least one of the first map or the second map is further based at least in part upon the amount of lateral translation.

19. A non-transitory computer-readable storage medium storing instructions that, upon being executed by a processor, cause the processor to:

obtain a first image and a second image each including a representation of an object and a respective representation of one or more background elements using a camera of an electronic device, the first image being captured at a first time and the second image being captured at a second time;

determine, using information generated by at least one embedded sensor, at least a rotational change in orientation of the electronic device between the first time and the second time;

estimate a first amount of image shift of the second image relative to the first image and a second amount of image shift of the first image relative to the second image based at least in part upon the rotational change in orientation;

substantially remove the respective representation of the one or more background elements in;

(a) the first image using a first mapping of respective image data of the first image and the second image and (b) the second image using a second mapping of the respective information of the first image and the second image, the first mapping being based at least in part upon the first amount of image shift, and the second mapping being based at least in part upon the second amount of image shift; and

determine motion of the object based on the first image and the second image each with the respective representation of the one or more background elements substantially removed.

20. The non-transitory computer-readable storage medium of claim 19, wherein the instructions upon being executed to cause the processor to determine the rotational change in orientation of the electronic device between the first time and the second time include causing the processor to:

receive angular velocity and timestamp information from the at least one embedded sensor; and

calculate an angular change for at least one axis of rotation.

21. The non-transitory computer-readable storage medium of claim 20, wherein the timestamp information does not match the first time and the second time, and the instructions, upon being executed, further cause the processor to:

interpolate respective orientation information for the first time and the second time using the rotational change in orientation for timestamps adjacent to the first time and the second time.