Selecting objects in a sequence of images

• US 8,953,843 B1
• Filed: 02/05/2013
• Issued: 02/10/2015
• Est. Priority Date: 07/17/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kwan, Gavan, Doshi, Rushabh Ashok, Lebrn, Federico Javier
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Lee, Jonathan S
• Application Number:
  US13/759,855
• Time in Patent Office: 735 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

The subject matter of this specification can be implemented in, among other things, a computer-implemented method including detecting positions of objects of a specific type within an ordered sequence of images. The method includes estimating one or more intermediate positions of one or more intermediate instances of an object in one or more intermediate images within the ordered sequence of images between an initial image and a subsequent image based on an initial position of an initial instance of the object in the initial image and a subsequent position of a subsequent instance of the object in the subsequent image. The method includes providing a list of the objects for presentation. The method includes receiving a selection of the object from the list. The method includes performing an operation on the initial instance, the intermediate instances, and the subsequent instance of the object.

Claims

1. A method comprising:

detecting positions of instances of objects of a specific type within an ordered sequence of images stored in a memory, wherein the ordered sequence of images comprises a video, and wherein the positions of the instances of the objects comprise at least one initial position of at least one initial instance of an object among the objects in at least one initial image within the ordered sequence of images and at least one subsequent position of at least one subsequent instance of the object in at least one subsequent image within the ordered sequence of images after the initial image in the ordered sequence of images;

determining that the ordered sequence of images comprises a number of intermediate images between the initial image and the subsequent image that is within a first threshold number of images, wherein the first threshold number of images is based on a first threshold amount of time;

estimating, by one or more processing devices, one or more intermediate positions of one or more intermediate instances of the object in the intermediate images based on the initial position and the subsequent position in response to determining that the number of intermediate images is within the first threshold number of images;

buffering the video for at least the first threshold amount of time;

performing an operation on the initial instance, the intermediate instances, and the subsequent instance of the object in real-time with a delay due to buffering the video for at least the first threshold amount of time and without performing the operation on the instances of one or more others of the objects; and

providing the video for presentation.

2. The method of claim 1, wherein estimating the intermediate positions comprises calculating a linear interpolation of the intermediate positions between the initial position and the subsequent position.

3. The method of claim 1, wherein performing the operation comprises blurring the object at the initial position in the initial image, the intermediate positions in the intermediate images, and the subsequent position in the subsequent image.

4. The method of claim 3, wherein blurring the object comprises lowering a resolution of the object at the initial position in the initial image, the intermediate positions in the intermediate images, and the subsequent position in the subsequent image.

5. The method of claim 1, further comprising determining, based on the initial position and the subsequent position, that the object moves across the intermediate images within a threshold amount and, in response, estimating the intermediate positions.

6. The method of claim 1, further comprising:

providing a list of the objects for presentation; and

receiving a selection of the object from the list, wherein performing the operation is in response to receiving the selection.

7. The method of claim 1, wherein detecting, determining, estimating, buffering, performing, and providing are performed by a server computing system, and wherein providing the video comprises providing the video from the server computing system to a client computing system.

8. The method of claim 1, further comprising estimating one or more additional positions of the object in one or more additional images within the ordered sequence of images, wherein the additional images are at one or more of before the initial image or after the subsequent image, and wherein the additional positions of the object in ones of the additional images before the initial image are based at least on the initial position and the additional positions of the object in ones of the additional images after the subsequent image are based at least on the subsequent position.

9. The method of claim 8, wherein the ones of the additional images before the initial image and the ones of the additional images after the subsequent image include up to a second threshold number of images.

10. The method of claim 9, wherein the second threshold number of images is based on a second threshold amount of time.

11. A system comprising:

a data storage to store an ordered sequence of images; and

one or more processing devices to;

detect positions of instances of objects of a specific type within the ordered sequence of images, wherein the ordered sequence of images comprises a video;

cluster the instances for an object among the objects, wherein the positions of the instances of the object comprise at least one initial position of at least one initial instance of the object in at least one initial image within the ordered sequence of images and at least one subsequent position of at least one subsequent instance of the object in at least one subsequent image within the ordered sequence of images after the initial image in the ordered sequence of images;

determine that the ordered sequence of images comprises a number of intermediate images between the initial image and the subsequent image that is within a first threshold number of images, wherein the first threshold number of images is based on a first threshold amount of time;

estimate one or more intermediate positions of one or more intermediate instances of the object in the intermediate images based on the initial position and the subsequent position in response to the determination that the number of intermediate images is within the first threshold number of images;

buffer the video for at least the first threshold amount of time;

perform an operation on the initial instance, the intermediate instances, and the subsequent instance of the object in real-time with a delay to buffer the video for at least the first threshold amount of time and without performing the operation on the instances of one or more others of the objects.

12. The system of claim 11, wherein the processing devices are to estimate the intermediate positions by calculating a linear interpolation of the intermediate positions between the initial position and the subsequent position.

13. The system of claim 11, wherein the processing devices are to perform the operation by blurring the object at the initial position in the initial image, the intermediate positions in the intermediate images, and the subsequent position in the subsequent image, and wherein the processing devices are to blur the object by lowering a resolution of the object at the initial position in the initial image, the intermediate positions in the intermediate images, and the subsequent position in the subsequent image.

14. The system of claim 11, wherein the processing devices are further to determine, based on the initial position and the subsequent position, that the object moves across the intermediate images within a threshold amount and, in response, estimate the intermediate positions.

15. The system of claim 11, wherein the processing devices are further to:

provide a list of the objects for presentation; and

receive a selection of the object from the list, wherein the processing devices perform the operation in response to receiving the selection.

16. The system of claim 11, wherein the processing devices are further to estimate one or more additional positions of the object in one or more additional images within the ordered sequence of images, wherein the additional images are at one or more of before the initial image or after the subsequent image, and wherein the additional positions of the object in ones of the additional images before the initial image are based at least on the initial position and the additional positions of the object in ones of the additional images after the subsequent image are based at least on the subsequent position.

17. The system of claim 16, wherein the ones of the additional images before the initial image and the ones of the additional images after the subsequent image include up to a second threshold number of images.

18. The system of claim 17, wherein the second threshold number of images is based on a second threshold amount of time.

19. A non-transitory computer-readable storage medium having instructions stored therein that, when executed by one or more processing devices, cause the processing devices to perform operations comprising:

detecting positions of instances of objects of a specific type within an ordered sequence of images stored in a memory, wherein the ordered sequence of images comprises a video, and wherein the positions of the instances of the objects comprise at least one initial position of at least one initial instance of an object among the objects in at least one initial image within the ordered sequence of images and at least one subsequent position of at least one subsequent instance of the object in at least one subsequent image within the ordered sequence of images after the initial image in the ordered sequence of images;

determining that the ordered sequence of images comprises a number of intermediate images between the initial image and the subsequent image that is within a first threshold number of images, wherein the first threshold number of images is based on a first threshold amount of time;

estimating, by the processing devices, one or more intermediate positions of one or more intermediate instances of the object in the intermediate images based on the initial position and the subsequent position in response to determining that the number of intermediate images is within the first threshold number of images;

buffering the video for at least the first threshold amount of time;

performing an operation on the initial instance, the intermediate instances, and the subsequent instance of the object in real-time with a delay due to buffering the video for at least the first threshold amount of time and without performing the operation on the instances of one or more others of the objects; and

providing the video for presentation.

20. The non-transitory computer-readable storage medium of claim 19, wherein performing the operation comprises blurring the object at the initial position in the initial image, the intermediate positions in the intermediate images, and the subsequent position in the subsequent image.

21. The non-transitory computer-readable storage medium of claim 19, wherein the operations further comprise:

providing a list of the objects for presentation; and

receiving a selection of the object from the list, wherein performing the operation is in response to receiving the selection.