METHOD AND SYSTEM FOR AUTOMATIC GENERATION OF CLIPS FROM A PLURALITY OF IMAGES BASED ON AN INTER-OBJECTS RELATIONSHIP SCORE

• US 20150131973A1
• Filed: 11/11/2014
• Published: 05/14/2015
• Est. Priority Date: 11/11/2013
• Status: Active Grant

• Current Assignee: Vimeo, Inc.
• Original Assignee: Magisto Ltd. (Vimeo, Inc.)
• Inventors: RAV-ACHA, Alexander, Boiman, Oren
• Granted Patent: US 9,324,374 B2
• Time in Patent Office: Days
• US Class Current: 386/285
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G11B 27/02 Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A method and a system for automatic generation of clips from a plurality of images based on inter-object relationships score are provided herein. The method may include: obtaining a plurality of images, wherein at least two of the images contain at least one object over a background; analyzing at least some of the images to detect objects; extracting geometrical meta-data of at least some of the detected objects; calculating an inter-object relationships score for at least some of the detected objects; and determining a spatio-temporal arrangement of at least some of the objects and at least some of the images based at least partially on the inter-object relationships score and the geometrical meta-data of at least some of the detected objects.

Claims

1. A method comprising:

obtaining a plurality of images, wherein at least two of the images contain at least one object over a background;

analyzing at least some of the images to detect objects;

extracting geometrical meta-data of at least some of the detected objects;

calculating an inter-object relationships score for at least some of the detected objects;

determining a spatio-temporal arrangement of at least some of the objects and image portions based at least partially on the inter-object relationships score and the geometrical meta-data of at least some of the detected objects; and

producing a clip based on the determined spatio-temporal arrangement.

2. The method according to claim 1, wherein the spatio-temporal arrangement further comprises duplicating at least one of the detected objects.

3. The method according to claim 1, wherein the spatio-temporal arrangement further comprises omitting at least one of the detected objects.

4. The method according to claim 1, wherein the spatio-temporal arrangement further comprises omitting at least one of the input images.

5. The method according to claim 1, wherein the determining of the spatio-temporal arrangement comprises cropping at least one of the images, substantially around at least one of the detected objects, based on the corresponding inter-object relationships score.

6. The method according to claim 1, wherein the calculating of the inter-object relationships score is carried out using a cost function.

7. The method according to claim 1, wherein the determining a spatio-temporal arrangement is further based on meta-data other than the geometrical meta-data of at least some of the detected objects.

8. The method according to claim 1, wherein the production further comprises generating transitions between the at least some of the extracted objects, based at least partially on the corresponding inter-object relationships score.

9. The method according to claim 1, wherein the production further comprises generating a music sequence along the sequence of the extracted objects, based at least partially on the corresponding inter-object relationships score.

10. The method according to claim 5, wherein the spatio-temporal arrangement results in at least two consecutive selections of the same object undergoing two different cropping operations.

11. The method according to claim 1, wherein the inter-object relationships comprises at least one of the following relationships:

the objects are looking at each other;

the objects are within a predefined time-space from each other;

the objects are talking to each other;

the objects are engaged in a similar activity.

12. The method according to claim 1, wherein the inter-object relationships score is affected positively by a preference for placing two objects close in time in a case said two objects are strongly related.

13. A system comprising:

a memory configured to store a plurality of images, wherein at least two of the images contain at least one object over a background; and

a computer processor configured to;

analyze at least some of the images to detect objects;

extract geometrical meta-data of at least some of the detected objects;

calculate an inter-object relationships score for at least some of the detected objects;

determine a spatio-temporal arrangement of at least some of the objects and image portions based at least partially on the inter-object relationships score and the geometrical meta-data of at least some of the detected objects; and

produce a clip based on the determined spatio-temporal arrangement.

14. The system according to claim 13, wherein the spatio-temporal arrangement further comprises duplicating at least one of the detected objects.

15. The system according to claim 13, wherein the spatio-temporal arrangement further comprises omitting at least one of the detected objects.

16. The system according to claim 13, wherein the spatio-temporal arrangement further comprises omitting at least one of the input images.

17. The system according to claim 13, wherein the determining of the spatio-temporal arrangement comprises cropping at least one of the images, substantially around at least one of the detected objects, based on the corresponding inter-object relationships score.

18. The system according to claim 13, wherein the calculating of the inter-object relationships score is carried out using a cost function.

19. The system according to claim 13, wherein the determining a spatio-temporal arrangement is further based on meta-data other than the geometrical meta-data of at least some of the detected objects.

20. The system according to claim 13, wherein the production further comprises generating transitions between the at least some of the extracted objects, based at least partially on the corresponding inter-object relationships score.

21. The system according to claim 13, wherein the production further comprises generating a music sequence along the sequence of the extracted objects, based at least partially on the corresponding inter-object relationships score.

22. The system according to claim 17, wherein the spatio-temporal arrangement results in at least two consecutive selections of the same object undergoing two different cropping operations.

23. The system according to claim 13, wherein the inter-object relationships comprises at least one of the following relationships:

the objects are looking at each other;

the objects are within a predefined time-space from each other;

the objects are talking to each other;

the objects are engaged in a similar activity.

24. The system according to claim 13, wherein the inter-object relationships score is affected positively by a preference for placing two objects close in time in a case said two objects are strongly related.