SYSTEM AND METHOD FOR TAGGING OBJECTS IN A PANORAMIC VIDEO AND ASSOCIATING FUNCTIONS AND INDEXING PANORAMIC IMAGES WITH SAME

• US 20080106594A1
• Filed: 08/10/2007
• Published: 05/08/2008
• Est. Priority Date: 11/07/2006
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Inventors: THRUN, SEBASTIAN
• Granted Patent: US 9,361,943 B2
• Time in Patent Office: Days
• US Class Current: 348/39
• CPC Class Codes: G06F 16/748 Hypervideo linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G11B 27/329 on a disc [VTOC]
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

A system and process for attaching tags to panoramic video. Tags provide information when viewing panoramic images, serve as references to specific actions and serve as reference for outside systems into a panoramic image database. Objects in a video can be tagged. It defines tags through 4-D time-space curves, which specify the 3-D location of a tagged object over time. It provides a user-friendly mechanism for defining said curves in panoramic video, which rely on curve fitting techniques to minimize the number of images to be labeled. It provides a mechanism for annotating tags with further information. When displaying tagged panoramic video, tags are graphically superimposed on the panoramic video feed using projective projection techniques. From this visualization, a user can then select a given tag and invoke an associated action. Additionally a mechanism whereby tags and associated user-provided information are used as index into panoramic image databases is provided.

Claims

1. A method for attaching a tag having a 4-D time-space function to time-sequenced panoramic images, comprising the steps of:

displaying a first panoramic image;

receiving a user selection of a tag location on the panoramic image;

determining the time-space function defining the location of the tag; and

storing the tag and its time-space function.

2. The method of claim 1, wherein the step of determining the time-space function includes the steps of:

extracting a plurality of panoramic images preceding and following said first panoramic image;

extracting a first patch around the user selected tag location;

identifying patches corresponding to said first patch in said plurality of preceding and following panoramic images; and

utilizing said corresponding patches to determine said time-space function.

3. The method of claim 1, wherein the step of determining the time-space function includes the steps of:

displaying at least one panoramic image preceding or following said first panoramic image;

receiving a user selection of a location corresponding to said selected tag location on at least one panoramic image preceding or following said first panoramic image; and

utilizing said selected tag location and said selected corresponding location to determine said time-space function.

4. The method of claim 3, wherein a plurality of panoramic images preceding or following said first panoramic image are displayed,

wherein for each displayed panoramic image a user selected corresponding location is received so that there are a plurality of selected corresponding locations, and

wherein said selected tag location and said plurality of selected corresponding tag locations are utilized to determine said time-space function.

5. The method of claim 1, wherein the step of determining the time-space function includes the step of:

defining a tag location to be a fixed distance from a reference.

6. The method of claim 5, wherein the reference is the image acquisition device.

7. The method of claim 1, wherein said tag includes a location component and an orientation component.

8. The method of claim 1, further comprising the step of:

receiving tag display criteria from a user, and

wherein said step of storing further stores said tag display criteria.

9. The method of claim 1, further comprising the step of:

receiving tag actions from a user, and

wherein said step of storing further stores said tag action.

10. A machine-readable medium having machine-executable instructions for performing a method for attaching a tag having a 4-D time-space function to time-sequenced panoramic images, the method comprising the steps of:

displaying a first panoramic image;

receiving a user selection of a tag location on the panoramic image;

determining the time-space function defining the location of the tag; and

storing the tag and its time-space function.

11. The machine-readable medium of claim 10, wherein the step of determining the time-space function includes the steps of:

extracting a plurality of panoramic images preceding and following said first panoramic image;

extracting a first patch around the user selected tag location;

identifying patches corresponding to said first patch in said plurality of preceding and following panoramic images; and

utilizing said corresponding patches to determine said time-space function.

12. The machine-readable medium of claim 10, wherein the step of determining the time-space function includes the steps of:

displaying at least one panoramic image preceding or following said first panoramic image;

receiving a user selection of a location corresponding to said selected tag location on at least one panoramic image preceding or following said first panoramic image; and

utilizing said selected tag location and said selected corresponding location to determine said time-space function.

13. The machine-readable medium of claim 12, wherein a plurality of panoramic images preceding or following said first panoramic image are displayed,

wherein for each displayed panoramic image a user selected corresponding location is received so that there are a plurality of selected corresponding locations, and

wherein said selected tag location and said plurality of selected corresponding tag locations are utilized to determine said time-space function.

14. The machine-readable medium of claim 10, wherein the step of determining the time-space function includes the step of:

defining a tag location to be a fixed distance from a reference.

15. The machine-readable medium of claim 14, wherein the reference is the image acquisition device.

16. The machine-readable medium of claim 10, wherein said tag includes a location component and an orientation component.

17. The machine-readable medium of claim 10, the method further comprising the step of:

receiving tag display criteria from a user, and

wherein said step of storing further stores said tag display criteria.

18. The machine-readable medium of claim 10, the method further comprising the step of:

receiving tag actions from a user, and

wherein said step of storing further stores said tag action.

19. A method for defining a tag utilized in displaying a series of panoramic images, comprising the steps of:

determining a 4-D time-space function for the tag;

receiving an annotation for the tag;

receiving an action to be performed upon selection of the tag; and

defining a pointer symbol for use in displaying the tag.

20. The method of claim 19, wherein the step of determining a time-space function includes the steps of:

displaying a first panoramic image;

receiving a user selection of a tag location on said first panoramic image;

extracting a plurality of panoramic images preceding and following said first panoramic image;

extracting a first patch around the user selected tag location;

identifying patches corresponding to said first patch in said plurality of preceding and following panoramic images; and

utilizing said corresponding patches to determine said time-space function.

21. The method of claim 19, wherein the step of determining the time-space function includes the steps of:

displaying a first panoramic image;

receiving a user selection of a tag location on said first panoramic image;

displaying at least one panoramic image preceding or following said first panoramic image;

receiving a user selection of a location corresponding to said selected tag location on at least one panoramic image preceding or following said first panoramic image; and

utilizing said selected tag location and said selected corresponding location to determine said time-space function.

22. The method of claim 21, wherein a plurality of panoramic images preceding or following said first panoramic image are displayed,

wherein for each displayed panoramic image a user selected corresponding location is received so that there are a plurality of selected corresponding locations, and

wherein said selected tag location and said plurality of selected corresponding tag locations are utilized to determine said time-space function.

23. The method of claim 19, further comprising the step of:

receiving an orientation indication of the tag.

24. A machine-readable medium having machine-executable instructions for performing a method for defining a tag utilized in displaying a series of panoramic images, the method comprising the steps of:

determining a 4-D time-space function for the tag;

receiving an annotation for the tag;

receiving an action to be performed upon selection of the tag; and

defining a pointer symbol for use in displaying the tag.

25. The machine-readable medium of claim 24, wherein the step of determining a time-space function includes the steps of:

displaying a first panoramic image;

receiving a user selection of a tag location on said first panoramic image;

extracting a plurality of panoramic images preceding and following said first panoramic image;

extracting a first patch around the user selected tag location;

identifying patches corresponding to said first patch in said plurality of preceding and following panoramic images; and

utilizing said corresponding patches to determine said time-space function.

26. The machine-readable medium of claim 24, wherein the step of determining the time-space function includes the steps of:

displaying a first panoramic image;

receiving a user selection of a tag location on said first panoramic image;

displaying at least one panoramic image preceding or following said first panoramic image;

receiving a user selection of a location corresponding to said selected tag location on at least one panoramic image preceding or following said first panoramic image; and

utilizing said selected tag location and said selected corresponding location to determine said time-space function.

27. The machine-readable medium of claim 26, wherein a plurality of panoramic images preceding or following said first panoramic image are displayed,

wherein for each displayed panoramic image a user selected corresponding location is received so that there are a plurality of selected corresponding locations, and

wherein said selected tag location and said plurality of selected corresponding tag locations are utilized to determine said time-space function.

28. The machine-readable medium of claim 24, the method further comprising the step of:

receiving an orientation indication of the tag.

29. A method for displaying a tag on a panoramic image, comprising the steps of:

loading a panoramic image of the desired item;

providing a tag for said panoramic image, said tag having a 4-D time-space function; and

projecting said tag onto said panoramic image from a selected viewpoint,

wherein the projection of said tag onto said panoramic image changes as the viewpoint changes.

30. The method of claim 29, wherein said tag includes an action to perform when selected, the method further comprising the step of:

performing an action indicated by said tag when said tag is selected.

31. A machine-readable medium having machine-executable instructions for performing a method for displaying a tag on a panoramic image, the method comprising the steps of:

loading a panoramic image of the desired item;

providing a tag for said panoramic image, said tag having a 4-D time-space function; and

projecting said tag onto said panoramic image from a selected viewpoint,

wherein the projection of said tag onto said panoramic image changes as the viewpoint changes.

32. The machine-readable medium of claim 31, wherein said tag includes an action to perform when selected, the method further comprising the step of:

performing an action indicated by said tag when said tag is selected.