Video image-based querying for video content

• US 8,185,543 B1
• Filed: 11/10/2005
• Issued: 05/22/2012
• Est. Priority Date: 11/10/2004
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Choudhry, Shahid, Marwood, David, Henzinger, Monika Hildegard
• Primary Examiner(s): TRUONG, CAM Y T
• Application Number:
  US11/271,558
• Time in Patent Office: 2,385 Days
• Field of Search: 707/3, 707/767, 707/770, 707/771, 382/305, 715/772
• US Class Current: 707/767
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 16/738 Presentation of query results

Abstract

An information retrieval system is provided for processing queries for video content. A server receives a query for video content and returns video articles, as received from broadcast systems or other content providers. Queries are formulated using text, video images, and/or visual content associated with a video article. Content data is associated with video article images and search queries are generated based on user selection of an image. The resulting set of video articles is ranked and a representation of each video article is displayed to the user. The user can play the video article either within the representation or independent of it.

Claims

1. A method of querying for video content using an image selected from a video article, comprising:

receiving user input indicating the image selected from the video article as a criterion for a search query, wherein the selected image is a first selected image;

substituting, by a system comprising a processor, text for the image selected as the criterion for the search query to form a text-based search query for video content, the substituted text comprising at least one categorical term describing the selected image and at least one term from text describing the selected image;

identifying a plurality of video articles with associated text matching at least a portion of the at least one categorical term and the at least one term from the text describing the selected image;

identifying in the plurality of video articles time stamps corresponding to the associated text matching at least a portion of the at least one categorical term and the at least one term from the text describing the selected image;

transmitting, for display, to a user results that include a plurality of images for one of the plurality of video articles, the plurality of images representing a plurality of segments in one of the plurality of video articles, wherein a visual indicator is displayed in association with at least one of the plurality of images that represents a segment including one of the identified time stamps, the visual indicator differentiating the at least one of the plurality of images from other images based on one from a group of color, shape, highlight and border, and the visual indicator displayed before the user selects the at least one of the plurality of images associated with the visual indicator;

storing the results;

receiving user input indicating an additional image selected from the video article;

and forming a second search query for video content with at least one categorical term and at least one term from the text associated with the selected additional image;

wherein the identifying a plurality of video articles further comprises identifying video articles with associated text matching at least a portion of the at least one categorical term and the at least one term from the text associated with either the first selected image or the selected additional image;

wherein the identifying in the plurality of video articles time stamps further comprises identifying time stamps corresponding to either the first selected image or the selected additional image; and

wherein the images transmitted to the user are associated with the time stamps corresponding to either the first selected image or the selected additional image.

2. The method of claim 1, wherein the at least one categorical term and the at least one term from the text associated with the selected image are received from a content provider.

3. The method of claim 1, wherein the image selected from the video article comprises a face, the method further comprising:

recognizing the face as associated with a name, wherein the name is included in the text associated with the selected image, the at least one categorical term and the name forming the text-based search query for video content; and

wherein transmitting to the user a plurality of images for one of the plurality of video articles further comprises transmitting images comprising faces.

4. The method of claim 3, wherein the faces are associated with the name.

5. The method of claim 1, wherein the image selected from the video article comprises a location, the method further comprising:

recognizing the location as associated with a name, wherein the name is included in the text associated with the selected image, the at least one categorical term and the name forming the text-based search query for video content; and

wherein transmitting to the user a plurality of images for one of the plurality of video articles further comprises transmitting images comprising locations.

6. The method of claim 5, wherein the name is selected from the group consisting of a name of a landmark, a name of a city, and a name of a region.

7. The method of claim 1, wherein the at least one categorical term and the at least one term from the text associated with the selected image correspond to the content of the image.

8. The method of claim 1, wherein the text associated with the selected image is selected from the group consisting of a video article title, a video article description, video article-associated metadata, video-article associated broadcast data, and video-article associated closed captioning data.

9. The method of claim 1, wherein the visual indicator differentiates the at least one of the plurality of images from other images based on color.

10. The method of claim 1, wherein the visual indicator differentiates the at least one of the plurality of images from other images based on shape.

11. The method of claim 1, wherein the visual indicator differentiates the at least one of the plurality of images from other images based on highlight.

12. The method of claim 1, wherein the visual indicator differentiates the at least one of the plurality of images from other images based on border.

13. The method of claim 1, wherein the substituted text includes an additional term, the method further comprising:

identifying an additional segment, in at least one video article, associated with the additional term, at least a portion of the at least one categorical term and the at least one term from the text describing the selected image; and

transmitting, for display, to the user an additional image representing the additional segment, the additional image displayed with an additional visual indicator, the additional visual indicator different from the visual indicator.

14. A non-transitory computer-readable storage medium comprising computer-readable instructions, coded on the medium, for querying for video content using an image selected from a video article, the instructions comprising:

receiving user input indicating the image selected from the video article as a criterion for a search query, wherein the selected image is a first selected image;

substituting, by a system comprising a processor, text for the image selected as the criterion for the search query to form a text-based search query for video content, the substituted text comprising at least one categorical term describing the selected image and at least one term from text describing the selected image;

identifying a plurality of video articles with associated text matching at least a portion of the at least one categorical term and the at least one term from the text describing the selected image;

identifying in the plurality of video articles time stamps corresponding to the associated text matching at least a portion of the at least one categorical term and the at least one term from the text describing the selected image;

transmitting, for display, to a user results that include a plurality of images for one of the plurality of video articles, the plurality of images representing a plurality of segments in one of the plurality of video articles, wherein a visual indicator is displayed in association with at least one of the plurality of images that represents a segment including one of the identified time stamps, the visual indicator differentiating the at least one of the plurality of images from other images based on one from a group of color, shape, highlight and border, and the visual indicator displayed before the user selects the at least one of the plurality of images associated with the visual indicator;

storing the results;

receiving user input indicating an additional image selected from the video article;

and forming a second search query for video content with at least one categorical term and at least one term from the text associated with the selected additional image;

wherein the identifying a plurality of video articles further comprises identifying video articles with associated text matching at least a portion of the at least one categorical term and the at least one term from the text associated with either the first selected image or the selected additional image;

wherein the identifying in the plurality of video articles time stamps further comprises identifying time stamps corresponding to either the first selected image or the selected additional image; and

wherein the images transmitted to the user are associated with the time stamps corresponding to either the first selected image or the selected additional image.

15. The computer-readable storage medium of claim 14, wherein the image selected from the video article comprises a face, the instructions further comprising:

recognizing the face as associated with a name, wherein the name is included in the text associated with the selected image, the at least one categorical term and the name forming the text-based search query for video content; and

wherein transmitting to the user a plurality of images for one of the plurality of video articles further comprises transmitting images comprising faces.

16. The computer-readable storage medium of claim 14, wherein the image selected from the video article comprises a location, the instructions further comprising:

recognizing the location as associated with a name, wherein the name is included in the text associated with the selected image, the at least one categorical term and the name forming the text-based search query for video content; and

wherein transmitting to the user a plurality of images for one of the plurality of video articles further comprises transmitting images comprising locations.

17. The computer-readable storage medium of claim 14, wherein the at least one categorical term and the at least one term from the text associated with the selected image corresponds to the content of the image.

18. A system for querying for video content using an image selected from a video article, the system comprising:

a processor;

means for receiving user input indicating the image selected from the video article as a criterion for a search query, wherein the selected image is a first selected image;

means for substituting text for the image selected as the criterion for the search query to form a text-based search query for video content, the substituted text comprising at least one categorical term describing the selected image and at least one term from text describing the selected image;

means for identifying a plurality of video articles with associated text matching at least a portion of the at least one categorical term and the at least one term from the text describing the selected image;

means for identifying in the plurality of video articles time stamps corresponding to the associated text matching at least a portion of the at least one categorical term and the at least one term from the text describing the selected image;

means for transmitting, for display, to a user results that include a plurality of images for one of the plurality of video articles, the plurality of images representing a plurality of segments in one of the plurality of video articles, wherein a visual indicator is displayed in association with at least one of the plurality of images that represents a segment including one of the identified time stamps, the visual indicator differentiating the at least one of the plurality of images from other images based on one from a group of color, shape, highlight and border, and the visual indicator displayed before the user selects the at least one of the plurality of images associated with the visual indicator;

means for storing the results;

means for receiving user input indicating an additional image selected from the video article;

and means for forming a second search query for video content with at least one categorical term and at least one term from the text associated with the selected additional image;

wherein the identifying a plurality of video articles further comprises identifying video articles with associated text matching at least a portion of the at least one categorical term and the at least one term from the text associated with either the first selected image or the selected additional image;

wherein the identifying in the plurality of video articles time stamps further comprises identifying time stamps corresponding to either the first selected image or the selected additional image; and

wherein the images transmitted to the user are associated with the time stamps corresponding to either the first selected image or the selected additional image.

19. The system of claim 18, wherein the image selected from the video article comprises a face, the system further comprising:

means for recognizing the face as associated with a name, wherein the name is included in the text associated with the selected image, the at least one categorical term and the name forming the text-based search query for video content; and

wherein transmitting to the user a plurality of images for one of the plurality of video articles further comprises displaying images comprising faces.

20. The system of claim 18, wherein the image selected from the video article comprises a location, the method further comprising:

means for recognizing the location as associated with a name, wherein the name is included in the text associated with the selected image, the at least one categorical term and the name forming the text-based search query for video content; and

wherein transmitting to the user a plurality of images for one of the plurality of video articles further comprises transmitting images comprising locations.

21. The system of claim 18, wherein the at least one categorical term and the at least one term from the text associated with the selected image correspond to the content of the image.

22. The system of claim 18, wherein the storing comprises caching.