Searching based on audio and/or visual features of documents

• US 8,799,257 B1
• Filed: 03/19/2012
• Issued: 08/05/2014
• Est. Priority Date: 03/19/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hubinette, Carl Fredrik
• Primary Examiner(s): DWIVEDI, MAHESH H
• Application Number:
  US13/423,702
• Time in Patent Office: 869 Days
• Field of Search: 707/705, 707/706
• US Class Current: 707/706
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

One or more computer devices may generate a keyword based on a visual feature or an audible feature of a document. The visual feature may include a color of a background of the document, a color of text in the document, a size of the text in the document, or a feature relating to a video in the document. The audible feature may relate to sound or music that is associated with the document. The one or more computer devices may further associate, in an index, the keyword with information identifying the document, receive a search query from a client, the search query including a term related to the keyword, identify documents based on the search query and using the index, the identified documents including the document, and provide a search results document to the client. The search results document may identify one or more of the identified documents.

Claims

1. A method comprising:

receiving, by one or more computer devices, a search query from a client,

the search query including a plurality of terms entered by a user of the client,

the plurality of terms including;

a first term related to a topic of a web page,

a second term related to a first keyword of a plurality of keywords,

a third term related to a second keyword of the plurality of keywords,

the plurality of keywords corresponding to a plurality of visual features,

the plurality of visual features including;

 

a color of a background of the web page,

 

a feature relating to an image in the web page, and

 

at least one of;

 

a color or a size of text in the web page, or

 

a feature relating to a video in the web page,

the first keyword relating to a first visual feature of the plurality of visual features,

the second keyword relating to a second visual feature of the plurality of visual features,

 

one of the first visual feature or the second visual feature including the color of the background of the web page, and

a fourth term to be searched in relation to the one of the first visual feature or the second visual feature,

an index associating the plurality of keywords with information identifying the web page;

converting, by the one or more computer devices, a format of the second term to a format of the first keyword in the index when the format of the second term does not match the format of the first keyword;

identifying, by the one or more computer devices, documents based on the search query and using the index,

the identified documents including the web page; and

providing, by the one or more computer devices, a search results document to the client,

the search results document identifying one or more of the identified documents.

2. The method of claim 1, further comprising:

generating the plurality of keywords based on the plurality of visual features,

the plurality of visual features including;

the color of the background of the web page,

the feature relating to the image in the web page,

the color or the size of the text in the web page, and

the feature relating to the video in the web page;

associating, in the index, the plurality of keywords with the information identifying the web page, and

where the plurality of terms further includes;

a fifth term to be searched in relation to another one of the first visual feature or the second visual feature.

3. The method of claim 1, further comprising:

generating the plurality of keywords based on the plurality of visual features, where generating the plurality of keywords comprises;

generating a keyword based on the feature relating to the image in the web page,

the feature relating to the image including information regarding an object in the image; and

associating, in the index, the keyword with the information identifying the web page,

where the plurality of terms further includes;

a fifth term identifying the keyword that is generated based on the feature relating to the image, and

a particular term to be searched in relation to the feature relating to the image,

the particular term identifying the object in the image.

4. The method of claim 3, where generating the keyword includes:

obtaining the image,

obtaining information based on analyzing the image, and

generating the keyword based on the obtained information.

5. The method of claim 1, further comprising:

generating a keyword based on an audible feature of the web page,

the audible feature corresponding to;

an indication of whether a sound is generated when the web page is rendered for display, or

an indication of whether music is generated when the web page is rendered for display,

a name of music that is generated when the web page is rendered for display, or

a name of an artist that is associated with music that is generated when the web page is rendered for display; and

associating, in the index, the keyword with the information identifying the web page,

where the plurality of terms further includes;

a term identifying the keyword that is generated based on the audible feature, and

a term to be searched in relation to the audible feature.

6. The method of claim 5, where generating the keyword further includes:

obtaining an audio file corresponding to the audible feature,

obtaining information based on analyzing the audio file, and

generating the keyword based on the obtained information,

where obtaining the information includes;

playing the audio file, and

obtaining the information based on playing the audio file.

7. The method of claim 1, further comprising:

generating at least one other keyword based on at least one of;

a length of the web page when the web page is rendered for display, or

a style of the web page when the web page is rendered for display; and

associating, in the index, the at least one other keyword with the information identifying the web page;

where the plurality of terms further includes;

a term related to the at least one other keyword, and

a term to be searched in relation to at least one of;

the length of the web page, or

the style of the web page.

8. The method of claim 1, further comprising:

generating the plurality of keywords based on the plurality of visual features,

where generating the plurality of keywords includes;

extracting information from hypertext markup language code associated with the web page, and

generating one or more of the plurality of keywords further based on the extracted information.

9. A system comprising:

one or more computer devices to;

receive a search query from a client,

the search query including a plurality of terms entered by a user of the client,

the plurality of terms including;

a first term related to a topic of a web page,

a second term related to a first keyword of a plurality of keywords,

a third term related to a second keyword of the plurality of keywords,

 

the plurality of keywords corresponding to a plurality of features of the web page,

 

the plurality of features including;

 

a color of a background of the web page,

 

a feature relating to an image in the web page, and

 

a feature relating to an audible feature of the web page,

 

the first keyword relating to a first feature of the plurality of features,

 

the second keyword relating to a second feature of the plurality of features, and

a fourth term to be searched in relation to one of the first feature or the second feature,

an index associating the plurality of keywords with information identifying the web page,

convert a format of the second term to a format of the first keyword in the index when the format of the second term does not match the format of the first keyword;

identify documents based on the search query and using the index,

the identified documents including the web page, and provide a search results document to the client,

the search results document identifying one or more documents of the identified documents,

the one or more documents including the web page.

10. The system of claim 9, where the one or more computer devices are further to:

generate the plurality of keywords based on the plurality of features of the web page, and

where, when generating the plurality of keywords, the one or more computer devices are to;

generate the plurality of keywords for;

the color of the background of the web page,

the feature relating to the image in the web page, and

the feature relating to the audible feature of the web page,

a color or a size of text in the web page, and

a feature relating to a video in the web page, and

where the plurality of terms further includes;

a term identifying another one of the plurality of keywords that is generated based on another one of the plurality of features, and

a term to be searched in relation to the other one of the plurality of features.

11. The system of claim 9, where the one or more computer devices are further to:

generate a keyword based on the feature relating to the image in the web page,

the feature relating to the image including a quantity of images in the web page, and

associate, in the index, the keyword with the information identifying the web page,

where the plurality of terms further includes;

a term identifying the keyword that is generated based on the feature relating to the image, and

a term to be searched in relation to the quantity of images in the web page.

12. The system of claim 11, where, when generating the keyword, the one or more computer devices are to:

obtain the image,

obtain information based on analyzing the image, and

generate the keyword based on the obtained information.

13. The system of claim 9, where the one or more computer devices are further to:

generate a keyword based on the audible feature of the web page, the audible feature corresponding to;

an indication of whether a sound is generated when the web page is rendered for display,

an indication of whether music is generated when the web page is rendered for display,

a name of music that is generated when the web page is rendered for display, or

a name of an artist that is associated with music that is generated when the web page is rendered for display; and

associate, in the index, the keyword with the information identifying the web page,

where the plurality of terms further includes;

a term identifying the keyword, and

a term to be searched in relation to the audible feature.

14. The system of claim 13, where, when generating the keyword, the one or more computer devices are further to:

obtain an audio file corresponding to the audible feature,

obtain information based on analyzing the audio file, and

generate the keyword based on the obtained information,

where, when obtaining the information, the one or more computer devices are to;

play the audio file, and

obtain the information based on playing the audio file.

15. The system of claim 9, where the one or more computer devices are further to:

generate at least one other keyword based on;

information identifying a length of the web page when the web page is rendered for display, and

information identifying a style of the web page when the web page is rendered for display; and

associate, in the index, the at least one other keyword with the information identifying the web page;

where the search query further includes;

a term related to the at least one other keyword, and

a term to be searched in relation to;

the length of the web page, and

the style of the web page.

16. The system of claim 9, where the one or more computer devices are further to:

generate the plurality of keywords based on the plurality of features of the web page, and

where, when generating the plurality of keywords, the one or more computer devices are to;

extract information from hypertext markup language code associated with the web page, and

generate one or more of the plurality of keywords based on the extracted information.

17. A non-transitory computer-readable medium for storing instructions, the instructions comprising:

one or more instructions, which when executed by one or more processors, cause the one or more processors to;

receive a search query from a client,

the search query including a plurality of terms entered by a user of the client,

the plurality of terms including;

a first term related to a topic of a document,

a second term related to one of a plurality keywords,

 

the one of the plurality of keywords being generated based on one of a plurality of features,

 

the plurality of features including;

 

a color of a background of the document,

 

a feature relating to an image in the document, and

 

a feature relating to an audible feature of the document, and

a third term to be searched in relation to the one of the plurality of features,

an index associating the plurality of keywords with information identifying the document,

convert a format of the second term to a format of the one of the plurality of keywords in the index when the format of the second term does not match the format of the one of the plurality of keywords,

identify documents based on the search query and using the index,

the identified documents including the document, and provide a search results document to the client,

the search results document identifying one or more documents of the identified documents,

the one or more documents including the document.

18. The computer-readable medium of claim 17, where the instructions further comprise:

one or more instructions to generate a keyword based on the feature relating to the image in the document,

where the feature relating to the image includes at least one of;

an object in the image, or

a quantity of images in the document, and

one or more instructions to associate, in the index, the keyword with the information identifying the document,

where the one or more instructions to generate the keyword include;

one or more instructions to obtain the image,

one or more instructions to obtain information based on analyzing the image, and

one or more instructions to generate the keyword based on the obtained information, and

where the plurality of terms further includes;

a term identifying the keyword that is generated based on the feature relating to the image, and

a term to be searched in relation to;

the object in the image, or

the quantity of images in the document.

19. The computer-readable medium of claim 17, where the instructions further include:

one or more instructions to generate a keyword based on the audible feature of the document,

the audible feature corresponding to;

an indication of whether a sound is generated when the document is rendered for display,

an indication of whether music is generated when the document is rendered for display,

a name of music that is generated when the document is rendered for display, or

a name of an artist that is associated with music that is generated when the document is rendered for display; and

one or more instructions to associate, in the index, the keyword with the information identifying the document,

where the plurality of terms further includes;

a term identifying the keyword that is generated based on the audible feature, and

a term to be searched in relation to;

the indication of whether a sound is generated when the document is rendered for display,

the indication of whether music is generated when the document is rendered for display,

the name of music that is generated when the document is rendered for display, or

the name of an artist.

20. The computer-readable medium of claim 19, where the one or more instructions to generate the keyword include:

one or more instructions to obtain an audio file corresponding to the audible feature,

one or more instructions to obtain information based on analyzing the audio file, and

one or more instructions to generate the keyword based on the obtained information,

where the one or more instructions to obtain the information based on analyzing the audio file include;

one or more instructions to play the audio file, and

one or more instructions to obtain the information based on playing the audio file.