System and method for using on-image gestures and multimedia content elements as search queries

• US 10,380,164 B2
• Filed: 03/10/2014
• Issued: 08/13/2019
• Est. Priority Date: 10/26/2005
• Status: Active Grant

• Current Assignee: Cortica Ltd.
• Original Assignee: Cortica Ltd.
• Inventors: Raichelgauz, Igal, Odinaev, Karina, Zeevi, Yehoshua Y.
• Primary Examiner(s): Nilsson, Eric
• Application Number:
  US14/203,047
• Publication Number:
  US 20140195513A1
• Time in Patent Office: 1,982 Days
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: H04H 20/103 Transmitter-side switching
• CPC Class Codes: H04H 20/38 Arrangements for distribution where lower stations, e.g. receivers, interact with the broadcast
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/33 Arrangements for monitoring the users' behaviour or opinions
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/56 Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
• CPC Class Codes: H04H 60/66 for using the result on distributors' side
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/466 Learning process for intelligent management, e.g. learning user preferences for recommending movies services using the results of monitoring in broadcast systems H04H60/61
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

A method and system for searching a plurality of information sources using a multimedia content element are provided. The method comprises receiving an input search query including the multimedia content element and at least one on-image gesture detected thereon; generating at least one signature for the multimedia content element; analyzing the at least one on-image gesture to determine at least one portion of the multimedia content element in which a user is interested; generating at least one signature for each of the at least one portion of the multimedia content element; generating a textual search query using the at least the one generated signature; searching the plurality of information sources using the generated textual search query; and causing the display of search results retrieved from the plurality of information sources on a user device.

Claims

1. A method for searching a plurality of information sources using a multimedia content element, comprising:

receiving an input search query including the multimedia content element and at least one on-image gesture detected thereon;

analyzing the at least one on-image gesture to determine at least one portion of the multimedia content element in which a user is interested;

generating at least one first signature for each of the at least one portion of the multimedia content element, wherein the at least one first signature is generated by a signature generator system including a plurality of at least partially statistically independent computational cores;

generating a textual search query using the at least the one first signature, wherein generating the textual search query further comprises matching the at least one first signature to a plurality of second signatures, wherein the textual search query is generated based on text associated with at least one matching second signature;

searching the plurality of information sources using the generated textual search query; and

causing the display of search results retrieved from the plurality of information sources on a user device.

2. The method of claim 1, further comprising:

creating an overlay on the at least one multimedia content element for receiving at least one search parameter representing an intent of the user, wherein the overlay is at least a search box.

3. The method of claim 2, further comprising:

receiving at least one of demographic information and location information related to a user submitting the input search query; and

optimizing the generated textual search query based on at least one of the demographic information, the location information, and the at least one search parameter.

4. The method of claim 1, wherein the on-image gesture is any one of:

a touch gesture, a sketch gesture, a scroll over at least a portion of the multimedia content element, a clicks over the at least a portion of the multimedia content element, wherein the touch gesture is detected on a user device having a touch screen display.

5. The method of claim 4, wherein analyzing the at least on-image gesture further comprises:

determining a type of the on-image gesture and a pattern of the on-image gesture with respect to the multimedia content element.

6. The method of claim 5, further comprising:

clustering the at least one first signature; and

searching the query database using the clustered signature.

7. The method of claim 1, wherein generating the textual search query further comprises:

searching a query database for a textual search query that matches the at least one first signature, wherein the query database maintains mappings of the second signatures to text queries, and wherein a plurality of the second signatures are mapped to a single textual search query.

8. The method of claim 1, wherein generating the textual search query further comprises:

matching the at least one first signature against a plurality of concept structures, and using a metadata of at least one matching concept structure as the textual search query;

wherein each concept structure comprises signature reduced clusters and metadata related to the signature reduced clusters.

9. The method of claim 1, further comprising:

determining a context of the multimedia content element respective of the at least one first signature; and

determining the textual search query based on the determined context.

10. The method of claim 1, wherein causing the display of search results retrieved from the plurality of information further comprises:

filtering the retrieved search results using the at least one first signature.

11. The method of claim 1, wherein the at least one multimedia content element is at least one of:

an image, graphics, a video stream, a video clip, a video frame, a photograph, images of signals, and portions thereof.

12. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 1.

13. The method of claim 1, wherein the at least one first signature is in a format useable to query a database of stored concepts to detect at least one concept stored in the database that matches the at least one first signature, wherein a concept is a collection of signatures representing elements of unstructured data and metadata describing the concept.

14. The method according to claim 1 wherein the plurality of at least partially statistically independent computational cores comprise neural networks and wherein the at least one first signature for each of the at least one portion of the multimedia content element represents a response of at least some of the neural networks to the the at least one portion of the multimedia content.

15. The method according to claim 14 wherein the plurality of at least partially statistically independent computational cores are implemented in one or more integrated circuits.

16. A system for searching a plurality of information sources using a multimedia content element, comprising:

a processor; and

a memory coupled to the processor, the memory contains instructions that when executed by the processor cause the system to;

receive an input search query including the multimedia content element and at least one on-image gesture detected thereon;

analyze the at least on-image gesture to determine at least one portion of the multimedia content element in which a user is interested;

generate, by a signature generator, at least one first signature for each of the at least one portion of the multimedia content element, wherein the signature generator includes a plurality of at least partially statistically independent computational cores;

generate a textual search query using the at least the one signature, wherein generating the textual search query further comprises matching the at least one first signature to a plurality of second signatures, wherein the textual search query is generated based on text associated with at least one matching second signature;

search the plurality of information sources using the generated textual search query; and

cause the display of search results retrieved from the plurality of information sources on a user device.

17. The system of claim 16, wherein the signature generator is communicatively connected to the processor.

18. The system of claim 16, wherein the system is further configured to:

create an overlay on the at least one multimedia content element for receiving at least one search parameter representing an intent of the user, wherein the overlay is at least a search box.

19. The system of claim 18, wherein the system is further configured to:

receive at least one of demographic information and location information related to a user submitting the input search query; and

optimize the generated textual search query based on at least one of the demographic information, the location information, and the at least one search parameter.

20. The system of claim 16, wherein the on-image gesture is any one of:

a touch gesture, a sketch gesture, a scroll over at least a portion of the multimedia content element, a clicks over the at least a portion of the multimedia content element, wherein the touch gesture is detected on a user device having a touch screen display.

21. The system of claim 20, wherein the system is further configured to:

determine a type of the on-image gesture and a pattern of the on-image gesture with respect to the multimedia content element.

22. The system of claim 21, wherein the system is further configured to:

search a query database for a textual search query that matches the at least one first signature, wherein the query database maintains mappings of the second signatures to text queries, and wherein a plurality of second signatures are mapped to a single textual search query.

23. The system of claim 22, wherein the system is further configured to:

cluster the at least one first signature; and

search the query database using the clustered signature.

24. The system of claim 16, wherein the system is further configured to:

match the at least one first signature against a plurality of structures, and use a metadata of at least one matching concept structure as the textual search query;

wherein each concept structure comprises signature reduced clusters and metadata related to the signature reduced clusters.

25. The system of claim 16, wherein the system is further configured to:

determine a context of the multimedia content element respective of the at least one first signature; and

determine the textual search query based on the determined context.

26. The system of claim 16, wherein the multimedia content element is at least one of:

an image, graphics, a video stream, a video clip, a video frame, a photograph, images of signals, and portions thereof.

27. The system of claim 16, wherein the at least one first signature is in a format useable to query a database of stored concepts to detect at least one concept stored in the database that matches the at least one first signature, wherein a concept is a collection of signatures representing elements of unstructured data and metadata describing the concept.

28. The system according to claim 16 wherein the plurality of at least partially statistically independent computational cores comprise neural networks and wherein the at least one first signature for each of the at least one portion of the multimedia content element represents a response of at least some of the neural networks to the the at least one portion of the multimedia content.

29. The system according to claim 28 wherein the plurality of at least partially statistically independent computational cores are implemented in one or more integrated circuits.