Indexing mobile onscreen content
First Claim
1. A system comprising:
- at least one processor;
an index that associates, for a user, key items with images of screens captured from a mobile device used by the user, the screens having content generated by a mobile application executing on the mobile device, each image being associated with a mobile application that generated the screen captured in the image; and
memory storing instructions that, when executed by the at least one processor, cause the system to;
determine, using the index, that a first key item is responsive to a text-based query initiated at the mobile device by the user and that the first key item relates to a class of mobile application,identify responsive images, the responsive images being associated with the first key item in the index,rank the responsive images, including boosting respective ranks of responsive images generated by mobile applications in the class,provide a search result to the mobile device that includes a portion of highest ranked responsive images, wherein each portion includes an area of the corresponding image that displays the first key item, andresponsive to receiving a selection of a first portion, open the mobile application that generated the first portion.
2 Assignments
0 Petitions
Accused Products
Abstract
Systems and methods for indexing onscreen content on a mobile device are provided. For example, a method can include determining, using an index for a user that associates key items with images of screens captured from a mobile device of the user, that a first key item is responsive to a query initiated at the mobile device, and identifying responsive images, the responsive images being associated with the first key item. The method may also include highlighting, in the responsive images, areas corresponding to the first key item, and providing a search result to the mobile device that includes portions of the responsive images. The portions of the responsive images may be areas corresponding to the first key item in the images.
-
Citations
26 Claims
-
1. A system comprising:
-
at least one processor; an index that associates, for a user, key items with images of screens captured from a mobile device used by the user, the screens having content generated by a mobile application executing on the mobile device, each image being associated with a mobile application that generated the screen captured in the image; and memory storing instructions that, when executed by the at least one processor, cause the system to; determine, using the index, that a first key item is responsive to a text-based query initiated at the mobile device by the user and that the first key item relates to a class of mobile application, identify responsive images, the responsive images being associated with the first key item in the index, rank the responsive images, including boosting respective ranks of responsive images generated by mobile applications in the class, provide a search result to the mobile device that includes a portion of highest ranked responsive images, wherein each portion includes an area of the corresponding image that displays the first key item, and responsive to receiving a selection of a first portion, open the mobile application that generated the first portion. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A method comprising:
-
determining, using an index for a user that associates key items with images of screens captured from a mobile device of the user, that a first key item is responsive to a query initiated at the mobile device and that the first key item relates to a class of mobile applications, each image being associated in the index with a mobile application that generated the screen captured in the image; identifying responsive images, the responsive images being associated with the first key item; ranking the responsive images, including boosting respective ranks of responsive images generated by mobile applications in the class; highlighting, in the responsive images, areas corresponding to the first key item; providing a search result to the mobile device that includes portions of highest-ranked responsive images; and responsive to receiving a selection of a first portion, open the mobile application that generated the first portion. - View Dependent Claims (10, 11, 12, 13, 14, 15, 16)
-
-
17. A mobile device comprising:
-
at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the mobile device to; capture an image of a screen on the mobile device, the screen being content generated by a mobile application that is displayed on a display of the mobile device, provide the image to a server, send a query to the server the query relating to a class of mobile applications, receive a search result from the server, the search result including portions of respective screen capture images that are responsive to the query ordered by respective rank, the screen capture images being previously provided by the mobile device to the server, and the respective ranks of images generated by mobile applications in the class represent a boosted rank, display the search result on the display of the mobile device, and responsive to receiving a selection of a first portion, open the mobile application that generated the content in first portion. - View Dependent Claims (18, 19, 20)
-
-
21. A mobile device comprising:
-
at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the mobile device to; receive an image of a screen captured on the mobile device, the screen being displayed on a display device of the mobile device, and the image including content generated by a mobile application executing when the image was captured, determine a plurality of key items in the image by performing recognition on the image, at least one of the plurality of key items being an entity in a graph-based data store, wherein key items located in areas of the image that are static for the mobile application executing when the image was captured are excluded from the plurality of key items, and associate, in an index, each of the plurality of key items with the image and the mobile application that generated the content in the image, and use the index to provide the image in a search result responsive to a key item of the plurality of key items being responsive to a query. - View Dependent Claims (22, 23, 24, 25, 26)
-
Specification