Telestration system for command processing

• US 9,384,217 B2
• Filed: 03/11/2013
• Issued: 07/05/2016
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Inventors: Ayoub, Ramy S., Mok, Swee M.
• Primary Examiner(s): ABDULSELAM, ABBAS I
• Application Number:
  US13/794,763
• Publication Number:
  US 20140253472A1
• Time in Patent Office: 1,212 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/254 Management at additional data server, e.g. shopping server, rights management server arrangements for maintenance or administration in data networks H04L41/00; network services using third party service providers H04L67/53
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/42224 Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region

Abstract

In one embodiment, a method detects a user input for a telestration on an image being displayed on a display device and determines a plurality of image portions of the image based on the telestration. The plurality of image portions are determined by a boundary around each image portion based on the telestration. The method then determines a set of tags for the plurality of image portions. The set of tags are determined based on image recognition of content in the plurality of image portions. An operator is determined based on the telestration where the operator characterizes an operation to perform for the plurality of image portions. The method determines a search query based on applying the operator to the set of tags and causes a search to be performed using the search query.

Claims

1. A method comprising:

detecting, by a computing device, a user input for a telestration on an image being displayed on a display device;

determining, by the computing device, a plurality of image portions of the image based on the telestration, wherein the plurality of image portions are determined by a boundary around each image portion based on the telestration;

determining, by the computing device, a set of tags for the plurality of image portions, wherein the set of tags are determined based on image recognition of content in the plurality of image portions;

determining, by the computing device, an operator based on the telestration, wherein the operator characterizes an operation to perform for the plurality of image portions;

determining, by the computing device, a search query based on applying the operator to the set of tags; and

causing, by the computing device, a search to be performed using the search query.

2. The method of claim 1, wherein detecting the user input comprises:

detecting, by the computing device, a first user input for a first telestration for a first image portion in the plurality of image portions, wherein the first telestration is used to form a boundary around the first image portion; and

detecting, by the computing device, a second user input for a second telestration for a second image portion in the plurality of image portions, wherein the second telestration is used to form a boundary around the second image portion.

3. The method of claim 2, wherein determining the operator comprises detecting, by the computing device, a third user input for a third telestration that defines the operator.

4. The method of claim 1, further comprising determining, by the computing device, a context for the operator, wherein the context is used in determining the search query.

5. The method of claim 4, wherein the context comprises a distance from one of the plurality of image portions or a location of the operator in the image.

6. The method of claim 1, further comprising determining, by the computing device, an intent for the operator, wherein the intent is used in determining the operation to perform for the operator.

7. The method of claim 6, wherein determining the intent for the operator comprises analyzing a telestration mark for the operator or performing image recognition on the telestration mark to determine the intent.

8. The method of claim 1, wherein determining the set of tags for the plurality of image portions comprises:

sending information for the plurality of image portions to a service that performs the image recognition of content in the plurality of image portions to generate the set of tags; and

receiving the set of tags from the service.

9. The method of claim 1, wherein the image recognition includes a text recognition or object recognition.

10. The method of claim 1, wherein causing the search to be performed comprises:

sending, by the computing device, the search query to a service configured to perform the search using the search query to generate a search result; and

receiving, by the computing device, the search result from the service.

11. The method of claim 1, wherein detecting the user input comprises receiving, by the computing device, the user input via an input device configured to cause the telestration to be displayed on the image being displayed on the display device.

12. The method of claim 11, wherein the input device is a remote control that includes a pointer that is used to form the telestration on the image.

13. The method of claim 11, wherein the input device includes a touchscreen that receives the user input for the telestration and causes display of the telestration on the image.

14. The method of claim 1, further comprising receiving a submit command, wherein the submit command causes submitting of the search query to determine a search result.

15. The method of claim 1, further comprising receiving a command operation, wherein the command operation causes an instruction associated with the command to be performed with respect to the telestration.

16. The method of claim 15, further comprising determining a context associated with the command operation, wherein the context is used to determine an action to perform for the instruction associated with the command.

17. The method of claim 1, further comprising outputting a search result for the search query.

18. A method comprising:

detecting, by a computing device, a first user input for a first telestration for a first image portion;

determining, by the computing device, a first boundary around the first image portion based on the first telestration;

detecting, by the computing device, a second user input for a second telestration for a second image portion;

determining, by the computing device, a second boundary around the second image portion based on the second telestration;

determining, by the computing device, an operator for the first telestration and the second telestration, wherein the operator characterizes an operation to perform for the first image portion and the second image portion;

detecting, by the computing device, a third telestration for a command; and

determining, by the computing device, an action to perform with respect to the first telestration, the second telestration, or the operator.

19. The method of claim 18, further comprising:

sending, by the computing device, the first image portion to an image processing service to have image recognition on the first image portion performed;

receiving, by the computing device, a first set of tags describing content of the first image portion based on the image recognition;

sending, by the computing device, the second image portion to the image processing service to have image recognition on the second image portion performed;

receiving, by the computing device, a second set of tags describing content of the second image portion based on the image recognition;

determining, by the computing device, a search query based on applying the operator to the first set of tags and the second set of tags, wherein the command is a search query command configured to cause a search to be performed; and

causing, by the computing device, the search to be performed using the search query.

20. An apparatus comprising:

one or more computer processors; and

a non-transitory computer-readable storage medium comprising instructions that, when executed, control the one or more computer processors to be configured for;

detecting a user input for a telestration on an image being displayed on a display device;

determining a plurality of image portions of the image based on the telestration, wherein the plurality of image portions are determined by a boundary around each image portion based on the telestration;

determining a set of tags for the plurality of image portions, wherein the set of tags are determined based on image recognition of content in the plurality of image portions;

determining an operator based on the telestration, wherein the operator characterizes an operation to perform for the plurality of image portions;

determining a search query based on applying the operator to the set of tags; and

causing a search to be performed using the search query.