Content image search

• US 20080226119A1
• Filed: 03/16/2007
• Published: 09/18/2008
• Est. Priority Date: 03/16/2007
• Status: Active Grant

• Current Assignee: Saturn Licensing LLC (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Hardacker, Robert, Candelore, Brant
• Granted Patent: US 8,861,898 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data

Abstract

An embodiment of the present invention includes a search technique for image query. A region of interest (ROI) containing an image in a scene of a displayed content is captured. The ROI is analyzed locally or remotely to search for related content associated with the image. The related content is received.

Claims

1. A method comprising:

capturing a region of interest (ROI) containing an image in a scene of a displayed content, the displayed content being from a broadcast content or a package media;

analyzing the ROI locally or remotely to search for related content associated with the image; and

receiving the related content.

2. The method of claim 1 wherein capturing the ROI comprises:

highlighting an object in the scene to correspond to the ROI using a remote control unit.

3. The method of claim 1 wherein analyzing the ROI locally comprises:

selecting image features of the image in the ROI; and

matching the image features to locate the related content.

4. The method of claim 3 wherein analyzing the ROI locally further comprises:

obtaining ancillary information associated with the scene, the ancillary information including at least one of metadata, Web link, scene timing information, caption, electronic program guide (EPG) data, content provider information, and audio segment; and

searching the related content using the ancillary information.

5. The method of claim 1 wherein analyzing the ROI remotely comprises:

transmitting the ROI over a network to a server for analysis, the server returning the related content.

6. The method of claim 5 wherein analyzing the ROI remotely further comprises:

obtaining ancillary information associated with the scene, the ancillary information including at least one of metadata, Web link, scene timing information, caption, electronic program guide (EPG) data, and audio segment; and

transmitting the ancillary information together with the ROI.

7. The method of claim 1 wherein receiving the related content comprises:

displaying the related content on one of a main display rendering the displayed content and a wireless device having a display.

8. The method of claim 7 wherein displaying the related content comprises:

displaying the related content on the displayed content using one of a picture-in-picture (PiP) mode, a graphic overlay, a transparency display, or a screen below or to side of the main display.

9. The method of claim 1 further comprising:

disabling the broadcast content if the related content corresponds to an inhibitory feature.

10. The method of claim 1 wherein analyzing the ROI comprises:

performing an optical character recognition (OCR) analysis if the image has textual information used to help identifying the image.

17. The article of manufacture of claim 1L wherein the data causing the machine to perform receiving the related content comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

displaying the related content on one of a main display rendering the displayed content and a wireless device having a display.

18. The article of manufacture of claim 17 wherein the data causing the machine to perform displaying the related content comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

displaying the related content on the displayed content using one of a picture-in-picture (PiP) mode, a graphic overlay, a transparency display, a screen below or to side of the main display.

11. An article of manufacture comprising:

a machine-accessible medium including data that, when accessed by a machine, cause the machine to perform operations comprising;

capturing a region of interest (ROI) containing an image in a scene of a displayed content, the displayed content being from a broadcast content or a package media.

analyzing the ROI locally or remotely to search for related content associated with the image; and

receiving the related content.

12. The article of manufacture of claim 11 wherein the data causing the machine to perform capturing the ROI comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

highlighting an object in the scene to correspond to the ROI using a remote control unit.

13. The article of manufacture of claim 11 wherein the data causing the machine to perform analyzing the ROI locally comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

selecting image features of the image in the ROI; and

matching the image features to locate the related content.

14. The article of manufacture of claim 13 wherein the data causing the machine to perform analyzing the ROI locally further comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

obtaining ancillary information associated with the scene, the ancillary information including at least one of metadata, Web link, scene timing information, caption, electronic program guide (EPG) data, content provider information, and audio segment; and

searching the related content using the ancillary information.

15. The article of manufacture of claim 11 wherein the data causing the machine to perform analyzing the ROI remotely comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

transmitting the ROI over a network to a server for analysis, the server returning the related content.

16. The article of manufacture of claim 15 wherein the data causing the machine to perform analyzing the ROI remotely further comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

obtaining ancillary information associated with the scene, the ancillary information including at least one of metadata, Web link, scene timing information, caption, electronic program guide (EPG) data, and audio segment; and

transmitting the ancillary information together with the ROI.

19. The article of manufacture of claim 11 wherein the data further comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

disabling the displayed content if the related content corresponds to an inhibitory feature.

20. The article of manufacture of claim 11 wherein the data causing the machine to perform analyzing the ROI comprises data that, when accessed by a machine, cause the machine to perform operations comprising:

performing an optical character recognition (OCR) analysis if the image has textual information used to help identifying the image.

21. A system comprising:

a display device to display a displayed content, the displayed content being from a broadcast content or a package media;

a remote control unit to allow a viewer to interact with the displayed content; and

an integrated receiver and decoder (IRD) coupled to the display device to receive the displayed content and communicatively coupled with the remote control unit, the IRD including an image content search processor to perform operations comprising;

capturing a region of interest (ROI) containing an image in a scene of the displayed content,

analyzing the ROI locally or remotely to search for related content associated with the image, and

receiving the related content.

22. The system of claim 21 wherein the remote control unit allows the viewer to select an area encompassing an object in the scene to correspond to the ROI.

23. The system of claim 21 wherein the image content search processor comprises a local search processor, the local search processor comprising:

a feature extractor to. extract image features of the image in the ROI;

an analysis engine coupled to the feature extractor to generate image features representing a query; and

a search database coupled to the analysis engine to match the image features to locate the related content.

24. The system of claim 23 wherein the analysis engine uses ancillary information associated with the scene to search the related content, the ancillary information including at least one of metadata, Web link, scene timing information, caption, electronic program guide (EPG) data, content provider information, and audio segment.

25. The system of claim 21 wherein the image content search processor comprises a remote search processor, the remote search processor comprising:

a query packager to package the ROI and ancillary information if available for transmission to a server via a network.

26. The system of claim 25 wherein the ancillary information includes at least one of metadata, caption, Web link, scene timing information, electronic program guide (EPG) data, content provider information, and audio segment.

27. The system of claim 21 wherein the related content is displayed on one of the display device and a wireless device having a display.

28. The system of claim 27 wherein the related content is displayed on the display device using one of a picture-in-picture (PiP) mode, a graphic overlay, and a transparency display, a screen below or to side of the main display.

29. The system of claim 21 wherein the image content search processor disables the displayed content if the related content corresponds to an inhibitory feature.

30. The system of claim 21 wherein the image content search processor performs an optical character recognition (OCR) analysis if the image has textual information used to help identifying the image.

31. A system comprising:

an integrated receiver and decoder (IRD) that provides a source of a displayed content;

a remote control unit to allow a viewer to interact with the broadcast content; and

a display device coupled to the IRD to display the displayed content, the display device including an image content search processor to perform operations comprising;

capturing a region of interest (ROI) containing an image in a scene of the displayed content,

analyzing the ROI locally or remotely to search for related content associated with the image, and

receiving the related content.

32. The system of claim 31 wherein the IRD is built into a television set.