Method and apparatus for conducting a search based on available data modes

• US 9,245,051 B2
• Filed: 09/20/2011
• Issued: 01/26/2016
• Est. Priority Date: 09/20/2011
• Status: Active Grant

• Current Assignee: WSOU Investments, LLC (WSOU Holdings, LLC)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Licata, Aaron, Bamidele, Adetokunbo, Fulks, Mark Travis
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US13/237,175
• Publication Number:
  US 20130073583A1
• Time in Patent Office: 1,589 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/9032 Query formulation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06V 20/635 Overlay text, e.g. embedded captions in a TV program
• CPC Class Codes: G06V 30/19167 Active pattern learning
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An approach is provided for conducting a search based on an extraction of a search term from available sensor data. The approach involves determining sensor data associated with at least one device, the sensor data determined from among a plurality of available data modes. The approach also involved processing and/or facilitating a processing of the sensor data to cause, at least in part, an extraction of one or more search terms for at least one query. The approach further involves determining one or more results of the at least one query based, at least in part, on context information associated with the at least one device, user profile information associated with the at least one device, or a combination thereof.

Claims

1. A method comprising facilitating a processing of and/or processing (1) data and/or (2) information and/or (3) at least one signal, the (1) data and/or (2) information and/or (3) at least one signal by a processor based, at least in part, on the following:

a creation of one or more bounding boxes around one or more estimated text portions of sensor data associated with at least one device;

an optical character recognition of the one or more estimated text portions within each of the one or more bounding boxes to determine recognized characters;

a merger of some of the one or more bounding boxes into at least one merged box along a bounding box line of some of the estimated text portions based, at least in part, on the recognized characters within the each bounding box;

an extraction, from some of the recognized characters that are within the remaining one or more bounding boxes that are not merged, of one or more search terms for at least one query based, at least in part, on the recognized characters within the at least one merged box; and

a search based, at least in part, on the at least one query and the one or more search terms to generate one or more results of the at least one query,

wherein the one or more bounding boxes are assigned an identification number and ranked in order from a top of an image to a bottom of the image sensed by way of an optical sensor.

2. The method of claim 1, wherein the sensor data is an audio input, and the audio input is converted to textual information.

3. The method of claim 1, wherein the (1) data and/or (2) information and/or (3) at least one signal are further based, at least in part, on the following:

at least one determination of a processing engine of one or more available processing engines for processing the sensor data based, at least in part, on a confidence value that the sensor data is of at least one data mode type that corresponds to a respective processing engine of the one or more available processing engines.

4. The method of claim 3, wherein the processing engines concurrently process one or more barcodes, one or more 2D planar objects, one or more 3D objects, or a combination thereof, to provide one or more of the textual strings.

5. The method of claim 3, wherein at least one of the processing engines processes pattern recognition for indexing from one or more photo search systems, to provide one or more of the textual strings.

6. The apparatus of claim 1, wherein the one or more results of the at least one query are a translation of the sensor data based, at least in part, on the user profile information associated with the at least one device.

7. The apparatus of claim 1, wherein the apparatus is further caused to:

determine information comprising one or more preferences and/or behavioral tendencies of the user by way of mining the user profile to cause, at least in part, the one or more results of the at least one query to be modified based on the information.

8. The method of claim 1, wherein the one or more results of the at least one query is further determined based, at least in part, on context information associated with the at least one device, user profile information associated with the at least one device, or a combination thereof.

9. The method of claim 8, wherein the context information associated with the at least one device includes user-to-device interaction data, historical data pertaining to the at least one device, or a combination thereof.

10. The method of claim 8, wherein the one or more results of the at least one query are a translation of the sensor data based, at least in part, on the user profile information associated with the at least one device.

11. The method of claim 8, wherein the (1) data and/or (2) information and/or (3) at least one signal are further based, at least in part, on the following:

information comprising one or more preferences and/or behavioral tendencies of the user by way of mining the user profile to cause, at least in part, the one or more results of the at least one query to be modified based on the information.

12. A method comprising facilitating a processing of and/or processing (1) data and/or (2) information and/or (3) at least one signal, the (1) data and/or (2) information and/or (3) at least one signal by a processor based, at least in part, on the following:

a creation of one or more bounding boxes around one or more estimated text portions of sensor data with at least one device;

a merger of some of the one or more bounding boxes into at least one merged box along a bounding box array, prior to optical character recognition;

an optical character recognition of the some estimated text portions within the at least one merged box and of the remaining one or more estimated text portions within the remaining one or more bounding boxes that are not merged;

an extraction of one or more search terms for at least one query based, at least in part, on optical character recognition results associated with the at least one merged box;

an extraction of one or more additional search terms for the at least one query based, at least in part, on optical character recognition results associated with the remaining one or more bounding boxes that are not merged; and

a search based, at least in part, on the at least one query and the one or more additional search terms to generate one or more results of the at least one query.

13. The method of claim 12, wherein the optical sensor is associated with an augmented reality display.

14. The method of claim 13, wherein the sensor data is detected and processed in real time with respect to a representation shown by the augmented reality display.

15. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs,

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,

cause, at least in part, a creation of one or more bounding boxes around one or more estimated text portions of sensor data associated with at least one device;

cause, at least in part, an optical character recognition of the one or more estimated text portions within each of the one or more bounding boxes to determine recognized characters;

cause, at least in part, a merger of some of the one or more bounding boxes into at least one merged box along a bounding box line of some of the estimated text portions based, at least in part, on the recognized characters within the each bounding box;

cause, at least in part, an extraction, from some of the recognized characters that are within the remaining one or more bounding boxes that are not merged, of one or more search terms for at least one query based, at least in part, on the recognized characters within the at least one merged box; and

a search based, at least in part, on the at least one query and the one or more search terms to generate one or more results of the at least one query,

wherein the one or more bounding boxes are assigned an identification number and ranked in order from a top of an image to a bottom of the image sensed by way of an optical sensor.

16. The apparatus of claim 15, wherein the optical sensor is associated with an augmented reality display.

17. The apparatus of claim 16, wherein the sensor data is detected and processed in real time with respect to a representation shown by the augmented reality display.

18. The apparatus of claim 15, wherein the apparatus is further caused to:

determine a processing engine of one or more available processing engines for processing the sensor data based, at least in part, on a confidence value that the sensor data is of at least one data mode type that corresponds to a respective processing engine of the one or more available processing engines.

19. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs,

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,

cause, at least in part, a creation of one or more bounding boxes around one or more estimated text portions of sensor data with at least one device;

cause, at least in part, a merger of some of the one or more bounding boxes into at least one merged box along a bounding box array, prior to optical character recognition;

cause, at least in part, an optical character recognition of the some estimated text portions within the at least one merged box and of the remaining one or more estimated text portions within the remaining one or more bounding boxes that are not merged;

cause, at least in part, an extraction of one or more search terms for at least one query based, at least in part, on optical character recognition results associated with the at least one merged box;

cause, at least in part, an extraction of one or more additional search terms for the at least one query based, at least in part, on optical character recognition results associated with the remaining one or more bounding boxes that are not merged; and

cause, at least in part, a search based, at least in part, on the at least one query and the one or more additional search terms to generate one or more results of the at least one query.