Sensor-based mobile search, related methods and systems

• US 9,234,744 B2
• Filed: 05/27/2014
• Issued: 01/12/2016
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rhoads, Geoffrey B., Rodriguez, Tony F., Shaw, Gilbert B.
• Primary Examiner(s): MIZRAHI, DIANE D
• Application Number:
  US14/287,933
• Publication Number:
  US 20140333794A1
• Time in Patent Office: 595 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06V 10/765 using rules for classification or partitioning the feature space
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1. A visual search method comprising the acts:

receiving image data corresponding to imagery of a scene, captured by a camera;

processing the received image data with at least first, second and third different baseline image processing operations, to thereby produce at least first, second and third different sets of processed data derived from the image data, said processing being performed using a particular apparatus, namely a portable system including said camera, a microphone, and a display;

opportunistically identifying, from a set of plural additional image processing operations, a smaller subset of one or more further image processing operations that should be automatically invoked, based at least in part on said processed data, at least one of the identified further processing operations having a computational cost greater than each of the first, second and third baseline image processing operations;

invoking said identified smaller subset of one or more further image processing operations; and

as a consequence of said further image processing operations, providing information concerning an object depicted in said imagery;

wherein different further image processing operations are identified and invoked, based on results of the baseline image processing operations, thereby providing an experience in which the portable system seems to respond intuitively and adapt image processing based on the scene from which the image data was captured.

2. The method of claim 1 wherein:

one of said baseline image processing operations outputs a frame of data derived from the image data;

another of said baseline image processing operations outputs a list of one-dimensional or two-dimensional regions or structures discerned from the image data; and

still another of said baseline image processing operations outputs histogram data.

3. The method of claim 1 that includes:

identifying and invoking a first further image processing operation when said baseline processing operations yield first results;

identifying and invoking a second further image processing operation when said baseline processing operations yield second results; and

identifying and invoking a third further image processing operation when said baseline processing operations yield third results;

wherein said first, second and third further image processing operations are different.

4. The method of claim 1 that further includes performing said subset of further image processing operations using, at least in part, a processor remote from the portable system, and returning information from said remote processor to the portable system.

5. The method of claim 1 in which at least one of said baseline image processing operations outputs a frame of data derived from the image data.

6. The method of claim 5 in which at least one of said baseline image processing operations remaps pixels in the image data.

7. The method of claim 5 in which at least one of said baseline image processing operations spatially remaps the image data.

8. The method of claim 5 in which at least one of said baseline image processing operations outputs a frame of output data derived from a sequence of frames of imagery.

9. The method of claim 1 in which at least one of said baseline image processing operations outputs a list of one-dimensional or two-dimensional regions or structures discerned from the image data.

10. The method of claim 1 in which at least one of said baseline image processing operations produces histogram output data.

11. The method of claim 1 that includes:

identifying and invoking a first further image processing operation when said baseline processing operations yield first results; and

identifying and invoking a second, different, further image processing operation when said baseline processing operations yield second results.

12. The method of claim 11 in which the first further image processing operation comprises an operation associated with barcode reading.

13. The method of claim 12 in which the second further image processing operation comprises an operation associated with optical character recognition.

14. The method of claim 12 in which the second further image processing operation comprises an operation associated with facial recognition.

15. The method of claim 1 in which said act of providing information includes decoding a payload of a barcode object depicted in said imagery.

16. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on context data.

17. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on geolocation context data.

18. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on geolocation context data indicating that the portable system is located in a grocery store.

19. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on historical context data.

20. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on historical context data concerning said user.

21. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on historical context data concerning one or more people different than said user.

22. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on historical social context data concerning said user.

23. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on historical data about portable system usage.

24. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on audio context data.

25. The method of claim 1 that includes selecting at least one of said baseline image processing operations based, at least in part, on a query previously received by the portable system from a user, the query being unrelated to image data.

26. The method of claim 1 that further includes enlarging said set of different baseline image processing operations, based on a contextual circumstance.

27. The method of claim 1 that further includes enlarging said set of different baseline image processing operations, based on a geolocation context trigger.

28. The method of claim 1 that further includes presenting a visual bauble to a user of said portable system, the bauble indicating a location within the imagery where the system has sensed a potential visual object of interest.

29. The method of claim 1 that further includes presenting a visual bauble to a user of said portable system, the bauble indicating that the system has sensed a potential visual object of interest, and conveying information about a potential type of said visual object.

30. The method of claim 1 that includes invoking one or more additional image processing operations as a consequence of received user input indicating user interest in a result of said one or more of said additional image processing operations, wherein the method includes both bottom-up and top-down processing.

31. The method of claim 1 that includes:

receiving plural images from a sequence of images captured over an interval of time;

processing each of said received images with the at least first, second and third different baseline image processing operations, to thereby produce —

for each of said received images —

at least first, second and third different sets of processed data; and

based on said processed data, identifying one of said received plural images to which said smaller subset of one or more further images processing operations should be applied.

32. The method of claim 1 in which one of said baseline image processing operations takes one or more image frames —

or parts thereof —

as input, and produces an image frame —

or partial frame —

as output.

33. The method of claim 32 in which another of said baseline image processing operations takes one or more input images —

or parts thereof —

as input, and produces a list of one-dimensional or two-dimensional regions or structures as output.

34. The method of claim 33 in which another of said baseline image processing operations produces a histogram as output.

35. The method of claim 32 in which another of said baseline image processing operations produces a histogram as output.

36. The method of claim 32 in which one of said baseline image processing operations takes one or more input images —

or parts thereof —

as input, and produces a list of one-dimensional or two-dimensional regions or structures as output.

37. The method of claim 36 in which another of said baseline image processing operations produces a histogram as output.

38. The method of claim 1 in which said identified smaller subset of one or more further image processing operations produces decoded barcode data.

39. The method of claim 1 in which said identified smaller subset of one or more further image processing operations produces optical character recognition data.

40. The method of claim 1 in which said identified smaller subset of one or more further image processing operations produces facial recognition data.

41. The method of claim 1 in which said baseline image processing operations are performed by a programmable microprocessor in said portable system.

42. The method of claim 1 that further includes:

making an assessment concerning an output of said further image processing operations; and

allocating resources to said further image processing operations in an amount based, at least in part, on said assessment.

43. The method of claim 42 in which making said assessment comprises making a detection state assessment relating to anticipated success of said further image processing operations in achieving a particular goal.

44. The method of claim 43 that includes making two or more of said detection state assessments spaced in time, and changing allocation of resources based on a change in said detection statement assessments.

45. The method of claim 42 that includes assessing a derivative of said detection state, to determine a speed or acceleration of said further image processing operations towards said goal, and allocating resources to said further image processing operations based, at least in part, on said assessed derivative.

46. The method of claim 42 in which making said assessment comprises making an assessment of user interest in the output of said further image processing operations.

47. The method of claim 42 that includes allocating an amount of memory to said further image processing operations based, at least in part, on said assessment.

48. The method of claim 42 that includes allocating a faster connection to said further image processing operations based, at least in part, on said assessment.

49. The method of claim 42 that includes allocating a graphics processing unit resource for use in said further image processing operations based, at least in part, on said assessment.

50. The method of claim 42 that further includes:

cyclically performing said first, second and third baseline image processing operations multiple times, each time performing said initial processing operations on a different frame of captured image data; and

invoking said further image processing operations on a particular frame of captured image data, only when output data from one of said cycles indicates that said particular frame of captured image data merits such further image processing operations.

51. The method of claim 42 in which said first, second and third different baseline image processing operations are performed by a graphics processing unit or a digital signal processor in said portable system, rather than by a general purpose microprocessor.

52. An article of manufacture including a non-transitory computer-readable medium having instructions stored thereon that, responsive to execution by a computing device, cause the computing device to perform operations comprising:

receiving image data corresponding to imagery of a scene;

processing the received image data with at least first, second and third different baseline image processing operations, to thereby produce at least first, second and third different sets of processed data derived from the image data;

opportunistically identifying, from a set of plural additional image processing operations, a smaller subset of one or more further image processing operations that should be automatically invoked, based at least in part on said processed data, at least one of the identified further processing operations having a computational cost greater than each of the first, second and third baseline image processing operations;

invoking said identified smaller subset of one or more further image processing operations; and

as a consequence of said further image processing operations, providing a user with information concerning an object depicted in said imagery;

wherein different further image processing operations are identified and invoked, based on results of the baseline image processing operations, thereby providing a user experience in which the computing device seems to respond intuitively and adapt image processing based on the scene to which the received image data corresponds.

53. A visual method comprising the acts:

receiving a set of image data corresponding to imagery captured by a camera;

processing the received image data with at least first, second and third different initial image processing operations, to thereby produce at least first, second and third different sets of processed data derived from the image data, said processing being performed using a particular apparatus, namely a portable system including said camera, a microphone, and a display;

based at least in part on said processed data, launching a further data processing operation;

making an assessment concerning an output of the further data processing operation; and

allocating resources to said further data processing operation in an amount based, at least in part, on said assessment;

wherein the method further includes;

cyclically performing the first, second and third initial processing operations multiple times, each time performing said initial processing operations on a different frame of captured imagery; and

launching the further data processing operation on a particular frame of captured imagery, only when output data from one of said cycles indicates that said particular frame of captured imagery merits such further data processing operation.

54. An article of manufacture including a non-transitory computer-readable medium having instructions stored thereon that, responsive to execution by a computing device, cause the computing device to perform operations comprising:

receiving image data corresponding to imagery;

processing the received image data with at least first, second and third different initial image processing operations, to thereby produce at least first, second and third different sets of processed data derived from the image data;

based at least in part on said processed data, launching a further data processing operation;

making an assessment concerning an output of the further data processing operation; and

allocating resources to said further data processing operation in an amount based, at least in part, on said assessment;

said operations further including;

cyclically performing the first, second and third initial processing operations multiple times, each time performing said initial processing operations on a different frame of captured imagery; and

pg,115

launching the further data processing operation on a particular frame of captured imagery, only when output data from one of said cycles indicates that said particular frame of captured imagery merits such further data processing operation.