Methods and arrangements employing sensor-equipped smart phones

• US 9,609,117 B2
• Filed: 09/22/2015
• Issued: 03/28/2017
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Davis, Bruce L., Rodriguez, Tony F., Rhoads, Geoffrey B., Conwell, William Y., Stach, John
• Primary Examiner(s): Han, Qi
• Application Number:
  US14/861,758
• Publication Number:
  US 20160028878A1
• Time in Patent Office: 553 Days
• Field of Search: 704/275, 704/270, 704/270.1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 2200/1636 Sensing arrangement for detection of a tap gesture on the housing
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 19/047 Vibrating means for incoming calls
• CPC Class Codes: H04M 2250/02 including a Bluetooth interface
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04M 2250/74 with voice recognition means
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

The present technology concerns improvements to smart phones and related sensor-equipped systems. Some embodiments involve spoken clues, e.g., by which a user can assist a smart phone in identifying what portion of imagery captured by a smart phone camera should be processed, or identifying what type of image processing should be conducted. Some arrangements include the degradation of captured content information in accordance with privacy rules, which may be location-dependent, or based on the unusualness of the captured content, or responsive to later consultation of the stored content information by the user. A great variety of other features and arrangements are also detailed.

Claims

1. A method employing a portable user system having a processor, a display, at least one microphone that captures audio, and at least one image sensor for capturing imagery, the method comprising the acts:

capturing imagery with the image sensor of the portable user system, the captured imagery depicting plural physical subjects within a physical environment of a user, and presenting the captured imagery to the user on the display;

the processor selecting a first of said depicted plural physical subjects as being of likely interest to the user, in accordance with stored rule data, and indicating said processor selection of the first depicted subject by a marking presented on the display;

capturing user speech with the microphone;

sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;

the processor employing information from the recognized user speech data as a verbal clue to help identify a second physical subject depicted within the captured imagery, different than the processor-selected first depicted subject, that is of actual interest to said user;

the processor causing said marking to move from the depiction of the first subject to the depiction of the second subject;

after receiving the recognized user speech data, performing an image processing operation concerning the second depicted subject; and

presenting on said display, as a graphical overlay on the captured imagery, a graphical indicia output, different than said marking, that depends on a result of said image processing operation.

2. The method of claim 1 in which said verbal clue is a color.

3. The method of claim 1 in which said verbal clue is a person'"'"'s name.

4. The method of claim 1 in which said verbal clue is a business'"'"' name.

5. The method of claim 1 in which said verbal clue comprises the word “

left,”

“

right,”

“

up,”

or “

down”

.

6. The method of claim 1 in which said recognized user speech data includes the word “

square”

.

7. A system comprising several elements including a processor, a memory, a camera, a display, and a microphone, at least certain of said elements being included in a face-worn apparatus, wherein the memory contains software instructions causing the system to perform the method of claim 1.

8. A non-transitory computer readable medium containing software instructions operative to cause a user'"'"'s portable computer system, equipped with a processor, a display, at least one microphone, and at least one image sensor, to perform acts including:

receiving camera imagery from the image sensor, depicting plural physical subjects within a physical environment of the user, and presenting the imagery to the user on the display;

selecting a first of said depicted plural physical subjects as being of likely interest to the user, in accordance with stored rule data, and indicating said selection of the first depicted subject by a marking presented on the display;

capturing user speech with the microphone;

sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;

employing information from the recognized user speech data as a verbal clue to help identify a second physical subject depicted within the imagery, different than the first depicted subject, that is of actual interest to said user;

causing said marking to move from the depiction of the first subject to the depiction of the second subject;

after receiving the recognized user speech data, performing an image processing operation concerning the second depicted subject; and

presenting on said display, as a graphical overlay on the imagery, a graphical indicia output, different than said marking, that depends on a result of said image processing operation.

9. A method employing a portable user device having a display, at least one microphone that captures audio, and at least one image sensor for capturing imagery, the method comprising the acts:

(a) capturing imagery with the image sensor, the captured imagery depicting plural physical subjects within an environment of said user, and capturing user speech with the microphone;

(b) sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;

(c) performing a verbally-clued computer-implemented cognition process, said cognition process employing information from the recognized user speech data as a verbal clue to help identify a physical subject within the captured imagery that is of interest to said user;

(d) displaying the captured imagery to the user on said display; and

(e) presenting on said display, as a graphical overlay on the captured imagery, a graphical indicia that is determined based on said verbally-clued cognition process;

wherein the presented graphical indicia varies based on said identified physical subject; and

wherein a set of stored rules establishes a priority order by which the cognition process ranks the plural subjects depicted in the captured imagery as being of probable interest to the user, wherein the recognized user speech data causes the cognition process to progress from one subject in said priority order, to a next subject in said priority order.

10. The method of claim 9 that further includes moving a bounding box from around said first subject, to said next subject, when the cognition process progresses from said first subject to said next subject.

11. The method of claim 9 in which said recognized user speech data includes the word “

not”

.

12. A method employing a portable user device having a display, at least one microphone that captures audio, and at least one image sensor for capturing imagery, the method comprising the acts:

(a) capturing imagery with the image sensor, the captured imagery depicting plural physical subjects within an environment of said user, and capturing user speech with the microphone;

(b) sending, to a speech recognition module, audio data corresponding to the user speech, and receiving recognized user speech data corresponding thereto;

(c) performing a verbally-clued computer-implemented cognition process, said cognition process employing information from the recognized user speech data as a verbal clue to help identify a physical subject within the captured imagery that is of interest to said user;

(d) displaying the captured imagery to the user on said display; and

(e) presenting on said display, as a graphical overlay on the captured imagery, a graphical indicia that is determined based on said verbally-clued cognition process;

wherein the presented graphical indicia varies based on said identified physical subject; and

wherein;

the device indicates a first subject of possible interest to the user, by presenting an indicia on the screen indicating said first subject;

in response to first recognized user speech data, the device indicates a second subject of possible interest to the user, by changing said indicia to indicate said second subject instead of said first subject; and

in response to second recognized user speech data, the device indicates a third subject of possible interest to the user, by changing said indicia to indicate said third subject instead of said second subject;

wherein the method emulates conversation, with the user directing, the device responding, and the user further-directing.