SYSTEMS AND METHODS FOR GAZE-BASED MEDIA SELECTION AND EDITING

• US 20180173319A1
• Filed: 12/08/2017
• Published: 06/21/2018
• Est. Priority Date: 12/31/2013
• Status: Active Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: MARGGRAFF, Lewis James, DRAKE, Eliot Francis
• Application Number:
  US15/835,891
• Publication Number:
  US 20180173319A1
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

Systems are presented herein, which may be implemented in a wearable device. The system is designed to allow a user to edit media images captured with the wearable device. The system employs eye tracking data to control various editing functions, whether prior to the time of capture, during the time of capture, or after the time of capture. Also presented are methods for determining which sections or regions of media images may be of greater interest to a user or viewer. The method employs eye tracking data to assign saliency to captured media. In both the system and the method, eye tracking data may be combined with data from additional sensors in order to enhance operation.

Claims

1-39. -39. (canceled)

40. A system for editing media images, the system comprising:

a wearable device;

a scene camera mounted on the device such that the scene camera captures media images of surroundings of a user;

a memory for storing the media images;

an eye tracking subsystem that projects a reference frame onto at least one eye of a user and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of the at least one eye of the user; and

one or more processors communicating with the scene camera and eye tracking subsystem for tagging media images captured by the scene camera based at least in part on the eye tracking data and for non-destructively editing the media images such that the media images are recoverable at a later time from the memory by the one or more processors.

41. The system of claim 40, wherein the one or more processors are configured to assign a validity weight to the eye tracking data based on a graceful degradation in eye tracking precision.

42. The system of claim 40, further comprising a transceiver or other wireless communication interface communicating with the one or more processors to transmit the media images to a remote location.

43. The system of claim 40, further comprising a transceiver or other wireless communication interface communicating with the one or more processors to receive data from one or more sensors remote from the wearable device.

44. A method for selecting or editing media images from a wearable device worn by a user, comprising:

capturing media images of the user'"'"'s surroundings at a scene camera of the wearable device;

capturing eye tracking data of at least one eye of the user at a user-facing camera of the wearable device;

projecting a reference frame onto the at least one eye of the user and associating the projected reference frame with a second reference frame of a display at an eye tracking subsystem of the wearable device;

at least one of selecting and editing the media images based at least in part on actions of the at least one eye identified from the eye tracking data;

detecting a threshold event based on an accelerometer of the wearable device; and

wherein at least one of selecting and editing the media images comprises automatically tagging the media images for a predetermined time immediately before the threshold event.

45. The method of claim 44, further comprising determining a saliency value of the media images from the scene camera based at least in part on eye movement of the user identified in the eye tracking data.

46. The method of claim 45, wherein detecting the threshold event selectively overrides other saliency determinations.

47. The method of claim 44, further comprising recognizing predetermined faces within the media images.

48. The method of claim 44, further comprising:

receiving, at a communication interface of the wearable device, metrics or heuristics from a remote location;

wherein at least one of selecting and editing the media images comprises assigning saliency to one or more segments of the media images based upon the metrics or heuristics.

49. The method of claim 45, further comprising using data from social media applications as input for partial determination of saliency in the media images.

50. The method of claim 44, further comprising cropping media images to boundaries suggested by a user'"'"'s hand motions to create a picture frame.

51. The method of claim 44, further comprising assigning saliency to media images based at least in part on input from multiple sensors, such as squinting, smiling or laughter.

52. The method of claim 44, further comprising automatically cropping media images to a region in which the at least one eye of the user is fixating.

53. The method of claim 44, further comprising automatically cropping media images to a region bounded by saccadic movements of the at least one eye of the user.

54. A method, comprising:

projecting a point of light onto a user'"'"'s eye;

capturing eye tracking data comprising a glint from a reflection of the point of light from the user'"'"'s eye at an eye tracking camera of a wearable device;

determining movement of the user'"'"'s eye relative to the glint;

capturing media images of the user'"'"'s surroundings at a scene camera of the wearable device; and

editing the media images based on movement of the user'"'"'s eye relative to the glint.

55. The method of claim 54, further comprising detecting a threshold event based on an accelerometer of the wearable device.

56. The method of claim 55, further comprising determining a saliency value of the media images from the scene camera based at least in part on eye movement of the user identified in the eye tracking data.

57. The method of claim 56, wherein detecting the threshold event selectively overrides other saliency determinations.

58. The method of claim 54, further comprising recognizing predetermined faces within the media images.

59. The method of claim 54, further comprising:

receiving, at a communication interface of the wearable device, metrics or heuristics from a remote location;

wherein editing the media images comprises assigning saliency to one or more segments of the media images based upon the metrics or heuristics.