SYSTEMS AND METHODS FOR GAZE-BASED MEDIA SELECTION AND EDITING

• US 20150220157A1
• Filed: 12/31/2014
• Published: 08/06/2015
• Est. Priority Date: 12/31/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Eyefluence, Inc. (Alphabet Inc.)
• Inventors: Drake, Eliot Francis, Marggraff, Lewis James
• Granted Patent: US 9,870,060 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

Systems are presented herein, which may be implemented in a wearable device. The system is designed to allow a user to edit media images captured with the wearable device. The system employs eye tracking data to control various editing functions, whether prior to the time of capture, during the time of capture, or after the time of capture. Also presented are methods for determining which sections or regions of media images may be of greater interest to a user or viewer. The method employs eye tracking data to assign saliency to captured media. In both the system and the method, eye tracking data may be combined with data from additional sensors in order to enhance operation.

Claims

1. A system for editing media images, comprising:

a wearable device;

a scene camera mounted on the device such that the scene camera captures media images of a user'"'"'s surroundings;

an eye tracking subsystem that projects a reference frame onto the eye and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of at least one eye of a user; and

one or more processors communicating with the scene camera and eye tracking subsystem for tagging media images captured by the scene camera based at least in part on the eye tracking data.

2. The system of claim 1, wherein the one or more processors are configured to utilize intentional or subconscious eye movement of the user identified in the eye tracking data to tag one or more media images from the scene camera.

3. The system of claim 2, wherein the one or more processors are configured for associating a tag with media images from the scene camera to define at least one of a starting point and an ending point of a video clip within the media images.

4. The system of claim 2, wherein the one or more processors are configured for determining a saliency value of media images from the scene camera based at least in part on eye movement of the user identified in the eye tracking images.

5. The system of claim 1, further comprising one or more sensors on the wearable device, the one or more processors communicating with the one or more sensors for tagging media images captured by the scene camera based at least in part on sensor signals from the one or more sensors.

6. The system of claim 5, wherein the one or more sensors comprise at least one of an inertial sensor, an image sensor, an audio sensor, a location sensor, an orientation sensor, a light sensor, a biometric sensor, a gesture sensor, an EEG sensor, wherein such sensors may be coupled locally or remotely.

7. The system of claim 5, wherein the one or more processors are configured to inform tagging or saliency of media images captured by the scene camera based at least in part on both the sensor signals and the eye tracking data.

8. The system of claim 5, wherein the one or more processors are configured to synchronize data from the one or more sensors with one or both of the media images and the eye tracking data for correlating the data and the images.

9. The system of claim 1, further comprising a display mounted on the wearable device such that the display is visible to the user.

10. The system of claim 9, wherein the one or more processors are configured for tagging media images captured by the scene camera based at least in part on the eye tracking data without activating the display.

11. The system of claim 9, further comprising one or more regions or indicators on or adjacent to the display, the one or more processors configured to analyze the eye tracking data to identify when the user gazes towards the one or more regions or indicators to associate a tag with the media images.

12. The system of claim 1, wherein the one or more processors are configured to select sections of the media images that are relevant to the user and to edit the sections of the media images by at least one of trimming, cropping, rotating, panning, zooming, and arranging the sections in a desired order.

13. The system of claim 1, wherein the one or more processors are coupled to the eye tracking subsystem to adjust the operating frequency of the eye tracking subsystem to correspond to a plurality of operating modes.

14. The system of claim 1, wherein the system is designed to accept one or both of input from a user or processed data to tag or determine saliency of a media image.

15-18. -18. (canceled)

19. A system to quantitatively assess comparative saliency in video images as determined by proximally-located wearable devices with the purpose of recording relevant events from different viewpoints, comprising:

a plurality of wearable devices configured to be worn by individual users, each wearable device including a scene camera mounted thereon such that the scene camera captures media images of the individual user'"'"'s surroundings, one or more sensors, a communication interface, and an eye tracking subsystem that projects a reference frame onto the user'"'"'s eye and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of the user'"'"'s eye; and

a server for communicating with the wearable devices via each wearable device'"'"'s communication interface.

20. The system of claim 19, further comprising one or more processors on each wearable device for monitoring the one or more sensors to identify a threshold event, whereupon the one or more processors communicate via the communication interface with other wearable devices to at least one of:

confirm that the threshold event has occurred, and

share media images related to the threshold event.

21. The system of claim 19, further comprising one or more processors on each wearable device for monitoring the one or more sensors to identify a threshold event, whereupon the one or more processors communicate via the communication interface with the server to at least one of:

a) confirm that the threshold event has occurred, and b) share media images related to the threshold event.

22. The system of claim 19, further comprising one or more processors on each wearable device for monitoring the one or more sensors to identify a threshold event, whereupon the one or more processors communicate via the communication interface with other wearable devices and with the server to at least one of:

confirm that the threshold event has occurred, and

share media images related to the threshold event.

23. The system of claim 19, wherein each eye tracking subsystem captures eye tracking data of the user'"'"'s eye;

each wearable device comprising one or more processors communicating with the scene camera and eye tracking subsystem for communicating with the communication interface with other wearable devices and with the server to incorporate eye tracking data into the saliency assessment and image selection.

24. A method for selecting or editing media images from a wearable device worn by a user, comprising:

capturing media images, using a scene camera on the wearable device, of the user'"'"'s surroundings;

capturing eye tracking data, using an eye tracking subsystem an eye tracking subsystem that projects a reference frame onto the eye and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of at least one eye of a user; and

at least one of selecting and editing the media images based at least in part on actions of the at least one eye events identified from the eye tracking data.

24-39. -39. (canceled)