Systems and methods for gaze-based media selection and editing

• US 9,870,060 B2
• Filed: 12/31/2014
• Issued: 01/16/2018
• Est. Priority Date: 12/31/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Marggraff, Lewis James, Drake, Eliot Francis
• Primary Examiner(s): Osorio, Ricardo L
• Application Number:
  US14/588,371
• Publication Number:
  US 20150220157A1
• Time in Patent Office: 1,112 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

Systems are presented herein, which may be implemented in a wearable device. The system is designed to allow a user to edit media images captured with the wearable device. The system employs eye tracking data to control various editing functions, whether prior to the time of capture, during the time of capture, or after the time of capture. Also presented are methods for determining which sections or regions of media images may be of greater interest to a user or viewer. The method employs eye tracking data to assign saliency to captured media. In both the system and the method, eye tracking data may be combined with data from additional sensors in order to enhance operation.

Claims

1. A system for editing media images, the system comprising:

a wearable device;

a scene camera mounted on the device such that the scene camera captures media images of surroundings of a user;

an eye tracking subsystem that projects a reference frame onto at least one eye of a user and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of the at least one eye of the user;

one or more processors communicating with the scene camera and eye tracking subsystem for tagging media images captured by the scene camera based at least in part on the eye tracking data; and

memory for storing the tagged media images.

2. The system of claim 1, wherein the one or more processors are configured to utilize intentional or subconscious eye movement of the user identified in the eye tracking data to tag one or more media images from the scene camera.

3. The system of claim 2, wherein the one or more processors are configured for associating a tag with media images from the scene camera to define at least one of a starting point and an ending point of a video clip within the media images.

4. The system of claim 2, wherein the one or more processors are configured for determining a saliency value of media images from the scene camera based at least in part on eye movement of the user identified in the eye tracking images.

5. The system of claim 1, further comprising one or more sensors on the wearable device, the one or more processors communicating with the one or more sensors for tagging media images captured by the scene camera based at least in part on sensor signals from the one or more sensors.

6. The system of claim 5, wherein the one or more sensors comprise at least one of an inertial sensor, an image sensor, an audio sensor, a location sensor, an orientation sensor, a light sensor, a biometric sensor, a gesture sensor, or an EEG sensor, wherein the one or more sensors may be coupled locally or remotely.

7. The system of claim 5, wherein the one or more processors are configured to inform tagging or saliency of media images captured by the scene camera based at least in part on both the sensor signals and the eye tracking data.

8. The system of claim 5, wherein the one or more processors are configured to synchronize data from the one or more sensors with one or both of the media images and the eye tracking data for correlating the data and the media images.

9. The system of claim 1, further comprising a display mounted on the wearable device such that the display is visible to the user.

10. The system of claim 9, wherein the one or more processors are configured for tagging media images captured by the scene camera based at least in part on the eye tracking data without activating the display.

11. The system of claim 9, further comprising one or more regions or indicators on or adjacent to the display, the one or more processors configured to analyze the eye tracking data to identify when the user gazes towards the one or more regions or indicators to associate a tag with the media images.

12. The system of claim 1, wherein the one or more processors are configured to select sections of the media images that are relevant to the user and to edit the sections of the media images by at least one of the group consisting of trimming, cropping, rotating, panning, zooming, and arranging the sections in a desired order.

13. The system of claim 1, wherein the one or more processors are coupled to the eye tracking subsystem to adjust an operating frequency of the eye tracking subsystem to correspond to a plurality of operating modes.

14. The system of claim 1, wherein the system is designed to accept one or both of input from a user or processed data to tag or determine a saliency value of a media image.

15. A system to quantitatively assess comparative saliency in video images as determined by proximally-located wearable devices with the purpose of recording relevant events from different viewpoints, the system comprising:

a plurality of wearable devices configured to be worn by individual users, each wearable device including a scene camera mounted thereon such that the scene camera captures media images of the individual user'"'"'s surroundings, one or more sensors, a communication interface, an eye tracking subsystem that projects a reference frame onto the individual user'"'"'s eye and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of the individual user'"'"'s eye, and one or more processors communicating with the scene camera, the one or more sensors, the communication interface and the eye tracking subsystem, the one or more processors for tagging media images captured by the scene camera based at least in part on the eye tracking data; and

a server for communicating with the wearable devices via each wearable device'"'"'s communication interface.

16. The system of claim 15, wherein the one or more processors on each wearable device are for monitoring the one or more sensors to identify a threshold event, whereupon the one or more processors communicate via the communication interface with other wearable devices to at least one of the group consisting of:

confirm that the threshold event has occurred, and

share media images related to the threshold event.

17. The system of claim 15, wherein the one or more processors on each wearable device are for monitoring the one or more sensors to identify a threshold event, whereupon the one or more processors communicate via the communication interface with the server to at least one of the group consisting of:

a) confirm that the threshold event has occurred, and b) share media images related to the threshold event.

18. The system of claim 15, wherein the one or more processors on each wearable device are for monitoring the one or more sensors to identify a threshold event, whereupon the one or more processors communicate via the communication interface with at least one of other wearable devices or the server to at least one of the group consisting of:

confirm that the threshold event has occurred, and

share media images related to the threshold event.

19. The system of claim 15, wherein each eye tracking subsystem captures eye tracking data of the user'"'"'s eye;

each wearable device comprising one or more processors communicating with the scene camera and eye tracking subsystem for communicating with the communication interface with other wearable devices and with the server to incorporate eye tracking data into a saliency assessment and image selection.

20. A method for selecting or editing media images from a wearable device worn by a user, comprising:

capturing media images, using a scene camera on the wearable device, of surroundings of the user;

capturing eye tracking data, using an eye tracking subsystem that projects a reference frame onto at least one eye of the user and associates the projected reference frame with a second reference frame of a display for capturing eye tracking data of the at least one eye of the user;

at least one of the group consisting of selecting and editing the media images based at least in part on one or more eye events identified from the eye tracking data; and

storing the at least one of selected and edited images in memory of the wearable device.

21. The method of claim 20, further comprising:

detecting a threshold event based at least in part on the one or more eye events; and

wherein at least one of selecting or editing the media images comprises automatically tagging the media images for a predetermined time immediately before the detected threshold event.