System and method for multimedia capture

• US 9,918,038 B2
• Filed: 02/29/2016
• Issued: 03/13/2018
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Ochiai, Yuichi, Nagata, Katsumi, Sasaki, Akira
• Primary Examiner(s): Shibru, Helen
• Application Number:
  US15/056,613
• Publication Number:
  US 20170251163A1
• Time in Patent Office: 743 Days
• Field of Search: 386201, 386223, 348148-149
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04N 9/8211 the additional signal being a sound signal H04N9/835 takes precedence
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal

Abstract

Methods, systems, and apparatus for multimedia capture. The system includes one or more storage devices connected through a network to a vehicle. The vehicle includes a memory for buffering video. The vehicle includes multiple cameras coupled to one or more video processors. The one or more video processors are configured to capture a first video of an interior view of the vehicle for a period of time, capture a second video of an exterior view of surroundings of the vehicle for the period of time, and synchronize the first video and the second video. The vehicle includes one or more microphones coupled to one or more audio processors for capturing audio and an electronic control unit. The electronic control unit includes a multimedia processor. The electronic control unit is configured to buffer the synchronized video, detect a triggering event, and store the synchronized video.

Claims

1. A multimedia capture system for a vehicle, comprising:

a plurality of cameras including a first camera and a second camera for capturing video;

a memory for buffering the video;

a video processor coupled to the plurality of cameras and the memory, the video processor configured to;

capture, using the first camera, a first video of an interior view of the vehicle for a period of time,

capture, using the second camera, a second video of an exterior view of surroundings of the vehicle for the period of time, and

synchronize the first video and the second video;

a microphone for capturing audio within the vehicle;

an audio processor for processing the captured audio; and

an electronic control unit including a multimedia processor that is configured to;

buffer, in the memory, the synchronized video,

detect a triggering event identified from the captured first video of the interior view of the vehicle or the captured audio within the vehicle, and

store, in a storage device, the synchronized video in response to the triggering event.

2. The multimedia capture system of claim 1,

wherein the captured audio includes one or more audio expressions;

wherein the audio processor is configured to measure a frequency level of the one or more audio expressions.

3. The multimedia capture system of claim 2, wherein to detect the triggering event, the multimedia processor is configured to:

determine that the triggering event occurred if the frequency level of the one or more audio expressions is within a particular range, wherein the particular range is user configurable or pre-determined.

4. The multimedia capture system of claim 3, wherein the multimedia processor is configured to synchronize the audio with the synchronized video.

5. The multimedia capture system of claim 1, wherein the multimedia processor is further configured to store the synchronized video beginning at a start time and ending at a finish time, the start time being at the time of the triggering event and the finish time being a threshold amount of time after the triggering event.

6. The multimedia capture system of claim 5, wherein at least one of the start time, the finish time, or the threshold amount of time after the triggering event is a user configurable setting.

7. The multimedia capture system of claim 1, wherein the triggering event is a selection of a user interface element on a user interface.

8. The multimedia capture system of claim 1, further comprising:

a navigation unit coupled to the electronic control unit for obtaining vehicle information including a current date and time and a current location of the vehicle;

wherein the multimedia processor is further configured to;

obtain, from the navigation unit, the vehicle information;

identify one or more landmarks being captured in the second video based on the vehicle information; and

tag the one or more landmarks in the synchronized video.

9. The multimedia capture system of claim 1, further comprising:

one or more external databases connected to the vehicle through a network, wherein the one or more external databases includes landmark information,

wherein the multimedia processor is further configured to;

obtain, from the one or more external databases, the landmark information;

identify one or more landmarks being captured in the second video based on the landmark information; and

tag the one or more landmarks in the synchronized video.

10. The multimedia capture system of claim 1, wherein to store the synchronized video, the multimedia processor is further configured to:

access one or more storage devices connected to the vehicle through a network; and

transmit the synchronized video to the one or more storage devices connected to the vehicle through the network.

11. A computer-implemented method for a road trip album for a vehicle, the method comprising:

capturing, in one or more video processors and using a first camera, a first video of an interior view of the vehicle for a period of time;

capturing, in the one or more video processors and using a second camera, a second video of an exterior view of a surrounding of the vehicle for the period of time;

synchronizing, using the one or more video processors, the first video and the second video;

buffering, in a memory, the synchronized video;

obtaining, using an audio processor and a microphone, audio within the vehicle;

detecting a triggering event identified from the captured first video of the interior view of the vehicle or the captured audio within the vehicle; and

storing the synchronized video in response to the triggering event.

12. The computer-implemented method of claim 11, further comprising:

obtaining vehicle information including a current date, location, and elevation of the vehicle;

identifying one or more landmarks being captured in the second video; and

tagging the one or more landmarks in the synchronized video.

13. The computer-implemented method of claim 12, wherein storing the synchronized video includes the tagged one or more landmarks, the vehicle information and one or more identities of occupants of the vehicle.

14. The computer-implemented method of claim 11, wherein the triggering event is user input that activates storing of the synchronized video.

15. The computer-implemented method of claim 11, wherein detecting the triggering event comprises:

determining, from the captured audio, an audio expression for an occupant of the vehicle; and

determining that the audio expression has a frequency that is within a particular range.

16. The computer-implemented method of claim 11, further comprising:

synchronizing, in a multimedia processor, the captured audio and the synchronized video; and

storing the captured audio with the synchronized video.

17. The computer-implemented method of claim 11, further comprising:

obtaining, from a user interface, user input, wherein the user input is configuration settings for the triggering event.

18. The computer-implemented method of claim 17, wherein detecting the triggering event comprises:

capturing, using the first camera, one or more facial features of an occupant;

determining one or more facial expressions of the occupant of the vehicle based on the one or more facial features; and

determining that the triggering event occurred based on the one or more determined facial expressions and the configuration settings for the triggering event.

19. A multimedia capture system for an autonomous vehicle, comprising:

a plurality of cameras including a first camera and a second camera for capturing video;

a memory for buffering the captured video;

a video processor coupled to the memory and the plurality of cameras, the video processor configured to;

capture a first video of an interior view of the autonomous vehicle for a period of time,

capture a second video of an exterior view surroundings of the vehicle for the period of time, and

synchronize the first video and the second video;

a microphones for capturing audio within the autonomous vehicle;

an audio processor for processing the captured audio; and

an electronic control unit including a multimedia processor that is configured to;

buffer, in the memory, the synchronized video;

detect a triggering event identified from the captured first video of the interior view of the vehicle or the captured audio within the vehicle; and

store the synchronized video in response to the triggering event.

20. The multimedia capture system of claim 19, wherein the autonomous vehicle operates using one or more sensors or a navigation unit and without human input.