US9594725B1 - Safety score using video data but without video - Google Patents

Safety score using video data but without video Download PDF

Info

Publication number
US9594725B1
US9594725B1 US14/012,136 US201314012136A US9594725B1 US 9594725 B1 US9594725 B1 US 9594725B1 US 201314012136 A US201314012136 A US 201314012136A US 9594725 B1 US9594725 B1 US 9594725B1
Authority
US
United States
Prior art keywords
safety score
data
maneuver
safety
determiner
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US14/012,136
Inventor
Bryon Cook
Niki Kypri
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lytx Inc
Original Assignee
Lytx Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US14/012,136 priority Critical patent/US9594725B1/en
Application filed by Lytx Inc filed Critical Lytx Inc
Assigned to DRIVECAM, INC. reassignment DRIVECAM, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: COOK, BRYON, KYPRI, NIKI
Assigned to LYTX, INC. reassignment LYTX, INC. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: DRIVECAM, INC.
Assigned to WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT reassignment WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT SECURITY AGREEMENT Assignors: LYTX, INC., MOBIUS ACQUISITION HOLDINGS, LLC
Assigned to LYTX, INC. reassignment LYTX, INC. RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME 032134/0756 Assignors: WELLS FARGO BANK, NATIONAL ASSOCIATION
Assigned to U.S. BANK NATIONAL ASSOCIATION, AS ADMINISTRATIVE AGENT reassignment U.S. BANK NATIONAL ASSOCIATION, AS ADMINISTRATIVE AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LYTX, INC.
Priority to US15/420,470 priority patent/US10068392B2/en
Publication of US9594725B1 publication Critical patent/US9594725B1/en
Application granted granted Critical
Assigned to HPS INVESTMENT PARTNERS, LLC, AS COLLATERAL AGENT reassignment HPS INVESTMENT PARTNERS, LLC, AS COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LYTX, INC.
Assigned to LYTX, INC. reassignment LYTX, INC. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: U.S. BANK, NATIONAL ASSOCIATION
Assigned to GUGGENHEIM CREDIT SERVICES, LLC reassignment GUGGENHEIM CREDIT SERVICES, LLC NOTICE OF SUCCESSOR AGENT AND ASSIGNMENT OF SECURITY INTEREST (PATENTS) REEL/FRAME 043745/0567 Assignors: HPS INVESTMENT PARTNERS, LLC
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G07CHECKING-DEVICES
    • G07CTIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C5/00Registering or indicating the working of vehicles
    • G07C5/02Registering or indicating driving, working, idle, or waiting time only
    • G07C5/06Registering or indicating driving, working, idle, or waiting time only in graphical form
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10Complex mathematical operations
    • GPHYSICS
    • G07CHECKING-DEVICES
    • G07CTIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C5/00Registering or indicating the working of vehicles
    • G07C5/008Registering or indicating the working of vehicles communicating information to a remotely located station
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/7867Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482Interaction with lists of selectable items, e.g. menus
    • GPHYSICS
    • G07CHECKING-DEVICES
    • G07CTIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C5/00Registering or indicating the working of vehicles
    • G07C5/08Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
    • G07C5/0816Indicating performance data, e.g. occurrence of a malfunction
    • GPHYSICS
    • G07CHECKING-DEVICES
    • G07CTIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C5/00Registering or indicating the working of vehicles
    • G07C5/08Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
    • G07C5/0841Registering performance data
    • G07C5/085Registering performance data using electronic data carriers
    • G07C5/0866Registering performance data using electronic data carriers the electronic data carrier being a digital video recorder in combination with video camera
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources

Definitions

  • Modern vehicles can include a vehicle event recorder in order to better understand the timeline of an anomalous event (e.g., an accident).
  • a vehicle event recorder typically includes a set of sensors, e.g., video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS (global positioning system), etc., that report data, which is used to determine the occurrence of an anomalous event. If an anomalous event is detected, then sensor data related to the event is stored for later review. Data from an inward facing video camera (e.g., containing an image of the driver) can be very useful for determining anomalous events. However, some vehicle event recorder users (e.g., drivers, fleet managers, etc.) do not allow the inward facing video camera data to be recorded or used for event detection.
  • FIG. 1 is a block diagram illustrating an embodiment of a system including a vehicle event recorder.
  • FIG. 2 is a block diagram illustrating an embodiment of a vehicle event recorder.
  • FIG. 3 is a block diagram illustrating an embodiment of a vehicle event recorder.
  • FIG. 4 is a block diagram illustrating an embodiment of a vehicle data server.
  • FIG. 5 is a diagram illustrating an embodiment of a user interface for a manual safety score determiner.
  • FIG. 6 is a flow diagram illustrating an embodiment of a process for building a maneuver vector.
  • FIG. 7 is a flow diagram illustrating an embodiment of a process for manually determining a safety score.
  • FIG. 8 is a flow diagram illustrating an embodiment of a process for automatically determining a safety score.
  • FIG. 9 is a flow diagram illustrating a process for determining overall safety scores.
  • FIG. 10 is a flow diagram illustrating a process for determining a safety score.
  • the invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.
  • these implementations, or any other form that the invention may take, may be referred to as techniques.
  • the order of the steps of disclosed processes may be altered within the scope of the invention.
  • a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task.
  • the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
  • a safety score using video data but without video is disclosed.
  • a system for determining a safety score comprises an input interface configured to receive sensor data.
  • the system for determining a safety score additionally comprises a processor configured to provide the sensor data to a safety score determiner, and determine a safety score using the safety score determiner without using inward facing video data, wherein the safety score determiner has been trained using a data set that includes sensor data with inward facing video data.
  • the system for determining a safety score additionally comprises a memory coupled to the processor and configured to provide the processor with instructions.
  • a vehicle event recorder mounted on a vehicle records vehicle data and anomalous vehicle events.
  • Anomalous vehicle event types include accidents, speed limit violations, rough road events, hard maneuvering events (e.g., hard cornering, hard braking), dangerous driving events (e.g., cell phone usage, eating while driving, working too long of a shift, sleepy driving, etc.), and any other appropriate kind of anomalous vehicle events.
  • the vehicle event recorder analyzes data from sensors (e.g., video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS, etc.) to determine when an anomalous event has occurred.
  • the vehicle event recorder applies one or more data processing techniques (e.g., filters, Fourier transform processing, neural networks, etc.) to sensor data to determine the probability of each of a set of hazardous events (e.g., hard maneuvering, collision, etc.).
  • a set of hazardous events e.g., hard maneuvering, collision, etc.
  • the set of probabilities is saved as a maneuver point. Multiple maneuver points occurring within a time threshold of one another are grouped together to form a maneuver vector.
  • the sensors associated with the vehicle event recorder include video recorders, any available video data from the video recorders corresponding to the time of the maneuver is associated with the maneuver vector.
  • the maneuver vector including video data, if available, is transmitted to a vehicle data server and analyzed.
  • the vehicle event recorder collects maneuver vectors over the course of a day and transmits them to the vehicle data server at the end of the day.
  • the vehicle event recorder analyzes the maneuver vector and any available video data to determine a safety score.
  • the safety score is determined by a manual safety score determiner comprising a human video reviewing station. A person views the available video data and indicates which of a set of dangerous behaviors are determined to be present. The safety score is computed from the behavior or behaviors indicated.
  • the safety score is provided to a safety score database, where it is associated with the driver, and used in later calculations of the driver's overall safety score.
  • the safety score is associated with the maneuver vector and provided to an automatic safety score determiner, which stores the maneuver vector and associated safety score.
  • an average safety score is computed, comprising the average of all safety scores received associated with the received maneuver vector.
  • the automatic safety score determiner builds a library of maneuver vectors and typical associated safety score values.
  • the automatic safety score determiner looks up the maneuver vector in the library of maneuver vectors and returns the associated safety score value.
  • the automatic safety score determiner determines the maneuver vector in the library that is most similar to the received maneuver vector (e.g., a library with source data from the same driver, from other drivers, etc.) and returns the safety score value associated with that maneuver vector.
  • FIG. 1 is a block diagram illustrating an embodiment of a system including a vehicle event recorder.
  • Vehicle event recorder 102 comprises a vehicle event recorder mounted in a vehicle (e.g., a car or truck).
  • vehicle event recorder 102 includes or is in communication with a set of sensors—for example, video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS, outdoor temperature sensors, moisture sensors, laser line tracker sensors, or any other appropriate sensors.
  • sensors for example, video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS, outdoor temperature sensors, moisture sensors, laser line tracker sensors, or any other appropriate sensors.
  • vehicle state sensors comprise a speedometer, an accelerator pedal sensor, a brake pedal sensor, an engine revolutions per minute (e.g., RPM) sensor, an engine temperature sensor, a headlight sensor, an airbag deployment sensor, driver and passenger seat weight sensors, an anti-locking brake sensor, an engine exhaust sensor, a gear position sensor, a cabin equipment operation sensor, or any other appropriate vehicle state sensors.
  • vehicle event recorder 102 comprises a system for processing sensor data and detecting events.
  • vehicle event recorder 102 comprises a system for detecting risky behavior.
  • vehicle event recorder 102 is mounted to vehicle 106 in one of the following locations: the chassis, the front grill, the dashboard, the rear-view mirror, or any other appropriate location.
  • vehicle event recorder 102 comprises multiple units mounted in different locations in vehicle 106 .
  • vehicle event recorder 102 comprises a communications system for communicating with network 100 .
  • network 100 comprises a wireless network, a wired network, a cellular network, a code division multiple access (CDMA) network, a global system for mobile communication (GSM) network, a local area network, a wide area network, the Internet, or any other appropriate network.
  • network 100 comprises multiple networks, changing over time and location.
  • Vehicle event recorder 102 communicates with vehicle data server 104 via network 100 .
  • Vehicle event recorder 102 is mounted on vehicle 106 .
  • vehicle 106 comprises a car, a truck, a commercial vehicle, or any other appropriate vehicle.
  • Vehicle data server 104 comprises a vehicle data server for collecting events and risky behavior detected by vehicle event recorder 102 .
  • vehicle data server 104 comprises a system for collecting data from multiple vehicle event recorders.
  • vehicle data server 104 comprises a system for analyzing vehicle event recorder data.
  • vehicle data server 104 comprises a system for displaying vehicle event recorder data.
  • vehicle data server 104 is located at a home station (e.g., a shipping company office, a taxi dispatcher, a truck depot, etc.).
  • events recorded by vehicle event recorder 102 are downloaded to vehicle data server 104 when vehicle 106 arrives at the home station.
  • vehicle data server 104 is located at a remote location.
  • events recorded by vehicle event recorder 102 are downloaded to vehicle data server 104 wirelessly.
  • a subset of events recorded by vehicle event recorder 102 is downloaded to vehicle data server 104 wirelessly.
  • FIG. 2 is a block diagram illustrating an embodiment of a vehicle event recorder.
  • vehicle event recorder 200 of FIG. 2 comprises vehicle event recorder 102 of FIG. 1 .
  • vehicle event recorder 200 comprises processor 202 .
  • Processor 202 comprises a processor for controlling the operations of vehicle event recorder 200 , for reading and writing information on data storage 204 , for communicating via wireless communications interface 206 , for determining a position using global positioning system 208 , and for reading data via sensor interface 210 .
  • Data storage 204 comprises a data storage (e.g., a random access memory (RAM), a read only memory (ROM), a nonvolatile memory, a flash memory, a hard disk, or any other appropriate data storage).
  • RAM random access memory
  • ROM read only memory
  • nonvolatile memory e.g., a flash memory, a hard disk, or any other appropriate data storage.
  • data storage 204 comprises a data storage for storing instructions for processor 202 , vehicle event recorder data, vehicle event data, sensor data, video data, map data, or any other appropriate data.
  • wireless communications interface 206 comprises one or more of a GSM interface, a CDMA interface, a WiFi interface, or any other appropriate interface.
  • Global positioning system 208 comprises a global positioning system (e.g., GPS) for determining a system location.
  • Sensor interface 210 comprises an interface to one or more vehicle event recorder sensors.
  • vehicle event recorder sensors comprise an external video camera, an internal video camera, a microphone, an accelerometer, a gyroscope, an outdoor temperature sensor, a moisture sensor, a laser line tracker sensor, vehicle state sensors, or any other appropriate sensors.
  • vehicle state sensors comprise a speedometer, an accelerator pedal sensor, a brake pedal sensor, an engine RPM sensor, an engine temperature sensor, a headlight sensor, an airbag deployment sensor, driver and passenger seat weight sensors, an anti-locking brake sensor, an engine exhaust sensor, a gear position sensor, a cabin equipment operation sensor, or any other appropriate vehicle state sensors.
  • sensor interface 210 comprises an on-board diagnostics (OBD) bus.
  • OBD on-board diagnostics
  • vehicle event recorder 200 communicates with vehicle state sensors via OBD bus.
  • FIG. 3 is a block diagram illustrating an embodiment of a vehicle event recorder.
  • vehicle event recorder 300 of FIG. 3 comprises vehicle event recorder 200 of FIG. 2 .
  • vehicle event recorder 300 comprises sensors 302 .
  • sensors 302 comprises a sensor interface and vehicle event recorder 300 is in communication with sensors via the sensor interface.
  • Sensors 302 deliver sensor data to data processing 304 .
  • sensor data comprises video camera data, microphone data, accelerometer data, gyroscope data, outdoor temperature sensor data, moisture sensor data, laser line tracker sensor data, vehicle state sensor data, or any other appropriate sensor data.
  • sensor data comprises calibrated accelerometer data from one or more accelerometers.
  • Data processing 304 processes sensor data to determine a set of hazard probabilities.
  • Hazard probabilities comprise the probabilities that one or more of a set of hazards are occurring.
  • data processing 304 comprises one or more of the following: digital filtering, digital Fourier transform processing, a neural network, or any other appropriate data processing technique.
  • hazard probabilities comprise a probability of hazardous acceleration, a probability of hazardous braking, a probability of a collision, a probability of hazardous left cornering, a probability of hazardous right cornering, a probability of a hazardous rough road, a probability of another hazardous condition, or any other appropriate hazard probability.
  • Data processing 304 provides hazard probabilities to hazard probability thresholds 306 .
  • Hazard probability thresholds 306 determines if any of the hazard probabilities received from data processing 304 are above their corresponding hazard probability threshold. If none of the hazard probabilities are above the corresponding hazard probability threshold, the set of hazard probabilities is discarded. If any of the hazard probabilities is above the corresponding hazard probability threshold, the set of hazard probabilities is labeled a maneuver point.
  • Hazard probability thresholds 306 provides maneuver points to maneuver vector time threshold 308 . Maneuver vector time threshold 308 determines if a received maneuver point is within a time threshold of a maneuver vector. In some embodiments, if two maneuver points are received within a time threshold of one another, the two maneuver points are determined to be a part of the same maneuver vector.
  • the maneuver points are added to the maneuver vector.
  • the time threshold comprises a time since the first received maneuver point in the maneuver vector. In some embodiments, the time threshold comprises a time since the last received maneuver point in the maneuver vector.
  • the maneuver vector is delivered to a vehicle data server. In some embodiments, the maneuver vector is stored and delivered to the vehicle data server at a later point in time (e.g., when the vehicle checks in at a depot, at the end of the day, at the end of the driver shift, etc.). In some embodiments, if sensors 302 combine video sensors (e.g., an inward-facing video camera, a forward-facing video camera), video data corresponding to the time period of a maneuver vector is delivered to the vehicle data server associated with the maneuver vector.
  • video sensors e.g., an inward-facing video camera, a forward-facing video camera
  • FIG. 4 is a block diagram illustrating an embodiment of a vehicle data server.
  • vehicle data server 400 comprises vehicle data server 104 of FIG. 1 .
  • vehicle data server 400 receives maneuver vectors from one or more vehicle event recorders (e.g., associated with one or more different vehicles). Some maneuver vectors received include associated video data, and some maneuver vectors received have no associated video. In some embodiments, some vehicle event recorders do not include or communicate with video recorders, and do not provide video data associated with maneuver vectors.
  • vehicle data server 400 receives a maneuver vector with associated video, it is provided to manual safety score determiner 402 .
  • Manual safety score determiner 402 manually determines a safety score associated with the maneuver vector with associated video.
  • manual safety score determiner 402 determines a safety score by displaying video to a maneuver reviewer (e.g., a human) and receiving from the maneuver reviewer a set of associated hazardous behaviors.
  • the safety score comprises a sum of safety scores associated with each of the set of hazardous behaviors.
  • Manual safety score determiner 402 provides the maneuver vector and the associated safety score to automatic safety score determiner 404 and driver safety score database 406 .
  • Automatic safety score determiner 404 stores maneuver vectors and associated safety scores received from manual safety score determiner in a library of maneuver vectors with associated safety scores.
  • automatic safety score determiner 404 if automatic safety score determiner 404 receives a maneuver vector for which it has already received one or more safety scores, an average safety score is computed, comprising the average of all safety scores received associated with the received maneuver vector.
  • automatic safety score determiner 404 builds a library of maneuver vectors and typical associated safety score values.
  • building a library of maneuver vectors and typical associated safety score values comprises training automatic safety score determiner 404 .
  • a third party data set of maneuver vectors with associated video data is used to train automatic safety score determiner 404 .
  • a third party data set of maneuver vectors with associated video data comprises a set of maneuver vectors with associated video data purchased from an external source.
  • Automatic safety score determiner 404 When vehicle data server 400 receives a maneuver vector without associated video, the maneuver vector is provided to automatic safety score determiner 404 .
  • Automatic safety score determiner utilizes its library of maneuver vectors and typical associated safety score values to determine a safety score value associated with the received maneuver vector.
  • automatic safety score determiner 404 determines the safety score by using a table, by interpolating between entries in a table, by using a neural network, by using machine learning techniques, by using a support vector machine, by leveraging ensemble modeling, or by using any other appropriate technique.
  • Automatic safety score determiner 404 provides the safety score to safety score database 406 .
  • Safety score database 406 comprises a database of safety scores of maneuver vectors.
  • safety score database 406 determines an overall safety score for a driver (e.g., by summing the safety scores associated with maneuver vectors over each day and determining the average safety score accumulated over a given week). In some embodiments, safety score database 406 determines an overall safety score for a fleet of drivers (e.g., by averaging the overall safety score for each driver in the fleet). In some embodiments, safety score database 406 determines a list of unsafe drivers (e.g., by taking the set of drivers with the worst safety scores, by taking the set of drivers with safety scores worse than a threshold, etc.).
  • a maneuver vector with associated video when received by vehicle data server 400 of FIG. 4 , it is provided to both manual safety score determiner 402 and automatic safety score determiner 404 .
  • safety score database 406 receives the safety scores from each of manual safety score determiner 402 and automatic safety score determiner 404 , it combines the two safety scores by averaging, weighted averaging, using a nonlinear combination, picking one of the two scores to use, or combining the scores in any other appropriate way.
  • FIG. 5 is a diagram illustrating an embodiment of a user interface for a manual safety score determiner.
  • the manual safety score determiner comprises manual safety score determiner 402 of FIG. 4 .
  • a maneuver reviewer e.g., a human uses the user interface for the manual safety score determiner to indicate a safety score.
  • user interface 500 comprises an inward camera display and a forward camera display. The inward camera display displays video from an inward-facing camera (e.g., facing the driver) and the forward camera display displays video from a forward-facing camera (e.g., facing the road ahead).
  • User interface 500 comprises a set of hazardous behavior checkboxes.
  • user interface 500 comprises checkboxes for cell phone usage, eating while driving, aggressive cornering (left), aggressive cornering (right), following too close, and collision.
  • user interface 500 comprises hazardous behavior checkboxes for Driving Fundamentals, Driver Distractions, Driver Awareness, Driver Condition, Road/Traffic Conditions, Weather Conditions, Seatbelt Usage, or any other appropriate hazardous behaviors.
  • User interface 500 additionally comprises a previous event button to return to review of a previous event, and a next event button to begin review of a next event.
  • a safety score is determined for the event associated with the displayed video data based on the hazardous behavior checkboxes selected by the maneuver reviewer.
  • FIG. 6 is a flow diagram illustrating an embodiment of a process for building a maneuver vector.
  • the process of FIG. 6 is executed by vehicle event recorder 300 of FIG. 3 .
  • sensor data is received.
  • sensor data comprises video camera data, microphone data, accelerometer data, gyroscope data, outdoor temperature sensor data, moisture sensor data, laser line tracker sensor data, vehicle state sensor data, or any other appropriate sensor data.
  • sensor data comprises calibrated accelerometer data from one or more accelerometers.
  • hazard probabilities are computed. In some embodiments, hazard probabilities are computed by processing the sensor data received in 600 .
  • hazard probabilities are computed by digital filtering, digital Fourier transform processing, a neural network, ensemble modeling, or any other appropriate technique.
  • hazard probabilities comprise a probability of hazardous acceleration, a probability of hazardous braking, a probability of a collision, a probability of hazardous left cornering, a probability of hazardous right cornering, a probability of a hazardous rough road, a probability of another hazardous condition, or any other appropriate hazard probability.
  • a new maneuver point is created from the hazard probabilities.
  • creating a new maneuver point from the hazard probabilities comprises storing the hazard probabilities as the maneuver point.
  • a maneuver point additionally comprises a date of creation, a time of creation, a driver identifier, a fleet identifier, a company identifier, a location, sensor data, a vehicle identifier, a maneuver point identifier, or any other appropriate maneuver point data.
  • it is determined whether the maneuver point is within a time threshold of an existing maneuver vector.
  • a time threshold comprises a time threshold since the first maneuver point in the maneuver vector. In some embodiments, a time threshold since the most recent point in the maneuver vector. If it is determined that the maneuver point is within the time threshold of an existing maneuver vector, control passes to 610 . In 610 , the maneuver point is added to the existing maneuver vector, and the process ends. If it is determined in 608 that the maneuver point is not within the time threshold of an existing maneuver vector, control passes to 612 . In 612 , a new maneuver vector is created with the maneuver point. In some embodiments, creating a new maneuver vector with the maneuver point comprises storing the maneuver point as the maneuver vector.
  • a maneuver vector additionally comprises a driver identifier, a vehicle identifier, a fleet identifier, a company identifier, a number of maneuver points, a time of creation, a time of most recent maneuver point, a duration, a location, sensor data, processed sensor data, statistical information on sensor data, indication of whether the maneuver vector comprises associated video, a maneuver vector identifier, or any other appropriate maneuver vector data.
  • FIG. 7 is a flow diagram illustrating an embodiment of a process for manually determining a safety score.
  • the process of FIG. 7 is executed by manual safety score determiner 402 of FIG. 4 .
  • a maneuver vector with associated video is received (e.g., from a vehicle event recorder).
  • the associated video is displayed (e.g., to a human maneuver reviewer).
  • a safety review is received.
  • a safety review comprises a set of associated hazardous behaviors.
  • a safety score is determined from the safety review.
  • determining the safety score comprises summing a set of safety scores associated with each of the set of hazardous behaviors.
  • the safety score is associated with the maneuver vector.
  • the maneuver vector and associated safety score are provided to an automatic safety score determiner and to a safety score database.
  • FIG. 8 is a flow diagram illustrating an embodiment of a process for automatically determining a safety score.
  • the process of FIG. 8 is executed by automatic safety score determiner 404 of FIG. 4 .
  • a maneuver vector is received.
  • the maneuver vector is associated with video data.
  • the maneuver vector is not associated with video data.
  • a safety score associated with the maneuver vector is determined.
  • the safety score is determined by using a table, by interpolating between values in a table, by using a neural network, by using machine learning techniques, by using a support vector machine, ensemble modeling or by using any other appropriate technique.
  • the safety score is provided to a safety score database.
  • FIG. 9 is a flow diagram illustrating a process for determining overall safety scores.
  • the process of FIG. 9 is executed by safety score database 406 of FIG. 4 .
  • the process of FIG. 9 is executed at predetermined intervals (e.g., every day, every week, etc.), upon a command (e.g., the command of the fleet supervisor, etc.), each time a safety score is received by the safety score database, or any other appropriate time.
  • a new driver safety score is determined.
  • determining a new driver safety score comprises summing a set of safety scores associated with the driver.
  • the set of safety scores comprises the set of safety scores received over a current period of time (e.g., the most recent day, the most recent month, the most recent 180 days, the most recent year, etc.).
  • the safety scores are weighted such that more recent safety scores affect the driver safety score more than older safety scores.
  • a new fleet safety score is determined.
  • determining a new fleet safety score comprises averaging the driver safety score for each driver in the fleet.
  • a new unsafe drivers list is determined.
  • a new unsafe drivers list comprises a list of drivers with driver safety score worse than a threshold, a list of the top 10% most unsafe drivers, a list of the top 25 most unsafe drivers (e.g., worst safety score drivers), a ranked list, or a list of any other appropriate unsafe drivers.
  • a new unsafe drivers list comprises a list of drivers that require coaching.
  • FIG. 10 is a flow diagram illustrating a process for determining a safety score.
  • maneuver data is received.
  • a safety score is determined from the maneuver data without inward facing video using a safety score determiner.
  • Maneuver data received is matched to a library of previously captured maneuvered data.
  • Each maneuver data stored in the library of previously captured maneuver data is stored with an associated safety score.
  • the associated safety score was assigned either automatically or manually and stored in the library associated with the maneuver data.
  • the maneuver data in the library is either with or without inward facing video data and is either directly associated with the driver associated with the received maneuver data or not at all associated with the driver associated with the received maneuver data.
  • the safety score is provided for the driver.

Abstract

A safety score using video data but without video comprises an input interface and a processor. The input interface is configured to receive sensor data. The processor is configured to provide the sensor data to a safety score determiner, and determine a safety score using the safety score determiner without using inward facing video data, wherein the safety score determiner has been trained using a data set that includes sensor data with inward facing video data. The system for determining a safety score additionally comprises a memory coupled to the processor and configured to provide the processor with instructions.

Description

BACKGROUND OF THE INVENTION
Modern vehicles (e.g., airplanes, boats, trains, cars, trucks, etc.) can include a vehicle event recorder in order to better understand the timeline of an anomalous event (e.g., an accident). A vehicle event recorder typically includes a set of sensors, e.g., video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS (global positioning system), etc., that report data, which is used to determine the occurrence of an anomalous event. If an anomalous event is detected, then sensor data related to the event is stored for later review. Data from an inward facing video camera (e.g., containing an image of the driver) can be very useful for determining anomalous events. However, some vehicle event recorder users (e.g., drivers, fleet managers, etc.) do not allow the inward facing video camera data to be recorded or used for event detection.
BRIEF DESCRIPTION OF THE DRAWINGS
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
FIG. 1 is a block diagram illustrating an embodiment of a system including a vehicle event recorder.
FIG. 2 is a block diagram illustrating an embodiment of a vehicle event recorder.
FIG. 3 is a block diagram illustrating an embodiment of a vehicle event recorder.
FIG. 4 is a block diagram illustrating an embodiment of a vehicle data server.
FIG. 5 is a diagram illustrating an embodiment of a user interface for a manual safety score determiner.
FIG. 6 is a flow diagram illustrating an embodiment of a process for building a maneuver vector.
FIG. 7 is a flow diagram illustrating an embodiment of a process for manually determining a safety score.
FIG. 8 is a flow diagram illustrating an embodiment of a process for automatically determining a safety score.
FIG. 9 is a flow diagram illustrating a process for determining overall safety scores.
FIG. 10 is a flow diagram illustrating a process for determining a safety score.
DETAILED DESCRIPTION
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
A safety score using video data but without video is disclosed. A system for determining a safety score comprises an input interface configured to receive sensor data. The system for determining a safety score additionally comprises a processor configured to provide the sensor data to a safety score determiner, and determine a safety score using the safety score determiner without using inward facing video data, wherein the safety score determiner has been trained using a data set that includes sensor data with inward facing video data. The system for determining a safety score additionally comprises a memory coupled to the processor and configured to provide the processor with instructions.
A vehicle event recorder mounted on a vehicle records vehicle data and anomalous vehicle events. Anomalous vehicle event types include accidents, speed limit violations, rough road events, hard maneuvering events (e.g., hard cornering, hard braking), dangerous driving events (e.g., cell phone usage, eating while driving, working too long of a shift, sleepy driving, etc.), and any other appropriate kind of anomalous vehicle events. The vehicle event recorder analyzes data from sensors (e.g., video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS, etc.) to determine when an anomalous event has occurred. In some embodiments, the vehicle event recorder applies one or more data processing techniques (e.g., filters, Fourier transform processing, neural networks, etc.) to sensor data to determine the probability of each of a set of hazardous events (e.g., hard maneuvering, collision, etc.). In the event that the probability of one or more of the hazardous events is above a corresponding threshold, the set of probabilities is saved as a maneuver point. Multiple maneuver points occurring within a time threshold of one another are grouped together to form a maneuver vector. In the event the sensors associated with the vehicle event recorder include video recorders, any available video data from the video recorders corresponding to the time of the maneuver is associated with the maneuver vector. The maneuver vector including video data, if available, is transmitted to a vehicle data server and analyzed. In some embodiments, the vehicle event recorder collects maneuver vectors over the course of a day and transmits them to the vehicle data server at the end of the day.
The vehicle event recorder analyzes the maneuver vector and any available video data to determine a safety score. In the event that video data is available, the safety score is determined by a manual safety score determiner comprising a human video reviewing station. A person views the available video data and indicates which of a set of dangerous behaviors are determined to be present. The safety score is computed from the behavior or behaviors indicated. The safety score is provided to a safety score database, where it is associated with the driver, and used in later calculations of the driver's overall safety score. In addition, the safety score is associated with the maneuver vector and provided to an automatic safety score determiner, which stores the maneuver vector and associated safety score. If the maneuver vector is already stored in the automatic safety score determiner with an associated safety score, an average safety score is computed, comprising the average of all safety scores received associated with the received maneuver vector. As a range of maneuver vectors with associated video data are received and processed manually, the automatic safety score determiner builds a library of maneuver vectors and typical associated safety score values. When a maneuver vector with no associated video data is received by the vehicle event recorder, it is provided to the automatic safety score determiner. The automatic safety score determiner looks up the maneuver vector in the library of maneuver vectors and returns the associated safety score value. In some embodiments, if the maneuver vector is not found in the library of maneuver vectors, the automatic safety score determiner determines the maneuver vector in the library that is most similar to the received maneuver vector (e.g., a library with source data from the same driver, from other drivers, etc.) and returns the safety score value associated with that maneuver vector.
FIG. 1 is a block diagram illustrating an embodiment of a system including a vehicle event recorder. Vehicle event recorder 102 comprises a vehicle event recorder mounted in a vehicle (e.g., a car or truck). In some embodiments, vehicle event recorder 102 includes or is in communication with a set of sensors—for example, video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS, outdoor temperature sensors, moisture sensors, laser line tracker sensors, or any other appropriate sensors. In various embodiments, vehicle state sensors comprise a speedometer, an accelerator pedal sensor, a brake pedal sensor, an engine revolutions per minute (e.g., RPM) sensor, an engine temperature sensor, a headlight sensor, an airbag deployment sensor, driver and passenger seat weight sensors, an anti-locking brake sensor, an engine exhaust sensor, a gear position sensor, a cabin equipment operation sensor, or any other appropriate vehicle state sensors. In some embodiments, vehicle event recorder 102 comprises a system for processing sensor data and detecting events. In some embodiments, vehicle event recorder 102 comprises a system for detecting risky behavior. In various embodiments, vehicle event recorder 102 is mounted to vehicle 106 in one of the following locations: the chassis, the front grill, the dashboard, the rear-view mirror, or any other appropriate location. In some embodiments, vehicle event recorder 102 comprises multiple units mounted in different locations in vehicle 106. In some embodiments, vehicle event recorder 102 comprises a communications system for communicating with network 100. In various embodiments, network 100 comprises a wireless network, a wired network, a cellular network, a code division multiple access (CDMA) network, a global system for mobile communication (GSM) network, a local area network, a wide area network, the Internet, or any other appropriate network. In some embodiments, network 100 comprises multiple networks, changing over time and location. Vehicle event recorder 102 communicates with vehicle data server 104 via network 100. Vehicle event recorder 102 is mounted on vehicle 106. In various embodiments, vehicle 106 comprises a car, a truck, a commercial vehicle, or any other appropriate vehicle. Vehicle data server 104 comprises a vehicle data server for collecting events and risky behavior detected by vehicle event recorder 102. In some embodiments, vehicle data server 104 comprises a system for collecting data from multiple vehicle event recorders. In some embodiments, vehicle data server 104 comprises a system for analyzing vehicle event recorder data. In some embodiments, vehicle data server 104 comprises a system for displaying vehicle event recorder data. In some embodiments, vehicle data server 104 is located at a home station (e.g., a shipping company office, a taxi dispatcher, a truck depot, etc.). In some embodiments, events recorded by vehicle event recorder 102 are downloaded to vehicle data server 104 when vehicle 106 arrives at the home station. In some embodiments, vehicle data server 104 is located at a remote location. In some embodiments, events recorded by vehicle event recorder 102 are downloaded to vehicle data server 104 wirelessly. In some embodiments, a subset of events recorded by vehicle event recorder 102 is downloaded to vehicle data server 104 wirelessly.
FIG. 2 is a block diagram illustrating an embodiment of a vehicle event recorder. In some embodiments, vehicle event recorder 200 of FIG. 2 comprises vehicle event recorder 102 of FIG. 1. In the example shown, vehicle event recorder 200 comprises processor 202. Processor 202 comprises a processor for controlling the operations of vehicle event recorder 200, for reading and writing information on data storage 204, for communicating via wireless communications interface 206, for determining a position using global positioning system 208, and for reading data via sensor interface 210. Data storage 204 comprises a data storage (e.g., a random access memory (RAM), a read only memory (ROM), a nonvolatile memory, a flash memory, a hard disk, or any other appropriate data storage). In various embodiments, data storage 204 comprises a data storage for storing instructions for processor 202, vehicle event recorder data, vehicle event data, sensor data, video data, map data, or any other appropriate data. In various embodiments, wireless communications interface 206 comprises one or more of a GSM interface, a CDMA interface, a WiFi interface, or any other appropriate interface. Global positioning system 208 comprises a global positioning system (e.g., GPS) for determining a system location. Sensor interface 210 comprises an interface to one or more vehicle event recorder sensors. In various embodiments, vehicle event recorder sensors comprise an external video camera, an internal video camera, a microphone, an accelerometer, a gyroscope, an outdoor temperature sensor, a moisture sensor, a laser line tracker sensor, vehicle state sensors, or any other appropriate sensors. In various embodiments, vehicle state sensors comprise a speedometer, an accelerator pedal sensor, a brake pedal sensor, an engine RPM sensor, an engine temperature sensor, a headlight sensor, an airbag deployment sensor, driver and passenger seat weight sensors, an anti-locking brake sensor, an engine exhaust sensor, a gear position sensor, a cabin equipment operation sensor, or any other appropriate vehicle state sensors. In some embodiments, sensor interface 210 comprises an on-board diagnostics (OBD) bus. In some embodiments, vehicle event recorder 200 communicates with vehicle state sensors via OBD bus.
FIG. 3 is a block diagram illustrating an embodiment of a vehicle event recorder. In some embodiments, vehicle event recorder 300 of FIG. 3 comprises vehicle event recorder 200 of FIG. 2. In the example shown, vehicle event recorder 300 comprises sensors 302. In some embodiments, sensors 302 comprises a sensor interface and vehicle event recorder 300 is in communication with sensors via the sensor interface. Sensors 302 deliver sensor data to data processing 304. In various embodiments, sensor data comprises video camera data, microphone data, accelerometer data, gyroscope data, outdoor temperature sensor data, moisture sensor data, laser line tracker sensor data, vehicle state sensor data, or any other appropriate sensor data. In some embodiments, sensor data comprises calibrated accelerometer data from one or more accelerometers. Data processing 304 processes sensor data to determine a set of hazard probabilities. Hazard probabilities comprise the probabilities that one or more of a set of hazards are occurring. In various embodiments, data processing 304 comprises one or more of the following: digital filtering, digital Fourier transform processing, a neural network, or any other appropriate data processing technique. In various embodiments, hazard probabilities comprise a probability of hazardous acceleration, a probability of hazardous braking, a probability of a collision, a probability of hazardous left cornering, a probability of hazardous right cornering, a probability of a hazardous rough road, a probability of another hazardous condition, or any other appropriate hazard probability. Data processing 304 provides hazard probabilities to hazard probability thresholds 306. Hazard probability thresholds 306 determines if any of the hazard probabilities received from data processing 304 are above their corresponding hazard probability threshold. If none of the hazard probabilities are above the corresponding hazard probability threshold, the set of hazard probabilities is discarded. If any of the hazard probabilities is above the corresponding hazard probability threshold, the set of hazard probabilities is labeled a maneuver point. Hazard probability thresholds 306 provides maneuver points to maneuver vector time threshold 308. Maneuver vector time threshold 308 determines if a received maneuver point is within a time threshold of a maneuver vector. In some embodiments, if two maneuver points are received within a time threshold of one another, the two maneuver points are determined to be a part of the same maneuver vector. As maneuver points are received within the time threshold of an existing maneuver vector, the maneuver points are added to the maneuver vector. In some embodiments, the time threshold comprises a time since the first received maneuver point in the maneuver vector. In some embodiments, the time threshold comprises a time since the last received maneuver point in the maneuver vector. When a maneuver vector is completed, it is delivered to a vehicle data server. In some embodiments, the maneuver vector is stored and delivered to the vehicle data server at a later point in time (e.g., when the vehicle checks in at a depot, at the end of the day, at the end of the driver shift, etc.). In some embodiments, if sensors 302 combine video sensors (e.g., an inward-facing video camera, a forward-facing video camera), video data corresponding to the time period of a maneuver vector is delivered to the vehicle data server associated with the maneuver vector.
FIG. 4 is a block diagram illustrating an embodiment of a vehicle data server. In some embodiments, vehicle data server 400 comprises vehicle data server 104 of FIG. 1. In the example shown, vehicle data server 400 receives maneuver vectors from one or more vehicle event recorders (e.g., associated with one or more different vehicles). Some maneuver vectors received include associated video data, and some maneuver vectors received have no associated video. In some embodiments, some vehicle event recorders do not include or communicate with video recorders, and do not provide video data associated with maneuver vectors. In the example shown, when vehicle data server 400 receives a maneuver vector with associated video, it is provided to manual safety score determiner 402. Manual safety score determiner 402 manually determines a safety score associated with the maneuver vector with associated video. In some embodiments, manual safety score determiner 402 determines a safety score by displaying video to a maneuver reviewer (e.g., a human) and receiving from the maneuver reviewer a set of associated hazardous behaviors. In some embodiments, the safety score comprises a sum of safety scores associated with each of the set of hazardous behaviors. Manual safety score determiner 402 provides the maneuver vector and the associated safety score to automatic safety score determiner 404 and driver safety score database 406. Automatic safety score determiner 404 stores maneuver vectors and associated safety scores received from manual safety score determiner in a library of maneuver vectors with associated safety scores. In some embodiments, if automatic safety score determiner 404 receives a maneuver vector for which it has already received one or more safety scores, an average safety score is computed, comprising the average of all safety scores received associated with the received maneuver vector. As a range of maneuver vectors with associated video data are received and processed by manual safety score determiner 402, automatic safety score determiner 404 builds a library of maneuver vectors and typical associated safety score values. In some embodiments, building a library of maneuver vectors and typical associated safety score values comprises training automatic safety score determiner 404. In some embodiments, in order to rapidly build a library of maneuver vectors and typical associated safety score values, a third party data set of maneuver vectors with associated video data is used to train automatic safety score determiner 404. In some embodiments, a third party data set of maneuver vectors with associated video data comprises a set of maneuver vectors with associated video data purchased from an external source.
When vehicle data server 400 receives a maneuver vector without associated video, the maneuver vector is provided to automatic safety score determiner 404. Automatic safety score determiner utilizes its library of maneuver vectors and typical associated safety score values to determine a safety score value associated with the received maneuver vector. In various embodiments, automatic safety score determiner 404 determines the safety score by using a table, by interpolating between entries in a table, by using a neural network, by using machine learning techniques, by using a support vector machine, by leveraging ensemble modeling, or by using any other appropriate technique. Automatic safety score determiner 404 provides the safety score to safety score database 406. Safety score database 406 comprises a database of safety scores of maneuver vectors. In some embodiments, safety score database 406 determines an overall safety score for a driver (e.g., by summing the safety scores associated with maneuver vectors over each day and determining the average safety score accumulated over a given week). In some embodiments, safety score database 406 determines an overall safety score for a fleet of drivers (e.g., by averaging the overall safety score for each driver in the fleet). In some embodiments, safety score database 406 determines a list of unsafe drivers (e.g., by taking the set of drivers with the worst safety scores, by taking the set of drivers with safety scores worse than a threshold, etc.).
In some embodiments, when a maneuver vector with associated video is received by vehicle data server 400 of FIG. 4, it is provided to both manual safety score determiner 402 and automatic safety score determiner 404. When safety score database 406 receives the safety scores from each of manual safety score determiner 402 and automatic safety score determiner 404, it combines the two safety scores by averaging, weighted averaging, using a nonlinear combination, picking one of the two scores to use, or combining the scores in any other appropriate way.
FIG. 5 is a diagram illustrating an embodiment of a user interface for a manual safety score determiner. In some embodiments, the manual safety score determiner comprises manual safety score determiner 402 of FIG. 4. In some embodiments, a maneuver reviewer (e.g., a human) uses the user interface for the manual safety score determiner to indicate a safety score. In the example shown, user interface 500 comprises an inward camera display and a forward camera display. The inward camera display displays video from an inward-facing camera (e.g., facing the driver) and the forward camera display displays video from a forward-facing camera (e.g., facing the road ahead). In some embodiments, one of the inward camera display and the forward camera display does not display video (e.g., video data from only one camera is present). User interface 500 comprises a set of hazardous behavior checkboxes. In the example shown, user interface 500 comprises checkboxes for cell phone usage, eating while driving, aggressive cornering (left), aggressive cornering (right), following too close, and collision. In various embodiments, user interface 500 comprises hazardous behavior checkboxes for Driving Fundamentals, Driver Distractions, Driver Awareness, Driver Condition, Road/Traffic Conditions, Weather Conditions, Seatbelt Usage, or any other appropriate hazardous behaviors. User interface 500 additionally comprises a previous event button to return to review of a previous event, and a next event button to begin review of a next event. In some embodiments, a safety score is determined for the event associated with the displayed video data based on the hazardous behavior checkboxes selected by the maneuver reviewer.
FIG. 6 is a flow diagram illustrating an embodiment of a process for building a maneuver vector. In some embodiments, the process of FIG. 6 is executed by vehicle event recorder 300 of FIG. 3. In the example shown, in 600, sensor data is received. In various embodiments, sensor data comprises video camera data, microphone data, accelerometer data, gyroscope data, outdoor temperature sensor data, moisture sensor data, laser line tracker sensor data, vehicle state sensor data, or any other appropriate sensor data. In some embodiments, sensor data comprises calibrated accelerometer data from one or more accelerometers. In 602, hazard probabilities are computed. In some embodiments, hazard probabilities are computed by processing the sensor data received in 600. In various embodiments, hazard probabilities are computed by digital filtering, digital Fourier transform processing, a neural network, ensemble modeling, or any other appropriate technique. In various embodiments, hazard probabilities comprise a probability of hazardous acceleration, a probability of hazardous braking, a probability of a collision, a probability of hazardous left cornering, a probability of hazardous right cornering, a probability of a hazardous rough road, a probability of another hazardous condition, or any other appropriate hazard probability. In 604, it is determined whether one or more of the hazard probabilities is above the corresponding hazard probability threshold. If it is determined that one or more of the hazard probabilities is not above the corresponding hazard probability threshold, the process ends. If it is determined that one or more of the hazard probabilities is above the corresponding hazard probability threshold, control passes to 606. In 606, a new maneuver point is created from the hazard probabilities. In some embodiments, creating a new maneuver point from the hazard probabilities comprises storing the hazard probabilities as the maneuver point. In various embodiments, a maneuver point additionally comprises a date of creation, a time of creation, a driver identifier, a fleet identifier, a company identifier, a location, sensor data, a vehicle identifier, a maneuver point identifier, or any other appropriate maneuver point data. In 608 it is determined whether the maneuver point is within a time threshold of an existing maneuver vector. In some embodiments, a time threshold comprises a time threshold since the first maneuver point in the maneuver vector. In some embodiments, a time threshold since the most recent point in the maneuver vector. If it is determined that the maneuver point is within the time threshold of an existing maneuver vector, control passes to 610. In 610, the maneuver point is added to the existing maneuver vector, and the process ends. If it is determined in 608 that the maneuver point is not within the time threshold of an existing maneuver vector, control passes to 612. In 612, a new maneuver vector is created with the maneuver point. In some embodiments, creating a new maneuver vector with the maneuver point comprises storing the maneuver point as the maneuver vector. In various embodiments, a maneuver vector additionally comprises a driver identifier, a vehicle identifier, a fleet identifier, a company identifier, a number of maneuver points, a time of creation, a time of most recent maneuver point, a duration, a location, sensor data, processed sensor data, statistical information on sensor data, indication of whether the maneuver vector comprises associated video, a maneuver vector identifier, or any other appropriate maneuver vector data.
FIG. 7 is a flow diagram illustrating an embodiment of a process for manually determining a safety score. In some embodiments, the process of FIG. 7 is executed by manual safety score determiner 402 of FIG. 4. In 700, a maneuver vector with associated video is received (e.g., from a vehicle event recorder). In 702, the associated video is displayed (e.g., to a human maneuver reviewer). In 704, a safety review is received. In some embodiments, a safety review comprises a set of associated hazardous behaviors. In 706, a safety score is determined from the safety review. In some embodiments, determining the safety score comprises summing a set of safety scores associated with each of the set of hazardous behaviors. In 708, the safety score is associated with the maneuver vector. In some embodiments, the maneuver vector and associated safety score are provided to an automatic safety score determiner and to a safety score database.
FIG. 8 is a flow diagram illustrating an embodiment of a process for automatically determining a safety score. In some embodiments, the process of FIG. 8 is executed by automatic safety score determiner 404 of FIG. 4. In 800, a maneuver vector is received. In some embodiments, the maneuver vector is associated with video data. In some embodiments, the maneuver vector is not associated with video data. In 802, a safety score associated with the maneuver vector is determined. In various embodiments, the safety score is determined by using a table, by interpolating between values in a table, by using a neural network, by using machine learning techniques, by using a support vector machine, ensemble modeling or by using any other appropriate technique. In some embodiments, the safety score is provided to a safety score database.
FIG. 9 is a flow diagram illustrating a process for determining overall safety scores. In some embodiments, the process of FIG. 9 is executed by safety score database 406 of FIG. 4. In various embodiments, the process of FIG. 9 is executed at predetermined intervals (e.g., every day, every week, etc.), upon a command (e.g., the command of the fleet supervisor, etc.), each time a safety score is received by the safety score database, or any other appropriate time. In the example shown, in 900, a new driver safety score is determined. In some embodiments, determining a new driver safety score comprises summing a set of safety scores associated with the driver. In some embodiments, the set of safety scores comprises the set of safety scores received over a current period of time (e.g., the most recent day, the most recent month, the most recent 180 days, the most recent year, etc.). In some embodiments, the safety scores are weighted such that more recent safety scores affect the driver safety score more than older safety scores. In 902, a new fleet safety score is determined. In some embodiments, determining a new fleet safety score comprises averaging the driver safety score for each driver in the fleet. In 904 a new unsafe drivers list is determined. In some embodiments, a new unsafe drivers list comprises a list of drivers with driver safety score worse than a threshold, a list of the top 10% most unsafe drivers, a list of the top 25 most unsafe drivers (e.g., worst safety score drivers), a ranked list, or a list of any other appropriate unsafe drivers. In some embodiments, a new unsafe drivers list comprises a list of drivers that require coaching.
FIG. 10 is a flow diagram illustrating a process for determining a safety score. In the example shown, in 1000 maneuver data is received. In 1002, a safety score is determined from the maneuver data without inward facing video using a safety score determiner. Maneuver data received is matched to a library of previously captured maneuvered data. Each maneuver data stored in the library of previously captured maneuver data is stored with an associated safety score. The associated safety score was assigned either automatically or manually and stored in the library associated with the maneuver data. The maneuver data in the library is either with or without inward facing video data and is either directly associated with the driver associated with the received maneuver data or not at all associated with the driver associated with the received maneuver data. The safety score is provided for the driver.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims (21)

What is claimed is:
1. A system for determining a safety score, comprising:
an input interface configured to receive maneuver data; and
a processor configured to:
determine a safety score from the maneuver data without inward facing video data using a safety score determiner; and
output the safety score, wherein the safety score is associated with the maneuver data and forms at least a partial basis to train the safety score determiner;
wherein inward facing video data is processable to identify anomalous events and train the safety score determiner and the safety score determiner has been trained using a data set that includes previously-collected maneuver data associated with inward facing video data.
2. A system as in claim 1, wherein the safety score determiner comprises a library of maneuver data and safety scores.
3. A system as in claim 1, wherein the safety score determiner determines the safety score by using a table.
4. A system as in claim 1, wherein the safety score determiner determines the safety score by interpolating between values in a table.
5. A system as in claim 1, wherein the safety score determiner determines the safety score by using a neural network.
6. A system as in claim 1, wherein the safety score determiner determines the safety score by using machine learning techniques.
7. A system as in claim 1, wherein the safety score determiner determines the safety score by using a support vector machine.
8. A system as in claim 1, wherein the system for determining the safety score additionally determines a driver safety score.
9. A system as in claim 8, wherein the driver safety score is recomputed periodically.
10. A system as in claim 1, wherein the system for determining the safety score additionally determines a fleet safety score.
11. A system as in claim 1, wherein the system for determining the safety score additionally determines an unsafe drivers list.
12. A system as in claim 11, wherein the unsafe drivers list is recomputed periodically.
13. A system as in claim 11, wherein the unsafe drivers list is a ranked list.
14. A system as in claim 11, wherein the unsafe drivers list comprises a list of drivers with safety scores worse than a threshold.
15. A system as in claim 11, wherein the unsafe drivers list comprises a list of drivers with top 10% worst safety scores.
16. A system as in claim 11, wherein the unsafe drivers list comprises a ranked list of drivers, the list including a pre-determined number of worst ranking drivers based on safety scores.
17. A system as in claim 1, wherein the system for determining the safety score additionally determines a list of drivers that require coaching.
18. A system as in claim 1, wherein the data set that includes maneuver data associated with inward facing video data comprises a third party data set.
19. A method for determining a safety score, comprising:
receiving maneuver data;
determining, using a processor, a safety score from the maneuver data without inward facing video data using a safety score determiner; and
outputting the safety score, wherein the safety score is associated with the maneuver data and forms at least a partial basis to train the safety score determiner;
wherein inward facing video data is processable to identify anomalous events and train the safety score determiner and the safety score determiner has been trained using a data set that includes previously-collected maneuver data associated with inward facing video data.
20. A computer program product for determining a safety score, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving maneuver data;
determining a safety score from the maneuver data without inward facing video data using a safety score determiner; and
outputting the safety score, wherein the safety score is associated with the maneuver data and forms at least a partial basis to train the safety score determiner;
wherein inward facing video data is processable to identify anomalous events and train the safety score determiner and the safety score determiner has been trained using a data set that includes previously-collected maneuver data associated with inward facing video data.
21. A method for determining a safety score, comprising:
receiving maneuver data, wherein the maneuver data does not include inward facing video data;
determining, using a processor, a safety score based on the maneuver data including:
determining whether the maneuver data matches at least one element of a set of previously-collected maneuver data, wherein the set of previously-collected maneuver data is trained by inward facing video data;
if the maneuver data matches at least one element of the set of previously-collected maneuver data, determining the safety score based on an average of scores associated with the at least one element of the set matching the received maneuver data; and
if the maneuver data does not match at least one element of the set of previously-collected maneuver data, determining the safety score based on a combination of scores associated with at least one element of the set meeting a threshold similarity compared with the received maneuver data; and
outputting the safety score.
US14/012,136 2013-08-28 2013-08-28 Safety score using video data but without video Active 2035-11-02 US9594725B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US14/012,136 US9594725B1 (en) 2013-08-28 2013-08-28 Safety score using video data but without video
US15/420,470 US10068392B2 (en) 2013-08-28 2017-01-31 Safety score using video data but without video

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US14/012,136 US9594725B1 (en) 2013-08-28 2013-08-28 Safety score using video data but without video

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/420,470 Continuation US10068392B2 (en) 2013-08-28 2017-01-31 Safety score using video data but without video

Publications (1)

Publication Number Publication Date
US9594725B1 true US9594725B1 (en) 2017-03-14

Family

ID=58227703

Family Applications (2)

Application Number Title Priority Date Filing Date
US14/012,136 Active 2035-11-02 US9594725B1 (en) 2013-08-28 2013-08-28 Safety score using video data but without video
US15/420,470 Active 2033-12-06 US10068392B2 (en) 2013-08-28 2017-01-31 Safety score using video data but without video

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/420,470 Active 2033-12-06 US10068392B2 (en) 2013-08-28 2017-01-31 Safety score using video data but without video

Country Status (1)

Country Link
US (2) US9594725B1 (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190047580A1 (en) * 2017-12-27 2019-02-14 Robert F. Kwasnick Safety inference engine for autonomous systems
US20200059626A1 (en) * 2018-08-20 2020-02-20 Omnitracs, Llc Fleet wide video search
US10594991B1 (en) 2018-01-09 2020-03-17 Wm Intellectual Property Holdings, Llc System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US10877783B2 (en) * 2017-06-14 2020-12-29 Wipro Limited System and method for alerting a user of risks
US10917690B1 (en) 2016-03-24 2021-02-09 Massachusetts Mutual Life Insurance Company Intelligent and context aware reading systems
US10919543B2 (en) * 2019-01-30 2021-02-16 StradVision, Inc. Learning method and learning device for determining whether to switch mode of vehicle from manual driving mode to autonomous driving mode by performing trajectory-based behavior analysis on recent driving route
US10986223B1 (en) * 2013-12-23 2021-04-20 Massachusetts Mutual Life Insurance Systems and methods for presenting content based on user behavior
US11373536B1 (en) 2021-03-09 2022-06-28 Wm Intellectual Property Holdings, L.L.C. System and method for customer and/or container discovery based on GPS drive path and parcel data analysis for a waste / recycling service vehicle
US11386362B1 (en) 2020-12-16 2022-07-12 Wm Intellectual Property Holdings, L.L.C. System and method for optimizing waste / recycling collection and delivery routes for service vehicles
US11475416B1 (en) 2019-08-23 2022-10-18 Wm Intellectual Property Holdings Llc System and method for auditing the fill status of a customer waste container by a waste services provider during performance of a waste service activity
US11488118B1 (en) 2021-03-16 2022-11-01 Wm Intellectual Property Holdings, L.L.C. System and method for auditing overages and contamination for a customer waste container by a waste services provider during performance of a waste service activity
US20220402505A1 (en) * 2020-05-18 2022-12-22 Verizon Connect Development Limited Systems and methods for detecting and classifying an unsafe maneuver of a vehicle
US11842404B2 (en) 2014-04-15 2023-12-12 Speedgauge, Inc. Enhancement using analytics based on vehicle kinematic data
US11928693B1 (en) 2021-03-09 2024-03-12 Wm Intellectual Property Holdings, L.L.C. System and method for customer and/or container discovery based on GPS drive path analysis for a waste / recycling service vehicle

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210094582A1 (en) * 2019-09-27 2021-04-01 Bluebox Labs, Inc. After-market vehicle copilot device

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070027583A1 (en) * 2003-07-07 2007-02-01 Sensomatix Ltd. Traffic information system

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070027583A1 (en) * 2003-07-07 2007-02-01 Sensomatix Ltd. Traffic information system

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10986223B1 (en) * 2013-12-23 2021-04-20 Massachusetts Mutual Life Insurance Systems and methods for presenting content based on user behavior
US11842404B2 (en) 2014-04-15 2023-12-12 Speedgauge, Inc. Enhancement using analytics based on vehicle kinematic data
US10917690B1 (en) 2016-03-24 2021-02-09 Massachusetts Mutual Life Insurance Company Intelligent and context aware reading systems
US10877783B2 (en) * 2017-06-14 2020-12-29 Wipro Limited System and method for alerting a user of risks
US10513272B2 (en) * 2017-12-27 2019-12-24 Intel Corporation Safety inference engine for autonomous systems
US20190047580A1 (en) * 2017-12-27 2019-02-14 Robert F. Kwasnick Safety inference engine for autonomous systems
US11616933B1 (en) 2018-01-09 2023-03-28 Wm Intellectual Property Holdings, L.L.C. System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US10594991B1 (en) 2018-01-09 2020-03-17 Wm Intellectual Property Holdings, Llc System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US10855958B1 (en) 2018-01-09 2020-12-01 Wm Intellectual Property Holdings, Llc System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US10750134B1 (en) 2018-01-09 2020-08-18 Wm Intellectual Property Holdings, L.L.C. System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US11128841B1 (en) 2018-01-09 2021-09-21 Wm Intellectual Property Holdings, Llc System and method for managing service and non service related activities associated with a waste collection, disposal and/or recycling vehicle
US11140367B1 (en) 2018-01-09 2021-10-05 Wm Intellectual Property Holdings, Llc System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US11172171B1 (en) 2018-01-09 2021-11-09 Wm Intellectual Property Holdings, Llc System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US11425340B1 (en) 2018-01-09 2022-08-23 Wm Intellectual Property Holdings, Llc System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US10911726B1 (en) 2018-01-09 2021-02-02 Wm Intellectual Property Holdings, Llc System and method for managing service and non-service related activities associated with a waste collection, disposal and/or recycling vehicle
US11902702B2 (en) * 2018-08-20 2024-02-13 Omnitracs, Llc Fleet wide video search
US20200059626A1 (en) * 2018-08-20 2020-02-20 Omnitracs, Llc Fleet wide video search
US10919543B2 (en) * 2019-01-30 2021-02-16 StradVision, Inc. Learning method and learning device for determining whether to switch mode of vehicle from manual driving mode to autonomous driving mode by performing trajectory-based behavior analysis on recent driving route
US11475416B1 (en) 2019-08-23 2022-10-18 Wm Intellectual Property Holdings Llc System and method for auditing the fill status of a customer waste container by a waste services provider during performance of a waste service activity
US11475417B1 (en) 2019-08-23 2022-10-18 Wm Intellectual Property Holdings, Llc System and method for auditing the fill status of a customer waste container by a waste services provider during performance of a waste service activity
US20220402505A1 (en) * 2020-05-18 2022-12-22 Verizon Connect Development Limited Systems and methods for detecting and classifying an unsafe maneuver of a vehicle
US11654924B2 (en) * 2020-05-18 2023-05-23 Verizon Connect Development Limited Systems and methods for detecting and classifying an unsafe maneuver of a vehicle
US11790290B1 (en) 2020-12-16 2023-10-17 Wm Intellectual Property Holdings, L.L.C. System and method for optimizing waste / recycling collection and delivery routes for service vehicles
US11386362B1 (en) 2020-12-16 2022-07-12 Wm Intellectual Property Holdings, L.L.C. System and method for optimizing waste / recycling collection and delivery routes for service vehicles
US11727337B1 (en) 2021-03-09 2023-08-15 Wm Intellectual Property Holdings, L.L.C. System and method for customer and/or container discovery based on GPS drive path and parcel data analysis for a waste / recycling service vehicle
US11373536B1 (en) 2021-03-09 2022-06-28 Wm Intellectual Property Holdings, L.L.C. System and method for customer and/or container discovery based on GPS drive path and parcel data analysis for a waste / recycling service vehicle
US11928693B1 (en) 2021-03-09 2024-03-12 Wm Intellectual Property Holdings, L.L.C. System and method for customer and/or container discovery based on GPS drive path analysis for a waste / recycling service vehicle
US11488118B1 (en) 2021-03-16 2022-11-01 Wm Intellectual Property Holdings, L.L.C. System and method for auditing overages and contamination for a customer waste container by a waste services provider during performance of a waste service activity

Also Published As

Publication number Publication date
US10068392B2 (en) 2018-09-04
US20170236338A1 (en) 2017-08-17

Similar Documents

Publication Publication Date Title
US10068392B2 (en) Safety score using video data but without video
US9996756B2 (en) Detecting risky driving with machine vision
US11059491B2 (en) Driving abnormality detection
US9650051B2 (en) Autonomous driving comparison and evaluation
US10210771B2 (en) Back-end event risk assessment with historical coaching profiles
US10255528B1 (en) Sensor fusion for lane departure behavior detection
US10545499B2 (en) Determining driver engagement with autonomous vehicle
US9626763B1 (en) Pothole detection
US9280857B2 (en) Dynamic uploading protocol
US10431089B1 (en) Crowdsourced vehicle history
EP2830023B1 (en) Managing the camera acquiring interior data
US10311749B1 (en) Safety score based on compliance and driving
EP3374981B1 (en) Traffic estimation
US10035514B1 (en) Dynamic configuration of event recorder content capture
US10015462B1 (en) Risk dependent variable compression rate for event storage
US20230385665A1 (en) Environmental condition-based risk level
US10040459B1 (en) Driver fuel score
US20230103670A1 (en) Video analysis for efficient sorting of event data
US10445603B1 (en) System for capturing a driver image
US9965907B1 (en) Running characteristic for frequent data readings
US11514733B1 (en) Extended time scale event detection
US11615654B1 (en) Signature use for efficient data transfer

Legal Events

Date Code Title Description
AS Assignment

Owner name: DRIVECAM, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:COOK, BRYON;KYPRI, NIKI;REEL/FRAME:031597/0039

Effective date: 20131008

AS Assignment

Owner name: LYTX, INC., CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:DRIVECAM, INC.;REEL/FRAME:032019/0172

Effective date: 20131104

AS Assignment

Owner name: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT,

Free format text: SECURITY AGREEMENT;ASSIGNORS:LYTX, INC.;MOBIUS ACQUISITION HOLDINGS, LLC;REEL/FRAME:032134/0756

Effective date: 20140124

AS Assignment

Owner name: U.S. BANK NATIONAL ASSOCIATION, AS ADMINISTRATIVE AGENT, NORTH CAROLINA

Free format text: SECURITY INTEREST;ASSIGNOR:LYTX, INC.;REEL/FRAME:038103/0508

Effective date: 20160315

Owner name: LYTX, INC., CALIFORNIA

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME 032134/0756;ASSIGNOR:WELLS FARGO BANK, NATIONAL ASSOCIATION;REEL/FRAME:038103/0328

Effective date: 20160315

Owner name: U.S. BANK NATIONAL ASSOCIATION, AS ADMINISTRATIVE

Free format text: SECURITY INTEREST;ASSIGNOR:LYTX, INC.;REEL/FRAME:038103/0508

Effective date: 20160315

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: HPS INVESTMENT PARTNERS, LLC, AS COLLATERAL AGENT, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNOR:LYTX, INC.;REEL/FRAME:043745/0567

Effective date: 20170831

Owner name: LYTX, INC., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:U.S. BANK, NATIONAL ASSOCIATION;REEL/FRAME:043743/0648

Effective date: 20170831

Owner name: HPS INVESTMENT PARTNERS, LLC, AS COLLATERAL AGENT,

Free format text: SECURITY INTEREST;ASSIGNOR:LYTX, INC.;REEL/FRAME:043745/0567

Effective date: 20170831

AS Assignment

Owner name: GUGGENHEIM CREDIT SERVICES, LLC, NEW YORK

Free format text: NOTICE OF SUCCESSOR AGENT AND ASSIGNMENT OF SECURITY INTEREST (PATENTS) REEL/FRAME 043745/0567;ASSIGNOR:HPS INVESTMENT PARTNERS, LLC;REEL/FRAME:052050/0115

Effective date: 20200228

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4