WO2005029264A2 - Tracking systems and methods - Google Patents

Tracking systems and methods Download PDF

Info

Publication number
WO2005029264A2
WO2005029264A2 PCT/US2004/030421 US2004030421W WO2005029264A2 WO 2005029264 A2 WO2005029264 A2 WO 2005029264A2 US 2004030421 W US2004030421 W US 2004030421W WO 2005029264 A2 WO2005029264 A2 WO 2005029264A2
Authority
WO
WIPO (PCT)
Prior art keywords
track
determining
correlating
objects
stopped
Prior art date
Application number
PCT/US2004/030421
Other languages
French (fr)
Other versions
WO2005029264A3 (en
Inventor
Gil J. Ettinger
Matthew Antone
W. Eric L. Grimson
Original Assignee
Alphatech, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alphatech, Inc. filed Critical Alphatech, Inc.
Priority to EP04788810A priority Critical patent/EP1668469A4/en
Publication of WO2005029264A2 publication Critical patent/WO2005029264A2/en
Publication of WO2005029264A3 publication Critical patent/WO2005029264A3/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the disclosed methods and systems relate generally to tracking methods and systems, and more particularly to tracking in unstructured environments.
  • VSAM video surveillance and monitoring
  • Such systems typically produce enormous quantities of data too overwhelming for human operators to process.
  • Video footage is often analyzed superficially, recorded without review, and/or simply ignored; however, high-coverage, continuous imaging provides a rich information source which, if used intelligently, can allow automatic characterization of normal site activities, detection of anomalous behaviors, and tracking of objects of interest.
  • Many video surveillance technology systems rely on face recognition or other biometrics, for example to screen airline passengers as they pass through heavily-trafficked areas.
  • variable viewing conditions under which the systems can operate include: (i) illumination (e.g., day/night, sunny/cloudy, sun angle, specularities); (ii) weather (e.g., dry/wet, seasonal changes, variable backgrounds (snow, leaves)); (iii) scene content variables including: (a) object density, speed, count; and, (b) size/shape/color within and across object classes; and, (iv) nuisance background clutter (e.g., shadows, swaying trees).
  • illumination e.g., day/night, sunny/cloudy, sun angle, specularities
  • weather e.g., dry/wet, seasonal changes, variable backgrounds (snow, leaves)
  • scene content variables including: (a) object density, speed, count; and, (b) size/shape/color within and across object classes; and, (iv) nuisance background clutter (e.g., shadows, swaying trees).
  • the disclosed methods and systems include monitoring applications in unstructured outdoor and/or indoor environments in which traffic of moving objects, such as cars and people, is characterized not only by motion triggers, but also by speed and direction of motion, size, shape, color of object, time of day, day of week, and time of year.
  • the methods and systems receive as input one or more camera and/or video streams and produce traffic statistics on objects of interest in locations of interest at times of interest. These statistics provide an object-oriented basis on which to characterize viewed scenes.
  • the resultant characterization can have a variety of uses, and in particular, large-scale applications in which many cameras monitor complex, unstructured locations.
  • scene characterization technology can be employed to prioritize video feeds for live review, raise alarms for selected behaviors of interest, and provide a mechanism to index recorded video sequences based on their content.
  • the correlating can include spatially correlating and temporally correlating, and correlating can include providing a model of at least one field of view, and, registering the video data to the model.
  • resuming track can include creating a new track.
  • the stopped object(s) properties can include kinematic properties, 2D appearance, and/or 3D shape, and in some embodiments, the stopped object(s) properties can include arrival time, departure time, size, color, position, velocity, and/or acceleration.
  • the video devices include at least two cameras having different fields of view.
  • the disclosed methods and systems can include providing one or more alerts based on determining the object(s) as a stopped object(s) and/or providing at least one alert based on a lapse of a time since determining the object is a stopped object.
  • the methods and systems can include comparing the object(s) track to a model track, and, providing an alert based on the comparison of the track to the model track.
  • an alert can be provided based on an object entering an area/region, a time at which an object enters an area/region of interest, and/or an amount of time that an object remains in a region (e.g., regardless of whether the object is stopped).
  • the disclosed methods and systems can include, based on determining that the stopped object is occluded, monitoring new tracks of objects emanating from the region occluding the object. Also included is selecting a new track consistent with the track of the occluded object prior to the occlusion, and, associating the track of the occluded object prior to the occlusion with the selected new track.
  • correlating video data can include detecting motion in the video data to identify objects, classifying objects from background, segmenting the background, detecting background regions with changes, and updating the background properties based on determining that the changes are due to at least one of illumination, spurious motion, and imaging artifacts.
  • correlating video data can include detecting moving objects, and, grouping moving objects based on object tracks.
  • Correlating video data can also and/or optionally include splitting groups of moving objects based on object tracks, where the splitting can include determining that at least one first object in a group is stopped, and, determining that at least one second object in the group is moving.
  • the methods and systems can include correlating the track trajectory of the at object(s) from a first video device, correlating the object properties of the object(s) from a second video device, and, determining, based on the correlation of the track trajectory and correlation of the object properties, to merge at least one track from the first video device and at least one track from the second video device.
  • the methods and systems can include determining, based on the correlation of the track trajectory and correlation of the object properties, to not merge at least one track from the first video device and at least one track from the second video device, and, based on such determination, ending a track of an object and/or starting a track of an object.
  • Figure 1 illustrates components of the disclosed methods and systems;
  • Figure 2 illustrates one embodiment of the disclosed methods and systems;
  • Figure 3 illustrates a video frame displayed by a graphical user interface (left) that is registered with a top-down schematic map of a surrounding region (right);
  • Figure 4 discloses a portion of one embodiment of the illustrated methods and systems;
  • Figure 5 illustrates a portable pixel map (PPM) image of an object and a corresponding portable gray map (PGM) image thereof;
  • Figures 6 and 7 illustrate two examples of move-stop-move object tracking;
  • Figure 8 illustrates one scheme for move-stop-move processing;
  • Figure 9 shows a processing scheme for occlusion tracking;
  • Figure 10 illustrates a dynamic background adaptation scheme; and,
  • Figure 11 illustrates a scheme for tracking an object across multiple views.
  • the disclosed methods and systems can detect, track, and classify moving objects and/or "objects of interest” (collectively referred to herein as "objects") in video sequences.
  • objects of interest can include vehicles, people, and animals, with such examples provided for illustration and not limitation.
  • the systems and methods include tracking objects of interest across changing and multiple viewpoints. Tracking objects of interest through pan/tilt/zoom transformations improves camera coverage and supports effective user interaction (for example, to zoom in on a suspicious person). Tracking across multiple camera views decreases the probability of occlusion and increases the range over which we can track a given object. Objects can be tracked within a single fixed video sequence, and the method and systems can also correlate trajectories across multiple variable-view sequences. [0022] The disclosed methods and systems can alert users to, and allow users and others to identify certain objects and events.
  • the methods and systems include a prioritization of multiple video feeds and an object-oriented indexing system to retrieve video sequences of objects of interest based on spatial and temporal properties of the objects.
  • Some processing and/or parameters of the disclosed methods and systems can include activity detection rate, activity characterization (speed, loitering time, etc.) rate, sensitivity to environmental conditions and activity types, tracking and classification through pan/tilt/zoom transformations, site-level reasoning, object tracking through stops, supervised classification learning, and integration of additional classifiers such as gait with existing size/shape/color criteria.
  • the methods and systems include a behavior-based video surveillance system robust to environmental factors that include, for example, lighting, rain, and blowing leaves.
  • FIG. 1 thus shows a block diagram of one embodiment of the disclosed methods and systems. As shown in Figure 1, the methods and systems can include one or more cameras 110 that can be understood to include one or more video devices.
  • the camera(s) 110 can be analog and/or digital devices, and can be positioned at one or more geographic locations and/or fields of view. For example, simultaneous parallel tracking of a single object from multiple cameras can be performed.
  • a quad- multiplexor can be used to concatenate four video streams into one composite stream. This composite stream can be divided and/or split back into four half-resolution streams, each of which can be provided to its own instance of a tracker object.
  • Four separate track databases can then be created and maintained as the stream progresses.
  • separate data streams can be employed directly from their respective sources.
  • a tracker can be instantiated for each feed, and tracking can proceed in parallel on the different streams.
  • the camera(s) can provide data and/or be in communications with one or more processor systems 112 that can include various features for processing the camera data (or data based on the camera data) in accordance with the disclosed methods and systems. It can thus be understood that some systems may not include all of the features of the illustrated system 112, and as provided previously herein, components of the illustrated system 112 can be combined, interchanged, separated, etc., without departing from the scope of the disclosed methods and systems.
  • the processor systems 112 includes a camera calibrator 114 for issues related to relative camera location, normalize illumination conditions, and compute intrinsic and extrinsic camera parameters, for example, and a camera stabilizer 116 that can accept data from the one or more cameras 110 and modify such data to account for camera motion, pan, tilt, etc. It can be understood that the cameras
  • a scheme for camera-to-site model registration processing scheme 118 can include a processing scheme for registering the camera data (e.g., stabilized and calibrated camera data) to a model of the site/location that is associated with a camera 110 and/or a field of view, and thus may include a transformation of camera coordinates to world coordinates.
  • the camera/video data can allow for the detection, classification, and tracking and/or processing of objects.
  • Such tracking and/or processing of objects can be correlated with time and location and recorded in one or more memories (e.g., database) that can further record physical features of the objects, including, for example, size, color, and shape of objects over time and location, which may also be recorded in a database 132. Accordingly, objects can be tracked and/or characterized based on object kinematics, 2D appearance, and/or 3D shape to allow for cross-track association of object data. Such data can be further correlated with other events that are not associated with the object(s) being tracked.
  • the Figure 1 embodiment thus includes a motion detection processing scheme
  • the motion detector 120 may detect objects of interest in cluttered and/or changing environments, such as people, vehicles, etc., while an object tracker 122 can maintain localization of moving objects within a camera's field of view to allow for continuous track through, for example, short occlusions and coverage lapses/gaps.
  • An object tracker 126 can also be used to characterize and/or otherwise associate tracked objects with physical features of the objects. Such object tracking can allow for object classification 126 amongst a class of objects. Such classification can provide robustness amongst class appearance variabilities. [0030] It can thus be understood that data from multiple cameras associated with a single site can be combined and/or fused by a camera data fusion processing scheme 124.
  • camera data fusion 124 can include fusion of camera data from multiple sites being provided to a fusion processing scheme 124 to allow for tracking between cameras/locations/fields of view and/or changing illumination conditions.
  • a spatial-temporal object movement characterization scheme 128 can allow for a development of motion pattern models of parameterized object trajectories to allow for an expression of a broad range of object trajectories.
  • Such trajectories can be utilized by the Figure 1 anomaly detector 130 which can include thresholds and/or other schemes (static and/or adaptive schemes) for determining whether an object's behavior, based on such tracking, may be considered an anomaly that should be associated with an alert 134.
  • Deviations from models provided by the disclosed object movement characterization scheme 128 can thus be detected by an anomaly detector 130, where such deviations can be user/system administrator defined and/or characterized based on the embodiment.
  • the disclosed methods and systems can allow for a tagging of objects 136 as such objects are tracked, such that an activity-indexed database 132 can be arranged for data retrieval by object and/or tag to allow retrospective inspection of historical object tracks.
  • the tagging of objects e.g., selection by a user/administrator/another
  • Figure 2 presents another embodiment of a system according to Figure 1 , which includes, for example, a camera processing module 210 associated with each camera 110, an activity extraction module 212 to extract data from an object's track, an activity database 214 that provides for data storage/retrieval/archiving, and an activity assessment module 216 that allows for an assessment of the object activity based on the object'(s) track.
  • a camera processing module 210 associated with each camera 110
  • an activity extraction module 212 to extract data from an object's track
  • an activity database 214 that provides for data storage/retrieval/archiving
  • an activity assessment module 216 that allows for an assessment of the object activity based on the object'(s) track.
  • the Figure 2 embodiment is also merely for illustration and the organization of modules is merely for convenience.
  • multiple cameras 110 can be positioned at geographically distinct locations and/or fields of view, where in the Figure 2 embodiment, each camera is associated with a camera stabilization 114 and camera calibration 116 processing scheme as provided previously herein.
  • the stabilized and calibrated data can be provided to a camera-to-site model registration processing scheme 118 before being provided to a motion detection scheme 120 to identify objects for tracking 122 and classification 126.
  • the tracked objects and classifications thereof from different cameras 110 can be provided to a single multi-fusion camera processing scheme 124 that can fuse data from multiple cameras at a single site andor different sites. The fused data can thus allow for object movement characterization of objects 128 as provided previously herein.
  • cross-camera tracking can include projection of each camera's tracks into a common reference frame, or site map, as shown in Figure 3, and correlating the tracks using the reference frame coordinates.
  • a mapping includes pre-calibration of each video stream with the map.
  • Several coordinate transformations can be used, and in one embodiment, a projective plane-to-plane model based on image homographies can be employed.
  • objects may be tracked according to their lowest point (e.g., bottom of a bounding box) rather than their center of mass. This is a more natural representation for object position with respect to the ground, since the scene is essentially projected onto the ground plane when transformed to map coordinates.
  • object tracks from the trackers can be transformed to map coordinates, and tracks can be associated across camera views based on kinematics.
  • the Figure 2 event database 218 can store events that are detected and/or recorded by the disclosed methods and systems, and such events can be stored/retrieved using the illustrated event storage and retrieval scheme 132 that can associate events and/or event data with activity descriptors.
  • the event database 218 can be accessed by a variety of processor controlled devices 220A, 220B, 220C, for example, that can be equipped with a tag-and-track user interface 136 that allows a user and/or another associated with the device 220 A-C to identify and/or select objects of interest for tracking.
  • the illustrated database 218 can allow for retrospective inspection of historical tracks, which may be accessed by and/or displayed on the processor-controlled devices 220A-C.
  • the processor devices 220A-C may communicate using wired and/or wireless networks. [0039] Communications can also be maintained between the processor devices 220A-C and the anomaly detection scheme 130 and/or the alert generation scheme 134. It can thus be understood that users of the processor devices 220A-C may configure the anomaly detection scheme 130 and/or the alert generation scheme 134 to allow, for example, conditions upon which alerts are to be generated, locations to which alerts should be directed transmitted, etc.
  • the processor devices 220A-C can thus be provided and/or otherwise configured with customized software that can display a site map, read target tracks as they are generated, and superimpose these tracks on the site map.
  • the customized software can also request current video frames, and generate audible and visual alerts while displaying image chips of objects as the objects cross virtual tripwires, for example.
  • Figure 4 depicts an example use of the disclosed methods and systems as provided herein as applied to detection of various behaviors within an office setting and at a mall entrance. In the top half of Figure 4, one embodiment of the system monitors people in a hallway and collects information on their dwell time. Alerts can be generated to notify the appropriate security personnel of suspicious behavior (e.g., loitering).
  • FIG 4 Also shown in Figure 4 is the use of a virtual "tripwire" to detect objects that cross a pre-defined threshold.
  • the system detects crossing events and motion direction to distinguish between a person/object entering and leaving an area of interest.
  • Statistics gathered as individuals cross virtual tripwires can reveal characteristics, such as, for example, the volume of traffic leaving the mall increases dramatically near, for example, a time associated with mall closing, can suggest that additional security personnel may be needed during that time.
  • Such an example includes tracking of moving objects, spatial and temporal activity characterization (e.g., object counts, speeds, trajectories), parameterization of activity patterns by time of day, day of week, time of year, and review of events of interest, as provided herein relative to Figures 1 and 2.
  • the methods and systems can employ virtual tripwires to detect pedestrian and vehicle traffic in the wrong direction(s). For example, in an aircraft/airport exemplary embodiment (an exemplary embodiment used herein for illustration and not limitation) while attendants and security personnel attempt to detect illegal movements through checkpoints and gates, automatic video-based detection and snapshots can complement such efforts. Virtual tripwires that incorporate directionality to provide an alert(s) when crossed in a specified direction can thus be employed.
  • Terrorist threats have expanded still further from the interior concourse to the exterior vehicle traffic circles.
  • the disclosed methods and systems can thus provide one or more alerts when vehicles exceeding a specified size drive through drop-off/pickup areas.
  • the disclosed methods and systems can learn "normal” vehicle size through long- term observation and flagging vehicles exceeding this "normal” size.
  • the methods and systems can be programmed and/or otherwise configured to identify and/or provide an alert regarding vehicles exceeding an explicit user-defined size.
  • the methods and systems include feature-based correlation and prediction techniques to match vehicles observed in upstream and downstream cameras, using statistical models to compare various object characteristics such as arrival time, departure time, size, shape, position, velocity, acceleration, and color.
  • Certain feature types can be output and/or provided for inspection and processing, such as object size and extent information (e.g., bounding box regions within the image), and object mask images, which are binary images in which zeros indicates background pixels and ones indicate foreground pixels.
  • Mask images have a one-to-one correspondence with "chips" that capture the pixel colors at a given time instant, for example stored in portable pixel map (PPM) format, as shown in Figure 5.
  • PPM portable pixel map
  • the disclosed methods and systems acknowledge that a robustness of adaptive background segmentation can be at the cost of object persistence in that objects that stop moving are eventually "absorbed" into the background and lost to a tracker. When these objects begin moving again, the system cannot re-associate to a previously seen track.
  • the disclosed methods and system address this "move-stop-move" problem by determining when a given object has stopped moving. This determination can be useful, for example, in abandoned luggage scenarios described herein. This determination can be accomplished by examining a pre-specified time window over which to monitor an object's motion history. If the object has not moved significantly during this time window, the object can be tagged or otherwise identified as "stopped” or still and saved as an image chip for later use. This saved image chip can be used to determine that a stopped object is still present in the video, and to associate the object with a new track(s) when it begins moving again.
  • Figures 6 and 7 illustrate a move-stop-move problem analysis, where in Figure
  • FIG. 6 illustrates a scenario to illustrate detection of abandoned luggage where a tracked individual abandons the luggage.
  • the tracked object of the person can be identified and associated with a shape, as can the luggage, where such objects can be tracked individually.
  • a retrospective of images prior to the determination can indicate that the luggage is a still object.
  • Properties of the still object/luggage can be monitored/updated with subsequent views of the area that contains the still object/luggage, and track can begin and/or resume when such properties change.
  • the Figure 7 example also provides an example of group tracking that can be employed in the disclosed methods and systems.
  • group tracking two or more objects (e.g., person and luggage, multiple people, etc.) can be tracked as a group, thereby allowing for tracking in high-traffic densities.
  • group tracking can include group splitting, and/or group merging.
  • Figure 8 illustrates a scheme for the aforementioned move-stop-move tracking in which an object can be tracked although the object stops moving, or becomes a "still" object.
  • video data can be provided from one or more video/camera sources and registered to a site model 810 such that motion can be detected and objects tracked 812 and correlated from multiple video sources 814.
  • object tracking 812 can continue; however, if it is determined that the object is still (e.g., non-moving) 816, then a second determination can be performed regarding the object's visibility 820. If the object is no longer visible 820, the track can be ended and/or suspended 822 until the object re-appears.
  • object properties e.g., kinematics, 2D appearance, and/or 3D shape
  • object properties e.g., kinematics, 2D appearance, and/or 3D shape
  • object properties can be stored/recorded 824 and monitored 826 with subsequent data 810 until it is determined 828 that the object is again moving.
  • the disclosed methods and systems can allow for a configuration in which an alert is provided to one or more locations (e.g., central location, individual locations, etc.) upon an object being tagged/characterized as "stopped", non-moving, still, etc., and/or being in such state for more than a specified time.
  • Other examples of alert conditions e.g., deviation from a model track are also possible.
  • Figure 9, like Figure 8, provides for an object that becomes occluded.
  • video data can be provided from one or more video/camera sources and registered to a site model 910 such that motion can be detected and objects tracked 912 and correlated across multiple video sources 914.
  • Based on the object track it can be determined whether an object is moving 916, and if the object is moving, object properties can be updated 918 and object tracking 912 can continue; however, if it is determined that the object is still (e.g., non-moving) 916, then a second determination can be provided regarding whether the object is occluded 920.
  • Object occlusion can be based on, for example, the site model and the track database by examining historical data prior to the object's still motion and/or occlusion.
  • Properties of the occl ⁇ ded object can be recorded/stored 922 and the occluded region can be monitored for new tracks originating from the occluded region and based on subsequent video data 924, until a new track appears that is consistent with the occluded object's track 926.
  • the track prior to the occlusion can be associated with the track subsequent to the occlusion 928, and a further determination can be made regarding the movement of the object 916.
  • Figure 9 thus indicates the continued process of tracking the object through the occlusions.
  • Figure 10 provides one example of a dynamic background adaptation scheme in which the video data is provided for the motion detection and object tracking 1010 as previously provided herein, where background segmentation 1012 can be performed to characterize background changes 1014. It can be understood that one or more of several segmentation schemes can be used based on the embodiment. If regions of change in the background (e.g., non-object areas) are determined, detected, and/or found 1016, the Figure 10 example processing scheme can determine if (e.g., classify) such background changes are illumination effects 1018, spurious motion effects 1020, and/or imaging artifacts 1022
  • the background properties can be updated 1024.
  • Figure 11 demonstrates one scheme for tracking an object from different video sources having different fields of view.
  • registered fracked objects from two video data sources 1105A, 1105B can be provided to one or more correlations schemes 1110, 1120 that correlate the object track trajectories and correlate the object properties from the two video data sources. Based on such correlations, if the tracks are the same 1130, the tracks are merged 1140, and otherwise, the tracks are viewed as distinct such that a particular track may end (e.g., an object track from a first video data source), while another track (e.g., an object track from a second video data source) may begin 1150.
  • a particular track may end (e.g., an object track from a first video data source), while another track (e.g., an object track from a second video data source) may begin 1150.
  • the methods and systems can be implemented in one or more computer programs, where a computer program can be understood to include one or more processor executable instructions.
  • the computer program(s) can execute on one or more programmable processors, and can be stored on one or more storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), one or more input devices, and/or one or more output devices.
  • the processor thus can access one or more input devices to obtain input data, and can access one or more output devices to communicate output data.
  • the input and/or output devices can include one or more of the following: Random Access Memory (RAM),
  • the computer program(s) can be implemented using one or more high level procedural or object-oriented programming languages to communicate with a computer system; however, the program(s) can be implemented in assembly or machine language, if desired.
  • the language can be compiled or interpreted.
  • the processor(s) can thus be embedded in one or more devices that can be operated independently or together in a networked environment, where the network can include, for example, a Local Area Network (LAN), wide area network (WAN), and/or can include an intranet and/or the internet and/or another network.
  • the network(s) can be wired or wireless or a combination thereof and can use one or more communications protocols to facilitate communications between the different processors.
  • the processors can be configured for distributed processing and can utilize, in some embodiments, a client-server model as needed. Accordingly, the methods and systems can utilize multiple processors and/or processor devices, and the processor instructions can be divided amongst such single or multiple processor/devices.
  • the device(s) or computer systems that integrate with the processor(s) can include, for example, a personal computer(s), workstation (e.g., Sun, HP), personal digital assistant (PDA), handheld device such as cellular telephone, laptop, handheld, or another device capable of being integrated with a processor(s) that can operate as provided herein. Accordingly, the devices provided herein are not exhaustive and are provided for illustration and not limitation.
  • workstation e.g., Sun, HP
  • PDA personal digital assistant
  • handheld device such as cellular telephone, laptop, handheld, or another device capable of being integrated with a processor(s) that can operate as provided herein. Accordingly, the devices provided herein are not exhaustive and are provided for illustration and not limitation.
  • references to "a microprocessor” and “a processor”, or “the microprocessor” and “the processor,” can be understood to include one or more microprocessors that can communicate in a stand-alone and/or a distributed environment(s), and can thus can be configured to communicate via wired or wireless communications with other processors, where such one or more processor can be configured to operate on one or more processor- controlled devices that can be similar or different devices.
  • Use of such "microprocessor” or “processor” terminology can thus also be understood to include a central processing unit, an arithmetic logic unit, an application-specific integrated circuit (IC), and/or a task engine, with such examples provided for illustration and not limitation.
  • references to memory can include one or more processor-readable and accessible memory elements and/or components that can be internal to the processor-controlled device, external to the processor-controlled device, and/or can be accessed via a wired or wireless network using a variety of communications protocols, and unless otherwise specified, can be arranged to include a combination of external and internal memory devices, where such memory can be contiguous and/or partitioned based on the application.
  • references to a database can be understood to include one or more memory associations, where such references can include commercially available database products (e.g., SQL, Informix, Oracle) and also proprietary databases, and may also include other structures for associating memory such as links, queues, graphs, trees, with such structures provided for illustration and not limitation.
  • references to a network can include one or more intranets and/or the internet.
  • References herein to microprocessor instructions or microprocessor-executable instructions, in accordance with the above, can be understood to include programmable hardware.

Abstract

A method, system and computer program for tracking an object including: identifying objects by correlating video data from at least one video device (812); determining that an object is stopped based on past motion data (816); determining if a stopped object is occluded (820); monitoring the properties of a stopped object (824); determining if a previously stopped object is moving (828); and resuming tracking if the previously stopped object is moving.

Description

TRACKING SYSTEMS AND METHODS CLAIM OF PRIORITY
[0001] This application claims priority to U.S.S.N. 60/504,583, filed on September 19, 2003, the contents of which are herein incorporated by reference in their entirety. BACKGROUND (1) Field
[0002] The disclosed methods and systems relate generally to tracking methods and systems, and more particularly to tracking in unstructured environments. (2) Description of Relevant Art
[0003] Wide availability and low cost allow incorporation of high-quality cameras and fast processors into high-coverage commercial video surveillance and monitoring (VSAM) systems. Such systems typically produce enormous quantities of data too overwhelming for human operators to process. Video footage is often analyzed superficially, recorded without review, and/or simply ignored; however, high-coverage, continuous imaging provides a rich information source which, if used intelligently, can allow automatic characterization of normal site activities, detection of anomalous behaviors, and tracking of objects of interest. [0004] Many video surveillance technology systems rely on face recognition or other biometrics, for example to screen airline passengers as they pass through heavily-trafficked areas. For a suspect to be identified, he/she must already be flagged as a potential risk and have a current feature set on file in the system's database. The effectiveness of such systems in correctly recognizing disguised or non-cooperative individuals is unclear at best. It is therefore desirable to augment identification systems with technologies that do not require a priori knowledge of specific individuals.
[0005] Robustness is thus an issue in such systems because of associated uncontrolled settings where viewing conditions and scene content may vary significantly. For example, variable viewing conditions under which the systems can operate include: (i) illumination (e.g., day/night, sunny/cloudy, sun angle, specularities); (ii) weather (e.g., dry/wet, seasonal changes, variable backgrounds (snow, leaves)); (iii) scene content variables including: (a) object density, speed, count; and, (b) size/shape/color within and across object classes; and, (iv) nuisance background clutter (e.g., shadows, swaying trees). SUMMARY [0006] The disclosed methods and systems include monitoring applications in unstructured outdoor and/or indoor environments in which traffic of moving objects, such as cars and people, is characterized not only by motion triggers, but also by speed and direction of motion, size, shape, color of object, time of day, day of week, and time of year. [0007] In one embodiment, the methods and systems receive as input one or more camera and/or video streams and produce traffic statistics on objects of interest in locations of interest at times of interest. These statistics provide an object-oriented basis on which to characterize viewed scenes. The resultant characterization can have a variety of uses, and in particular, large-scale applications in which many cameras monitor complex, unstructured locations.
[0008] In one embodiment, scene characterization technology can be employed to prioritize video feeds for live review, raise alarms for selected behaviors of interest, and provide a mechanism to index recorded video sequences based on their content.
[0009] Disclosed are methods, systems, and computer/processor program products for tracking an object(s), including identifying the object(s) by correlating video data from at least one video device, based on motion data of the object(s) for a previous time, determining that the object(s) movement is stopped, based on determining that the stopped object(s) is not occluded, monitoring the stopped object(s) properties, determining from the monitoring that the stopped object(s) is moving, and, resuming track of the object(s). The correlating can include spatially correlating and temporally correlating, and correlating can include providing a model of at least one field of view, and, registering the video data to the model. [0010] For the disclosed methods and systems, resuming track can include creating a new track. Further, the stopped object(s) properties can include kinematic properties, 2D appearance, and/or 3D shape, and in some embodiments, the stopped object(s) properties can include arrival time, departure time, size, color, position, velocity, and/or acceleration. In the disclosed methods and systems, the video devices include at least two cameras having different fields of view.
[0011] In some embodiments, the disclosed methods and systems can include providing one or more alerts based on determining the object(s) as a stopped object(s) and/or providing at least one alert based on a lapse of a time since determining the object is a stopped object. In an embodiment, the methods and systems can include comparing the object(s) track to a model track, and, providing an alert based on the comparison of the track to the model track. In some embodiments, an alert can be provided based on an object entering an area/region, a time at which an object enters an area/region of interest, and/or an amount of time that an object remains in a region (e.g., regardless of whether the object is stopped).
[0012] The disclosed methods and systems can include, based on determining that the stopped object is occluded, monitoring new tracks of objects emanating from the region occluding the object. Also included is selecting a new track consistent with the track of the occluded object prior to the occlusion, and, associating the track of the occluded object prior to the occlusion with the selected new track.
[0013] In an example embodiment, correlating video data can include detecting motion in the video data to identify objects, classifying objects from background, segmenting the background, detecting background regions with changes, and updating the background properties based on determining that the changes are due to at least one of illumination, spurious motion, and imaging artifacts. In some embodiments, correlating video data can include detecting moving objects, and, grouping moving objects based on object tracks. Correlating video data can also and/or optionally include splitting groups of moving objects based on object tracks, where the splitting can include determining that at least one first object in a group is stopped, and, determining that at least one second object in the group is moving.
[0014] In some embodiments, the methods and systems can include correlating the track trajectory of the at object(s) from a first video device, correlating the object properties of the object(s) from a second video device, and, determining, based on the correlation of the track trajectory and correlation of the object properties, to merge at least one track from the first video device and at least one track from the second video device. Similarly, the methods and systems can include determining, based on the correlation of the track trajectory and correlation of the object properties, to not merge at least one track from the first video device and at least one track from the second video device, and, based on such determination, ending a track of an object and/or starting a track of an object.
[0015] Also disclosed are systems and processor program products having processor- readable instructions for performing the disclosed methods. [0016] Other objects and advantages will become apparent hereinafter in view of the specification and drawings. BRIEF DESCRIPTION OF DRAWINGS [0017] Figure 1 illustrates components of the disclosed methods and systems; Figure 2 illustrates one embodiment of the disclosed methods and systems; Figure 3 illustrates a video frame displayed by a graphical user interface (left) that is registered with a top-down schematic map of a surrounding region (right); Figure 4 discloses a portion of one embodiment of the illustrated methods and systems; Figure 5 illustrates a portable pixel map (PPM) image of an object and a corresponding portable gray map (PGM) image thereof; Figures 6 and 7 illustrate two examples of move-stop-move object tracking; Figure 8 illustrates one scheme for move-stop-move processing; Figure 9 shows a processing scheme for occlusion tracking; Figure 10 illustrates a dynamic background adaptation scheme; and, Figure 11 illustrates a scheme for tracking an object across multiple views. DESCRIPTION [0018] To provide an overall understanding, certain illustrative embodiments will now be described; however, it will be understood by one of ordinary skill in the art that the systems and methods described herein can be adapted and modified to provide systems and methods for other suitable applications and that other additions and modifications can be made without departing from the scope of the systems and methods described herein. [0019] Unless otherwise specified, the illustrated embodiments can be understood as providing exemplary features of varying detail of certain embodiments, and therefore, unless otherwise specified, features, components, modules, and/or aspects of the illustrations can be otherwise combined, separated, interchanged, and/or rearranged without departing from the disclosed systems or methods. Additionally, the shapes and sizes of components are also exemplary and unless otherwise specified, can be altered without affecting the scope of the disclosed and exemplary systems or methods of the present disclosure.
[0020] The disclosed methods and systems can detect, track, and classify moving objects and/or "objects of interest" (collectively referred to herein as "objects") in video sequences. Objects of interest can include vehicles, people, and animals, with such examples provided for illustration and not limitation.
[0021] The systems and methods include tracking objects of interest across changing and multiple viewpoints. Tracking objects of interest through pan/tilt/zoom transformations improves camera coverage and supports effective user interaction (for example, to zoom in on a suspicious person). Tracking across multiple camera views decreases the probability of occlusion and increases the range over which we can track a given object. Objects can be tracked within a single fixed video sequence, and the method and systems can also correlate trajectories across multiple variable-view sequences. [0022] The disclosed methods and systems can alert users to, and allow users and others to identify certain objects and events. Given the volume of video imagery collected in monitoring applications, most processing must be performed automatically and in real time, so that users need only review a small set of machine-flagged events and can cue to footage or objects of interest. An indexed database of activity can be maintained alongside the raw video data to facilitate such interaction. Accordingly, the methods and systems include a prioritization of multiple video feeds and an object-oriented indexing system to retrieve video sequences of objects of interest based on spatial and temporal properties of the objects. [0023] Some processing and/or parameters of the disclosed methods and systems can include activity detection rate, activity characterization (speed, loitering time, etc.) rate, sensitivity to environmental conditions and activity types, tracking and classification through pan/tilt/zoom transformations, site-level reasoning, object tracking through stops, supervised classification learning, and integration of additional classifiers such as gait with existing size/shape/color criteria. [0024] In one embodiment, the methods and systems include a behavior-based video surveillance system robust to environmental factors that include, for example, lighting, rain, and blowing leaves. By extracting spatio-temporal features such as color, size, shape, position, velocity, and growth rate, and integrating behavioral modeling therewith, statistics and alerts can be generated based on a detection of unusual activities (as determined by the embodiment). In some embodiments, an alert can be provided based on an object entering an area/region, a time at which an object enters an area/region, and/or an amount of time that an object remains in a region (e.g., regardless of whether the object is stopped). [0025] Figure 1 thus shows a block diagram of one embodiment of the disclosed methods and systems. As shown in Figure 1, the methods and systems can include one or more cameras 110 that can be understood to include one or more video devices. The camera(s) 110 can be analog and/or digital devices, and can be positioned at one or more geographic locations and/or fields of view. For example, simultaneous parallel tracking of a single object from multiple cameras can be performed. In one embodiment, a quad- multiplexor can be used to concatenate four video streams into one composite stream. This composite stream can be divided and/or split back into four half-resolution streams, each of which can be provided to its own instance of a tracker object. Four separate track databases can then be created and maintained as the stream progresses. Additionally and optionally, in an embodiment, separate data streams can be employed directly from their respective sources. A tracker can be instantiated for each feed, and tracking can proceed in parallel on the different streams. [0026] As shown in Figure 1, the camera(s) can provide data and/or be in communications with one or more processor systems 112 that can include various features for processing the camera data (or data based on the camera data) in accordance with the disclosed methods and systems. It can thus be understood that some systems may not include all of the features of the illustrated system 112, and as provided previously herein, components of the illustrated system 112 can be combined, interchanged, separated, etc., without departing from the scope of the disclosed methods and systems.
[0027] In the Figure 1 embodiment, the processor systems 112 includes a camera calibrator 114 for issues related to relative camera location, normalize illumination conditions, and compute intrinsic and extrinsic camera parameters, for example, and a camera stabilizer 116 that can accept data from the one or more cameras 110 and modify such data to account for camera motion, pan, tilt, etc. It can be understood that the cameras
110 can be fixed, moving, and/or pole-mounted, for example. Such calibration and stabilization schemes can be based on the embodiment, and the disclosed methods and systems are not limited to a particular scheme. Also shown in Figure 1 is a scheme for camera-to-site model registration processing scheme 118 that can include a processing scheme for registering the camera data (e.g., stabilized and calibrated camera data) to a model of the site/location that is associated with a camera 110 and/or a field of view, and thus may include a transformation of camera coordinates to world coordinates. [0028] As provided herein, and as shown in Figure 1, the camera/video data can allow for the detection, classification, and tracking and/or processing of objects. Such tracking and/or processing of objects can be correlated with time and location and recorded in one or more memories (e.g., database) that can further record physical features of the objects, including, for example, size, color, and shape of objects over time and location, which may also be recorded in a database 132. Accordingly, objects can be tracked and/or characterized based on object kinematics, 2D appearance, and/or 3D shape to allow for cross-track association of object data. Such data can be further correlated with other events that are not associated with the object(s) being tracked. [0029] The Figure 1 embodiment thus includes a motion detection processing scheme
120 and a moving object tracker 122, both of which can be of various forms based on the embodiment. For example, the motion detector 120 may detect objects of interest in cluttered and/or changing environments, such as people, vehicles, etc., while an object tracker 122 can maintain localization of moving objects within a camera's field of view to allow for continuous track through, for example, short occlusions and coverage lapses/gaps.
An object tracker 126 can also be used to characterize and/or otherwise associate tracked objects with physical features of the objects. Such object tracking can allow for object classification 126 amongst a class of objects. Such classification can provide robustness amongst class appearance variabilities. [0030] It can thus be understood that data from multiple cameras associated with a single site can be combined and/or fused by a camera data fusion processing scheme 124. In some of the disclosed embodiments, camera data fusion 124 can include fusion of camera data from multiple sites being provided to a fusion processing scheme 124 to allow for tracking between cameras/locations/fields of view and/or changing illumination conditions. Such object tracking over time and/or location can thus allow for a spatial- temporal object movement characterization 128 that can determine, for example, whether an object has moved between two locations in an exceptionally fast and/or an exceptionally slow manner, with such examples provided for illustration and not limitation. Accordingly, one embodiment of a spatial-temporal object movement characterization scheme 128 can allow for a development of motion pattern models of parameterized object trajectories to allow for an expression of a broad range of object trajectories. Such trajectories can be utilized by the Figure 1 anomaly detector 130 which can include thresholds and/or other schemes (static and/or adaptive schemes) for determining whether an object's behavior, based on such tracking, may be considered an anomaly that should be associated with an alert 134. Deviations from models provided by the disclosed object movement characterization scheme 128 can thus be detected by an anomaly detector 130, where such deviations can be user/system administrator defined and/or characterized based on the embodiment.
[0031] As indicated in Figure 1, the disclosed methods and systems can allow for a tagging of objects 136 as such objects are tracked, such that an activity-indexed database 132 can be arranged for data retrieval by object and/or tag to allow retrospective inspection of historical object tracks. The tagging of objects (e.g., selection by a user/administrator/another) can further allow for processing resources to be dedicated to tagged objects rather than non-tagged objects.
[0032] Queries to an activity-indexed database 132 can thus assist in the determination of anomaly behavior. The event data can further be stored using activity descriptors to maintain high transaction volume based on spatio-temporal parameters. [0033] Figure 2 presents another embodiment of a system according to Figure 1 , which includes, for example, a camera processing module 210 associated with each camera 110, an activity extraction module 212 to extract data from an object's track, an activity database 214 that provides for data storage/retrieval/archiving, and an activity assessment module 216 that allows for an assessment of the object activity based on the object'(s) track. As provided relative to Figure 1, the Figure 2 embodiment is also merely for illustration and the organization of modules is merely for convenience.
[0034] As shown in the Figure 2 embodiment, multiple cameras 110 can be positioned at geographically distinct locations and/or fields of view, where in the Figure 2 embodiment, each camera is associated with a camera stabilization 114 and camera calibration 116 processing scheme as provided previously herein. As Figure 2 indicates, the stabilized and calibrated data can be provided to a camera-to-site model registration processing scheme 118 before being provided to a motion detection scheme 120 to identify objects for tracking 122 and classification 126. The tracked objects and classifications thereof from different cameras 110 can be provided to a single multi-fusion camera processing scheme 124 that can fuse data from multiple cameras at a single site andor different sites. The fused data can thus allow for object movement characterization of objects 128 as provided previously herein. [0035] Accordingly, in one embodiment, cross-camera tracking can include projection of each camera's tracks into a common reference frame, or site map, as shown in Figure 3, and correlating the tracks using the reference frame coordinates. As indicated herein, such a mapping includes pre-calibration of each video stream with the map. Several coordinate transformations can be used, and in one embodiment, a projective plane-to-plane model based on image homographies can be employed. A 3x3 homography matrix, H, can transform an image point in homogeneous coordinates p to a map point m according to: Hp m = . Hp - z
[0036] The eight parameters of the homography, hy, can be estimated by computing the least-squares solution to constraints of the form: hnx + hl2y + h13 - h31xu — h32yu = u h2lx + h22y + h23 - h3lxv - h32yv = v where p = (x,y) and m = (u,v) are known from manually-specified point pairs between the video imagery and the map. At least four such pairs are needed for a unique solution. [0037] To support this projection of inherently 3D objects onto 2D surfaces, objects may be tracked according to their lowest point (e.g., bottom of a bounding box) rather than their center of mass. This is a more natural representation for object position with respect to the ground, since the scene is essentially projected onto the ground plane when transformed to map coordinates. In an embodiment, object tracks from the trackers can be transformed to map coordinates, and tracks can be associated across camera views based on kinematics.
[0038] With further reference to Figure 2, the Figure 2 event database 218 can store events that are detected and/or recorded by the disclosed methods and systems, and such events can be stored/retrieved using the illustrated event storage and retrieval scheme 132 that can associate events and/or event data with activity descriptors. The event database 218 can be accessed by a variety of processor controlled devices 220A, 220B, 220C, for example, that can be equipped with a tag-and-track user interface 136 that allows a user and/or another associated with the device 220 A-C to identify and/or select objects of interest for tracking. As provided previously herein, the illustrated database 218 can allow for retrospective inspection of historical tracks, which may be accessed by and/or displayed on the processor-controlled devices 220A-C. As indicated in Figure 2, the processor devices 220A-C may communicate using wired and/or wireless networks. [0039] Communications can also be maintained between the processor devices 220A-C and the anomaly detection scheme 130 and/or the alert generation scheme 134. It can thus be understood that users of the processor devices 220A-C may configure the anomaly detection scheme 130 and/or the alert generation scheme 134 to allow, for example, conditions upon which alerts are to be generated, locations to which alerts should be directed transmitted, etc.
[0040] The processor devices 220A-C can thus be provided and/or otherwise configured with customized software that can display a site map, read target tracks as they are generated, and superimpose these tracks on the site map. The customized software can also request current video frames, and generate audible and visual alerts while displaying image chips of objects as the objects cross virtual tripwires, for example. [0041] Figure 4 depicts an example use of the disclosed methods and systems as provided herein as applied to detection of various behaviors within an office setting and at a mall entrance. In the top half of Figure 4, one embodiment of the system monitors people in a hallway and collects information on their dwell time. Alerts can be generated to notify the appropriate security personnel of suspicious behavior (e.g., loitering). Also shown in Figure 4 is the use of a virtual "tripwire" to detect objects that cross a pre-defined threshold. The system detects crossing events and motion direction to distinguish between a person/object entering and leaving an area of interest. Statistics gathered as individuals cross virtual tripwires can reveal characteristics, such as, for example, the volume of traffic leaving the mall increases dramatically near, for example, a time associated with mall closing, can suggest that additional security personnel may be needed during that time. Such an example includes tracking of moving objects, spatial and temporal activity characterization (e.g., object counts, speeds, trajectories), parameterization of activity patterns by time of day, day of week, time of year, and review of events of interest, as provided herein relative to Figures 1 and 2.
[0042] As further described relative to Figures 1 and 2, for a system and method such as that of Figure 3, different cameras can be mounted at different locations, and thus the features that the different cameras observe can thus differ. Under ideal conditions, differences between camera observations are small, so that each camera can correctly and consistently identify a given object; however, effects such as lighting changes and perspective projection can hinder multiple-view fusion, such that the aforementioned camera models and computer vision techniques can be allowed to address this problem. [0043] Further, as objects pass behind one another, the objects can be partially or fully hidden from view. Object tracks are commonly lost and must be reacquired when the object reappears. Partial occlusion may also undermine object identification, for example, when an individual on an escalator is visible only from the waist up. Such difficulties can be ameliorated by using multi-hypothesis tracking combined with kinematics modeling and classification. The use of overhead cameras can also assist in minimizing occlusion effects. [0044] The methods and systems can employ virtual tripwires to detect pedestrian and vehicle traffic in the wrong direction(s). For example, in an aircraft/airport exemplary embodiment (an exemplary embodiment used herein for illustration and not limitation) while attendants and security personnel attempt to detect illegal movements through checkpoints and gates, automatic video-based detection and snapshots can complement such efforts. Virtual tripwires that incorporate directionality to provide an alert(s) when crossed in a specified direction can thus be employed. [0045] Further, and continuing with an airport exemplary embodiment, with an increased threat of explosive devices that has expanded from aircraft to the concourse, heightened security measures dictate immediate confiscation and in some instances, destruction of unattended baggage. Such items are generally located visually by patrolling security personnel or reported by travelers, but may remain unnoticed for unacceptably long periods. The disclosed methods and systems thus provide airport security with automatic alerts when an individual places an item at a location and walks more than a specified distance away; and/or, when an item is observed unattended for more than a specified period of time.
[0046] Terrorist threats have expanded still further from the interior concourse to the exterior vehicle traffic circles. The disclosed methods and systems can thus provide one or more alerts when vehicles exceeding a specified size drive through drop-off/pickup areas.
For example, trucks and cargo vans are rarely observed and may constitute suspicious activity. The disclosed methods and systems can learn "normal" vehicle size through long- term observation and flagging vehicles exceeding this "normal" size. In some embodiments, the methods and systems can be programmed and/or otherwise configured to identify and/or provide an alert regarding vehicles exceeding an explicit user-defined size.
[0047] Since no single fixed-view camera can view entire large sites such as airports, individuals and vehicles can be tracked over long temporal extents by camera-to-camera handoff using the multiple camera scenarios illustrated herein. Such a capability, optionally together with tag-and-track capability can allow an operator to graphically indicate an object of interest, and track its movement across coverage gaps and occlusions, also obtaining its previous motion history.
[0048] Further, the gathering of statistics such as average queue lengths, traffic flow, and wait times in various locales can allow, for instance, re-allocation of staff at different times of day, or re-routing of traffic to address increased congestion. [0049] The methods and systems include feature-based correlation and prediction techniques to match vehicles observed in upstream and downstream cameras, using statistical models to compare various object characteristics such as arrival time, departure time, size, shape, position, velocity, acceleration, and color. Certain feature types can be output and/or provided for inspection and processing, such as object size and extent information (e.g., bounding box regions within the image), and object mask images, which are binary images in which zeros indicates background pixels and ones indicate foreground pixels. Mask images have a one-to-one correspondence with "chips" that capture the pixel colors at a given time instant, for example stored in portable pixel map (PPM) format, as shown in Figure 5.
[0050] The disclosed methods and systems acknowledge that a robustness of adaptive background segmentation can be at the cost of object persistence in that objects that stop moving are eventually "absorbed" into the background and lost to a tracker. When these objects begin moving again, the system cannot re-associate to a previously seen track.
Accordingly, the disclosed methods and system address this "move-stop-move" problem by determining when a given object has stopped moving. This determination can be useful, for example, in abandoned luggage scenarios described herein. This determination can be accomplished by examining a pre-specified time window over which to monitor an object's motion history. If the object has not moved significantly during this time window, the object can be tagged or otherwise identified as "stopped" or still and saved as an image chip for later use. This saved image chip can be used to determine that a stopped object is still present in the video, and to associate the object with a new track(s) when it begins moving again. [0051] Figures 6 and 7 illustrate a move-stop-move problem analysis, where in Figure
6, a segment of video footage was digitized in which a tracked vehicle stops for a length of time before continuing. Using the disclosed methods and systems, track of the object/vehicle is not lost because the object is not "absorbed" into the background, but rather, marked and monitored based on an examination of a pre-specified time window and the aforementioned recording of an image chip corresponding to the object vehicle. As provided herein, when the tracked vehicle resumes movement, the track can be continued. [0052] Figure 7 illustrates a scenario to illustrate detection of abandoned luggage where a tracked individual abandons the luggage. With reference to Figure 7, the tracked object of the person can be identified and associated with a shape, as can the luggage, where such objects can be tracked individually. Using the methods and systems described herein, based on determining that the luggage is a still object (e.g., non-moving object), a retrospective of images prior to the determination can indicate that the luggage is a still object. Properties of the still object/luggage can be monitored/updated with subsequent views of the area that contains the still object/luggage, and track can begin and/or resume when such properties change.
[0053] The Figure 7 example also provides an example of group tracking that can be employed in the disclosed methods and systems. In group tracking, two or more objects (e.g., person and luggage, multiple people, etc.) can be tracked as a group, thereby allowing for tracking in high-traffic densities. As also shown by the example of Figure 7, group tracking can include group splitting, and/or group merging.
[0054] Figure 8 illustrates a scheme for the aforementioned move-stop-move tracking in which an object can be tracked although the object stops moving, or becomes a "still" object. As Figure 8 indicates, and as previously provided in Figures 1 and 2, video data can be provided from one or more video/camera sources and registered to a site model 810 such that motion can be detected and objects tracked 812 and correlated from multiple video sources 814. Based on the object track, it can be determined whether an object is moving 816, and if the object is moving, object properties (e.g., kinematics, 2D appearance, and/or 3D shape) can be updated 818 and object tracking 812 can continue; however, if it is determined that the object is still (e.g., non-moving) 816, then a second determination can be performed regarding the object's visibility 820. If the object is no longer visible 820, the track can be ended and/or suspended 822 until the object re-appears. Alternatively, if the object is visible 820, object properties (e.g., kinematics, 2D appearance, and/or 3D shape) can be stored/recorded 824 and monitored 826 with subsequent data 810 until it is determined 828 that the object is again moving. As previously described herein, the disclosed methods and systems can allow for a configuration in which an alert is provided to one or more locations (e.g., central location, individual locations, etc.) upon an object being tagged/characterized as "stopped", non-moving, still, etc., and/or being in such state for more than a specified time. Other examples of alert conditions (e.g., deviation from a model track) are also possible. '
[0055] Figure 9, like Figure 8, provides for an object that becomes occluded. With reference to Figure 9, and as provided with respect to Figure 8, video data can be provided from one or more video/camera sources and registered to a site model 910 such that motion can be detected and objects tracked 912 and correlated across multiple video sources 914. Based on the object track, it can be determined whether an object is moving 916, and if the object is moving, object properties can be updated 918 and object tracking 912 can continue; however, if it is determined that the object is still (e.g., non-moving) 916, then a second determination can be provided regarding whether the object is occluded 920. Object occlusion can be based on, for example, the site model and the track database by examining historical data prior to the object's still motion and/or occlusion. Properties of the occlμded object can be recorded/stored 922 and the occluded region can be monitored for new tracks originating from the occluded region and based on subsequent video data 924, until a new track appears that is consistent with the occluded object's track 926. Upon determination of a new track that is consistent with the occluded track 926, the track prior to the occlusion can be associated with the track subsequent to the occlusion 928, and a further determination can be made regarding the movement of the object 916. Figure 9 thus indicates the continued process of tracking the object through the occlusions.
[0056] As also provided herein, the disclosed methods and systems allow for tracking through viewpoint changes and lighting changes using a dynamic background adaptation scheme. Figure 10 provides one example of a dynamic background adaptation scheme in which the video data is provided for the motion detection and object tracking 1010 as previously provided herein, where background segmentation 1012 can be performed to characterize background changes 1014. It can be understood that one or more of several segmentation schemes can be used based on the embodiment. If regions of change in the background (e.g., non-object areas) are determined, detected, and/or found 1016, the Figure 10 example processing scheme can determine if (e.g., classify) such background changes are illumination effects 1018, spurious motion effects 1020, and/or imaging artifacts 1022
(e.g., noise, glint, etc.), such that the background properties can be updated 1024.
[0057] Figure 11 demonstrates one scheme for tracking an object from different video sources having different fields of view. As Figure 11 indicates, and with continued reference to Figures 1 and 2, registered fracked objects from two video data sources 1105A, 1105B can be provided to one or more correlations schemes 1110, 1120 that correlate the object track trajectories and correlate the object properties from the two video data sources. Based on such correlations, if the tracks are the same 1130, the tracks are merged 1140, and otherwise, the tracks are viewed as distinct such that a particular track may end (e.g., an object track from a first video data source), while another track (e.g., an object track from a second video data source) may begin 1150.
[0058] What has thus been described are methods, systems, and computer program products for tracking an object(s), including identifying the object(s) by correlating video data from at least one video device, based on motion data of the object(s) for a previous time, determining that the object(s) movement is stopped, based on determining that the stopped object(s) is not occluded, monitoring the stopped object(s) properties, determining from the monitoring that the stopped object(s) is moving, and, resuming track of the object. [0059] The methods and systems described herein are not limited to a particular hardware or software configuration, and may find applicability in many computing or processing environments. The methods and systems can be implemented in hardware or software, or a combination of hardware and software. The methods and systems can be implemented in one or more computer programs, where a computer program can be understood to include one or more processor executable instructions. The computer program(s) can execute on one or more programmable processors, and can be stored on one or more storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), one or more input devices, and/or one or more output devices. The processor thus can access one or more input devices to obtain input data, and can access one or more output devices to communicate output data. The input and/or output devices can include one or more of the following: Random Access Memory (RAM),
Redundant Array of Independent Disks (RAID), floppy drive, CD, DVD, magnetic disk, internal hard drive, external hard drive, memory stick, or other storage device capable of being accessed by a processor as provided herein, where such aforementioned examples are not exhaustive, and are for illustration and not limitation. [0060] The computer program(s) can be implemented using one or more high level procedural or object-oriented programming languages to communicate with a computer system; however, the program(s) can be implemented in assembly or machine language, if desired. The language can be compiled or interpreted. [0061] As provided herein, the processor(s) can thus be embedded in one or more devices that can be operated independently or together in a networked environment, where the network can include, for example, a Local Area Network (LAN), wide area network (WAN), and/or can include an intranet and/or the internet and/or another network. The network(s) can be wired or wireless or a combination thereof and can use one or more communications protocols to facilitate communications between the different processors. The processors can be configured for distributed processing and can utilize, in some embodiments, a client-server model as needed. Accordingly, the methods and systems can utilize multiple processors and/or processor devices, and the processor instructions can be divided amongst such single or multiple processor/devices.
[0062] The device(s) or computer systems that integrate with the processor(s) can include, for example, a personal computer(s), workstation (e.g., Sun, HP), personal digital assistant (PDA), handheld device such as cellular telephone, laptop, handheld, or another device capable of being integrated with a processor(s) that can operate as provided herein. Accordingly, the devices provided herein are not exhaustive and are provided for illustration and not limitation.
[0063] References to "a microprocessor" and "a processor", or "the microprocessor" and "the processor," can be understood to include one or more microprocessors that can communicate in a stand-alone and/or a distributed environment(s), and can thus can be configured to communicate via wired or wireless communications with other processors, where such one or more processor can be configured to operate on one or more processor- controlled devices that can be similar or different devices. Use of such "microprocessor" or "processor" terminology can thus also be understood to include a central processing unit, an arithmetic logic unit, an application-specific integrated circuit (IC), and/or a task engine, with such examples provided for illustration and not limitation.
[0064] Furthermore, references to memory, unless otherwise specified, can include one or more processor-readable and accessible memory elements and/or components that can be internal to the processor-controlled device, external to the processor-controlled device, and/or can be accessed via a wired or wireless network using a variety of communications protocols, and unless otherwise specified, can be arranged to include a combination of external and internal memory devices, where such memory can be contiguous and/or partitioned based on the application. Accordingly, references to a database can be understood to include one or more memory associations, where such references can include commercially available database products (e.g., SQL, Informix, Oracle) and also proprietary databases, and may also include other structures for associating memory such as links, queues, graphs, trees, with such structures provided for illustration and not limitation. [0065] References to a network, unless provided otherwise, can include one or more intranets and/or the internet. References herein to microprocessor instructions or microprocessor-executable instructions, in accordance with the above, can be understood to include programmable hardware.
[0066] Unless otherwise stated, use of the word "substantially" can be construed to include a precise relationship, condition, arrangement, orientation, and/or other characteristic, and deviations thereof as understood by one of ordinary skill in the art, to the extent that such deviations do not materially affect the disclosed methods and systems. [0067] Throughout the entirety of the present disclosure, use of the articles "a" or "an" to modify a noun can be understood to be used for convenience and to include one, or more than one of the modified noun, unless otherwise specifically stated. [0068] Elements, components, modules, and/or parts thereof that are described and/or otherwise portrayed through the figures to communicate with, be associated with, and/or be based on, something else, can be understood to so communicate, be associated with, and or be based on in a direct and/or indirect manner, unless otherwise stipulated herein. [0069] Although the methods and systems have been described relative to a specific embodiment thereof, they are not so limited. Obviously many modifications and variations may become apparent in light of the above teachings.
[0070] Many additional changes in the details, materials, and arrangement of parts, herein described and illustrated, can be made by those skilled in the art. Accordingly, it will be understood that the following claims are not to be limited to the embodiments disclosed herein, can include practices otherwise than specifically described, and are to be interpreted as broadly as allowed under the law.

Claims

What is claimed is:
1. A method for tracking at least one object, the method comprising: identifying the at least one object by correlating video data from at least one video device, based on motion data of the at least one object for a previous time, determining that the at least one object movement is stopped, based on determining that the at least one stopped object is not occluded, monitoring the at least one stopped object properties, determining from the monitoring that the at least one stopped object is moving, and, resuming track of the at least one object.
2. A method according to claim 1, where the correlating includes spatially correlating and temporally correlating.
3. A method according to claim 1, where resuming track includes creating a new track.
4. A method according to claim 1 , where the at least one stopped object properties include at least one of: kinematic properties, 2D appearance, and 3D shape.
5. A method according to claim 1, where the at least one stopped object properties include at least one of: arrival time, departure time, size, color, position, velocity, and acceleration.
6. A method according to claim 1, where the at least one video device includes at least two cameras having different fields of view.
7. A method according to claim 1 , where correlating data includes: providing a model of at least one field of view, and, registering the video data to the model.
8. A method according to claim 1, further comprising providing at least one alert based on determining the at least one object is located in a region of interest.
9. A method according to claim 8, where providing an alert includes determining a time that the at least one object entered the region of interest.
10. A method according to claim 1, further comprising providing at least one alert based on a lapse of a time since determining the at least one object entered a region of interest.
11. A method according to claim 1 , further comprising: comparing the at least one object track to a model track, and, providing an alert based on the comparison of the track to the model track.
12. A method according to claim 1, further comprising: based on determining that the at least one stopped object is occluded, monitoring new tracks of objects emanating from the region occluding the at least one object.
13. A method according to claim 12, further comprising: selecting a new track consistent with the track of the at least one occluded object prior to the occlusion, and, associating the track of the at least one occluded object prior to the occlusion with the selected new track.
14. A method according to claim 1, where correlating video data includes: detecting motion in the video data to identify objects, classifying objects from background, segmenting the background, detecting background regions with changes, and, updating the background properties based on determining that the changes are due to at least one of illumination, spurious motion, and imaging artifacts.
15. A method according to claim 1, where correlating video data includes: detecting moving objects, and, grouping moving objects based on object tracks.
16. A method according to claim 1, where correlating video data includes: detecting moving objects, and, splitting groups of moving objects based on object tracks.
17. A method according to claim 16, where splitting groups of moving object based on tracks includes: determining that at least one first object in a group is stopped, and, determining that at least one second object in the group is moving.
18. A method according to claim 1, where correlating data from at least one video device includes: correlating the track trajectory of the at least one object from a first video device, correlating the object properties of the at least one object from a second video device, and, determining, based on the correlation of the frack trajectory and correlation of the object properties, to merge at least one track from the first video device and at least one track from the second video device.
19. A method according to claim 18, where determining includes: determining, based on the correlation of the track trajectory and correlation of the object properties, to not merge at least one track from the first video device and at least one track from the second video device, and, based on determining, performing at least one of: ending a frack of an object, and, starting a track of an object.
20. A processor program product having processor-readable instructions for performing a method according to claim 1.
PCT/US2004/030421 2003-09-19 2004-09-17 Tracking systems and methods WO2005029264A2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP04788810A EP1668469A4 (en) 2003-09-19 2004-09-17 Tracking systems and methods

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US50458303P 2003-09-19 2003-09-19
US60/504,583 2003-09-19

Publications (2)

Publication Number Publication Date
WO2005029264A2 true WO2005029264A2 (en) 2005-03-31
WO2005029264A3 WO2005029264A3 (en) 2007-05-18

Family

ID=34375525

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2004/030421 WO2005029264A2 (en) 2003-09-19 2004-09-17 Tracking systems and methods

Country Status (3)

Country Link
US (1) US20050073585A1 (en)
EP (1) EP1668469A4 (en)
WO (1) WO2005029264A2 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008100358A1 (en) * 2007-02-16 2008-08-21 Panasonic Corporation Method and apparatus for efficient and flexible surveillance visualization with context sensitive privacy preserving and power lens data mining
US7944468B2 (en) 2005-07-05 2011-05-17 Northrop Grumman Systems Corporation Automated asymmetric threat detection using backward tracking and behavioral analysis
EP2580738A4 (en) * 2010-08-10 2018-01-03 LG Electronics Inc. Region of interest based video synopsis
CN112153341A (en) * 2020-09-24 2020-12-29 杭州海康威视数字技术股份有限公司 Task supervision method, device and system, electronic equipment and storage medium

Families Citing this family (138)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9038108B2 (en) * 2000-06-28 2015-05-19 Verizon Patent And Licensing Inc. Method and system for providing end user community functionality for publication and delivery of digital media content
US9892606B2 (en) * 2001-11-15 2018-02-13 Avigilon Fortress Corporation Video surveillance system employing video primitives
US8564661B2 (en) * 2000-10-24 2013-10-22 Objectvideo, Inc. Video analytic rule detection system and method
US8972862B2 (en) * 2001-06-27 2015-03-03 Verizon Patent And Licensing Inc. Method and system for providing remote digital media ingest with centralized editorial control
US20070089151A1 (en) * 2001-06-27 2007-04-19 Mci, Llc. Method and system for delivery of digital media experience via common instant communication clients
US20060236221A1 (en) * 2001-06-27 2006-10-19 Mci, Llc. Method and system for providing digital media management using templates and profiles
US7970260B2 (en) * 2001-06-27 2011-06-28 Verizon Business Global Llc Digital media asset management system and method for supporting multiple users
US8990214B2 (en) * 2001-06-27 2015-03-24 Verizon Patent And Licensing Inc. Method and system for providing distributed editing and storage of digital media over a network
US7073158B2 (en) * 2002-05-17 2006-07-04 Pixel Velocity, Inc. Automated system for designing and developing field programmable gate arrays
EP1563686B1 (en) * 2002-11-12 2010-01-06 Intellivid Corporation Method and system for tracking and behavioral monitoring of multiple objects moving through multiple fields-of-view
US20050102183A1 (en) * 2003-11-12 2005-05-12 General Electric Company Monitoring system and method based on information prior to the point of sale
US20050110634A1 (en) * 2003-11-20 2005-05-26 Salcedo David M. Portable security platform
US20060018516A1 (en) * 2004-07-22 2006-01-26 Masoud Osama T Monitoring activity using video information
WO2006012645A2 (en) * 2004-07-28 2006-02-02 Sarnoff Corporation Method and apparatus for total situational awareness and monitoring
US20060095317A1 (en) * 2004-11-03 2006-05-04 Target Brands, Inc. System and method for monitoring retail store performance
US7356425B2 (en) * 2005-03-14 2008-04-08 Ge Security, Inc. Method and system for camera autocalibration
DE602006020422D1 (en) * 2005-03-25 2011-04-14 Sensormatic Electronics Llc INTELLIGENT CAMERA SELECTION AND OBJECT PURSUIT
US7583815B2 (en) * 2005-04-05 2009-09-01 Objectvideo Inc. Wide-area site-based video surveillance system
US20080291278A1 (en) * 2005-04-05 2008-11-27 Objectvideo, Inc. Wide-area site-based video surveillance system
US9076311B2 (en) * 2005-09-07 2015-07-07 Verizon Patent And Licensing Inc. Method and apparatus for providing remote workflow management
US8631226B2 (en) * 2005-09-07 2014-01-14 Verizon Patent And Licensing Inc. Method and system for video monitoring
US9401080B2 (en) 2005-09-07 2016-07-26 Verizon Patent And Licensing Inc. Method and apparatus for synchronizing video frames
US20070107012A1 (en) * 2005-09-07 2007-05-10 Verizon Business Network Services Inc. Method and apparatus for providing on-demand resource allocation
JP4488996B2 (en) * 2005-09-29 2010-06-23 株式会社東芝 Multi-view image creation apparatus, multi-view image creation method, and multi-view image creation program
US10878646B2 (en) 2005-12-08 2020-12-29 Smartdrive Systems, Inc. Vehicle event recorder systems
US20070150138A1 (en) 2005-12-08 2007-06-28 James Plante Memory management in event recording systems
US9201842B2 (en) 2006-03-16 2015-12-01 Smartdrive Systems, Inc. Vehicle event recorder systems and networks having integrated cellular wireless communications systems
US8996240B2 (en) 2006-03-16 2015-03-31 Smartdrive Systems, Inc. Vehicle event recorders with integrated web server
DE102006013474B4 (en) * 2006-03-23 2019-01-31 Siemens Healthcare Gmbh Method for real-time reconstruction and representation of a three-dimensional target volume
JP4473232B2 (en) * 2006-04-26 2010-06-02 株式会社日本自動車部品総合研究所 Vehicle front environment detecting device for vehicle and lighting device for vehicle
US20080129824A1 (en) * 2006-05-06 2008-06-05 Ryan Scott Loveless System and method for correlating objects in an event with a camera
US8269617B2 (en) * 2009-01-26 2012-09-18 Drivecam, Inc. Method and system for tuning the effect of vehicle characteristics on risk prediction
US8508353B2 (en) * 2009-01-26 2013-08-13 Drivecam, Inc. Driver risk assessment system and method having calibrating automatic event scoring
US8849501B2 (en) * 2009-01-26 2014-09-30 Lytx, Inc. Driver risk assessment system and method employing selectively automatic event scoring
US7468662B2 (en) * 2006-06-16 2008-12-23 International Business Machines Corporation Method for spatio-temporal event detection using composite definitions for camera systems
US7685014B2 (en) * 2006-07-28 2010-03-23 Cliff Edwards Dean Bank queue monitoring systems and methods
US20080036864A1 (en) * 2006-08-09 2008-02-14 Mccubbrey David System and method for capturing and transmitting image data streams
US7940959B2 (en) * 2006-09-08 2011-05-10 Advanced Fuel Research, Inc. Image analysis by object addition and recovery
JP4558696B2 (en) * 2006-09-25 2010-10-06 パナソニック株式会社 Automatic body tracking device
US8649933B2 (en) 2006-11-07 2014-02-11 Smartdrive Systems Inc. Power management systems for automotive video event recorders
US8989959B2 (en) 2006-11-07 2015-03-24 Smartdrive Systems, Inc. Vehicle operator performance history recording, scoring and reporting systems
US8868288B2 (en) 2006-11-09 2014-10-21 Smartdrive Systems, Inc. Vehicle exception event management systems
US8396250B2 (en) * 2006-11-20 2013-03-12 Adelaide Research & Innovation Pty Ltd Network surveillance system
US20080151049A1 (en) * 2006-12-14 2008-06-26 Mccubbrey David L Gaming surveillance system and method of extracting metadata from multiple synchronized cameras
US7719568B2 (en) * 2006-12-16 2010-05-18 National Chiao Tung University Image processing system for integrating multi-resolution images
KR20080073933A (en) * 2007-02-07 2008-08-12 삼성전자주식회사 Object tracking method and apparatus, and object pose information calculating method and apparatus
US8760519B2 (en) * 2007-02-16 2014-06-24 Panasonic Corporation Threat-detection in a distributed multi-camera surveillance system
US8587661B2 (en) * 2007-02-21 2013-11-19 Pixel Velocity, Inc. Scalable system for wide area surveillance
JP2008227689A (en) * 2007-03-09 2008-09-25 Seiko Epson Corp Coder and image recorder
JP5080333B2 (en) * 2007-04-06 2012-11-21 本田技研工業株式会社 Object recognition device for autonomous mobile objects
US8239092B2 (en) 2007-05-08 2012-08-07 Smartdrive Systems Inc. Distributed vehicle event recorder systems having a portable memory data transfer system
WO2009006605A2 (en) * 2007-07-03 2009-01-08 Pivotal Vision, Llc Motion-validating remote monitoring system
CN101098461B (en) * 2007-07-05 2010-11-17 复旦大学 Full shelter processing method of video target tracking
US8131010B2 (en) * 2007-07-30 2012-03-06 International Business Machines Corporation High density queue estimation and line management
JP5079480B2 (en) * 2007-12-07 2012-11-21 ソニー株式会社 Information processing apparatus, information processing method, and program
US8525825B2 (en) 2008-02-27 2013-09-03 Google Inc. Using image content to facilitate navigation in panoramic image data
TWI489394B (en) * 2008-03-03 2015-06-21 Videoiq Inc Object matching for tracking, indexing, and search
US8325976B1 (en) * 2008-03-14 2012-12-04 Verint Systems Ltd. Systems and methods for adaptive bi-directional people counting
JP5264582B2 (en) * 2008-04-04 2013-08-14 キヤノン株式会社 Monitoring device, monitoring method, program, and storage medium
CA2719504A1 (en) * 2008-04-14 2009-10-22 Thomson Licensing Technique for automatically tracking an object
WO2009137616A2 (en) * 2008-05-06 2009-11-12 Strongwatch Corporation Novel sensor apparatus
US9224425B2 (en) * 2008-12-17 2015-12-29 Skyhawke Technologies, Llc Time stamped imagery assembly for course performance video replay
US8854199B2 (en) 2009-01-26 2014-10-07 Lytx, Inc. Driver risk assessment system and method employing automated driver log
US8180107B2 (en) * 2009-02-13 2012-05-15 Sri International Active coordinated tracking for multi-camera systems
US9541505B2 (en) 2009-02-17 2017-01-10 The Boeing Company Automated postflight troubleshooting sensor array
US9418496B2 (en) * 2009-02-17 2016-08-16 The Boeing Company Automated postflight troubleshooting
US8812154B2 (en) * 2009-03-16 2014-08-19 The Boeing Company Autonomous inspection and maintenance
US20100293173A1 (en) * 2009-05-13 2010-11-18 Charles Chapin System and method of searching based on orientation
US9046892B2 (en) * 2009-06-05 2015-06-02 The Boeing Company Supervision and control of heterogeneous autonomous operations
US20100318588A1 (en) * 2009-06-12 2010-12-16 Avaya Inc. Spatial-Temporal Event Correlation for Location-Based Services
US20110115909A1 (en) * 2009-11-13 2011-05-19 Sternberg Stanley R Method for tracking an object through an environment across multiple cameras
US8577083B2 (en) 2009-11-25 2013-11-05 Honeywell International Inc. Geolocating objects of interest in an area of interest with an imaging system
MY150414A (en) * 2009-12-21 2014-01-15 Mimos Berhad Method of determining loitering event
US8358808B2 (en) * 2010-01-08 2013-01-22 University Of Washington Video-based vehicle detection and tracking using spatio-temporal maps
US8817094B1 (en) 2010-02-25 2014-08-26 Target Brands, Inc. Video storage optimization
US8773289B2 (en) 2010-03-24 2014-07-08 The Boeing Company Runway condition monitoring
US8599044B2 (en) 2010-08-11 2013-12-03 The Boeing Company System and method to assess and report a health of a tire
US8712634B2 (en) 2010-08-11 2014-04-29 The Boeing Company System and method to assess and report the health of landing gear related components
US9172913B1 (en) * 2010-09-24 2015-10-27 Jetprotect Corporation Automatic counter-surveillance detection camera and software
US8982207B2 (en) * 2010-10-04 2015-03-17 The Boeing Company Automated visual inspection system
IL208910A0 (en) * 2010-10-24 2011-02-28 Rafael Advanced Defense Sys Tracking and identification of a moving object from a moving sensor using a 3d model
JP5893634B2 (en) * 2010-11-05 2016-03-23 コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. Image forming apparatus for image forming object, method of operating image forming apparatus, and computer program for image forming
US9147260B2 (en) * 2010-12-20 2015-09-29 International Business Machines Corporation Detection and tracking of moving objects
US9154747B2 (en) * 2010-12-22 2015-10-06 Pelco, Inc. Stopped object detection
US8566325B1 (en) 2010-12-23 2013-10-22 Google Inc. Building search by contents
US8533187B2 (en) 2010-12-23 2013-09-10 Google Inc. Augmentation of place ranking using 3D model activity in an area
US8744123B2 (en) 2011-08-29 2014-06-03 International Business Machines Corporation Modeling of temporarily static objects in surveillance video data
US8606492B1 (en) 2011-08-31 2013-12-10 Drivecam, Inc. Driver log generation
US8744642B2 (en) 2011-09-16 2014-06-03 Lytx, Inc. Driver identification based on face data
US8996234B1 (en) 2011-10-11 2015-03-31 Lytx, Inc. Driver performance determination based on geolocation
US9298575B2 (en) 2011-10-12 2016-03-29 Lytx, Inc. Drive event capturing based on geolocation
US8675917B2 (en) 2011-10-31 2014-03-18 International Business Machines Corporation Abandoned object recognition using pedestrian detection
WO2013065121A1 (en) * 2011-11-01 2013-05-10 アイシン精機株式会社 Obstacle alert device
US8989914B1 (en) 2011-12-19 2015-03-24 Lytx, Inc. Driver identification based on driving maneuver signature
US8676428B2 (en) 2012-04-17 2014-03-18 Lytx, Inc. Server request for downloaded information from a vehicle-based monitor
US9240079B2 (en) 2012-04-17 2016-01-19 Lytx, Inc. Triggering a specialized data collection mode
US9728228B2 (en) 2012-08-10 2017-08-08 Smartdrive Systems, Inc. Vehicle event playback apparatus and methods
JP6046948B2 (en) * 2012-08-22 2016-12-21 キヤノン株式会社 Object detection apparatus, control method therefor, program, and storage medium
US9344683B1 (en) 2012-11-28 2016-05-17 Lytx, Inc. Capturing driving risk based on vehicle state and automatic detection of a state of a location
US20140184803A1 (en) * 2012-12-31 2014-07-03 Microsoft Corporation Secure and Private Tracking Across Multiple Cameras
CN103971359A (en) * 2013-02-05 2014-08-06 株式会社理光 Method and device for locating object through object detection results of multiple stereo cameras
US9454827B2 (en) * 2013-08-27 2016-09-27 Qualcomm Incorporated Systems, devices and methods for tracking objects on a display
CN104574433A (en) * 2013-10-14 2015-04-29 株式会社理光 Object tracking method and equipment and tracking feature selection method
US9501878B2 (en) 2013-10-16 2016-11-22 Smartdrive Systems, Inc. Vehicle event playback apparatus and methods
US9610955B2 (en) 2013-11-11 2017-04-04 Smartdrive Systems, Inc. Vehicle fuel consumption monitor and feedback systems
US10586570B2 (en) 2014-02-05 2020-03-10 Snap Inc. Real time video processing for changing proportions of an object in the video
US8892310B1 (en) 2014-02-21 2014-11-18 Smartdrive Systems, Inc. System and method to detect execution of driving maneuvers
US9972182B2 (en) 2014-04-07 2018-05-15 William J. Warren Movement monitoring security devices and systems
US9955050B2 (en) * 2014-04-07 2018-04-24 William J. Warren Movement monitoring security devices and systems
KR102152725B1 (en) * 2014-05-29 2020-09-07 한화테크윈 주식회사 Control apparatus for camera
US9754178B2 (en) 2014-08-27 2017-09-05 International Business Machines Corporation Long-term static object detection
US9663127B2 (en) 2014-10-28 2017-05-30 Smartdrive Systems, Inc. Rail vehicle event detection and recording system
US11069257B2 (en) 2014-11-13 2021-07-20 Smartdrive Systems, Inc. System and method for detecting a vehicle event and generating review criteria
US10609270B2 (en) 2014-11-18 2020-03-31 The Invention Science Fund Ii, Llc Devices, methods and systems for visual imaging arrays
US20180063372A1 (en) * 2014-11-18 2018-03-01 Elwha Llc Imaging device and system with edge processing
US10438277B1 (en) 2014-12-23 2019-10-08 Amazon Technologies, Inc. Determining an item involved in an event
US10552750B1 (en) 2014-12-23 2020-02-04 Amazon Technologies, Inc. Disambiguating between multiple users
US10475185B1 (en) 2014-12-23 2019-11-12 Amazon Technologies, Inc. Associating a user with an event
US9710712B2 (en) * 2015-01-16 2017-07-18 Avigilon Fortress Corporation System and method for detecting, tracking, and classifiying objects
US10116901B2 (en) * 2015-03-18 2018-10-30 Avatar Merger Sub II, LLC Background modification in video conferencing
US9754413B1 (en) 2015-03-26 2017-09-05 Google Inc. Method and system for navigating in panoramic images using voxel maps
US11829945B1 (en) * 2015-03-31 2023-11-28 Amazon Technologies, Inc. Sensor data fusion for increased reliability
US9679420B2 (en) 2015-04-01 2017-06-13 Smartdrive Systems, Inc. Vehicle event recording system and method
US10044988B2 (en) 2015-05-19 2018-08-07 Conduent Business Services, Llc Multi-stage vehicle detection in side-by-side drive-thru configurations
US10262293B1 (en) 2015-06-23 2019-04-16 Amazon Technologies, Inc Item management system using multiple scales
US9959468B2 (en) 2015-11-06 2018-05-01 The Boeing Company Systems and methods for object tracking and classification
US10217001B2 (en) * 2016-04-14 2019-02-26 KickView Corporation Video object data storage and processing system
TWI633497B (en) 2016-10-14 2018-08-21 群暉科技股份有限公司 Method for performing cooperative counting with aid of multiple cameras, and associated apparatus
US11545013B2 (en) * 2016-10-26 2023-01-03 A9.Com, Inc. Customizable intrusion zones for audio/video recording and communication devices
JP6961363B2 (en) * 2017-03-06 2021-11-05 キヤノン株式会社 Information processing system, information processing method and program
CN109427074A (en) 2017-08-31 2019-03-05 深圳富泰宏精密工业有限公司 Image analysis system and method
TWI656512B (en) * 2017-08-31 2019-04-11 群邁通訊股份有限公司 Image analysis system and method
US10482572B2 (en) * 2017-10-06 2019-11-19 Ford Global Technologies, Llc Fusion of motion and appearance features for object detection and trajectory prediction
CN108090414A (en) * 2017-11-24 2018-05-29 江西智梦圆电子商务有限公司 A kind of method for capturing face tracking trace immediately based on computer vision
CN107944960A (en) * 2017-11-27 2018-04-20 深圳码隆科技有限公司 A kind of self-service method and apparatus
WO2020028040A1 (en) 2018-07-30 2020-02-06 Carrier Corporation Method for activating an alert when an object is left proximate a room entryway
CN110706251B (en) * 2019-09-03 2022-09-23 北京正安维视科技股份有限公司 Cross-lens tracking method for pedestrians
US11074460B1 (en) * 2020-04-02 2021-07-27 Security Systems, L.L.C. Graphical management system for interactive environment monitoring

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6816184B1 (en) * 1998-04-30 2004-11-09 Texas Instruments Incorporated Method and apparatus for mapping a location from a video image to a map
US6570608B1 (en) * 1998-09-30 2003-05-27 Texas Instruments Incorporated System and method for detecting interactions of people and vehicles
US6690374B2 (en) * 1999-05-12 2004-02-10 Imove, Inc. Security camera system for tracking moving objects in both forward and reverse directions
US6424370B1 (en) * 1999-10-08 2002-07-23 Texas Instruments Incorporated Motion based event detection system and method
EP1472870A4 (en) * 2002-02-06 2006-11-29 Nice Systems Ltd Method and apparatus for video frame sequence-based object tracking
US7242423B2 (en) * 2003-06-16 2007-07-10 Active Eye, Inc. Linking zones for object tracking and camera handoff

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of EP1668469A4 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7944468B2 (en) 2005-07-05 2011-05-17 Northrop Grumman Systems Corporation Automated asymmetric threat detection using backward tracking and behavioral analysis
WO2008100358A1 (en) * 2007-02-16 2008-08-21 Panasonic Corporation Method and apparatus for efficient and flexible surveillance visualization with context sensitive privacy preserving and power lens data mining
EP2580738A4 (en) * 2010-08-10 2018-01-03 LG Electronics Inc. Region of interest based video synopsis
CN112153341A (en) * 2020-09-24 2020-12-29 杭州海康威视数字技术股份有限公司 Task supervision method, device and system, electronic equipment and storage medium

Also Published As

Publication number Publication date
US20050073585A1 (en) 2005-04-07
EP1668469A4 (en) 2007-11-21
WO2005029264A3 (en) 2007-05-18
EP1668469A2 (en) 2006-06-14

Similar Documents

Publication Publication Date Title
US20050073585A1 (en) Tracking systems and methods
US11594031B2 (en) Automatic extraction of secondary video streams
US10664706B2 (en) System and method for detecting, tracking, and classifying objects
Collins et al. Algorithms for cooperative multisensor surveillance
US7280673B2 (en) System and method for searching for changes in surveillance video
KR101085578B1 (en) Video tripwire
Tian et al. IBM smart surveillance system (S3): event based video surveillance system with an open and extensible framework
US8620028B2 (en) Behavioral recognition system
US20100150403A1 (en) Video signal analysis
Zabłocki et al. Intelligent video surveillance systems for public spaces–a survey
US20100165112A1 (en) Automatic extraction of secondary video streams
KR20070101401A (en) Video surveillance system employing video primitives
Park et al. A track-based human movement analysis and privacy protection system adaptive to environmental contexts
US10643078B2 (en) Automatic camera ground plane calibration method and system
Chan A robust target tracking algorithm for FLIR imagery
Gupta et al. Suspicious Object Tracking by Frame Differencing with Backdrop Subtraction
Ali et al. Advance video analysis system and its applications
Hafiz et al. Event-handling based smart video surveillance system
Hassan Video Analytics for Security Systems
Dyer Application of scene understanding to representative military imagery
Awate et al. Survey on Video object tracking and segmentation using artificial neural network in surveillance system

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BW BY BZ CA CH CN CO CR CU CZ DK DM DZ EC EE EG ES FI GB GD GE GM HR HU ID IL IN IS JP KE KG KP KZ LC LK LR LS LT LU LV MA MD MK MN MW MX MZ NA NI NO NZ PG PH PL PT RO RU SC SD SE SG SK SY TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): BW GH GM KE LS MW MZ NA SD SZ TZ UG ZM ZW AM AZ BY KG MD RU TJ TM AT BE BG CH CY DE DK EE ES FI FR GB GR HU IE IT MC NL PL PT RO SE SI SK TR BF CF CG CI CM GA GN GQ GW ML MR SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
WWE Wipo information: entry into national phase

Ref document number: 2004788810

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 2004788810

Country of ref document: EP