Video analysis, archiving and alerting methods and apparatus for a distributed, modular and extensible video surveillance system   

This application hereby claims priority from provisional U.S. patent application Ser. Nos. ______ and ______ filed on Oct. 12, 2004 and assigned to the same assignee as the present application. These provisional applications are hereby incorporated by reference in their entirety as if fully restated herein. This application is also related to U.S. patent application Ser. No. ______ filed on even date and assigned to the same assignee as the present application. U.S. patent application Ser. No. ______ is also incorporated by reference in its entirety as if fully restated herein.

TECHNICAL FIELD

The present invention generally concerns video surveillance systems and more particularly concerns methods and systems for analyzing, managing and cataloguing of video tracks. The present invention also more particularly concerns systems and methods for organizing video tracks and associated event and statistical information in well-indexed databases whereby video tracks containing certain types of, for example, objects; events; and activities, are easily managed and accessed through web services and browsers.

In the present environment of heightened concern over criminal and terrorist activities, those skilled in the art have recognized the need for a more widespread application of video surveillance systems. This recognition has resulted in the desire for a more flexible suite of physical video surveillance assets (for example, highly programmable video cameras having the innate ability to handle a multitude of surveillance situations) and firmware for integrating and managing a distributed system of video surveillance assets. U.S. patent application Ser. No. ______, assigned to the same assignee as the present invention, discloses such methods and systems and is hereby incorporated by reference in its entirety as if fully restated herein.

The desire for flexibility also extends to the establishment and management of video surveillance activities. Current systems for establishing and managing video surveillance activities lack flexibility and therefore are difficult to modify to respond to evolving threat environments. In particular, current systems for establishing and managing video surveillance activities were often constructed to respond to a relatively limited number of threats and to perform little or no video analysis. In order to take advantage of the flexibility and programmability of video surveillance cameras and video analysis engines, new video surveillance system architectures are required.

Those skilled in the art desire video surveillance system architectures that adopt a modular approach to video analysis applications and operations. For example, those skilled in the art desire a software and middleware framework that accommodates the rapid addition of video analysis applications to a suite of pre-existing video analysis applications. Such a framework would make it far easier to tailor video surveillance system assets to evolving threat environments.

In addition, state-of-the-art video surveillance systems, particularly those with large numbers of a video surveillance cameras and video analysis engines, create a great deal of video and data. Obviously, it would be prohibitively expensive and impractical to employ a human user to monitor the output each video camera of such a highly integrated and distributed video surveillance system. In addition, it would thwart one of the objectives of such systems, that is, to give a cadre of surveillance system analysts a holistic view of a particular surveillance environment through highly distributed video analysis operations. Instead, the surveillance system analysts would develop only a tunnel vision view of a particular surveillance environment.

Another problem results simply from the number of video cameras that comprise such a distributed and large video surveillance system. Since it would be prohibitively expensive to employ security personnel to monitor each video camera, there must be a system for recording, analyzing and cataloguing the output of the video surveillance system, and for alerting surveillance system analysts in dependence on evolving threats revealed by analysis. Otherwise, the data created by such a system would simply go to waste as so much un-reviewed data.

An additional problem arises from the fact that security system analysts in next generation video surveillance systems will not be continuously monitoring a relatively limited number of video feeds giving them the ability to develop a contextual understanding of fields of view as events evolve in the field of view. Rather, due to the large number of available views, it is simply impossible to monitor all of them. As a result, some way must be devised to analyze, manage, catalogue and present video tracks so that the situational awareness of surveillance system analysts is substantially improved over current video surveillance systems.

Thus, those skilled in the art desire a modular, extensible and distributed video surveillance system architecture that easily accommodates the addition and management of video analysis applications. Those skilled in the art also desire a system for analyzing, managing, cataloguing and presenting video tracks. In particular, those skilled in the art desire a system for assigning meaningful track identifying tags to simplify and ease the cataloguing of such tracks. In addition, those skilled in the art desire a suite of video analysis applications capable of generating a wide degree of track information from key frames suitable for improving the situational awareness of surveillance analysts, to statistical information summarizing monitored object activities and events.

SUMMARY

The foregoing and other problems are overcome, and other advantages are realized, in accordance with the presently preferred embodiments of these teachings. The present invention comprises methods and apparatus for performing and managing video surveillance activities.

In particular, a first alternate embodiment of the present invention comprises a video surveillance system comprising: at least one video surveillance system module comprising: a video surveillance camera, wherein the video surveillance camera is remotely programmable; a video encoder to encode video gathered by the video surveillance camera; and a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera and to create data derived from the video; at least one video surveillance system network coupled to the at least one video surveillance system module; at least one database and application system coupled to the at least one video surveillance system network to store video gathered by the video surveillance camera and data created by the video analysis engine, and to receive and process queries for the video and data; a video surveillance system control interface coupled to the at least one video surveillance system network to control the operations of the video surveillance system; and a video surveillance application and control interface coupled to the at least one video surveillance system network for controlling video surveillance activities, the video surveillance application and control interface comprising: a program for controlling video surveillance activities; a display for displaying video gathered by the video surveillance camera and data created by the video analysis engine; and at least one input device for creating queries of the database and application system.

A second alternate embodiment of the present invention comprises: a signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus of a computer system to perform operations comprising management of video surveillance activities using an electronic video surveillance system, the operations comprising: applying a video analysis application tool to video gathered by a video surveillance camera in real time, wherein the video analysis application tool comprises at least one user-specified video analysis criterion, and where application of the video analysis application tool to the video generates a data stream related to the at least one user-specified video analysis criterion; transmitting the data stream to an electronic database and application system; storing information comprising the data stream in the electronic database and application system; selecting at least a portion of the information comprising the data stream for forwarding to a video surveillance analyst performing video surveillance activities, wherein the selection is made based on at least one user-specified information forwarding criterion; and forwarding the selected information to the video surveillance analyst for display on an interface device of the electronic video surveillance system.

Thus it is seen that embodiments of the present invention overcome the limitations of the prior art. In particular, the methods and apparatus of the present invention provide a highly flexible middleware architecture for adding new video analysis applications in order to accommodate the availability of new technology or to respond to new threat environments.

In addition, methods and apparatus of the present invention achieve continuous situational awareness through around-the-clock automated object and event detection; object tracking and object classification. In prior video surveillance systems, the systems often depended on the constant vigilance of a video surveillance analyst, a situation fraught with danger due to the inherent limitations of human beings, for example the inability to remain at a high degree of attentiveness. In contrast to the prior art, in the present invention computer-controlled automated systems perform most of the object and event detection, object tracking and object classification, meaning that the lack of attentiveness of a video surveillance analyst will be much less of an issue.

Continuing, the methods and apparatus of the present invention heighten the situational awareness of video surveillance analysts by performing video surveillance in a more structured manner. In the prior art, video surveillance often consisted of setting up a bank of video cameras and associated video displays and monitoring them. Such a low-level approach did not take into consideration likely threats; instead the video monitors are simply observed for “anything which might happen”. In contrast, the methods and apparatus of the present invention institute highly regimented video surveillance procedures and activities which take into consideration likely threats and desired responses. For example, video analytics are selected based on the threat environment. In addition, alarm events are pre-instituted, meaning that the potential negative effect of human judgment is reduced.

Further, the methods and apparatus of the present invention apply video analytics continuously, and sometimes supply the results in real time, thereby providing the video surveillance analyst with a wealth of focused information that simply was not available in prior-art systems. The systematic way in which video analytics is performed and the resulting data saved means that information is available both for real-time alarm situations and near- and far-term investigative situations. This is particularly important as video surveillance analysts simply cannot observe patterns which only become visible over many months or years.

In addition, the continuous recording and cataloguing of objects and events means that a video surveillance analyst always has available just-occurred-event information. Often the most important information for deciding how to respond to a threat situation is event information that is only minutes or tens of minutes old. In prior art situations where the information simply was not recorded or, if recorded, was not immediately or easily accessible (for example, video tape) the ability to make decisions based on an almost immediate review of just-occurred-event information was either very difficult or impossible. In the present invention, just-occurred-event information is saved continuously and is immediately available in response to simple queries to assist a decision maker in deciding how to respond to a threat situation.

In summary, the foregoing alternate embodiments of the present invention are exemplary and non-limiting. For example, one of ordinary skill in the art will understand that one or more aspects or steps from one alternate embodiment can be combined with one or more aspects or steps from another alternate embodiment to create a new embodiment within the scope of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other aspects of these teachings are made more evident in the following Detailed Description of the Preferred Embodiments, when read in conjunction with the attached Drawing Figures, wherein:

FIG. 1A depicts a block diagram of a front end of an electronic video surveillance camera system organized in relation to video surveillance cameras comprising the video surveillance camera system and made in accordance with the present invention;

FIG. 1B depicts a block diagram of a front end of a video surveillance camera system organized in relation to programmed views of video surveillance cameras comprising the video surveillance camera system and made in accordance with the present invention;

FIG. 2 depicts the data flow architecture of a distributed video surveillance camera system made in accordance with the present invention;

FIG. 3 depicts the control flow architecture of a distributed video surveillance camera system made in accordance with the present invention;

FIG. 4 depicts the data flow architecture of an alternate distributed video surveillance camera system made in accordance with the present invention;

FIG. 5 depicts an alternate video surveillance system architecture made in accordance with the present invention;

FIG. 6 depicts the steps of a method for automating video analysis operations and data collection and distribution activities of a distributed video surveillance camera system;

FIGS. 7A and 7B depict video analysis criteria and information forwarding criteria available for automating video analysis and data collection and distribution activities of a distributed video surveillance camera system;

FIG. 8 depicts a schematic representation of a track data model in accordance with the present invention;

FIG. 9 depicts a track summary document in accordance with one embodiment of the present invention;

FIG. 10 depicts a track data document in accordance with one embodiment of the present invention;

FIG. 11 depicts a background image document in accordance with one embodiment of the present invention;

FIG. 12 depicts a track identification format in accordance with the present invention;

FIG. 13 depicts a temporal-based hierarchical data structure for organizing video track data in accordance with the present invention;

FIG. 14 depicts a conceptual block diagram illustrating the control flow in establishing the schematic architecture of an electronic database system for use in a distributed video surveillance camera system made in accordance with the present invention;

FIG. 15 depicts the steps of a method in establishing the architecture of an electronic database system for use in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 16 depicts additional operations available for managing the schematic architecture of an electronic database system for use in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 17 depicts a schematic representation of the data organization of an electronic database system operating in accordance with the present invention;

FIG. 18 depicts in schematic representation of data flow from a video analysis engine to an electronic database system in an electronic video surveillance system operating in accordance with one embodiment of the present invention;

FIG. 19 depicts the flow of data from a video analysis engine to an electronic database system that in certain embodiments may occur in real- or near-real-time in a video surveillance camera system operating in accordance with the present invention;

FIG. 20 depicts the flow of data from a video analysis engine to an electronic database system that in certain embodiments may occur in background mode or off-peak hours in a video surveillance camera system operating in accordance with the present invention;

FIG. 21 depicts a conceptual block diagram of the control flow in an event retrieval module incorporated in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 22 depicts steps of a method for retrieving event information from an electronic database system of a video surveillance camera system operating in accordance with the present invention;

FIG. 23 depicts additional operations available for managing the retrieval of event information from an electronic database system of a video surveillance camera system operating in accordance with the present invention;

FIG. 24 depicts a conceptual block diagram of the control and data flow in a track information retrieval service incorporated in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 25 depicts the steps of a method for managing track information retrieval in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 26 depicts additional operations available for managing track information retrieval in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 27 depicts the control and data flow of a data transformation service incorporated in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 28 depicts the steps of a method for transforming the format of data retrieved from the electronic database system in a distributed video surveillance camera system operating in accordance with the present invention;

FIG. 29 depicts additional operations available in the transformation service of the present invention;

FIG. 30 is a block diagram depicting the control and data flow of an instant alert management service incorporated in a video surveillance system operating in accordance with the present invention;

FIG. 31 is a block diagram depicting the track data flow in an instant alert management service operating in accordance with the present invention;

FIG. 32 depicts the steps of a method performing the instant alert management service of the present invention;

FIG. 33 depicts additional operations available in the instant alert management service of the present invention;

FIG. 34 depicts the establishment of a speeding alert definition for use by the instant alert management service of the present invention;

FIG. 35 depicts the establishment of a loitering alert definition for use by the instant alert management service of the present invention;

FIG. 36 depicts a browser page comprising an instant alert summary generated by the instant alert management service of the present invention;

FIG. 37 is a block diagram depicting the control and data flow of a summary management service operating in accordance with the present invention;

FIG. 38 depicts the steps of a method performing the summary management service of the present invention;

FIG. 39 depicts additional operations available in the summary management service of the present invention;

FIG. 40 depicts a chart and browser page of keyframes generated by the summary management service of the present invention summarizing human activity;

FIG. 41 depicts a chart and browser page of keyframes generated by the summary management service of the present invention summarizing hourly automobile activity;

FIG. 42 depicts charts generated by the summary management service of the present invention summarizing daily object activity;

FIG. 43 depicts a chart generated by the summary management service of the present invention summarizing arrival and departure times;

FIG. 44-46 are background images depicting object movement tracks generated by the summary management service of the present invention;

FIG. 47 is a browser page comprised of keyframes associated with human activity generated by the summary management service of the present invention; and

FIG. 48 is a browser page comprised of keyframes associated with automobile activity generated by the summary management service of the present invention.

DETAILED DESCRIPTION

Particular embodiments and features of the present invention are illustrated in the schematic block diagrams depicted in FIGS. 1-5. The block diagram in FIG. 1A depicts a portion of a video surveillance system front end 100 comprised of a root video surveillance cluster 102, which is in turn comprised of individual video surveillance camera clusters 112, 114. Each cluster 112, 114 is in turn comprised of at least one video surveillance camera; for example, cluster 1 comprises four video surveillance cameras 122, 124, 126, 128, while cluster 114 comprises one video surveillance camera 130.

Individual cameras comprising the videos surveillance camera clusters are preferably highly programmable with programmable pan, tilt, zoom and digital zoom features. The individual cameras also have view-programming capability, wherein a view comprised of a specific set of operations or movements selected from at least one of the pan, tilt, zoom or digital zoom options are programmed into a sequence of operations or movements. In certain embodiments, each camera may have multiple programmed views as illustrated by camera 124 in FIG. 1A, which has views 142, 144, and 146, camera 128 which has views 148 and 150; and camera 130 which has views 152 and 154.

In addition to view programming, the video corresponding to a particular view 146 gathered by a video surveillance camera 124 is analyzed by a video analysis engine 214 according to one or more user-specified criteria. The video analysis 214 engine creates track data 162, 164 by analyzing the video gathered by the video surveillance cameras, for example, video surveillance camera 124. As is shown in FIG. 1A, the video analysis occurs on a specific view 146 of video surveillance camera 124. The track data comprises at least a pointer to video satisfying the user-specified criteria. For example, if the criteria correspond to the event of an automobile entering the view of the camera, the video analysis engine will create track data comprising at least a pointer to each segment of video recording the occurrence of an automobile entering the field of view.

In contrast to the embodiment depicted in FIG. 1A, the embodiment depicted in FIG. 1B is organized in clusters of programmed camera views. As in the case of the embodiment depicted in FIG. 1A, the video surveillance camera front end 170 comprises a root cluster 171 further comprised of clusters 172 and 173. Each of the clusters 172, 173 is comprised of programmed camera views selected from different cameras. For example, cluster 172 is comprised of a view 1 (174) selected from camera 1; view 3 (175) selected from a camera 2; and a view 2 (176) selected from a camera 5; and cluster 173 is comprised of a view 4 (177) from a camera 7 and a view 3 (178) from a camera 9. In addition, video analysis operations can be applied to a cluster of views, or to individual views comprising the cluster. Further, data and alarm conditions derived from video analysis applied to a cluster of views can be reported from a cluster-centric perspective.

This feature illustrates a particular advantage of the present invention in that video surveillance and analysis activities can be easily and continually customized to a particular threat environment. A video surveillance analyst is not confronted with a situation as in the case of the prior art where video surveillance options were predetermined at the establishment of the video surveillance system; rather, when practicing the methods and apparatus of the present invention the video surveillance analyst can continually adjust the system to evolving threat situations.

FIG. 2 depicts data management details and FIG. 3 depicts control management details of a video surveillance system 200 made in accordance with the present invention. In particular, video surveillance system 200 comprises, in part, multiple video surveillance system modules 210 comprising video cameras 212; video analysis engines 214; video encoders 216; and camera controllers 218. At least one video surveillance system module comprises a video surveillance camera cluster in the embodiment depicted in FIGS. 2 and 3. The video surveillance system modules 210 are a particular advantage of the present invention because they provide a high degree of programmability, modularity, flexibility and extensibility. Each video surveillance camera 212 in a video surveillance system module is remotely programmable through camera controller 218. In addition, the video analysis engine 214 is remotely programmable, permitting a user to specify video analysis criteria remotely. In the data flow depicted in FIG. 2, data derived from the analysis of video gathered by the video surveillance cameras 212 is transmitted by the video analysis engine to the electronic database system 220 over a video surveillance system network 215. Although the video surveillance system module is depicted as incorporating both a video analysis engine and video encoder, in other embodiments video analysis engines and video encoders may be distributed in other ways; for example, a video encoder may perform encoding operations for multiple cameras.

In other embodiments the video surveillance system module further comprises a keyframe generation system to create a keyframe of a specific field of view viewed by the video surveillance camera. Keyframes are generated whenever a user-specified video analysis criterion is satisfied, and generally correspond to a still image of a field of view best capturing an object or event being observed. In further embodiments, a keyframe may be a headshot of a person entering a facility. The headshot may be created by momentarily zooming the video surveillance camera on the person\'s face.

A unitary system network 215 is depicted in FIGS. 2 and 3, but in various alternate embodiments of the present invention the video surveillance system network can be segmented into various component networks operating in accordance with various protocols. For example, portions of the video surveillance system network can be hard-wired while other portions can be wireless. In addition, different wired and wireless network protocols can be used at various levels of the system.

The electronic database system 220 comprises a database server 222 for receiving and cataloguing data generated by the video analysis engines 214; a video manager for receiving video encoded by video encoders 216; and an application server 226 for managing video surveillance applications. In the particular embodiment depicted in FIGS. 2 and 3 both implementation and management of the video surveillance system 200, and video surveillance activities are managed through the electronic database system 220 of the video surveillance system 200. In alternate embodiments, these activities can be managed through separate instrumentalities.

In addition, the database system is depicted in FIGS. 2 and 3 as integrating both a database server 222 and video manager 224. In alternate embodiments, these can be separated; and in still other embodiments there may be multiple database servers 222 and video managers 224 in a large-scale distributed video surveillance system operating in accordance with the present invention. In fact, the use of globally-unique identifiers for identifying video surveillance camera clusters; video surveillance cameras; video surveillance camera views and track data greatly reduces database management overhead since there is no need to centrally manage identification activities, and makes it possible to implement a distributed video surveillance system.

In the embodiment depicted in FIGS. 2 and 3 two applications 228 and 230 interact with the video surveillance system 200 through the electronic database system 220. In particular, application 228 is a control application for implementing and managing the video surveillance system, while application 230 is a surveillance application for controlling surveillance activities accomplished using the video surveillance system 200. For example, in various embodiments, the surveillance application 230 would automatically disseminate video and data gathered by the video surveillance system 200 to video surveillance analysts performing surveillance activities. In other embodiments, the surveillance application would receive queries of the electronic database system 200 formulated by the video surveillance analysts; locate the data and video satisfying the queries, and display the data and video in browser format 240 as shown in FIGS. 2 and 3. In further embodiments, the surveillance application 230 would generate alarm conditions for alerting the video surveillance analysts performing video surveillance activities. These various options can also be simultaneously available in a single embodiment.

Particular advantages of the methods and apparatus of the present invention are apparent in FIGS. 2 and 3. For example, video analysis can be performed continuously by the video analysis engines 214, thereby creating a rich database of object and event data that can be accessed either in near-real-time in aid of, for example, policing activities, or much later in aid of investigative activities. In addition, it is apparent the simplicity with which a video surveillance system having the architecture depicted in FIGS. 2 and 3 can be extended through the addition of modular video surveillance system modules 210. This is further eased by the assigning of unique identifiers to video surveillance camera clusters; video surveillance cameras; and video surveillance camera views. The assignment of unique identifiers aids not only in the cataloguing and archiving of data and video in the electronic database system 220, but also in the development of situational awareness by video surveillance analysts whose awareness is increased through the association of video surveillance cameras and views with unique identifiers.

In addition, the unique identifiers assigned to the clusters, cameras and views can also function as web address in a web-based system, greatly simplifying interactions with the electronic video surveillance system. For example, if a video surveillance analyst desired to see a live feed associated with a particular camera view, the analyst need merely enter the address of the camera view using the unique identifier to form a portion of the address to access the camera view.

An alternate video surveillance system operating in accordance with the methods and apparatus of the present invention is depicted in FIG. 4. In FIG. 4, the video analysis engine 214 is programmed with an object detection application 250 to perform object detection; an object tracking application 252 to perform object tracking; an object classification application 254 to perform object classification; and an activity analysis application 256 to perform activity analysis. In the embodiment depicted in FIG. 4, the analysis activities are applied to video gathered from a single video surveillance camera 212, illustrating that multiple video analysis activities can be simultaneously applied to the video.

The video encoder 216 comprises multiple video encoding applications, providing a wide range of options for video surveillance analysts using the video surveillance system. In particular, video encoder 216 comprises an MPEG4 encoder 260; a Windows Media encoder 262; and a Verint Encoder 264.

The database server 222 depicted in FIG. 4 is programmed to perform various alerts in response to data received from the video analysis engine 214. For example, the programming of the database server comprises a speeding alert application 270 to issue speeding alerts; a vehicle alert application 272 to issue vehicle alerts; a loitering application 274 to issue loitering alerts; and another application 276 to issue other alerts. In other embodiments (for example, those depicted in FIGS. 2-3) alarms can also be implemented locally to be performed by at least one of the video analysis engines 214.

Additional details of a video surveillance system 200 made in accordance with the present invention are depicted in the conceptual block diagram depicted in FIG. 5. The video surveillance system 200 includes at least one data processor 301 coupled to at least one system bus 302 through which the data processor 301 may address a memory sub-system 303, also referred to herein simply as the memory 303. The memory 303 may include RAM, ROM and fixed and removable disks and/or tape. The memory 303 is assumed to store at least one program comprising instructions for causing the data processor 301 to execute methods in accordance with the teachings of the invention. The data processor 301 is also coupled through the bus 302 to a user interface, preferably a graphical user interface (“GUI”) 305 that includes a user input device 306, such as one or more of a keyboard, a mouse, a trackball, a voice recognition interface, as well as a user display device 307, such as a high resolution graphical CRT display terminal, an LCD display terminal, or any suitable display device. With these input/output devices, a user can perform the steps of the methods of the present invention where user-specified values are required.

The data processor 301 may also be coupled through the bus 302 to a network interface 308 that provides bidirectional access to the video surveillance system network 215, such as an intranet and/or the internet. In various embodiments of the present invention, both system implementation and management, as well as surveillance activities can be performed through the display 307 and input devices 306.

In general, these teachings may be implemented using at least one software program running on a personal computer, a server, a microcomputer, a mainframe computer, a portable computer, an embedded computer, or by any suitable type of programmable data processor 301 operating in combination with video surveillance system management application 228 and video surveillance application 230 residing on the electronic database system 200. A video surveillance system technician can also manage the video surveillance system modules 210 comprising at least one video surveillance camera cluster 211 through the display 305 and input device 306.

One of the advantages of the present invention is the ease in which a distributed video surveillance system can be quickly and easily constructed. This is the result of the modular concept embodied in the system, reflected in FIGS. 3-5, wherein the firmware and software for establishing and managing the video surveillance system are designed to operate with video surveillance cameras having a high degree of programmability and video analysis engines providing a wide range of video analytics.

The foregoing description generally concerned the architecture of a video surveillance system operating in accordance with the methods and apparatus of the present invention. The following description will concern establishment of video surveillance activities—particularly the establishment of automatic video analysis operations—and the handling of data created during video surveillance operations.

For example, FIG. 6 depicts a block diagram illustrating the steps of a method for establishing video analysis operations and data distribution in a distributed video surveillance system operating in accordance with the present invention. In particular, the first step 410 comprises selecting a view of a video surveillance camera. Next, at step 420 the video surveillance analyst selects a video analysis criterion to be applied to video gathered by the view of the video surveillance camera. Then, at step 430, the video surveillance analyst selects an information forwarding criterion, related to the video analysis criterion selected at step 420, which governs when and what categories of data generated by the video analysis engine 214 and, in some instances the database server 220, will be forwarded to the video surveillance analyst. Then, the video analysis engine 214 performs video analysis on video gathered by the video surveillance camera at step 440, and transmits a data stream created by the video analysis operations to the database server 222 of the electronic database system 220 for storage at step 450. Next, at step 460, the database server 220 applies the information forwarding criterion selected by the video surveillance analyst to the data stream received from the video analysis engine 214, and selects information meeting the information forwarding criterion established by the video surveillance analyst. Information meeting the criterion is then forwarded to the video surveillance analyst at step 470.

FIGS. 7A and 7B depict various video analysis criteria and information forwarding criteria available for selection by video surveillance analysts performing video surveillance activities aided by a video surveillance system operating in accordance with the present invention. In particular, available video analysis criteria listed in FIG. 7A are object detection 481; object tracking 482; object classification 483; activity detection 484; activity monitoring 485; and activity analysis 486. Available information forwarding criteria are depicted in FIG. 7B and comprise a speeding alert 491; an accident alert 492; a vehicle appearance alert 493; a loitering alert 494; and a view monitor 495 for forwarding general information derived from monitoring a particular view of a video surveillance camera. Many other video analysis criteria and information forwarding criteria can be selected for use in a video surveillance system operating in accordance with the present invention. FIGS. 7A and 7B depict the modularity of the present invention at the video analysis and alert levels. In particular, a video surveillance analyst can easily reprogram a video surveillance system to accommodate new threat environments by adding new video analysis applications and alert criteria.

The immediately preceding description concerned the selection of video analysis activities and general information distribution details of a system operating in accordance with the present invention. The following description will concern details associated with data identification and storage in a distributed video surveillance system operating in accordance with the present invention.

In general, in video surveillance systems operating in accordance with the present invention, video gathered by a particular programmed view of a video surveillance camera is continuously analyzed by a video analysis engine 214 of the video surveillance system to generate tracks, and the data derived from the analysis is then transmitted to the electronic database system 220 for storage in the database server 222. Each track is comprised of track data and captures the activity of an object inside the field of view of the video surveillance camera. The track data comprises at least a pointer to a video segment containing the object, and can further comprise in various embodiments location data, trajectory data, motion data, visual features (for example, color texture and shape) and object class or identity. In addition to pointers to video segments, the track data can also comprise digital video clips, keyframes, foreground models and background models. FIG. 8 depicts one possible data model for an electronic video surveillance system operating in accordance with the present invention. In the particular data model depicted in FIG. 8, track data 510 can be comprised of four parts: video media 512; background information 514; object information 516 and track summary information 518.

In one alternate embodiment of the present invention, primitive tracks output from a video analysis engine 214 are represented as XML documents. In this particular alternate embodiment there are three types of XML documents: track summary documents, track data documents and background image documents. FIG. 9 depicts an example of a track summary document 520; FIG. 10 depicts an example of a track data document 530; and FIG. 11 depicts an example of a background image document 540. The track summary document 520 depicted in FIG. 9 is structured as follows:

<?xml version=“1.0” encoding=“UTF-8”?> <Tracks> <TrackSummary> . . . </TrackSummary> * * * <TrackSummary> . . . </TrackSummary> </Tracks> This document type allows multiple occurrences of <TrackSummary> so that a video analysis engine 214 can send an optimal number of track summary documents to the database server 222 of the electronic database system in dependence on load conditions and the urgency of real time data delivery. Each track summary document 520 is associated with a particular video surveillance camera view that is identified through use of a globally-unique identifier.

Track data documents 530 each contain a root tag <TrackData> . . . </TrackData> and are structured as follows:

<?xml version=“1.0” encoding=“UTF-8”?> <TrackData> <TrackDataFragment> . . . </TrackDataFragment> * * * <TrackDataFragment> . . . </TrackDataFragment>

Download full PDF for full patent description/claims.




You can also Monitor Keywords and Search for tracking patents relating to this Video analysis, archiving and alerting methods and apparatus for a distributed, modular and extensible video surveillance system patent application.

Patent Applications in related categories:

20130113935 - Active vision system with subliminally steered and modulated lighting - An active vision system includes an image capture device for capturing images in a region exterior of a vehicle and a headlamp control unit for controlling a vehicle headlamp beam for illuminating an environment exterior of a vehicle. The headlamp control unit is configured to selectively illuminate between making a ...

20130113937 - Entry control point device, system and method - The present invention provides an entry control device, system and method which can recognize a vehicle by its undercarriage image only. The present invention compares a new vehicle undercarriage image with those stored in a reference database to facilitate the automatic search of a vehicle's undercarriage. The present invention can ...

20130113936 - Method and system for managing a parking lot based on intelligent imaging - To manage a plurality of parking spaces, one or more images are acquired, with each parking space appearing in at least one image. Periodically acquired images of occupancy and identity are used in directing a customer to a parked vehicle. Periodically acquired images of just occupancy are used in controlling ...


###


Other recent patent applications listed under the agent :