WO2003065180A2

WO2003065180A2 - System and method for creating a distributed network architecture

Info

Publication number: WO2003065180A2
Application number: PCT/US2003/003251
Authority: WO
Inventors: John Fairweather
Original assignee: John Fairweather
Priority date: 2002-02-01
Filing date: 2003-02-03
Publication date: 2003-08-07
Also published as: WO2003065177A3; WO2004002044A2; AU2003225542A1; US20080016503A1; US7533069B2; WO2004002044A3; WO2003065634A2; US7369984B2; AU2003214975A1; US20030171911A1; WO2003065252A1; WO2003065171A3; US7103749B2; US8099722B2; AU2003269798A1; AU2003269798A8; US7685083B2; US20030200531A1; AU2003217312A1; WO2003065634A3

Abstract

A large scale distributed multimedia system comprising the following: I) one or more server(s), wherein each server includes a main server thread (210), one or more threads capable of monitoring incoming data feeds (221) and instantiating the resultant data into the system (monitoring threads), one or more threads capable of handling incoming data requests (211) (request threads), and cache memory (250); ii) a mass storage system (MSS) (250), wherein the MSS stores information concerning the arrangement of one or more server(s) and is capable of controlling one or more robotic autoloader systems (robots); iii) a types system for defining data types at a binary level and for associating those data types with one or more server(s) such that the mapping maps data types with the servers that must be addressed to obtain the corresponding data; and iv) a query system for executing data queries on servers mapped to the data type being queried. Additional features are also supported including registration on one or more servers of customized commands and functions, and input and output folders for transmitting data to or from data storage.

Description

SYSTEM AND METHOD FOR CREATING A DISTRIBUTED NETWORK

ARCHITECTURE ^' '

INVENTOR: JOHN FAIRWEATHER

BACKGROUND OF THE INVENTION

Current solutions to solving the distributed information problem fall into two main classes: the client server approach and the peer-to-peer approach. In a client/server system, there is generally' a large powerful central server, usually a relational database, to which a set of clients talks in order to fetch, update, and query information. Such a network architecture can be viewed as a star with the clients forming the edges of the star and the large central server the center. The current state of the art in this field adds limited ability to distribute the central database to multiple server machines with • perhaps the most sophisticated form being the "Information Warehouse" or "Corporate Information Factory" (CIF) approach described in the book "Corporate Information Factory" by W.H Inmon, Claudia Imhoff, and Ryan Sousa. Unfortunately, the CIF approach falls far short of what is necessary to handle more sophisticated systems such as those required for intelligence purposes, for example.

The peer-to-peer approach overcomes many of the limitations of the centralized system by allowing many machines on a network to cooperate as peers, however it does this by removing the concept of the specialized server, which limits its applicability in the area of intelligence systems, , where the need for powerful distributed clusters of machines operating as a single logical 'server' for the purposes of processing an incoming 'feed' remains. Furthermore, neither approach addresses the needs of multimedia data and the consequent storage and 'streaming' demands that it places on the server architecture. Once the purpose of a system is broadened to acquisition of unstructured, non- tagged, time-variant, multimedia information (much of which is designed specifically to prevent easy capture and normalization by non-recipient systems), a totally different approach is required. In this arena, many entrenched notions f information science and database methodology must be discarded to permit the problem to be addressed. We shall call systems that attempt to address this level of problem, 'Unconstrained Systems' (UCS). An unconstrained system is one in which the source(s) of data have no explicit or implicit knowledge of, or interest in, facilitating the .capture and subsequent processing of that data by the system. " .'

What is needed, then, is an architecture that embodies concepts from both the client/server and the peer-to-peer approach, but which is modified to reflect the unique needs of a distributed multimedia intelligence system. SUMMARY OF INVENTION

The present invention provides a network architecture that embodies the best of both of these approaches while still providing robust support for multimedia distribution. The invention is comprised of the following components:

a) A customizable (preferably via registered plug-ins) multi-threaded and distributed server implementation containing a main server thread 210, one or more built-in threads for monitoring incoming 'feeds' and instantiating the resultant data into the system, and one or more threads for handling incoming client requests. This server implementation can be extended in a hierarchical manner so that each server has a 'drone' server (to any number of levels) which transparently operate as part of the logical server cluster and which are tasked by the main server machine. All communication between clients, servers, and drones occurs via a standard protocol, such as TCP/IP, so that the logical server cluster, including associated mass storage, can be physically distributed over a wide area.

b) A^" tightly integrated mass storage (MSS) framework that is cognizant of the arrangement . of server^'machines in the cluster and the storage devices to hich they are attached, and

.which is capable of controlling one or more possibly heterogeneous connected robotic autoloader systems in order to ensure that whenever any machine of the cluster requires information stored on a robot, the media that contains the information is automatically mounted into a drive that is connected to that machine. In the preferred embodiment, automatic creation of archive media and its migration from cache to robotic storage is tightly integrated with the server architecture, and the mass storage system may be extended to support new mass storage devices by defining device drivers that map the - -- - .- - logical operations performed by -the MSS- onto the physical commands necessary to implement them for a given device.

c) A standardized framework for defining data types at a binary level and for associating those data types with one or more server clusters on the network such that the mapping between data types and the servers that must be addressed to obtain the corresponding data can be made automatically from anywhere in the environment.

d) A standardized framework for defining and executing queries on servers, for distributing portions of those queries to the servers involved, and for reassembling the results of such distributed queries into a unified "hit list" for the client, including the ability to save and execute these queries on a continuous basis against any new information acquired from the feed. A number of extentions.and enhancements are also disclosed but given this architecture; any number of additional improvements and permutations would be obvious to those skilled in the art. For example, the systen could include an 'output' folder into- which the multimedia portion of any data acquired by the server is placed and which serves as the framework within which the server builds up batches of information to be migrated to the mass storage subsystem as appropriate. .. _, , . . ,^" -

. BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 illustrates a basic configuration of an intelligence system.

Figure 2 illustrates a sample structure of a server in the mass storage system architecture (MSS) of the present invention. ^■ ; • _; . •

Figure 3 shows a sample screen shot illustrating a typical set of server windows.'

Figure 4 illustrates a' sample image showing the results of clicking on the button.

Figure 5 illustrates a sample client server status window.

Figure 6 illustrates a sample embodiment of a master server (in this case, a video server) and a cluster of drone machines. - ; •

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

The descriptions given below may refer to a number of other key technologies and concepts, with which the reader is assumed to be familiar, and will be helpful to fully appreciate the material presented herein. These various building-block technologies have been previously described in the followmg patent applications (which have been expressly incorporated herein):

1) Appendix 1 - Types Patent : ^• ' ^■ . ^•

2) -Appendix 2 - Collections Patent

3) Appendix 3 - Ontology Patent ^{■ ■}

4) Appendix 4 - MitoMine Patent ^' • , .

Referring now to Figure 1, a basic configuration of an intelligence system is illustrated.

Digital data of diverse types flows through the (distributed) intake pipe 110 and some small quantity is extracted, normalized, and^' transferred into the system environment 120 and persistent storage 130.

Once in the environment 120, the data is available for analysis and intelligence purposes. Any intercepted data that is not sampled as it passes the environment intake port, is lost.

Because of the vast amounts of data that will be acquired and stored, a layered mass storage system (MSS) architecture is provided wherein data initially arrives in a (distributed) cache and is then automatically and transparently migrated onto deep storage media^'. In the preferred embodiment, the ability to interface to, and control, heterogeneous robotic mass storage farms is provided.^' In such a system, all data remains on-line or, if archived data is requested, the system must be able to cause the appropriate robot to load it. Because the migration, media, and retrieval process is dependant on not only the data type involved but also on the physical distribution of the servers, as well as other factors, the MSS architecture is intimately tied to the server framework provided by the UCS. The system also advantageously uses the fact that, in most cases, the number of accesses to a given datum tends to follow an exponential decay with its age. Thus recent data, which is accessed far more heavily, will often reside in the cache. Archival data, on the other hand, migrates over time to the robot farm. Because older data may become the subject of intense interest, however, the MSS architecture also transparently handles the temporary migration of this data back into cache.

. The environment also .supports the ability to customize migration strategies on a per-server basis to support other approaches, perhaps based on frequency of access, item content, or other considerations. Because on-line storage is costly, the system has been adapted to use and access low cost random-access media (such as CDs and DVDs) and is also capable of smoothly migrating to newer, denser, media as it becomes available. Manual media loading by an operator is also provided transparently by the system when necessary (such as when truly massive amounts of data or being - requested). The system provides this functionality by routing media load requests to an operator station(s) and can guide to the operator through the loading (and un-loading) sequence. It is anticipated access time increases as data moves deeper and deeper into the archive, however, the primary goal of this system is to permit transparent access to data no matter where it is stored. Again, in the preferred embodiment, robots are loaded with blank media, connected and configured, and then left alone to automatically become part of the archive over time. ' .

Given the scale of the problem, in the preferred embodiment, even individual servers are implemented as distributed clusters. The environment also provides extensive support for the reconfiguration of any system parameter that might change.

^• \ ^'

Before going further, it is important to understand what is meant by a "server" and a "client" in such a system. In conventional client/server architectures, a server is essentially a huge repository for storing, searching, and retrieving data. Clients tend to be applications or veneers that access or supply server data in order to implement the required system functionality. In this architecture, servers must sample from the torrent o 'data going though the (virtual) intake pipe. Thus it is clear that, unlike the standard model, the servers in this system will automatically create and source new normalized data gleaned from the mtake pipe and then exsLrnine that data to see if it may be of interest to one or more users. For these reaons, every server has a built in client capable of sampling data in the pipe and instantiating it into the server and the rest of persistent storage as necessary. Thus, the- present system discards use of the standard 'server' and instead uses server-client pair(s).

In the preferred embodiment, since each server will specialize in a different kind of multimedia or ontological data, and because the handling of each and every multimedia type cannot be defined beforehand, the basic behaviors of a server (e.g., talking to a client, access to storage, etc.) are provided by the architecture. In the event that it is desirable to customize server behaviors, the server calls a^'plug-in API that allόws^'system programmers to define these behaviors. For example,^{' "*} certain specialized servers may have to interface directly to legacy or specialized external systems and will have to seamlessly (and invisibly) utilize the capabilities of those external systems while still providing behaviors and an interface to the rest of 'the environment. An example of such an external system might be a face, voice, or fingerprint recognition system. Furthermore, new servers may be brought on line to the system at any time and must be able to be found and used by^' the rest of the system as they are added. While this discussion has referenced "servers" throughout, there is no reason (and the system supports) use of a 'client' machine that can declare its intent to 'serve' data into the environment. Indeed, in a large community, of analysts, this ability is essential if analysts are to be able to build on and reference the work of others. Thus every client is also potentially a server. The only remaining distinction between a mostly-server and a mostly-client is that a server tends to source a lot more data on an on-going basis than does a client. Finally, the present architecture permits application code mnning within the system to remain unaware of the existence of such things as a relational database or servers in general thereby permitting many "general utility"- applications. As this description makes clear, this UCS architecture is more like a peer-to-peer network than it is a classic client server model^". • . _ •

Referring now to Figure 2, a diagram illustrating a sample structure of a server in the mass storage system architecture (MSS) of the present mvention is shown. The construction of a single machine server within the architecture of this invention will first be described and then this approach will be generalized to the distributed case. The server itself consists of three types of processes (threads), the "Main server thread 210" 210, the "Favorite Flunky" 220, and one or more standard^' flunkies 230, 235. The main server thread 210 is responsible for receiving 211 and routing 212 client requests 205 and otherwise coordinating the activities of all processes within the server. The favorite flunky 220 is primarily responsible for monitoring the stream of data arriving from the data feed 221, interpreting its contents, and writing 222 the data into server storage 240, 250 where it may be accessed 241, 251 by other flunkies 235 in response to client requests 211. In the preferred embodiment, the standard flunky processes 230, 235 are created on an "as needed" basis, and are responsible for handling client requests (as routed 212 by the main server thread 210), and transmitting 231, 236 the results back to the client processes 205. The maximum number of standard flunkies 230, 235 within a server will thus be equal to the maximum number of simultaneous client requests 205 that the server has experienced because the main server thread 210 will only create a new flunky thread when all currently available flunky threads 230, 235 are already busy.- When a' client request 205 ' completes, the flunky thread 230, 235 responsible for handling it is entered into a list (not shown) of available threads so that it can be re-tasked by the main server thread 210 when the next client request 205 arrives. The favorite flunky 220 is also utilized by the main server thread 210 to accomplish various other housekeeping or batch tasks in order to insure that the main server thread 210 remains responsive to new client requests 205 coming in. •There is therefore communication 213 and close' ^l - ' coordination between the activity of the favorite flunky 220 and the needs of the main server thread 210. It is for this same reason that the main server thread 210 would preferably pass off all client requests 205 to the standard flunkies 230, 235 since the main server thread 210 cannot know how long the processing of the client request 205 will take.

^' In the preferred embodiment, the server package provides support for up to 5. distinct directories (not all of which may be required for a particular server) that may be used by servers (or • registered server plug-ins) for distinct purposes as follows: • >

Input Folder - This is the directory where incoming multimedia data arrives from the feed (or elsewhere). The input folder may be hierarchical (i.e., contain folders within it to any level), and when any item is moved from the input folder to the output folder, the

SUBSTITUTE S EET corresponding portion of the folder hierarchy will be replicated there thus allowing multimedia data to be organized based either on feed(s) or on any other system chosen by the client processes (see API below). The favorite flunky's primary task is to monitor the input folder for any sign of change or new files, and when such occurs, to process whatever is new as determined by the registered plug-ins.

Output Folder - The output folder is the place where all the multimedia data arriving'at the input folder is moved once it has been ingested by the favorite flunky 220. Its structure may be hierarchical as determined by the input folder structure. It is within the output folder, which is essentially a multimedia cache 250 in the case of servers witn associated robotic mass storage, where the various MSS media 'chunk' images are built up prior to being sent to' be moved to (or otherwise stored in) mass storage.

Aliases Folder - The aliases folder provides the ability for a server to 'pull' information from sources or feeds on other machines, rather than the more conventional 'push' feed architecture provided by the input folder. An alias may be created and placed into the aliases folder (either manually or through the API) which will cause the favorite flunky 220 to remotely mount the corresponding disk and directory from another machine on the network and, if anything new is found therein, to copy it over to the input folder and optionally delete the original. This means, for example, that users can drop files into a directory in their local machine and they will be fetched and processed in a scheduled manner by the server using the alias mechanism. Like the input and output folders, this folder may be hierarchical, and any hierarchy will be reflected as the data is moved to subsequent folders of the server during processing.

Rejects Folder -'If an error occurs during the process of data in the input folder by the favorite flunky 220, the data file(s), instead of being moved to the output folder! will^" be moved to the reject folder. The purpose is to allow system operators to examine such rejected material and determine why it was rejected. If the reason for the rejection can be corrected (perhaps by altering the data mining script used for injestion), then the reject files can be dragged back into the input folder where they will then be processed correctly thus avoiding data loss. ^' •

Collections Folder - This folder contains the extracted ontological data (see Ontology Patent), which is referred to in a server context as the 'descriptors', for the items extracted from the feed. This folder contains collections often organized in a directory hierarchy that matches the system ontology. These collections are referred to in a server context as the server database 240. The server performs many functions; especially in terms of querying, by perforating operations on the database 240. For this reason, it is only necessary to retrieve, material from mass storage when the full multimedia content is requested by the client (e.g., movie playback). For certain server types that do not relate directly to the onto logy,, a ' - . simplified database based on the ET StringList type may optionally be used.

For further clarification, Appendi -A provides sample code illustrating the processing of mcoming events and performance of scheduled tasks by the main server thread 210. Appendix A also provides sample code for the .command receipt and response process performed by the favorite flunky 220. ^■"' . ^' . v - ^{' ' • '} . ^{• '} ' '

Referring now to Figure 3, a sample' screen shot illustrating^'a typical set of server windows 310, 320, 330, 340, 350, 360, 370 is shown. This figure shows a typical set of server windows 310, 320, 330, 340, 350, 360, 370 running on a single machine (although servers would often run on different machines). ^"In the preferred embodiment, all servers created in this framework would ■ essentially look the same, the only difference(s) between them being the nature of the plug-ins that have been registered on the server in order to customize the server behavior for the data type concerned. " • ■• - _. . -

The server window 310 has been expanded to show the contents of a typical server activity log 311 and also indicates the use of the 'picon' feature, in this case the last image fetched from the Image server 310 was of the flag of Afganistan so this flag is displayed in the picon area 312. Other servers may display different things in this area, depending on the nature of the "picon maker" plug-in registered. In this server window, a button 313 is provided such that clicking on this button 313 causes the server's maintenance window to be displayed.

. Referring now to Figure 4,^' a sample image showing the results of clicking on the button 313 is shown. -The pop-up menu 411 of the maintenance. window 410 allows the user, to choose ^'from one of a number of registered logical 'pages! containing information and allowing operations that relate to a particular class of maintenance actions on the server 310. The maintenance window API (see below) allows the registration of a set of maintenance 'buttons" with any defined page. In the illustrated case, the page mapped to the pop-up menu 411 is "Server Items." In the preferred embodiment, a text area 412 is provided which provides description/help information for the button whenever an input device, such as a mouse, is over the button concerned. In the illustrated embodiment, the server maintenance window 410 also includes a "server items" area^' 413. In this case, the server items area 413 provides unique item ID along with the time the item was captured by the system and the path in the output folder where the file containing the multimedia content of the item has been placed. For example, the list in this case indicates that the images are from an output folder titled "WorldFactBook" and that they are part of archive batch "@0001" (see below for details). If the items had already been archived to robotic storage, the path could be modified to reflect this fact. Three maintenance pages are predefined by the server implementation itself: ' „, - ' . -

Server Items - This page preferably allows examination of, and operations on, any item in the server. . ;• " . '

Archive items - This page preferably permits examination of the state of any archive 'batch' and operations thereon. The process of building up a batchj sending it to be burned, moving it to robotic storage, and then purging the original from the output folder, may be quite complex in some instances, and a number of maintenance actions may be required should anything go wrong in this sequence. ^' .

Archive Devices - This page preferably allows direct control over the robotic storage devices attached to the server. For example one could move media within and between robots, initiate re-calibration of the robot contents, etc. ^"Again the number of possibilities here is quite large and varies from one robot type to another. For this reason, the present invention provides an open-ended registry for robotic maintenance actions. The need for tight integration of robotic storage activity and> the state information held in the server is clear. - '

The dispatching of incoming requests from clients to the standard flunkies that will handle , them occurs in the communications handler associated with the main server thread 210, that is it is asynchronous to thread operation and occurs essentially at interrupt level. There are a huge number of possible client commands that must be routed in this manner (see API below). The processing performed in this context for any given command is essentially as follows: ^{' •}

case Comman Type : ip = server status record . fp = SV_FindFreeFlunky ( ip) / . / / find/make flunky to

Handle cmd ' copy client parameters to flunky buf sr area ^• if ( command was proxy roυ ted )

, • ' strip of proxy routing tag in flunky buffer area issue command to flunky break ; - ;- _ ,

Where : ^' _ int32 SV_FindFreeFlunky ' ( // find/create a free flunky

ET_CSStatePtr ip ^' // 10 : Pointer to IP

Server Status - ^' ^•

) . • // R: Index for the flunky -selected ,

{ for ( i = 0 ; i < max simultane ous users ; i++ )

{ // scan all our flunky records fp = &flunky [i] if ( flunky is free )

{ ^' '

^• return i; } else if ( !fp->flags ) // uninitialized record, use it

{ ^' fp = create a new. flunky and flunky buffer area return i; ' -

} ^•

} ' ^{' '} - log "too many simultaneous users!" error return error ^"

A sample list of the major commands that are preferably supported by the architecture are as follows:

ttdefine kCStrigger 'csTR' // Trigger an IP (server->client)

#define kCSabort OsAB" // aborted server command (server-

>client) .

#define kCSoneltem 'csll¹ // data for single item requested

(server- >client) .'

#define kCSitemChunk 'csln' // data for chunk of items (server-

>client) •

#define kCSitemResponse 'csRs' // respose data for a command

■(server- >client) - ¹

#define kCSforwardCmd ^■csFW // Forward a client command (server-

>server)

#define kCSAcknowledge 'csAK' // Acknowledge

^define kCSCollectionC d 'csCO' // collection command (client-

>server) -

#define kCSunload •^■ 'CSUI,' // unload an IP (client->server)

#define kCSstart ' csGO ' . // start a server (client->server)

^'. #define _- CSstop - . ^■csOF' -// stop a server (client->server) ^•

#define kCSload 'CSLD' // load an IP (client->server)

#define kCSfetch . ^■csFT' // fetch an IP (client->server) ffdefine kCSkill •csK ' // kill all IPs for this machine

.(client - >server)

#define kCSuserCmd • σsUS ' // user defined command (client-

>server) ,

#define kCSgetPreviewList •csPL' _,// get an IP 'item preview list

(client- >server) -

#define kCSwakeServer 'CSW ' // wake server to scan inputs

(client- >server) ,

#define kCSgetFiieBasedltem 'CSFI- // get a file-based item (client-

>server)

#define kCSputFileBasedltem 'CSFP' // put a file-based item to input

(client- >server)

#define kCSarchiveCmd 'csAC // archive user defined command

(client- >server) .

#define kCSfetchChunkID •csFC // fetch archive chunk ID for ID

(client - server) #defιne kCSgetServerStatus 'csST' // Fetch server status (client-

_>server)

#defme kCSgetNextSequencelD ' csNI ' // Get next ID in server sequence

(client- >server) . ftdefine kCSisServerRunnmg ¹ csRS ' ^"// Check if -the server is running

(client- >server) ttdefine kCSdeclareEvent 'csDE' // Declare an event has occured

(client- >server) ,

#defme kCSacquireDBlock 'csLK' // acquire a DB lock (drone->master)

#defιne kCSreleaseDBlock 'csUK' // release a DB lock (drone->master)

#define kCSsendDroneStatus ¹ csDS ' // send drone status to master

(drone->master)

#defme kCSdoesIDexist ^■csIX' // Does an item ID exist (client-

>server) idefme kCScueltemFile 'CsQF' // Cue Item File (clιent->server)

#defme kCSCountServer ems ^•csCI // Count items in the server DB

(client- >server) _' *

#def e kCSAddressToDrone ^■csAD' // Convert IP address to drone ID

(client- >server) • • fdefme kCSDroneτoAddress ^•csDA' ^"// Convert drone ID to IP address

(client- >server) '

#define kCSStandardQuery 'csSQ' // Standard Query (MitoQuest™)

(client- >server) -

#defme kCSClientStatusMessage'csMG' // Display client status message

(client- >server)

Two of the commands above deserve further discussion here. The 'kCSCollectionCmd' is a collection command that enables all servers in this architecture to inherently support the full suite of server-based collection operations as described in the Collections Patent. This means that all data residmg in the server collections is available for transparent use by any client without the need to be aware of the server communications process.

The 'kCSuserCmd' allows the open-ended definition of server-specific custom commands ('kCSarchiveCmd' performs the same function for archive related commands). This mechanism enables the customization of the basic set of server operations m order to support the features and operations that are specific to any particular data or server type. This functionality is preferably ^{" *} invoked in the client via CL_CallCustomFuncQ - see API description below. In the embodiment descπbed below, this capability is used by registering a custom function handler (descnbed below) that packs up the parameters passed to the function into a command block. This command block is sent to the server and the results are "un-packed" by the server as required by the caller. The custom function handler on the client side is registered (via CL_SpecifyCallBack) using the selector 'kFnCustomFuncHdlr '. On the server side, the corresponding server plug-in is registered using 'kFnCustomCmdHdlr'. Each of these are further described below. Within the custom command handler in the server, the pseudo code logic is essentially as follows:

static EngErr myServerSideFn ( // custom command handler

12

SUBSTITUT long aϋse Command, // I: Command type void "buffer, . . // I:The client supplied data buffer charPtr anldentifyingName, // I: identifying Text string

Boolean wantReply ' // I: RUE if client ,- wants a reply ^• // R:2ero for success, else error # err - 0 ,^{■ ■} cmd = . (my command record*) buffer; ^• switch ( aUserCommand ) . case command type 1: . extract command parameters from cmd ^' perform the custom command if ( .wantReply ) ^• • ^■ . ^{' •}

{ ^' . - . .. ^{■ " ■ '} rply = (my reply record*) allocate buffer sis = sizeof (allocated buffer) fill in rply record with results

} ^" - : break; „ '. • case command type 2: ... etc. default: - report "unknown command error" break;

if ( rply ) ..

{

SV_SetReply(sp, ... , rply, siz) ; // send reply back to the client

Dispose of (rply);

} return err;

} " •

Thus, in addition to all the logical plug-ins supported for the basic server functionality, this invention allows full control and extension of the functionality of the server itself (in addition to the client). This is in marked contrast to conventional client/server architectures that do not support this level of server customization.

The following is a partial list of the standard logical plug-ins that^'are supported by the client server architecture of this invention and a brief description of their purpose. Some callbacks are required but most are optional depending on the nature of a particular server and data type.^' These logical callbacks are sufficient to implement most kinds of multimedia servers without the need to resort to custom functions. In general, for any given server type, only a very small number of the possible callbacks defined below will- be registered, since the server infrastructure provides default behaviors that in most cases perform what is necessary given the standardized environment in which the system operates. Optional callbacks are marked with a '*' below: ^" , ^■ ■

SELECTOR FN. TYPEDEF NAME ■• DESCRIPTION kFnStatus ET_IPserverStatus aStatusFn* ^■ get srvr status kFnDBrecIrut kFnDBrecIrut ET_IPdbsReclnιtTerm aDBSrecIrutFn* init db descnptor record kFnDBrecTerm ET_IPdbsReclratTeπn aDBSrecTermFn* •, clean-up/ erm DBS record kFnFileDelete ET_IPfileDeleter aFileDeleteFn* delete file fr input fldr FnIDgetter ' " - ^" ET_DBSidGetter aDBSidGetterFn get unique ID fr db recrd kFnDBSattacher ET_DBSattacher aDBSattachFn attach data to db record kFnDBSadder ET_IPdbsAdder aDBadderFn* add item to the database kFnFileProcessor ETJPfileProcesser aFileProcesserFn* process file in input fldr kFnFUeTypeChecker ETJsItemFileFn aFileTypeCheckFn* . check if file of req type kFnCustomCmdHdlr ET_CustomCmdFn aCustomCmdFn* Srvr call on rcpt of cmnd kFnCustomFuncHdlr ET_CustomFuncFn aCustomFuncFn* Clnt call process cust fhs kFnPiconMaker ET_HandleToPicon aPiconMakerFn* . convert item data handle kFnDBSfetcher ET_DBSfetcher aDBSfetchFn get ET_DBInvokeRec fields kFnExprEvaluator ET_ExprEvaluate anExprEvalFn* evaluate an IP expression . ^kFnFUePathMover ET_DBSFilemover aDBSFileUpdateFn* . . update data item file path kFnArc rveGetter ET_ArchiveGetter anArchiveGetterFn* get archv creat fldr path FnArchiveCopier ET_Arc iveCopier anArchiveCopierFn* copy file to an archive kFnArchiveS tarter ET_ArchiveStarter , anArc iveStartFn* kick-off archive procss FnArc iveEiider ET_ArchiveEnder anArc iveEndFn* • clean up after archiv proc kFnArchivePoUer ET_ArchivePoller ^" anArcbαvePoUFn* archiv process complete? kFnNetCopyResponder ET_NetCopyResponder aNέtCopyRespFn* ' srvr copy file over nt r . kFnNetCopyStarter ET_NetCopyStarter aNeCopyStartFn* init nt k file cpy at clnt kFn etCopyldler ET_NetCopyldler aNetCopyldleFn* sust ntwk file cpy at clnt kFnNetCopyEnder ET_NetCopyEnder aNetCopyEndFn* clean up aftr ntwork copy kFnOpAliasResolver ET_AlιasResoIver anAliasResoIverFn* ^' reslve alias' in outpt pth kFnOpAIiasModifier ET_AliasModifier anAliasModiflerFn* mod/ cust alias' on archive kFtiArchiveRecIniter ET_ArchiveRecIniter anArchiveRecIniterFn* . init/ archiv db record kFnCustomArchiveCmdHdlr ET_CustomCmdFn aCustomArc iveCmdFn* cust arch comnd srvr call ^"kFnCustomArchiveFuncHdlr ET_CustomFuncFn aCustomArchiveFuncFn* process cust arch fns dt ' kFnltemlnfoGetter ETJtemlnfόGetter anltemlnfoGetterFn* get info related to item kFnArchivelnfoGetter ETJtemlnfoGetter anArchivelnfoGetterFn* get arch chnk/dev info^* kFnArchiveLUNGetter ET_ArdιiveLUNGetter anArchiveLUNGetterFn* get list of archive LUNs kFnRepartitioπNfyer ET_RepartitionNfyer aRepartitionN yerFn* notify repartn begin/end kFnDBRecGutsClone ET_DBRecGutsCloneFn aDBRecGutsαonerFn* clone non-flat desc attchs fcFnServerldler ET_ServerIdler aServerIdleFn*_ call when server is idle . ; kFnFilePutRouter^* ET_FilePutRouter aFilePutRouterFn* route input fls to drones kFnFileGetRouter ET_FUeGetRoutβr aFUeGetRouterFn* rte don't care fl fetches kFnBusyCounter ET_BusyCounter aBusyCounterFn* bus/#clnts load in server kFnQueryHandler ET_Query Handler aQueryHandlerFn* handle standard queries kFnClienCWiper ΞT_ClientWιper aClient ipβFn* call when clnts strε/stσp kFnStatus - In the preferred embodiment, this function may be used to add additional status information over and above that provided internally by the server environment. For certain specialized servers, additional information may be required in order to fully understand the- state of the server. In addition, this plug-in may utilize additional custom option flags in order to return information that may be used'functionally by a client for special purposes. Normally status information is simply displayed in the Client/Server Status window and allows a user to determine the status of the system servers. - . Referring now to Figure 5, a sample of a client/server status window 500 is shown.

kFnDBrecInit - If additional initialization for a newly allocated (and zeroed) descπptor record is required, this function may be used to accomplish this. An example might be when descriptors sent to the client have a different format to those used internally by the server plug-ins. In such a case, this function would allow those fields that are valid for the client to be set up from the true descriptor.

kFnDBrecTerm - If additional termination or memory disposal is required for a descriptor record over and above simply de-allocating the descriptor memory (for example if the descriptor contains references to other memory allocated), this function may be used to accomplish this in order to avoid memory leaks.

kFnFileDelete - In many cases, the information relating to a particular item coming from a feed may span multiple files. For example, many newswire feeds produce a separate file containing the meta-data relating to the story content which is held in a text file. The file processing function (see below) can access this additional information. When the server has processed the file and wishes to delete it from the input folder, this function (if present) may be called in order to clean up any other files that are associated with the item and which the server itself is not aware of.

kFnlDgetter - This function is used by the server code to extract the unique ID value from a descriptor record. Since descπptor record structure is normally opaque to the server architecture itself, this function is required in order to determine the ID to which a descriptor refers.

kFnDBSattacher - Many descriptor records contain opaque references to the multimedia content that is associated with the item (for example, the image for an image server) and this - multimedia information is packaged up and sent automatically to the client on request. Since descriptor records are opaque to the framework, however, this function is required in order to attach the reference back to the descπptor on the client side. ,

kFnDBSadder - If an external database, unknown to the server architecture itself, is associated with a given server, this routine can be used to cause the necessary information to be written to that external database at the same time that the server itself adds the item to its own database. This allows external databases (such as relational databases) to be kept in synchronization with the activity of the server. This may be critical in cases where the system is being interfaced to other external systems via the external DB. kFnFileProcessor - Once a file^' in the input folder has been identified as being of the correct type for processing by the server, this function may be called in order to actually process the contents of the file in order to extract whatever information is pertinent and make it available to the server. As is well-known to those in the art, the specific processing performed on a file tends to be different for each kind of server and is heavily dependant on the form of the ^'information being processed. ' . - '

kFnFUeTypeChecker - When scanning the input folder for new input, the server may call ^' this function for each file in the folder(s) in order to determine if the file should be processed by the server, ignored, or rejected. In cases where multiple files are required to process a new input, this function would preferably handle this by indicating that the file(s) should be ignored until all required members of the file set have arrived. This behavior is common in ^■ commercial feed situations where the arrival of all files relating to a given item maybe spread over a considerable time.

kFnCustomCmdHdlr - This is the client side of the custom command functionality described above. ,

kFnCustomFuncHdlr- This is the server side of the custom command functionality described above. .. .

kFnPiconMaker - This function can be used to create a 'picon' to be shown in the server . window 312 whenever a particular item is processed or retrieved by the server. The image shown in the Image Server window 310 is one example, however many server types can usefully display such images such as video servers, map servers, etc. _ '

kFnDBSf etcher - When invoking a client side user interface to display/edit an item of a given type, the system may call this function in order to fill out the fields of the invocation record with the necessary information for the item concerned, passing it the preview data obtained from the server. The additional fields required to invoke a handler for different types may vary and, in the preferred embodiment, it is this function's responsibility to ensure that all necessary fields are filled out.

kFnExprEvaluator - This function may be called by the server in order to evaluate a given "interest profile" against any new item that has been, processed. Because the querying functionality available for a particular type may be quite different than that for others (e.g., in ^■ an image server "looks like", in a sound server "sounds like"), this functionality would ideally be registered via this mechanism. In this way, the server, through the MitoQuest™ querying - interface, is able to respond to and evaluate type specific queries, not only as part of interest profile processing but also within the generalized querying framework., , « . . . _^

' ^" kFnFilePathMover - This function may be used to take whatever additional actions (if any) are required whenever the server moves the physical location of a multimedia file from one place to another. , - .. - ^'

^■ ' kFnArchiveGetter - This function can be used to determine the network/file path to which the files of a newly created archive batch should be copied. This function is normally / registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required.

kFnArchiveCopier - This function can.be used to perform the actual copying of archive chunks to the designated path in preparation for archive burning. This function is normally registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required.

kFnArchiveStarter - This function can be used to start burning of an archive chunk. This function is normally registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required. Media burning devices (such as CD-ROMs) frequently require dedicated machines and scripting of commercial software to burn. Also, the burning process may cause the machine involved to lose the ability to communicate for some time. All this logic is supported by the Logical MSS layer via its own registry API (described below).

kFriArchiveEnder - This function can be used to complete the burning of an archive chunk , to handle the transfer of that chunk to the roboticautoloader selected, and to initial flushing of the original data from the output folder cache. This function is normally registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required.

kFnArchivePoUer - This function can be used to poll for archive bum completion. This function is normally registered by the Logical MSS layer and need not be specified unless . ^" custom behavior not supported by the Logical MSS layer is required.

kFnNetCopyResponder - The communications model used by the client/server facility for copying multimedia files across the network to/from clients consists of the client calling a 'copy.starter' function and then entering a loop calling a 'copy idler' until the transfer is complete at which time the 'copy ender' function is called. At the server end, the 'copy

17

TITUTE SHEET RULE 2β responder' function is responsible for transferring the data. Normally all of these functions are pre-defined by the client/server environment and no additional registration is required. For specialized media types (e.g., video), however, the associated server may wish to override the default suite in order to take advantage of other more appropriate techniques (e.g., video streaming).

IcFnNetCopyStarter - See the discussion for 'kFnNetCopyResponder' ■ .

kFπNetCopyldler- See^" me discussion for 'kFnNetCopyResponder' . ^' •

IcFnNetCopyEnder - See the discussion for ' kFnNetCopyResponder'

kFnOpAliasResolver - When a multimedia item has b.een moved to robotic storage, an ' ' 'alias' file is left in its place within the output folder. This alias file contains all information necessary to allow the server to mount the storage if required and copy the file contents to the client as desired. This function is preferably registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required.

kFnOpAliasModifier - This function can be used to modify the alias file associated' with accessing multimedia data held on robotic storage devices. This function is normally registered by the LogicaTMSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required. ^' . • ^' ■.

IcFnArchiveRecIniter - This function can be used to initialize an archive database descriptor record for a given archive batch. This function is preferably registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS^' layer is required. , ^' , ^'

kFnCustomArchiveCmdHdlr - This function provides the server side of the archive custom command capability discussed above. This function is preferably registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required. ^• ■ ^{• ' •} . '

kFnCustom&rchiveFuncHdlr - This function provides the client side of the archive custom command capability discussed above. This function is preferably registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required: ^{' •} • ^{• '}

18

SUBSTITUTE 'SHEET RULE 26 KJb'niteπunloϊjerter - i his function can be used to supply additional textual information relating to a given item in the server (as displayed in the "Server Items" _^maintenance window page). ^" ■ . • ^' ' _, ^" . ^•

kFnArchiveLTJNGetter - This function can be used to obtain a series of lines of text giving the Logical Unit Number (LUN) information for all the archive devices available for a given server. This function is preferably registered by the Logical MSS layer and need not be specified unless custom behavior not supported by the Logical MSS layer is required.

kFnRepartitionNfye^'r - This function can be used to take whatever action is required in ' . response to repartitioning of the server output folder. It is often the case that during the life of a server, the mass storage media in the attached robotic storage may be changed (for example upgrading from CD-ROM to DVD) in order to increase capacity or improve performance. When this occurs, the server output folder would preferably be re-partitioned (initiated from the maintenance window). The re-partitioning process involves the steps of retrieving all the batches from the old robotic media to cache, reconstructing a new batch based on the ^• parameters of the new storage device (which probably involves re-allocating the contents of multiple batches to a new batch set), and then re-burning a new copy of the data and moving it to the new storage device. This function can be used to notify external systems of what is going on in cases where this is required. '

kFnDBRecGutsClone - This function can be used to perform any special processing required when the contents of a database descriptor record is replicated. If the descriptor is 'flat', this is probably. not required, however, in the case where the descriptor contains ' ^' references to other memory, these external allocations must be cloned also and the new descriptor set up to reference the new allocations.

kFnServerldler - Many types of servers require regular idle processes to be run and this function can be used to define the activity to be performed during the^' idling process.

kFnFilePutRouter- When transferririg files from a client to the server for processing, in the case of a distributed server cluster, a default algorithm is used to choose which server drone should be selected to handle the new- task. The default selection criteria is preferably based^' (at least in part) on how busy each of the available drones is at the time. If custom routing is required then this function can be used to achieve that as well. Because all server communication occurs via the IP protocols, servers themselves may be widely distributed physically and may have different network bandwidths available to access them. In such cases, it is often preferable to register¹ custom get and put routers. kFnFileGetRouter -This router function is the converse of the file put router descπbed above. That is it can be used to override the default algoπthm used to select a drone to handle the transfer. For example m a distributed video server, it may be advisable to stream from a machine that is physically closer to the client. Defining these algoπthms requires consideration and examination of the connectivity state of the robotic storage (if present) to determine which machines have access to the actual data involved.

kFnBusyCounter - Preferably, a server determines how many of its flunkies are busy by examining which ones are still m the process of executing a client command. In certain rare circumstances, however, this may not be appropπate and this function can be used to obtam a more accurate measure. An example occurs m the case of a video server, which is usmg an external video strearmng package (e.g., QuickTime Streaming Server) in order to actually stream the video to the clients. In this case, the load on the machine is actually a function of the number and bandwidth of the streams being sent by the streaming package and so a custom 'kFnBusyCounter' would be used to determine this in order that routing of new streaming tasks would go to the most appropπate drone. This is another mechanism that can be used to impact task routing over and above that provided by the routers descπbed above.

kFnQueryHandler - This function can be used to process standard server quenes oπgmatmg from MitoPlex™ (see patent ref. 3).

kFnClientWiper - This function can be used to perform whatever resource de-allocation and cleanup is required whenever a server client quits or first registers. This is only necessary if resources outside those manipulated directly by the server and its plug-ms are involved.

Another opportunity for customizing of the standard server behaviors occurs when the server is first created via the CLJDefineDataTypeO API call. One of the parameters to this function is a 'tweak' record that is defined in the preferred embodiment as follows:

typedef struct ET _IPserverTweakRec II record for tweaking server pararaetfϊrs

{ ιnt32 serverOp ions ,- II logical options for server short serverlconID; II Icon ID for server view short filesPerAliasFetch; II files fetched per alias scan , mt32 NeedProgressBar; II use progress bar for more items

1I.C32 MaxI erns; II max # of items sent mt32 MaxS multaneousUsers; II maximum # of simultaneous users mt32 CheckTime, II Scanning interval for new inputs int32 Loadlncrement ; // time between load updates int32 ^{• •} I emChunkSize; II A of items in chunk sent from server ' ; int32 IPServerTimeout; // Timeout for 'server on" client int32 ' ClientRecvBufferSize; // size of client item receive buffer ,_ int32 ServerRecvBufferSize ; /'/ size of server request buffer . int32 .. IPClientTimeout; // Client timeout on server int32 favFlunkyStackSize; // Stack size_, for favorite . flunky > int32'_| ^' . flunkySt ckSize; // Stack size for other flunkies , uns ong minHeapMemory; // Minimum heap for server to run ^' , • .

OSType startThisSer erBefore // Control server startup order ^•. unsLong • idlerlnterval ; // ticks between idler calls . unsLong textBackupInterval ; //^' -text backup interval .

} ΞT_IPserverTweakRec,-

Where defined server options are:

#define kServerScansInputFld .0x00000010 // automated scanning of input folder . _, ftdefine kServerScansAliases ^{' '} , 0x00000020 // automated capture via alias folder • '

#define kServerAutoS rt 0x00000040 // server should start automatically , ' . '

Sdefine kServerHasTimeHistor ^' . 0x00000080, // server has a time history field

#define kAliasesCanAlsoBeToClumps 0x00000100 // aliases can also be to clump files ^"

#define kSendDesc ithPrev.iew 0x00000200 // send descriptor with _ preview

#define kDontCreateOutputFiles ' 0x00000400 // server does not create output files

Sdefine^'kArchiveOutputFolder ^■ 0x00000800 // archive output folder

#define kServerStaysInFgnd 0x00001000 // don't automatically move to^' background

#define kServerDatalsDistributed -0x00002000 // the server's data is distributed

#defιne kUseDefaultArchiveSuite 0x00004000 // use the -default archiving suite

#define kUseArchiveltemsPage 0x00008000 // use the default archive items page

#define kUseArchiveDevicesPage 0x00010000 // use the default archive devices page ^■ ' ^* -

#defi^ήe kServerlPsHaveTldeMark ^• 0x00020000 // ^'server retroactively evaluates IPs '

#define kServerHasNoDatabase ' 0x00040000 // This server has no database . ' . '

#define kDontOptimize Pexprs •_ 0x00080000 //^• Evaluate every IP expression ^■ . . .' ; . ^"

#define kPullByCommonPrefix 0x00100000 // aliase fetches grouped by prefix

#define kNoAutoWindowShow ' 0x00200000 //"Don" show the server when started ^' By setting up the various fields in the tweak record prior to creating the'server, a wide variety of different server behaviors can be customized to the needs of a particular server type. In a similar manner, the parameters of the archiving subsystem associated with a server can be customized via an archive tweak record. An example follows: ^•

typedef struct ET _ArchiveTweak .

{ unsLong archiveBlockSize; . //. Block size of the archive volume , .

. unsLong archiveChunkSize ; // number of blocks on the archive "volume . , int32 sizeOfArchiveDesc; , // size of archive DB desriptor record

" unsLong mountTimeoutlnTicks ; // disk mount timeout

• char configStringCSTRINGBUFFSIZE] ; . // archive configuration string char droneList [STRINGBUFFSIZE] ; . // list of drone machines

} ET ArchiveTweak '

The descriptions above have referred to the process of creating archive 'batches' for multimedia content and the means whereby those batches are transferred to mass storage. Whenever a server processes an input feed, two types of information result. The first type of information is descriptive information relating to the item itself, when it was captured, where it came from, etc. This information, as discussed above, finishes up in the server database and is the primary means whereby the server can be queried as to its content. Descriptive information like this tends to be relatively small in volume (no more than Gigabytes) and thus is easily handled on the local disk(s) associated with the server. The other type of information is termed multimedia information examples being^' images, video, maps, sounds, etc. This information is generally encoded in specialized formats and requires specialized software to interpret it. In addition, multimedia information tends to be vastly larger than the descriptive information that goes with it, In this case, the server's job is to transparently handle two diverse access and storage requirements of these two types of information so that when presented to clients, they are unified. Because of the unique requirements of multimedia" data it becomes necessary to tightly integrate a sophisticated mass storage system with every stage of server operation and to allow the operation of that mass storage system to be configured on a per-data- type basis. The present invention provides this integration by creating batches of multimedia data that can be archived to mass storage while remaining accessible to the server as a whole. A batch is a set of-multimedia items, possibly distributed throughout the hierarchical server directory, whose total size ^" is just less than the storage size defined for the archival media associated with the robotic storage attached to the server. Possible archival media include hard' disks, CD-ROMs, optical disks, tapes, DVDs, etc. Each of these media types has a fixed storage capacity, known as the chunk size. The process of building up an archive batch in the output folder involves accumulating the total size of all

22

SUBSTITUTL oHEET RULE 26 files that are part of the batch until 'they add up to enough to warrant the batch being moved to an archive media device. When a batch reaches this stage, it is transferred to an archive location, usually the disk of another machine associated with the server cluster, and then burning or writing of the batch is initiated. Assuming the storage media is a CD-ROM, then the burning process would mvolve inserting a blank CD-ROM into a CD burner associated with the burning machine and then scπpting a commercial CD writing program to actually bum the CD with the contents of the information held in the archive image. When burning is complete, the CD-ROM is removed and preferably placed into the robotic storage device associated with the server. At this point, the media is verified and then the original data for the batch is deleted. More specifically, each file so deleted is replaced by an alias to the location of the archive material in the robot/media. This means that when the server subsequently tries to access the file, it encounters an alias instead and automatically initiates loading of the corresponding media in the robot to a drive associated with the server so that it can be accessed and sent to the client.

In the preferred embodiment, the entire archiving operation occurs automatically (without the need for real time human intervention). This means that in the prefened embodiment, the writer(s) associated with a server as well as the readers are connected to a number of different media devices, all of which are within the robot (so that they can be automatically loaded and un-loaded). Most robotic autoloaders allow a variable number of drives to be installed within them and the connection to each of these drives to be made to any of a number of external computers via SCSI, firewire, USB, or any other logical connection permitting structured communications. The computers that form this cluster (other than the main server machine itself -the master) are referred to herein as drones. In large installations, the drones themselves may have drones and this tree can be repeated to any number of levels. To simplify management of the drones, it is often the case that each such computer is running an identical copy of the environment, including a server for the data type involved. Control over the autoloaders themselves is generally provided via a serial connection, a SCSI connection, or more recently via an Internet/network link. Thus an archiving server tends to be a cluster of machines (possibly physically distributed) all associated with one or more (possibly dissimilar) robots each of which is under the direct or indirect control of the master machine of the cluster.

Referring now to Figure 6, a sample embodiment of a master server 610 (in this case, a video server) and a cluster of drone machines 615, 620, 625, 630, 635, 640, 645 is shown. In the preferred embodiment, each drone machine 615, 620, 625, 630, 635, 640, 645 has an associated cache area that can store data. When the server 610 choses a drone which will, for example, stream video to a client 601, 602, 603, not only must it consider which drones 615, 620, 625, 630, 635, 640, 645 are busy and which are not, but also which drones are connected to drives that a robot can control (615, 620, 625, 630, 635) and contain the media for the batch being requested. Additionally, the data required may already be stored in the cache associated with any one of the server's drones and wherever possible access to the^' data in cache is preferable to accessing it from mass storage. The process of streaming a video thus becomes as follows: ^'

1)' Select the drone 615, 620, 625, 630, 635, 640, 645 that will perform the task. , 2) Command the associated robot to mount the disk in a drive connected to the selected drone. ; ^'

3) ' Command the selected drone to begin streaming from the disk (optionally simultaneously . - cacheing). ^'

4) When the stream is done,-- the disk becomes 'free' and can be dismounted by the robot to . make room for another if required. . • ' ^• ' ' •

Similar logical sequences are required for burning media and many other actions that the server needs to perform. The very tight integration between the server state and database(s) and the mass storage implmentation is a key benefit of the present invention as it addresses the kinds of • problems that occur in large scale systems, such as the distributed video server depicted in Figure 6.

The mass storage issues described above are just one aspect of the more general problem of creating distributed server clusters that is addressed by this architecture. Even in the absence of mass^' storage issues, distribution of server functionality across a set of drones may be desirable for performance reasons alone. Consequently in the API descriptions below and in the underlying server implementation, the concept of distribution viathe server-drone mechanism is inherent. In a distributed server, the drone servers perform a similar function to the server flunkies described in the single server discussion above and the logic to implement server distribution across the drones is in many ways similar to that described for flunkies. Each drone server is identical to the single server - described above, the only difference being that clients of the cluster cannot directly initiate cornmunicatio^'ή with a^'drone, they must go^" through the master server in order to have a drone allocated to them. Thereafter, the drone server and the client communicate directly to accomplish the_^ desired task much like the behavior of individual server flunkies. Many of the built-in server commands and functions listed above are associated purely with the business of coordinating activity between drones and the master server. • ^" '

Another aspect provided by the present invention is the use of "Interest^'Profiles". As mentioned at the outset, the present server architecture is continuously acquiring new inputs from one or more feeds and exarnining items extracted from those feeds to see if they are of interest to any of the server clients. In such a case, a client would preferably be notified that a match has been detected so that the client can take whatever action is appropriate. The mechanism for performing this continuous monitoring on behalf of the clients is the interest profile. An interest profile consists of a • standard format MitoQuest™ query (see^'the Ontology^' Patent materials incoφorated herein) that is

• applied to each new item as it is acquired rather than searching on all available data. ^' The logic associated with executing these queries witihin the context of the favorite flunky has been given above. In the preferred embodiment, when clients start up (or when a new interest profile is defined), the client registers all interest profiles/queries defined by the current user with the appropriate - server(s) so that while the server is running it has a list of all interest profiles that must be checked against each new item. While startup and interest profile creation are the preferred way of triggering the registration function, many other events could also be defined to trigger the registration process. In the preferred embodiment, when the processing of a new input iteni completes successfully, the server instructs the favorite flunky to iterate through this list, testing the item just acquired against each query in the list. If a match is found, the server sends a notification to the corresponding client machine indicating that a hit has occurred. On the client machine, the user is notified (either audibly, visibly, or as. otherwise dictated by the client) that such a hit has occurred. By taking the appropriate action in the user interface, the data corresponding to the hit can be immediately displayed. Because there may be commonality between the interest profiles defined by different users, the server framework may also be programmed to include logic for eliminating multiple executions of queries that are wholly or partially identical and this behavior can considerably reduce the load implied by interest profile processing. Since most users may be busy doing something else, and will only access data from the server if it matches their profiles, the processing of interest profiles may represent the majority of the computational load on any given server. Because of the heavy loads created by interest profile(s), there is often a need to distribute the process of one or more interest profiles on any given server over multiple drones so that each drone may independently process a part of the incoming feed and execute the associated interest profiles without the need to involve other machines.

, In addition to registering interest profiles when new items are added to a server, the present invention also permits registration of an "Event", which may be one of many different logical events that relate to the data type involved. For example, users may also register interest profiles on server behavior such as when items are updated, deleted, or even read. This capability has many implications in the monitoring of system usage either for tuning or security purposes.

While it has been stated that the environment being described is in many ways similar to a peer-to-peer architecture, the discussion so far has described the invention purely in terms of servers and clients. In the preferred embodiment, every client machine also has an automatic (preferably invisible) server that is run whenever the client software is'active. This server is available to other machines, including servers, to allow them to query and interact with the client. One of the key behaviors provided by this built-in client -server is support for 'publishing' collections that exist in the client machine (see the Collection Patent mateπals that have been incorporated herein). This built-in server also allows the client to act as a server to any other machine on the network for one or more data ' types. This is identical to the behavior of machines in a peer-to-peer network except that it is implemented in this case as a special case where the client has a built-in server. In the' preferred embodiment, there is in fact no difference whatsoever between the architectural software running in client machines and that running in servers. Indeed, the only difference between the two is caused by system configurations that cause certain rhachines to initiate as servers and others as clients. This exact match between the software running on the client and that ranning on the server in a data mangement framework is unique to this invention and provides extreme flexibility as the network can be rapidly reconfigured. ^: • . • , ^•

Client Functionality

The API function .list given below illustrates the basic publicly accessible calls available to a client process to support operations within this architecture. As described previously this API can be extended (or truncated) as required for a given server type by use of custom functions or extention of the functions provided below. For additional clarification, the pseudo code associated with the API function list below is provided in Appendix B.

The function CL_GetServerClass() may be used to convert a ClientServer data type (an . OSType e.g., 'FAKE') to the corresponding server type used when sending to the server. To do this, it . switches the four characters in place such that the client type is the rnirror image of the server type (i.e., for a client type 'FAKE', server type is ΕKAF). This distinction is made in order to avoid the possibility of client and^' server events getting confused in cases where both the client and the server reside on the same machine. Events sent from the client to the server must use the server class, those from the server to the client use the unaltered data type.

The function CL_DoesItemExistO can be used to ask a server if a given Item ID exists in its database. . ^" '. .^' _ . . - ,

The function CLJDispIayMessageOnServerO can be used to display a simple one-line message on the status window of the server. Clients should use this to indicate the problem that caused them to fail to execute a given server request

The function CL_GetServerLocation() can be used to get the server location (IP address) for a given data type. Returns TRUE' for success, otherwise FALSE. . .' . "•

The function CL_DroneToLocation() can be used to get the drone location.(IP address) for a given server and drone data type. Preferably- the function returns TRUE for success, otherwise FALSE ^{' • ■} The function CL_LocationToDroneO can be used to determine if a given server type has a drone server (not necessanly running) at the IP address specified. .

The function CL_ClrServerAddress() can be used to clear the server address for a given server data type. Preferably, this function is used when a communications error occurs with that server so that the system will continue to attempt to re-establish communication with the server until it returns. TRUE is returned for success, otherwise FALSE. In addition, this routine may be explicitly called before attempting to send a command- to a server if it is suspected that the server may have gone off line for any reason. Making this explicit call will force a check before attempting to send any further server commands. _ .^• .. . , ^{' "} - '

' The function CL_SetDefaultNetCopyO can be used to specify the default (non-isochronous) file data transfer callback functions to be used by ClientServer when transferring files across the network from server to client in response to a CL_FetchItemFile() call. These defaults can be overriden on a per server/data-type basis by calling CL_SpecifyCallBack() for the data type concerned. The purpose of this API function is to allow the ClientServer architecture to be modified to utilize whatever file transfer protocols may be most appropriate for the environment and data type concerned. If no alternatives are specified by CL_SpecifyCallBackO for a given data type, the default transfer suite specified in this call will be used. The file transfer suite is defined as an abstract set of four procedures that are called by ClientServer as follows:

ET_NetCopyStarter — In the preferred embodiment, this function, called in the client, is passed two strings. The first is the fully expanded file path to the local disk where the output file is to be copied. This file may already exist or the file itself may not yet be created. This function would preferably perform whatever actions are necessary to set-up the transfer at the client end, returning whatever parameters are required by the responder function in the server in the output string parameter. A sample embodiment of a copy starter function is as follows: ^'

EngErr myCopyStarter ( // network copy starter f (client) charPtr ' outputFilePath, // Irlcl output file path to be used charPtr paramBuff, // 0:Outpt param bffr long ^♦context // 0:Context value (if required)

. - ^' )^{' •} // R:0 for success, else error #

ET_NetCopy Idler- In the preferred embodiment, this function, called' in the client, is passed' the same two strings as the copy starter function, the purpose of this function is to take whatever actions are necessary to sustain the transfer or abort it if it fails. When the copy process is complete (passed or failed) this function would preferably return TRUE in 'allDone', otherwise it should return

27

SUBS FALSE in order to request additional calls. The idler function would also preferably be implemented to carefully use the processor time as^' it will be called regularly by the calling code until such time as it returns TRUE. The idler is not called if the starter function returns an error. If the idler function wishes to pass information to the 'Ender¹ function, it can do so by modifying the para Buff buffer contents or the context record. The file transfer process actually completes, not when the idler returns TRUE, but when the ClientServer response is returned from the server. This means that the idler does not-need to support the 'allDone' parameter if it does not want to. It also means that the transmission may abort for reasons other than a true response from the idler, so the 'ender' function must account for this. A sample embodiment is provided below: ^• '

EngErr myCopyldler ; ( . • , ■ II network copy idler fn (client) charPtr ^' , ^'• outputFilePath, // I:Lcl output file path to be used charPtr ^■ paramBuff, // 0:Output parameter buffer

Boolean -^' *allDone, // 0:TRUE to indicate copy completion long *context // IO:Gontext value (if required)

' ' . ET_FileNameModifier fNameModFn . // I:Fn modify fie names - uniqueness y ^{■ ■} y . // R:0 for success, else error #

ET_NetCopyEnder - In the preferred embodiment, this function is called in the client and is passed the same two strings as the copy starter function. The purpose of this function is to take whatever actions are necessary to tear down and clean up a transfer process (either passed or failed). A sample embodiment is provided below: , ^■ . ^'

EngErr myCopyEnder . . ( . // network copy ender fn (client) charPtr outputFilePath, // I:Lcl output file path to be used

. charPtr paramBuff, // 0:Output parameter buffer long • ""context ^'// IO:Context value (if required)

) // R:0 for success, else error #

'

ET_NetCopyResponder- In the preferred embodiment, this function is called by the server and is passed the¹ contents of the paramBuff string created by the ET_NetCopyStarter called by the . client and the local^'file path on the server for the item ED specified in the CL_FetchItemFileO call. The responder is preferably called within one of the flunkies of a standard server and as such can maintain control until completion. Once again, in the present mvention, this function would manage its processor use so that other server activities can continue simultaneously. A sample embodiment is provided below: EngErr myCopyResponder ( ■ ^•' // network copy responder fn (srvr)

^• charPtr , • ^■ sourceFilePath, // LLocl source file path to be used-

.^' charPtr ^' ' paramBuff // I:Parameter buffer from in client

• . . ' ) ^' ' // R:0 for success, else error #

In the preferred embodiment, the function CL_GetDefauItNetCopyO returns the address of the registered default net copy-routines. This might be used by custom data type net copy funtions ' that wish to use the basic capability but then add additional features in a 'wrapper layer.

In the preferred embodiment, the function CL_AddResourceToFile() adds the database descriptor structure as a resource to the file in^'place. A database descriptor record is created and the ^•registered function to initialize a record is called. The file name is extracted from the source string and the item is copied to the name given. The local descriptor is disposed. This function could be used to pre-assign database descriptors to an item that will subsequently be put in the input folderT This might be performed when simultaneously creating other references to the item which contains a database ID, and must be kept consistent. - _. ' . . ''

In the preferred embodiment, the function CL_GetResourceFromFϋe() recovers the database descriptor resource from a file recovered from a server. 'It is the opposite of .

CL_AddResourceToFile(). - . '

In the preferred embodiment, the function CLJPutlnlternlnputFolderO takes a file full path name and a database descriptor pointer (whatever that may be) and copies the specified file to the used server input directory, adding the database descriptor structure as a resource to the file as it does so. During the process, a temporary file of the same name is created in the temporary directory. , Preferably, the name and path of the output.file are conveyed to the-caller and the -handle is saved to the requested temporary file name so that the resource can be added before the copy process (to avoid server interference before completion of process). On completion, the temporary files are deleted.

. In the preferred embodiment, the function CL JPutHInlterπlnputFolderO performs a similar function to CL_PutInItemInputFolderQ except the data for the item is passed in as a handle, not in the form of an existing file. It is often more convenient to have the ClientServer package create the file to be. sent automatically. -

In the preferred embodiment, the function CL_AddItemUsingServerAliasO takes a file full path name and an database descriptor pointer (whatever that may be) and copies the specified file to the used server input directory, adding the database descriptor structure as a resource to the file as it does so. During the process, a temporary file of the same name is created in the temporary directory. The name and path of the output file are conveyed to the caller and the handle is saved to the requested temporary file name so that the resource can be added before the copy process (to avoid server interference before completion of process). On completion, the temporary files are deleted.

In the preferred embodiment, the function CL_GetServerStatus() checks to see if the server for a given data type is running. It preferably returns 0 if the server is running else the system error. This function could also be used to check to see if the used server disks are mounted on the client side (if used). In the preferred embodiment, the optionFlags argument has bits for getting the used information as follows: bit klsServerRunning: if 1 check to see if server is running. If running, bit is CLEARED else it is set to 1 bit klsDiskMounted: if 1 check to see if necessary server disks are mounted. If mounted, bit is CLEARED else it is set to 1 bit kListServerProblems:ιf 1 check for any problems that might impede use. If problems, bit is CLEARED else it is set to 1. If a problems exist, a description would preferably be placed in the 'explanation' buffer. For example, the followmg could be used: bit kGetServerSummaryTextr if 1 produce textual server status summary. If supported, bit is CLEARED else it is set to 1. Additional bit masks could also be defined to determine other status information about a particular server.

In the preferred embodiment, the function CL_IsServerLocal() returns TRUE if the current machine is the same'as that designated as the server machine for the specified type. This means that the server is running on the local copy of the environment. If the server can run on that machine, TRUE is returned, otherwise, FALSE is returned.

In the preferred embodiment, the function CL_DataTypeHasServerO returns TRUE if specified data type has an associated server, FALSE otherwise.

In the preferred embodiment, the function CL_GetNextIDFromServer() returns the next -unique ED in the sequence for a given server.

In the preferred embodiment, the function CL_GetTimeoutTicksO given an options word as passed to CL_SendServerCommand(), this routine returns the time in ticks for the associated communications timeout period. There are three possible values, short, normal, and long. Short and long timeouts can also be obtained by specifying options, such as T ShortCommsTimeout' and T LongCommsTimeout' options respectively.

* _'

In the prefened embodiment, the function CLjSendServerCommandO is used to send commands from an client to a server for a given data type. Inihaily it verfies the server that is communicating. The caller's data is preferably added to the'host and an Interest Profile fetch/load/unload may be requested. If the operation is successful, a 0 is returned, otherwise an error is returned. In the preferred embodiment, the function CL_RelayCommandToDrone() is identical to CL_SendServerCommandO but allows commands to be relayed/routed to a specified drone. Unless implementing specialized protocols or engaging in an extended 'session', there is generally no need to such direct request as the system automatically handled such routing (as described above).

In the prefened embodiment, the function CL_vCalICustomFuncO allows any registered ' custom client function to be called and the appropriate parameters passed/returned. By accessing functionality for a given data type through this mechanism, it becomes possible to remove the , requirement for Unking the library associated with the type directly to the calling code.^' This allows calling code to be designed so that if a given data type/library is installed, it will use it (see CL DoesTypeExisf), otherwise it will not. In either case however, the calling code can be built and linked without needing the library to be present. This is a key benefit provided by modular systems. Most often the custom functions will amount to preparing arecόrd to be passed to the data type server using the CL_SendServerCommand() function; however, they could also be local function's and need not communicate with the server. Any 'aCommand' values supported by a given type would preferably be defined as char constants (e.g., 'cmdl') so that they are easy to recognize in server error messages etc. Furthermore, these values will preferably correspond one-for-one with 'command type' to be sent to the server so as to simplify things; the custom server commands and the function designators could also be treated as a single constant 'set'. Libraries may wish to declare macros for each custom function call in order to ensure that the correct number of arguments is passed. In order to process custom functions, a type would preferably define a custom function handler. A sample API is provided below:

In the example above, the calls to va_start() and va_end() occur within GL_CallCustomFuncQ and are not used in myCustomFunc. The routine CL_vCallCustomFunc can be used to pass a variable argument list through from the caller to the registered custom client function for the type specified. In this latter case, the calling code is responsible for calling va_startQ and-va_end().

In the preferred embodiment, the function CL_CallCustornFunc() allows a- registered custom client function to be called^' and the appropriate parameters passed/returned. See CL_vCallCustomFuncO for details. ^' ,_. . . . ^{• •} -

' In the preferred embodiment, the function CL_NeedServerComms() invokes all initialization functions registered using CL_SetEPinitTermCallO- CL_xDumpServerComms disconnects from server communications with the specified data type. The routine is preferably called when the IP notifier widget receives its first command. It may also be called by any other widget that uses direct access to server commands. Finally, the handler records and bufferare de-allocated and disposessed. . ^•. " ^" . ' . '

In the preferred embodiment, the function CLJDumpServerCornmsO' invokes all termination functions registered using CL_SetEPinitTermCallO. It is preferably called when the EP notifier widget terminates. It may also be called prior to termination by any other widget that has . called CL_NeedServerComms for the same data type. . ,- _, ^{■ '} .

In the preferred embodiment, the function CL SetlnitTermCallO allows 'other libraries to register functions to- call whenever the LP notifier (or any other widget) uses access to'the server for a given data type. . . - . ^' >

In the preferred embodiment, the function CL_ClrInitTermCallO allows other libraries to remove initialization/termination functions registered with CL_SetIPinitTermCallO-

In the preferred embodiment, the function CL_DefϊneIP() defines/creates/updates an Interest Profile record. First, the function verifies the data type and then checks the record size and adjusts it accordingly. Ultimately, ^'the IPs files will be updated and the result is returned.' Preferably, the result will be zero for success, and an error code otherwise.

In the preferred embodiment, the function CL_GetIPlistHdr() returns the first IP in the LP list, NULL if there is none. By use of repeated calls to CL_GetIPfϊeld, each element in the EP hst will be examined. . ' . - In the preferred embodiment, the function C _UnDefϊneIP() deallocates/disposes of an Interest Profile record. The Interest Profile is removed from the server and the hnk is disposed of. Any associated managers (such as a notifier widget and/or Data Type manager) are informed and the ^' IP name is removed from the recognizer and the IPs file is updated

In the preferred embodiment, the function CL_RegisterIP() registers ah Interest Profile by name. When an interest profile is 'registered¹, it becomes known to the environment's IP notifier view/widget. This causes the IP to be loaded into the server and handles the display of any hits that occur for the IP by posting them to the pending views window and the view menu itself. Once an IP has been registered in this fashion, it effectively becomes the property of the IP notifier widget; the widget or code making the registration will preferably not de-register or undefme the IP concerned. This is true even on termination unless for some reason the LP hits can no longer be properly processed when the original registering widget is no longer mnning. Other than this case, the environment's IP notifier preferably takes over all responsibility for handling and disposing of a registered interest profile. - ^'" •

In the preferred embodiment, the function CL_DeRegisterIPO deregisters an Interest Profile by name to prevent subsequent accesses via CL_ResolveEP0. Initially, the routine checks to verify the IP record. It then informs the widget notifier and Data Type manager and updates the IPs files.

In the preferred embodiment, the^' function CL_ResolveD?0 resolves an Interest Profile registry name into the corresponding P record pointer. In the preferred embodiment, the function returns the Interest Profile pointer or NULL for an error. . . ^'

In the preferred embodiment, the function CL_GetD?fϊeld() allows various fields of an - Interest Profile record to be retrieved. Zero is returned for success or an error otherwise. The field to .be obtained js.specified by the 'fieldSelector' parameter, while the 'fieldValue' parameter would be a pointer to a. variable or buffer that is appropriate to the field being recovered. An example embodiment of such a field follows: ^'

SELECTOR ^• VALUE TYPE ' DESCRIPTION

kIPoptionsFLD . int32' Value passed in for 'options' when the IP was defined kLPeventTypFLD ^; int32 32 bit event mask - ' . kIPdataTypFLD ^• . OSType The ClientServer data type for this IP kIPurgencyFLD ^""■ char The urgency level for the IP . kIPvoicelndexFLD char The voice index to be used when speaking IP name etc. kIPiconldFLD short Icon ED used in Pending Views window to ED this IP'

33

ΠTUTE SHEET kIPcontextFLD void* value passed for 'context' when the IP was created kIPnameFLD char[256] The name of the IP - , . _, - . ■ kIPexpressionFLD ■ charHdl* MitoQuest™ expr. defining the IP match criteria kIPmachineNamFLD char[256] machine name IP belongs to (rel for srvrs only) ' kffuserNamFLD char[256] user name IP belongs to (rel for servers only) kIPnextlPFLD ^' ET_LPRecordPtr* Address of the next LP in the IP list

In the preferred embodiment, the function CL_GetDataTypeOptions() obtains the options specified when a data type was created using CL_DefineDataType().

In the preferred embodiment, the function CL_FetchItemPreviews(), when given an array of 'database descriptor records' for a particular client/server data type, retrieves the 'preview' data associated with those records (if available) and attaches it to the records. For example, if the data type were an image, the 'preview' data might be a picon of the image whereas if the data type was text, the preview might be the text itself. Some data types may not have preview data. This function can also be used to obtain the database descriptor records (for those servers that support this) by using the _ "kWantDescRecs¹ option. In this case, only the 'unique ID' field of the descriptor records needs to be set up prior to the call, the entire descriptor will be filled in from the server (optionally also with the' preview data). This function serves the dual purpose of associating preview data with existing descriptor records, or of initializing empty descriptor records (other than ID) and optionally the associated preview data. Zero is returned for success, otherwise error number is returned.

In the preferred embodiment, the function CL_IsFiIeCached() determines if a file is already cachedjocally and if so returns the file path in the cache. ^■

In the preferred embodiment, the function CL_AddFileToCache() adds an arbitrarily named _% file, to the internal database of files that are in the local ClientServer file cache.^' This function is .. . preferably called automatically by CL_FetchItemFileO, in which case the file names will conform to the syntax returned by CLJjetCacheFilePathQ. If the file is being moved by some other means, however, this function can be called after the file has been moved in order for the moved file to subsequently be recognized as cached. In this latter case, the file name can be arbitrary. If a subsequent call^' attempts to access the file at the location specified and it is no longer there, CL_FetchItemFileQ will re-fetch the file from the server and delete the missing file from the database of cached files, adding the new one. ' ,

In the preferred embodiment, the function CL_AddDataToCache() adds an arbitrarily memory resident data handle, to the intemal database of files that are in the local ClientServer file cache. This mechanism is intended to allow the ClientServer cache to be used to store additional data types that do not have a correspondmg CLientServer server associated with them. To make use ot the facility, a unique data type and an item ID withm that type must be supplied. This function takes the data handle supplied and saves if to a path within the cache that is constructed based on the data type and ID. Items saved in this manner can be manipulated using the other ClientServer cache functions in this package. The only difference is that since there is no corresponding server, only items in the cache will actually be accessible.

In the preferred embodiment, the function CL_ScanCacheForNewFiIes() scans the cache all folders (recursively) referenced from the main cache folder looking for "CachedFileDB" files. When it finds one, it opens it and attempts to add and files referenced by it to the main "CachedFileDB" file of the cache folder. What this means is that to add a previously saved cache to the current one (e.g.^', burned onto a CD-ROM and then referenced via an alias in the main cache), all that is required is . invocation of this function. In the preferred embodument, this function is called automatically followmg system startup and can also be initiated from the Administration window after a new alias has been added to the cache. This makes it relatively trivial to augment a machine's cache with additional cached files.

In the preferred embodiment, the function CL_PurgeCache() is called regularly by the environment. If it discovers that the disk(s) containing the cache folder is becoming full, it purges old files from the cache until disk utilization falls below acceptable limits. This routine could be called explicitly to ensure a certain amount of free disk space. Files are purged starting with the oldest and file purges could be further limited by other limits. For example, this function could be implemented such that files less than 2 hours old are not purged unless the purge ratio is set to 100%.

- In the preferred embodiment, the function CL_DeleteFileFromCacheO deletes a file from the ClientServer file cache and removes its entry from the cache database. The file may be specified either by DataType and itemID or by file path. For obvious reasons, the former is considerably more efficient.

In the preferred embodiment, the function CL_GetCacheFilePath() returns the file path that CL_FetchItemFile() would use to cache the correspondmg data if it were requested. This function could be used in the event that the file is placed into the cache by some other means while still permitting retreival by CL_FetchItem().

In the preferred embodiment, the function CL_CueItemFiIeO causes a server to take whatever action is necessary to cue a file for subsequent playback, this could include mounting the corresponding archive 'chunk' in the server (or one of its drones). The main purpose of the 'cue' command is to allow clients to predict that an item file may be requested by the user and thus begin the process of loading it in order to save time should a subsequent file fetch be issued. In the preferred embodiment, unless the 'aNetworkFilePath' parameter is set to non-NULL, the cue command returns immediately. By setting 'aNetworkFilePath' non-NULL, the caller will wait until the cue operation is complete at which time the full network path where the file was placed is known. Since servers are free to move media that is not currently in use in the preferred embodiment, the path returned should only be considered valid for a short peπod of time (i.e., seconds). Otherwise, the path should be venfied again by issumg a fresh 'cue' command.

In the preferred embodiment, the function CL_FetchItemFileO can be used to move the contents of a file-based item to a local file. It returns zero for success, otherwise an error number.

In the preferred embodiment, the function CLJDefineEventType 0 allows the names and bit masks for data specific server events to be defined to the server so that Interest Profiles and searches can be specified m terms of these events. Certain event types are predefined by the standard server package, notably:

"kMatchltemAdd' ~ "Add" triggered when a new item is added to the data set T MatchltemWπte' - "Wπte" triggered when an existing item is wπtten to "kMatchltemDelete' -- "Delete" triggered when an item is deleted 'kMatchltemRead' - "Read" triggered when an item is read/accessed

Additional data specific actions can be defined usmg this function and then used m creating Interest Profiles. The caller supplies a textual descnption for the event (such as those in quotes above) together with a bit mask specifying which bit (0..23) is to be used to specify and check this condition. By makmg a seπes of such calls, a complete descnption of all available event types for a given server and data type can be defined. TRUE is returned for success and FALSE otherwise.

In the prefened embodiment, the function CL_MaskToEventType() translates an (single bit) event type mask associated with a ChentServer data type mto the conesponding descπptive stπng. Only the lowest bit set in the mask is extracted. The association between these bit masks and the conesponding event type is made by calling CL_DefineEventType(). TRUE is returned for success, otherwise FALSE is returned.

In the prefened embodiment, the function CL_EventTypeTo askO translates a descπptive event type stπng to the conesponding mask for a given ClientServer data type A search is performed for the exact match to the stπng. The association between the bit masks and the coπesponding event type is made by calling CL_Def eEventType(). Success returns TRUE, otherwise FALSE is returned. , In the prefened embodiment, the function CL isfE vents 0 returns an alphabetized, carriage return (<nl>) separated hst of all the IP event types supported by the server for given data type. ^' Initially the data type is verified. Once a string list is obtained, it is sorted alphabetically. The descπptive field is extracted and the string list unsorted. NULL is returned for an enor, otherwise handle to event type hst. •. , ..^'

In the prefened embodiment, the function CL_GetServerItemCount() returns the count of the number of items for a given server. . ^• . .

In the prefened embodiment, the function CLJDecIar^'eEventO can be called either within a server or from a client in order to declare that a specific event has occured for the data type , concerned. The effect of making such a declaration is to request the server to check all interest ^• ' profiles to evaluate if any of them should trigger as a result of the declaration and if so to inform any interested clients (as described above). Interest profiles can be defined based on the action that the interest profile is interested in. One such action -is "Add" which occurs automatically when a server adds an item to the server database. In the prefened embodiment, predefined actions such as "Add" do not use explicit calls to CL_DeclareEvent() since these are made automatically by ClientServer code. . ^{• •} ' ^' . ' .

In the prefened embodiment, the function CL_SpecifyCallBackO can be used to specify one of the various callback functions used by the ClientServer architecture. See the description above for a full list of the possible call-backs that can be registered and their purposes.

In the prefened embodiment, the function CL_ObtainCallBack0 can be used to obtain the address of one of the various callback functions used by the ClientServer architecture. Some callbacks are used, others are optional depending on the nature of a particular server and data type. ,See CL_Spec-fyCa-lBackO for details. . . ^• . .^" . ^{' '} .

In the prefened embodiment, the function CL_DefineDataType() allows the data type and server information for a particular server data type to be defined. Using this information, this function is able to construct appropπate dialogs, determine server state, and communicate with the server both as part of Interest Profiles and also for retrieving data of the type given. Initially, data type verification is performed and subsequently the new record is added to the^' linked Hst and tweaking parameters are handled. A string list is created for the event types and all the tags are set equal to the first field. The value TRUE is returned for success, otherwise FALSE is returend. ' .

In the prefened embodiment, the function CL_StandardHitLfstQueryO can be used^' to issue a query command (kCSStandardQuery) for the express purpose of obtainmg a standard hit list from a specified server or from multiple servers. The hit list obtained is in the form of a standard hit list (of type ET_HitList). Preferably this function is invoked invisibly via the MitoPlex™ API, however, it could also be used to allow more specialized direct uses. • ' , . _: -

Server Functionality .^{* •}

The API listing below gives the basic suite of public calls available for customizing code that is runnmg within a server or which is manipulating server state directly. Most of these calls will fail if invoked outside the context of a server or one of its drones/flunkies. Again, while the followmg is •^'the prefened embodiment, any number of different or related API calls could also be provided. For additional clarity, Appendix B includes a sample pseudo code embodiment of the header files associated with such functions (which are described below).

In the prefened embodiment, the function SV_ShowFlunkyWindows() ensures all flunky windows are showing. • _, •

,' In the prefened embodiment, the function SVJWidgetModalEnterO and SV_Widget odaLLeaveO are called as a widget modal window is show or hidden. These function allow the client server package to take certain speciaUzed actions in the case where the widget modal belongs to a flunky of a standard server. More specifically, in the prefened embodiment; the action ^' taken is to ^'register a symbolic function so that the widget modal will always be re-shown when the main server window is brought forewards. .This function prevents loss of such a window when clicking on another widget. . ^{" '} - .^'" ^' • .^'

In the prefened embodiment, the function SV_ServerInitComplete() allows flunky threads • within a server to determine if the main server thread has completed initialization yet. It is often important to ensure that no other activity occurs before the main thread has established a consistent logical environment within which such activity can take place. .

In the prefened embodiment, the function SV_AllServerInitsComplete() allows code in any context to determine if all servers in the cunent machine have completed their startup sequence. If there are no local servers, this routine returns true immediately. ^' ' .

In the prefened embodiment, the function SV_SetArchiveSuite() is used to specify the default suite of functions to be used for archiving. In the prefened embodiment of this function, a standard archiving software suite is registered using this function so that individual servers need not consider the requirements of archiving other than to set the necessary option. Specialized archive • suites could also be registered on a particular server.

In the prefened embodiment, the function SV_PutInitemRejectFolder() takes a file path and moves the specified file to the Reject directory, optionally deleting the source file. The file name is extracted frβm the source string and concatenated to the directory where the file's existence is verified. • • . ' ^■ .

- In the prefened embodiment, the function SV_GetServerFoIder() can be used to obtain the full file path to one of the four server folders. Return Zero for success, otherwise enor number.

In the prefened embodiment, the function SV_GetArchiveTweakRecord() returns a pointer to the archive tweak record for a given server type. The result is a pointer to the archive tweak record, or NULL in the case of an enor. . • - ^• ■

In the prefened embodiment, the function SV_GetTweakRecord() returns a pointer to the tweak record (the tweak record should not change when the server is ranning) for a given server type. The result is a pointer to the tweak record, or NULL in the case of an enor.

^• In the prefened embodiment, the function SV_GetCurrentBatchDetailsQ is used to obtain - details of the batch cunently being processed by a given server. This function is preferably called from within server plugin. • , - . . '

' Given the size of the resource and data forks of a file and the blocksize for the target device, • SV_GetBIocksQ returns the number of blocks that will be consumed by the file on that device. ^'

In the prefened embodiment, the function SV_FetchArchiveChunkTJD() fetches the archive 'chunk' ED associated with a given item in the server. For example if the server archiving device , includes a CD-ROM based device, the chunk LD would be a unique (within this server) reference to the particular CD-ROM on which the item resides. The chunk ID value returned can be utilized in subsequent archive specific server commands.

In the prefened embodiment, the function SV_Index2FolderNameO creates a base 36 folder ^• name string encoded into 5 characters as follows "@XXXX" where each 'X' character can be 'A' to 'Z' or '0' to '9'. In the prefened embodiment, the function SV_FolderName2Index() can be used to convert.these folder names back to a numberic value. The reason for such a function is avoid limits. on the number of output folders that can be created by the ClientServer package. If a decimal encoding were used, only 10,000 output/archive folders would be allowed. By using base 36' encoding, the number is nearly 1-.7 million. Again, additional or modified encoding schemes could also be used depending upon the application.

-In the prefened embodiment, the function SV_FolderName2IndexO creates abase 36 folder name string encode into 5 characters as follows "@XXXX" where each 'X character can be Α' to 'Z' or '0' to '9'., In the prefened embodiment; the function SV_FolderName2Index() can be used to convert these folder names back to a numberic value. The rationale for this approach is set forth above. ^" . ^{• '•} - ^• . ' . ,

In the prefened embodiment, the function SVJDefiπeDroneTypeO is used to define a 'drone' data type. A 'drone' data type is a data type for which the server is not usually directly addressed but is associated with a primary data type server. This is similar to the way that flunkies are associated with a server. Preferably, drone' servers are treated exactly like primary server flunkies except that drones servers often reside on another machine. This may be necessary if the task being assigned is very computationally intensive or if the drone is being used to access and deliver data from storage devices that are actually connected to the drone machine, not the prime. As an example, a video server might consist of a single prime server with a number of drones, each capable of sustaining a certain number of streams direct to the original clients, but all of which are actually accessed via requests to the prime. In actual operation, the primary server can assign a task to one of its own flunkies, which in turn can re-assign the task to a drone server, hanging up until the drone sends a completion message. A complete API would also preferably provided for server plugins in order to search and assign available drones of a primary server. In theory, drone servers themselves could have drones. , . ^• . ' .

In the prefened embodiment, the function SV_CountDrones() returns a count of the number drone servers/data types that are associated with a given master type.

. In the prefened embodiment, the function SV_GetDroneType() returns the drone type conesponding to a given index within the drone types list of a given master data type.

In the prefened embodiment, the function SV_IsAssociatedDroneType() returns true if the specified drone type is associated with the cunent server, false otherwise. ^•

In the prefened embodiment, the function SV_FindLeastBusyDrones() returns the least busy drone(s) out of a list of drones.

In the prefened embodiment, the function SV_GetStarusWord() returns the status word conesponding either to a server itself, or to a given index within the drone types list of a given master data type. Servers can declare their 'status', which in the prefened embodiment is a 16-bit flag word, by calling SVJDeclareStatus(). In the case of drones, this status is relayed to the master where it can be examined using this routine. This feature is intended to support applications where various drones of a server may go in and out of certain 'states' that need to be communicated to plugins in the master (e.g., routers etc.). The prefened use of the status word is as an anay of bits where a server would update any state bit individually. An example follows:. ' , status = SV_GetStatusWord(myDataType,-l) - // get cuπent status

SV DeclareStatus(status | aNewStatusBif) // set the bit we're concerned with

In the prefened embodiment, the function SV_DeclareStatus() sets the contents of a servers status word. See SV_GetStatusWord() for additional details.

In the prefened embodiment, the function SVJVlakeServerViewQ adds an instance of the built in 'LP Server¹ widget to a view. This widget preferably provides multi-user server capabilities for a particular data type, including support for notification of clients based on the triggering of interest profiles. Initially the data type is verified. The server view is created in such a way that it cannot be halted other than by software termination. The return value is the launch code of the created view or 0 for enor.

In the prefened embodiment, the function SV_ArchiveThisFileQ returns TRUE if the specified file in the server output folder should be archived, FALSE if not. For non archiving servers, the result is always FALSE. For archiving servers, the result may be FALSE in cases where archiving has been turned off, such as be using SV_SetFolderArchiveEnb() for some folder ancestral to the file.

In the prefened embodiment, the function SV_InitiateBatchScan initiates an output folder scan for the purpose of sending the files in the specified batch to a bum device.

In the prefened embodiment, the function SV_PerformArchiveBurnO, through various registered archiving callbacks, performs an archive bum sequence for the batch ED specified.

In the prefened embodiment, the function SV_GetStatePtr() can be called from any context to discover the local server state pointer for a given data type. This is not the prefened methodology other than rare cases. ' ^{• ■} ■

In the prefened embodiment, the function SV_GetClientOptions(), when called from within a server flunky, returns the options specified on the cunent command at the time of the call to CL_SendServerCornmand(). In addition to the standard options bits for this function, the prefened embodiment also provides eight custom option bits (starting from TcCustomCmdOption ) that can be used to pass parameters to custom command functions in the server, for example. This function is intended to allow retrieval of these options.

In the prefened embodiment, the function SV_GetDatabasePtrO can be called from within a standard server and gets the database record pomter address associated with the specified server. This pointer may be used by ClientServer callbacks in order to make calls to the Database API to manipulate the ChentServer database directly. _; ^■ ' In the prefened embodiment, the function SV_GetArchiveDBPtr() can be called from within a standard server and gets the database record pointer address associated with the archive for a specified server. This pointer may be used by ClientServer callbacks in order to make calls to the Database API to manipulate the ClientServer archive database directly. If the server does not support archiving, the return value will be NULL.

In the prefened embodiment, the function SV_AddArchiveBatchIDO ^{can De} called within an archiving server (or its plugins) to add a new archive batch to the archive DB and return its ; batchED. .• • ' - ^" . ' • .- ^' . ^' - ^'

In the prefened embodiment, the function SV_SetFolderArchiveEnbQ can be used to enable/disable archiving for a specified server output sub-folder (and any folder hierarchy it contains). The routine (as presently implemented) is only valid when called from within a standard server or one of its flunkies. In certain situations, especially those where a server output sub-folder contains a limited set or may be subject to frequent updates, it is often desirable to rum off archiving for that portion of the server hierarchy since, once archived, updating of the content may no longer be possible (depending on the nature of the archive device — such as 'write once' devices).

In the prefened embodiment, the function SV_SetupMessage() can be used by standard server callback functions, and intemal environment code, to display messages both in the momentary activity display in the server window, and also in the time history display (if that option is enabled). If the 'log' parameter is true, the string is added to the time history window, otherwise it is not. This is useful when code displays status messages that you do not wish recorded as well as exception condition messages that you do. In the prefened embodiment, this function behaves just as an 'sprintf; that is, string substitution parameters may be specified as "normal" (up to a total message size of 256 characters). •The flunky index and state pointer parameters necessary for this call may also be obtained by calling SV^GetStatePtrO. ^'In the preferred embodiment, if the ^' 'kServerHasTimeHistory' option is not selected when the server is created, the time history information is not recorded but all strings are still displayed to the momentary display (such as in the server window). • .

In the prefened embodiment, the function SVJPingClient() can be called .from within a server context and will look for the cunent client widget, on the appropriate machine, for the calling flunky. If the client machine or widget cannot be found, this function returns FALSE, otherwise it returns TRUE. No actual communication with the client widget occurs; rather, its presence is simply sensed. In cases where a server flunky may stay connected to a client for long peπods of time and where there may be no obvious activity from the client, it is wise to pmg the client on some regular

42

^'SϋB - - - basis (e.g., once a minute) so that it the client has actually gone away, the server process can be aborted and any allocated resources released. " . • -

In the prefened embodiment, the function SV_AutoPingCIieπt() causes the client associated with a flunky to be auto-pinged at the specified interval. If the client goes away, the specified wake handler will be called so that the flunky can take whatever action is appropriate to tear down allocated client resources. ' ^' - .

In the prefened embodiment, the function SV_SetPicon() displays (or erases) a new picon in the server picon window. . ^' '

In the^" prefened embodiment, the function SVjSendToClientO can be called within an non- favorite flunky of a server in order to send a message to the client that caused the flunky to be instantiated. This permits client notification of the progress of a request etc. A reply to this message will generally not be returned from the client unless the reply is handled in another flunky. In other words, in the prefened embodiment, this is a one-way communication scheeme.

In the prefened embodiment, the function SV_ForwardCommand() can be called from within a flunky of an existing server to forward a command received unmodified to another server. This mechanism allows one server to server as a router to other servers since the command is sent by 'proxy' and thus the reply goes back to the originating client, not to the caller of ' SV_ForwardCommand(). -Commands can be re-routed multiple times using this proxy mechanism thus allowing the creation of large networks of servers where the topography of a particular server cluster is hidden from external clients via a 'router' server. The ability to re-route commands by proxy (in conjunction with the server 'drone' mechanism) is important to the creation of servers that are implemented as a 'cluster' of machines all of which appear as a single logical server.

In the prefened embodiment, the function SVjPrepareFϋeForXferO can be called to prepare a file for transfer to a client without actually doing the transfer. If files are being transmitted to the client by some other means, this function could be called to ensure the file is available/mounted ^", before you begin the transfer. In the preferred embodiment, this function can be called either in the master server or a drone and in the former case may route the original command to the selected drone (returning -1). For these reasons, the code that calls this function would preferably appear in both the drone and main server custom command functions.

In the prefened embodiment, the function SV_RepIaceCommandO can be called prior to a - call to SV_ForwardCommandO in order to alter the actual command that is being forwarded. The oπginal command is disposed and replaced by the new command supplied, a subsequent call to SV_ForwardCommand() will cause the new command to be forwarded. Parameters are identical to CL_SendServerCommandO except for the first two and ^"the fact that 'aDataType' is not supplied since it is implicitly the data type of the enclosing server.^' - . ^' •

In the prefened embodiment, the function SV_GetOptions is only valid within the context of a ClientServer server or it's flunkies and callbacks. SV_GetOptions() returns the options parameter specified when the server way created (See SV_MakeServerView), SV_SetOptions() allows the options to be dynamically modified. " - .• ^{■ ' • '}

In the prefened embodiment, the function SV_SetOptions() is only valid within the context of a ClientServer server or it's flunkies and callbacks. SV_GetOptionsQ returns the options parameter specified when the server way created (See SV_MakeServerView), SV_SetOptions() allows the options to be dynamically modified. ^' " '^' . _, ^'

In the prefened embodiment, the function SV_GetContextQ returns the value of the server context field set by a prior call to SV_SetContext(). In general, if server flunkies or callbacks need a^' global context record, a server function will be registered (using SV_SetLPinitTermCall) in order to- ' create the context when the first widget within the server is created. Similarly, another function is preferably registered to terminate the. allocation. Thereafter, any subsequent callback code may obtain the context by calling SV_GetStatePtr() to obtain the server state pointer, and then SV_GetContext to obtain the context value. The value of the context field is returned.

In the prefened embodiment, the function SN_SetContextQ sets the value of the context field wit m a server state record. This context value may be obtained from any flunky or callback function ^■ by calling SV_GetContext(): ' . •• ^{' " '}. - , '

In the prefened embodiment, the function SV_GetArchiveContext() returns the context value associated with the archiving process for a given server. A seperate context location provided for archiving (in addition the that provided by SV_GetContexf) in order to ensure that the normal context location is still available for custom use by each server independant of the archiving software registered for that server. - , - - " -

In the preferred embodiment, the function SV_SetArchiveContext() sets the value of the context field within a server state record. This context value may be obtained from any flunky or callback function by calling SV_GetContextQ. . ' •

Calls to SV_SetReplyO are preferably made within a custom command handler callback, function ( ET_CustomCmdF ) for a ClientServer server. In the prefened embodiment, the function is used to set a pointer to a block of data to be returned to the client as the reply. If the 'wantReply' parameter to the ET_CustomCmdFn function is FALSE, SV_SetReply should not be called since the

reply data will simply be discarded. Replies may be passed back to the client either synchronously or asynchronously and each reply is uniquely tagged using a value supplied by the, original caller. In the prefened embodiment, the buffer supplied will be duplicated and attached to internal server structures so that it may be a local variable. Its size may be anything from a single byte up to the maximum size of the client receive buffer. The return value for success is Zero, otherwise an enor number is returned. . ' - • ^{'• '} r

In the prefened embodiment, the function SV_SetItemListRepIy() may be called from a server callback function. Its purpose is to cause the reply event from a custom server command to be an item list (see SV_RequestItemsList()). This call will fail if the client did not specify that it was anticipating an items list reply by setting the 'kResponselsItemList' option. The item list sent will consist of a series of descriptor records as defined by the data type involved, however, if necessary, the item list will be sent as a series of 'chunks' of items which are automatically reconstructed at the other end into a contiguous list. This mechanism should be used in any case where the number of items resulting from a callback function is unknown and might possibly overflow the client receive buffer if an attempt to send them as a normal reply record were made. By using this scheme, it is possible to keep the size of the reply buffer that must be allocated by each client to a reasonable value even if hundreds or thousands of items may be sent in response to a given command. The client end code (presumably in a custom function procedure), issues the appropriate command with the "kResponselsItemList' option set, the data appearing in the reply record will consist simply of a handle to the item hst. The 'aMarkupFn' parameter may be used to specify a function that is called for each descriptor/data item in the list (after it has been recovered but before it has been sent). This function is passed the 'aMarkupContext' to allow it to establish context, its purpose is presumably to alter the descriptor or data record being sent in whatever way might be appropriate given the context of the call. Thus, if there is additional information to be inserted into the descriptor record or data for this command type, the 'aMarkupFn' function can be used to do so. . . . - - -. '._ .

In the prefened embodiment, the function SV_Dont aitForMeO can be called from a server flunky when that flunky is engaged in some lengthy action which should not inhibit normal operation of the 'favorite flunky' (which is responsible for acquiring and processing new input from the input folder). In the prefened embodiment, the default behavior of the invention is to inhibit activity by the favorite flunky whenever there is any outstanding activity for other flunkies. This is based on assumption that providing rapid responses to client requests is more important than it is to acquire new items for the server database. Note that if a flunky calls this function, the server will not wait for that flunky when being 'stopped'. The same is true when the environment is 'Quit' so if this causes a problem for the flunky, it should either not call this function, or should register a termmation function in order to perform the necessary cleanup. In the prefened embodiment, the function SV_GetClientLocation() can be called from a server flunky (other than the favonte flunky) or callback code invoked from such a server flunky. In the prefened embodiment, the function returns the network path of the client machme that the flunky is cuwently handling.

In the prefened embodiment, the function SV_GetOutputFilePath() can only be called from withm a standard server. It gets the partial output file path (if any) for a given item ID In order to construct the full file path, this partial path must be appended to the output folder base path which may be obtained using SV_GetServerFolderO-

In the prefened embodiment, the function SV_DefineMaintenancePageO can only be called from withm a standard server. Usmg the 'book' API, this routme allows the names, handler functions, etc. for custom views/pages in the server maintenance (or wrench) wmdow to be specified to the ClientServer package. In order to add a new page to the maintenance w dow, its name must be specified and an appropπate handler function supplied. The ClientServer itself registers one page called "Server Items" which allows limited examination of and control over the state of the items contamed in the server. In addition, if archiving is enabled, an "Archive Items" page is automatically registered, which allows examination and manipulation of the 'chunks' of storage (e.g., CD-ROMs) that make up an archive. The "Server Items" and "Archive Items" pages are supported by additional ChentServer APIs which allow servers and archive suites to register iconic buttons and the conespondmg action functions so that it is possible to provide all the basic server manipulation functionality without having to explicitly wπte any UI code. Often, archive suites will also register an "Archive Devices" page, which allows explicit access to, and control over the autoloader devιce(s). Other custom server types may wish to register a "Server Status" page which can be used to display additional status over and above that provided by the standard window. Other uses of the maintenance wmdow for purposes unique to a particular custom server are envisaged. TRUE is returned for success and FALSE otherwise. A typical maintenance page handler might appear as follows:

Boolean my ;nntPageHdlr (

ET_CSStatePtr ip. // I Server state pointer charPtr pageName, // I name of the page to be handled

ET_UIEventPtr anEvent , // I UI event/msg to be handled short action, // I Action (see notes)

ET_ControlRef aControlH // I Control handle for page ) // R:TRUE if the event was handled t if ( action == kHandleϋlevent ) switch ( anEvent ->messageType )

46

SUB l case kEventMsg : _% , ' // R event received

'break; : , '

■ , case kWindowMsg : ' , // Window or' control hit

• switch ( anEvent ->controlLabel ) break;

} else return YES ; ;,

}

' Only one view/page is visible in the maintenance window at any given time. When the user clicks on the 'wrench' control in the server main view, the maintenance floating window would be shown. In the prefened embodiment, control label numbers below 100 are reserved exclusively for ClientServer use within the maintenance window. The title of the view control you supply must match the 'pageName' parameter exactly and the view's dimensions must be 408h*168v.

' In the prefened embodiment, the function SV_GetActiveMaintPage() allows you to obtain the maintenance page that is cunently active (if any).

In the prefened embodiment, the function SV_;GetPageRegistryO gets the page registry for the mainenance window. The ChentServer maintenance^' page capability supports the ability to create^' an arbitrary ET_StringList registry associated with each page of the maintenance window. This capability is available for custom use, however, in the case of the built in maintenance pages (i.e., "Server Items" and "Archive Items"), the registry is used to register custom buttons, icons and action functions associated with the page. See SV_DefineItemPageButtonO for details. When the -maintenance page is destroyed, the conesponding ET_StringList that makes up the registry is also ^" deleted. The string list is automatically set to be the context value for the book page, ^"i.e., it will be^" - passed to any page handler in the context field, replacing any value set in the UI_DefineBookPageQ call. ClientServer initially sets the context value to zero when creating a page via SV_DefineMaintenancePage().

In the prefened embodiment, the function SV_DefineItemPageButton() is associated with the predefined "Server Items" and usually with the "Archive Items" pages in the server maintenance window. It utilizes the registries (see SV_GefPageRegistry) associated with these pages to store the details of a series of buttons that are displayed whenever <a given item in the list of server/archive items is selected.. For other maintenance pages, the registry may be used for other purposes and calls to this routine may not succeed. This routine inserts entries into the ETJStringList (using the kStdDelimiter delimiter) where the field contents are as follows: "buttonName" kStdDelimiter" ".buttonHelp" kStdDelimiter "buttonType" kStdDelimiter "buttonData" kStdDelimiter "buttonHandler" You may recover any of this information later using SV_GefltemPageButton(). In particular, you should ensure that during termination of your page, you dispose of any memory associated with the 'buttonData' handle since this is not automatic. You generally define the possible set of buttons for a page on the 'klnitializePage' call to the page handler, and dispose of the button data on the "kTerminatePage' call. . ^'

In the prefened embodiment, the function SV_GetItemPageButtonQ recovers the . information for a button associated with a maintenance window registry. See - SV_DefmeItemPageButton() for details. ' ^"

In the prefened embodiment, the function SV_GetFavoriteFlunkyID() can be called from anywhere within the context of a server 'cluster' and returns the widget ID of the. favorite flunky of the cluster. This allows code running in other flunkies or the main widget (especially in maintenance pages) to issue commands to the favorite flunky using wake events. In this manner the maintenance page code can cause lengthy operations to occur in the favorite flunky context thus avoiding hanging up the server itself. . . ' ' . • ^• r . -

In the prefened embodiment, the function SV_CustomCommandFavFlunkyO is used to issue a command to the favorite flunky of a ClientServer server while ensuring that commands do not overrun the flunky and maintaining all necessary server states. This function can be used within a ClientServer server cluster. -^• ' -

In the prefened embodiment, the function Sy_CustomCommandFlunkyO can be used to issue a custom command call to any available flunky (other than the favorite flunky) of a ClientServer . server. This function can be used within a ClientServer server cluster. The call is asynchronous and no reply is anticipated. This routine would preferably only be used for commands that are synthesized , internally to the server, not for relaying commands originating from a client. The purpose of using this routme (rather than creating a seperate non-server flunky explicitly) is that any flunkies created in this manner become part of the server accounting for how many 'clients' it is busy handling. This ^•behavior may be necessary if the original command handler must complete and reply to the client but the processing of the clients command is not yet fully complete.

In the prefened embodiment, the function SV_GetDefaultQueryFunc() returns the address of the default query handler function provided by the server package. For example, in cases where a custom query handler function has been registered for a server, it could perform code similar to the following: if ( queryText starts with custom flag sequence (e.g., "SQL:") or 'funcName' is our special type ) hitList = perform the custom query using whatever method is applicable else . qFn = SV_GetDefaultQueryFuncO hitList = qFn(...) - return hitList

Alternatively, the standard query could be allowed to execute and modify the resultant hit list.

Logical Mass Storage Functionality

^• . Although the server API allows the direct registration of all plug-ins necessary to handle robotic mass storage,in the prefened embodiment the Logical Mass Storage layer also provides these plug-ins for an archiving server. This is primarily because the complexity of actually implementing mass storage conectly means that a higher level, simpler API is required that performs all the MSS functions and requires only the registration of a number of 'driver' plug-ins in order to allow control of a new robot or media type. A number of support functions are provided to make the definition of new robotic drivers a relatively easy process. To do this, the abstraction defines a robotic storage device to be comprised of a number of standardized components, and the registered driver maps the actual robotic arrangement into this logical model. In the prefened embodiment, a storage robot is logically (though not necessarily physically) comprised of the following:

Media Slots - Each robot has some number of slots into which media can be placed and from which the media can be retrieved. Slots may be either empty or full depending on whether they contain media at the time or not. If media exists in a slot, it may be either blank (i.e., not yet written) or not. Many robots divide slots into a set of shelves and thus the driver must convert the logical slot number into an X,Y form in order to address the actual physical aπangement of slots within the machine.

Shelves - As mentioned above, media slots may be organized into groups or shelves. Sonie robots support the removal and re-insertion of an entire shelf containing many media items. In certain robots, media is held in caddies. In this case, the concept of a shelf is mapped to _,' . the caddy system and the robot may provide a separate import export mechanism for caddies. ^'

• Because of this, slots themselves have an additional logical state that they either 'exist' or they do not (i.e., the shelf that contains them has been removed). ^■ -^'

Drives - Robots may contain some number of drives which may be divided into two types, - 'writer' drives, i.e., those capable of burning new media, and 'reader' drives which are capable only of reading the burned media. Some drives may be both read and write capable.

Import/Export Slot - Most robots support some kind of mechanism for inserting and removing media to/from the robot, usually refened to as a 'mailbox'. Logically, the MSS system assumes the existence of an import/export slot in order to perform actions requiring 'the insertion or removal of media.

Transport - When media is moved from one location to another within the robot, some kind of trasport device or 'picker' is used. Thus logically, media may be in the transport during the time it is being moved and thus the transport is unavailable for other moves until the media concerned leaves the transport. ^• • .

Controller - In order to issue commands to the robot, the software 'talks' to a controller device that supports a command set allowing media movement of various types. Communication with the controller may be either by serial, SCSI, Internet, or any other logical communication means. ^'

^■ In the prefened embodiment, in order to logically refer, to the various -possible locations that media may be moved to/from, the logical MSS layer assigns the following numeric values to various locations which must be translated by the driver into the conesponding locations in the physical robot:

-- 0 - 16383 -Media Slot locations ' ' - -

16384 - 16384 + 256 - Drive locations

• 32767 -The transport ■ . 32766 - The import/export slot

In this abstraction, to add a new robot type, the following plug-ins may be used: FN. TYPEDEF cm^'dName - REQD DESCRIPTION

~.

ET_MSSChunkMover "Move" ^' Y Y ^' Move chunk from one location to another

ET_MSSSlotIniter "CheckSlotStatus" Y Phys chck if range of slots empty/full

ETJvlSSRobotlniter "InitRobot" ^' , Y Initialize each autoloader (robotNum)

ET_MSSReadyForImporteι ^• "MailBoxLockUlock" N Prep for import/export of a chunk

ET_MSSSlotStarusFn ^' "IsEmptyElement" Y Is a drive or slot is empty or full?

ET_MSSSlotStatusFn "DoesElementExist" N Determine if a drive or slot exists

ET_MSSRobotIniter , ,^: "TermRobot" ^' N Terminate each autoloader (robotNum)

ET_MSSShelfIO "ImportExportShelf N EjectRestore entire contents of shelf

Move - The move command is can be issued by the logical MSS layer in order to move media from one location to another. It is passed a logical source and^' destination address. The particular robotic driver translates the logical addresses to the conesponding physical addresses and then executes the move. Ln many cases, a single logical move command may translate to a series of physical moves in a particular robot. For example, a move from a slot to the import export tray may equate to the sequence "move transport to slot, pick media, move transport to import export slot, place media". '. ' '• " •

CheckSlotStatus - Many robots have the ability to sense if a given slot or range of slots is empty or full. Some do this by moving the picker to each slot and attempting a pick. If the pick fails, the slot(s) is empty, otherwise it is full. Other robots maintain an internal map of slot status and this is used to respond to this command. In either case, this command can be issued by the logical MSS layer to obtain this status information.

"InitRobot - This command is issued in order to initialize the state of the robot when the' system first starts up. Many robots perform some form of initialization or calibration of picker position before they are ready to operate and if this is required, it should be invoked in the InitRobot call. Also, this call is the time when the driver first establishes communication with the controller and confirms that the robot is of the type expected and that it is operating conectly.

MailBoxLockUlock - This command can be issued to lock/un-lock the mailbox slot. In the prefened embodiment, the logic of the MSS layer is that the mailbox slot is preferably locked and that in order to insert/remove media from it it must be temporarily un-locked and then re- locked when the transfer is complete. This logic is prefened because frequently in actual robotic implementations, the action of un-locking the mailbox may cause a physical obstruction to the picker mechanism and any attempt to move the picker while in this state may result in physical collision with the mailbox hardware. This same logic frequently exists - for drives that are 'ejected' in that ejecting a dπve usually means that the drive tray. is sticking out (ready for media to be put on it) and this also will result in a physical collision if the picker is moved while the drive is in this state. For this reason, the logical MSS layer preferably maintains knowledge of the state of all drives and ensures that they are retracted before attempting a move in-the robot.

IsEmptyElement - This command is similar to "CheckSlotStatus" above but requests that the robot physically verify the empty/full status of the slot. This is prefened in cases where the integrity of the logical MSS layer's map of slot contents, or that of the robot itself, may be suspect. This can occur when an operator physically opens the door of the robot and manually inserts or removes media. In this case, unless the media includes a tag or other form of ID, neither the robot nor the software has any way to be sure of what media is where in the robot. In this case, the software initiates a complete inventory for the slots and drives that is known as a "Cortical Wipe".

DoesElementExist - For robots that permit removal of entire shelves, it is usually the case that the robot has sensors that tell it which shelves are installed and which are not. This command is used to determine the existence of a slot based on this mechanism.

TermRobot - This command may be issued when the archiving software is shutting down and allows the driver to take whatever actions are necessary to bring the robotics to a stable state prior to shutdown. In the prefened embodiment, any communications path that was established to the robot controller would be torn down in the terminate call.

ImportExportShelf - In robots supporting the insertion/removal of entire shelves of media, this plug-in can be used to perform whatever specialized action is required to do this. If the robot is caddy based and provides an import/export device for caddies, this function would access and control this functionality.

To bring the discussion above more into focus, we will use some of the commercially available autoloaders to examine what is necessary to implement an actual robotic driver withm this architecture. Many different autoloaders may be implemented in this architecture. The examples chosen are intended to Olustrate die diversity of autoloader architectures, and the issues involved in implementing drivers for them:

The Pioneer P5004. This is a 500 CD autoloader containing up to four dπves that may be any mixture of readers and reader/wπters. Media is grouped mto five (removable) shelves of 100 CDs

52 each ananged vertically. There is no import export slot so this function must be accomplished by loading media to/from the picker mechanism itself which can be locked/unlocked to grip/release the media. Control ofthe P5004 is via SCSI. • ^" '

The Cygnet MiniDISC autoloader. This autoloader consists of a number of rotating carousels, each of which contains up to 250 CD DVD disks. The InfiniDisc is compπsed of a number of different 19" rack mountable modules that can be installed in various configurations in order to make up a complete system. In the InfiniDisc coordinate system, X refers to the slot number (1..250) and Y refers to the DSU number (0..11). Up to 25 racks of equipment may be serially daisy chamed and controlled via a single serial port. All I finiDISC commands must be prefixed by the rack number (A to Y) to which they are being addressed. The dπves in the mfiniDisc are numbered such that the bottom most drive is tray 0 and tray numbers increase upwards. The possible modules are:

1) CU (Control Unit) This is a 2U module containing the robotics controller. This unit is connected via various connectors on the back of the module to all the other components in the rack in order to allow them to be controlled. Control is via a serial (RS-422) link.

2) DU (Drive Unit) The drive unit is a 2U module containing two tray-loading CD/DVD drives. The DU is entirely controlled via SCSI.

3) DSU (Disk Storage Unit) The DSU is a 4U module that contains a rotating carousel holding up to 250 disks, each rack system can have Up to 12 DSUs for a total of 3,000 disks per rack. The carousel is divided into 5 equal sections (to even out weight distribution) thus every time the slot number is incremented, it will spin one fifth the circumference of the carousel.

4) ATU (Arm Tower Unit) The ATU is a vertical disk earner attached to the left side of the rack. Once a disk is picked from a DSU's extractor, it is vertically carried to any drive tray or vice-verca. No import/export slot is provided so this function is implemented by inserting removing media from the ATU.

5) The rack. ^• All modules are mounted in a standard 19" rack. v

The TiltRac DVD-RAM DAU (Digital Archive Unit). This is nominally a 480 CD/DVD autoloader which may contain up to six drives and also provides a mail slot. Control of this autoloader is via a serial link. The autoloader also supports a bar-code reader which may be used to verify media disk identity. P-5004 • . ^',

The P-5004 driver is a SCSI driver and thus makes use of the standard SCSI command set defined for autoloader devices, which makes implementation of the driver relatively simple. The driver registers the following plug-ins: ' .. '

Move - The DRM-5004 move function is recursive since logical moves often translate into a series of physical moves. The logic is as follows: ^■ , - . , '

54 CheckSlotStatus - This function simply issues a SCSI slot check command for each slot in the range, allowing the user to abort the process (via a progress bar) at any time, since it may be lengthy. ^{' ■} ' ' -' .

InitRobot -^' This function issues a SCSI "test unit ready" command and then a SCSI "inquiry" command to retrieve the unit identification and firmware version.

^" TermRobot - No action required. , ' , . ' ^'

MailBoxLockUlock - This function moves the transport to a mid position in the autoloader and allows media to be inserted/removed. , _ . '

IsEmptyElement - Issues a SCSI "element status" command

ImportExportShelf - This function first disables the robot to ensure no other movement occurs during shelf removal, then prompts the user to perform the removal, then re-enables the robot.

^• InfiniDISC

The InfiniDISC is unusual in that the controller itself is directly capable of ejecting and retracting drive trays in addition to the the ability to control drive tray position via SCSI commands issued by- the drone that is physically attached to the drive (the normal mechanism). The state of the drive trays is critical in this architecture since the picker arm may easily collide with an inconectly ejected drive tray. Communication with the controller is via serial commands. The logic used to translate logical slot numbers to physical X,Y addresses (IND_SlotToXY) is simply:

aYvalue = (slot-1) / 250; // DSU numbering starts from 0

, anXvalue = 1 + ((slot-1) % 250); .. // each DSU holds 250 CDs in this robot, slots start from 1 ' - - - -- - ^...

Move - The InfiniDISC move function is recursive since logical moves often translate into a series of physical moves. ' ^•

CheckSlotStatus -This function takes advantage of the fact that each DSU in the Irif uDISC maintains a directory of the contents and thus this capability can be implemented simply by retrieving this directory and marking the slots empty/full as indicated by the directory. This is much faster than the more obvious approach of performing a series of attempted picks from the carousel.

InitRobot - This function issues a "system boot" command followed by an "Init System" command and then an "Inquiry" command to obtain the robot identification and firmware version.

55

SUBSTITUTE SH ET RULE TermRobot -No action re uϊre^'d"

MailBoxLockUlock -No action required.

IsEmpryElemenf- Utilizes "CheckSlotStatus" above.

TiltRac DAU

In this autoloader, communication with the controller is via serial commands. The logic used to translate logical slot numbers to physical X,Y addresses (TL2_SlotToXY) is simply:

¹ - anXvalue = 1 + ((slot-1) / 60); _, // each shelf holds 60 CDs in this robot aYvalue = 1 + ((slot-1) % 60); ^' // numbering starts from 1 not 0

Move - The logic for this function is virtually identical to that given for the InfiniDISC above. - ^•

CheckSlotStatus - This function operates by attempting a pick from the slot, however, because the timeout on a failed pick could be considerable becuase the robot makes repeated attempts, this function first puts the robot into diagnostic mode so that only one attempt is made.

InitRobot - This function is similar to that for other drivers but because the DAU is capable of sensing installed drives and returning their number and location, this function makes a number of . consistency checks to ensure that the installation of the actual robot matches that specified via the archive preferences. ' , • - , . ^'

TermRobot - There are two serial connections to be torn down here, one to the controller and the other to the bar-code reader (if present). _, ^{' *} - , ^' •

MailBoxLockUlock - Issues a mail-box lock/unlock command.

^• IsEmptyElement - Utilizes "CheckSlotStatus" above.

Exhibit C provides a listing of the basic API necessary to gain access to the functionality provided by the logical MSS layer. _, ,

^' The function ML_SetPlayerState() updates a player state.

The function ML_ResolveFuncQ resolves a plug-in registry name into the conesponding function pointer or NULL for an enor. ^' The functions ML JYIarkS.o tBlaVkθ7MLJVΪar- SlotNo tBlankO,and MLJsSlotBlankO mark a slot as containing/not containing a blank media chunk. The function MLJsSlotBlankO determines a slots status as far as blank media is concerned

The functions ML_MarkSlotPresentO,ML_MarkSIotAbsentO,ML_IsSIotAbsentO mark a slot as being present/absent in/from the robot. Certain robot types allow slots or entire shelves - (caddies?) to be removed at which time all slots within the shelf become unavailable. ML_IsSlotPresentO returns the state present/absent status of the slot.

The function MLJsSlotPresentQ marks a slot as being present absent in from the robot. Certain robot types allow slots or entire shelves (caddies?) to be removed at which time all slots within the shelf become unavailable. ML_IsSlotPresent() returns the state present/absent status of the slot. , . . ^" '•

The function ML_MarkSIotNotEmpty() marks a slot as occupied empty. The function ^' MLJsSlotEmptyO determines if a slot is empty or not. ' • '

The function ML_MarkSlotEmpty() marks a slot as occupied/empty. The function MLJsSlotEmptyO determines if a slot is empty or not.

¹ The function MLJsSlotEmptyO marks a slot as occupied/empty. The function '

MLJsSlotEmptyO determines if a slot is empty or not.

The function ML_CanControlDrive determines if a drive matching the options and drone specified exists and is controllable. If it does, the conesponding robot and drive number are returned.

The function ML_VoINametoBatchIDO given a batch ID, set up the conesponding volume name and vice-verca. This function is only valid within a server/drone. _

The function ML_CallMoveFuncO is a local function used to invoke registered move functions and update other stuff as necessary.

The function ML_GetArchiveContextO is a wrapper for CL_GetArchiveContext .

The function MLJRegisterGIobalFuncO registers a global (i.e., non-autoloader specific) logical function by name. Defined logical functions are as follows: ' , '

FN. TYPEDEF cmdName REQD DESCRIPTION

ET_MSSMediaTyper "PrintMediaType" N produces C strng w readble media type ET_MSSImtTermRobotTypes "IriϊtRobotTypes" Y called once to init avail robot types

ET_MSSInitTermRobotTypes "TermRobotTypes" Y called once to term avail robot types

ETJvlSSArchiveGetter "GetAnArchive" Y get/reserve an archive partition

ET_MSSArchiveRelease "DropAnArchive" , Y Release attached bum stn for archive

ET_MSSArchiveKickOff " ickOffArchive". ^" Y Initiate the archivmg process

An examples of the function types above is provided below:

Boolean myReleaseArchive ( // Release the attached burn station >

OSType droneServerType, // I : type for drone srvr to be used charPtr volumeName, // I:Name of the volume to grab unsLong ba chID // I:Batch ID assoc w vol to release

) // R:TRUE for success,

FALSE otherwise

Boolean myGetAnArchive { // get/reserve an archive partition > unsLong batchID, // I: the batch ID

OSType *droneTyρe, // 0:Contains drone type for partitn charPtr volName // 0:ptr to returned partition name

^" ) // R:TRUE for success.

FALSE otherwise

A standard implementation of archive burning for common devices (such as CD-ROMs and DVDs) is registered internally so most of the plug-ins above will often not be required.

In the prefened embodiment, the function MLJRegisterFunc 0 registers a logical function by name for a given autoloader type. Defined logical functions are listed above.

- -hrthe prefened embodiment, the function ML_OpenCloseTray uses the function UF_OpenCloseTray to Open Close the media tray for a removeable dπve without having to have the drive mounted or contain any media. In addition, this routine updates the robot->Players[].scsiXX fields of the archive context record to match the device information found as a result of finding a mounted volume in the device specified. This latter behavior should be unnecessary but may be useful in the event that the information matching SCSI devices to players is entered wrongly. In such a case, this function would conect such a problem.

In the prefened embodiment, the function ML_CurrentItemLocation returns the server data type of the drone or master server in which the media 'chunk' associated with a given item m the server DB is currently loaded. If the item is not cunently loaded or is on the local machine, this routine returns 0. This routme can only be called from within an archive plugin of a ChentServer server. - ^{" '} -.

In the prefened embodiment, the function ML_SetDroneCallbacksO causes certain archiving callbacks to be altered for callbacks that instead send a request to the specified 'parent' server in order to control the autoloader. This is necessary in cases where a server is distributed over . more than one machine and is controlled centrally by the parent server.

In the prefened embodiment, the function ML_GetCurrentBatchDetailsO can be used to obtain details of the batch cunently being processed by a given drone/master server. The only anticipated need for such information might be for use in implementing custom file put routers (see ClientServer). ■ ,^' . . ^"

If use of the default file get/put routers provided by the MSS layer in a distributed server is ^• desired, the function MLJJseDefaultFileRoutersO could be called during the definition of the server and drone callbacks. The default routers are data type independant and for data types with specialized needs ot may be desirable to provide alternative routers. A sample embodiment of a 'put' router algorithm is provided below: ,

1) Only route to drones mat are cunently running. ._,. ^■ • •

2) If any batch is already less than 90% without adding the cunent file and adding the mcoming file (described by nP) will make any batch be > 90% complete (but not > 100%) then pick the least busy drone out of all drones meeting this criteria. If there is more than one least busy drone, pick at random from them. • ^{• ■}'^■

3) If and only if (2) is not met, then if there are any drones available for which the cunent batch will not go over 100% by adding the incoming file, pick the least busy drone from all drones meeting this criteria, if more than one such drone, pick at random from them. - ' - ^•

4) Otherwise pick the least busy from all available drones, if more than one available drone, pick at random from them. .

The goals of this algorithm are: . ^• • a) To ensure that batches are created that exceed the 90% media utilization criteria and are as close to 100% as possible. b) Once a batch reaches the 90% level, it will be burned in preference to adding to a less complete batch in order to ensure that batches do not all build up and then get flushed at around the same time thus overloading burner resources. . c) If it is possible to avoid burning a batch with less than 90% utilization, the system will do so by 'adding to an incomplete batch. . . . . d) Wherever a choice exists after applying the rules above, pick the least busy drone (based on # of clients). ^• -' - . , • • - e)- If a choice still exists pick at random. . ^• > ^"

The ClientServer file 'get* router plugin used to route file fetches from the server OUTPUT folder to the appropriate drone server. This function can be called in either the master or the drone servers whenever the marker value (see DB_GetMarkerValue) for an item is equal to ~0. In the master server call (which can be identified via the data type implied by the 'ip' parameter) the router function should select from the available active drone types to handle the request and return that type as the result, thus causing the request to be forwarded to the drone selected. In this case, any changes made to 'aFullFilePath' will have no effect. In the drone call (again identified by use of the 'ip' parameter), the router would preferably alter the 'aFullFilePath' value if necessary and then return zero as the result (implying no further forwarding is required). Alternatively further routing may be acheived by returning some other legal drone type. The default file 'get' router implements the following strategy: ^' -

1) If the media chunk for a given item is already mounted in one of the available drones or if the item is within the server output folder of such a drone, route to that drone. ' ^'

2) Otherwise pick the least busy available drone (determined by cunent number of clients).

3) If more than' one 'least busy' drone, pick at random from them.

In the prefened embodiment, the function MLJUriloadAllMediaO would return archived media chunks that are cunently loaded in a drive back to their home slots and their status is updated to reflect the move. The function is preferably implemented as a custom command and is normally called from the test program. The function is a useful testing aid.

In the prefened embodiment, the function MLJEnbDisRobotO allows Enable/Disable of movement in a specified robot. ,

In the prefened embodiment, the function MLJEnbDisAllRobotsO allows Enable/Disable of movement of all robots.

In the prefened embodiment, the function ML__IsRobotEnabled() determines if a given robot it enabled or not.^' '

The foregoing description of the prefened embodiments of the invention has been presented for the purposes of illustration and description. For example, although described with respect to the C programming language, any progran rning language could be ^"used to implement this invention. Additionally, the claimed system and method should not be limited to the particular API disclosed.

60

SUBSTPTE SHEET RULE26 The descriptions of the header structures should also not be urniteα to ne emeodiments descπbed. While the sample pseudo code provides examples of the code that may be used, the plurality of implementations that could in fact be developed is nearly limitless. For these reasons, this description is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

APPENDIX A ip = server status record switch ( event type ) '

^{ ' ^{' • •} - ^' case kSecondlD: ' // woken once per second... update time display in server window perform maintenance window tasks (if any) ' if ( no flunkies are busy )

{ - ' .^" " . if ( 5 seconds have elapsed since last activity )

{^'-

SV_SetupMessage(ip, -2,0, "Idle... ") ; if ( idler function registered & favorite flunky not busy ) SV_CommandFavFlunky(ip,kFlunkyCallServerIdler, ... ) ;

} if ( tweak record idlerlnterval has elapsed )

SV_CommandFavFlunky(ip,kFlunkyCallServerIdler, ... ) ; if ( server scans input folder & scan time has elapsed )

{ if ( favorite flunky not busy )

{ ^•' // process another file from input

SV_SetupMessage(ip, -2,kLogToTimeHistory, "Get item from %s", ...) res β SV_CommandFavFlunJcy(ip, FlunkyProcessFilePassl, ...) ;

} } else if ( server scans alias folders & scan time elapsed )

{ '

SV__SetupMessage(ip, -2, Q, "Scanning Alias Folder (s) ...") ; res = SV_CommandFavFlunky (ip,kFlunkyScanAliasFolder, ...) ; } else if ( server archives output folder & poll time elapsed ) res = SV_CommandFavFlunky(iρ,kFlunkyArch.iveCσmρletePoll, ...) ;

' } , if ( loading display update time has elapsed ) update user,cpu load, etc. values in server window

^"} • if ( stop request received ) initiate server stop _ break;„ . > cas kNotifyllD: switch ( no ifyMsg->nfyType )

^• ( case kNfylntProfile: id = notifyMsg->nfyItem;

{ . ^" case kltemCSnfyFavCmd: // re-issue to favorite flunky ccP = (ET_CustomCmdFlunkPtr)notify sg->message; SV_CustomCommandFavFlunk (ip, ccP->context, ... ) ; __^ break; case kltemCSnfyWakeServer: // client sent server wake request command favorite flunky to re-scan input folder break;

default: break;

} " ^{•' • •} break; default: break;

FAVORITE FLUNKY SAMPLE CODE ip = server status record switch (cmd)

{ ^{• • ■'} case kFlunkyProcessFilePassl: // 1st pass, process the input file ^• . . - ^{' ~ '}• . - ^• if ( ip->dataHandle ) dispose (ip->dataHandle) ; ^'err = (dp->aFileProcesserFn) (ip->sourceFile, &ip->dataHandle) ; if ( err )

^{• '} f ^• -. ." . ^{" "■} result = 1; ' ^■ // Reject item if things go wrong ^' . _ ^' ' . ' "

^" . SV_PutInItemRej ectFolder (DB_GetDBdataType (&άp->db) , ip- '

>sourceFile , YES) ; ' ' .

• if ( dp->aFileDeleteFn_ ) . . // need special delete?

. (dp >aFileDeleteFn) (ip->sourceFile, . . . ) ; - break; . ^{• '• •} '•

}

, if ( dp->aPiconMakerFn ) • // picon maker registered?

. if ( ip->^'piconHandle ) dispose (ip->ρiconHandle) ,- ip->piconHandle = (dp- >aPicon akerFn) (ip->pRef ID, ip->dataHandle) ;

;^'} - ^{• •} - ^'. ^• • break; ^' , . ' case kFlunkyProcessFilePass2 : - • ' if (ip->dataHandle) • - .''. , ^"• ' . ' . II Try to log the item

{ _, . ^'. ' ^'// If logging fails, reject item myDescriptor = create new descriptor^' if ( !SV_AddItemtoDB (ip, myDescriptor) ) result « 1;_ else . • . . ' ^' // add to the item DB { . err = DB_AddItem(&ip->db, ... ) ; // add item to server DB if ( err ) move item to reject folder

} if ( ip->descriρtor ) • ' // we had a descriptor left over

{ if ( dp->aDBSrecTermFn ) // descriptor terminator. presents err = (dp->aDBSrecTermF,n) (ip->descriptor, ...) ;

}

} else

^■ {

SV_PutInItemRejectFolder (...); • .- if ( dp-?aFileDeleteFn ) - -

^" (dp->aFileDeleteFn) (ip->sourceFile, ...) ; ^' ■ } . . ' .- . .' • _ , break; ^{• " ' •} case kFlunkyChecklnterestProfiles : // check item against all

IPs hits = SV_ScanAllIPsFor atch(ip, ... ) ; break; „ .

65

S E SHEET (RULE 26 WLj 11 'UJ :^;.".-U. ii " ^■••■.m^,.:"H

. case kFlunkyReScanOutputFolder: // re-scan output to check for batch if ( ip->scanBatchID | |. SV_ScanAllFiles (ip) ⁾ ))

, - SV_CopyMarkedFiles dp) ; // Copy files to archive volume ^' . result = 0; break; ' case kFlunkyCallServerldler.* // call the server idler function if ( dp->aServerIdleFn )

(dp->aServerIdleFn) 0 ; ^'SV_PdllPingClientList^'(ip) ;^' //^' ping auto-ping client list ~ . • result = 0; _ . ^' ._ break; . case kFlunkyScanlnputFolder: .// scan input folder for new files ^• . - ^{• ■}

•^{" •} SV_CheckForFiles (ip) ; ^'• . // Chec ^'for a new item - - , result = 0; brea ; '^' ' case kFlunkyScanAliasFolder: // scan alias folder for new files . result = 0; - break; , ' case kFlunkyArchiveCompletePoll: ^"// poll for archive completion . - if ( dp->anArchivePollFn )

^"result = (dp->anArchivePollFn) () if ( result ) , .

^: clean up batch ^" '' • •^{• • '}, break; • ' • case kFlunkyRepartitionOutputFolder: // re-partition output folders ^■

SV_RecomputeOutputFblders (ip) ; • / ■occurs i 'media type changes , ,' • ^' . result = 0; ' ,

. break; ^' .^"'' ^' . - case kFlunkyCustomCommand: // custom command favorite flunky if (^,_ip->customCmdPrσc ) ^' result = dp->customCmdProc) (ip->customContext, options) ; - ip->customCmdProc = NULL; ip->customContext = 0; ^" . break; ■ case ... : '

^■■ many more cases ' ' . break; ., •^• . ^' .

APPENDIX B

OSType CL_GefcServerClass . ( ' // cnvrt ClntSrvr type to srvr AE clss ~. OSType aDataType // I:The client data type ) ; // R:The: mirror imaged server AE class . : ', '

Boolean CL_Does emExist ( // Ask server if recognizes item ID OSType aDataType, // I:The IS data type server required

ET_ϋniqueID . anltemlD, •_ . // I:The item ID you want charPtr ileName ^• // I: If non-NULL, holds item file name

' ⁾ •^• . ' ^• //' R:TRUE if recognized, else

FALSE ^' .

Boolean CL_DisρlayMessageOnServer( // Display msg in srvr status window , OSType aDataType, // Iiϋnigue type code for srvr type charPtr , aMessage • , ' // I:The message to display

⁾•-.1 . ." // R.TRUE for success, FALSE otherwise ' ^■ \

Boolean CLjSetServerLocation ( // Get the server locn for IP dat type ', OSType aDataType, // I:data type code for IP data type charPtr location // 0:The server locn (2SS chars min.)

); // R:TRUE for success, FALSE otherwise

Boolean CL__DroneToLoca^' ion ( // Get the drone location osτype aDataType, // I: Unique type code for this server : ^• OSType aDroneType, // I:The drone server data type charPtr location // 0:Drone locn (at least 2SS chars) ); - // R:TRUE for success, FALSE otherwise , _, ,

OSType CL_^LocationToDrone _, ( // Srvr have drone at speed IP addrss? - , OSType aDataType, // I:Unique data type code for srvr charPtr ' location // I: he drone IP address ); // R:Drone type/name, 0 if no drone ^■ . Boolean CL_ClrServerAddress ( // Clear srvr address assoc w dat type OSType aDataType // I:Data type code for srvr data type ); . // R:TRUE for success, FALSE otherwise void CL_SetDefaultNetCopy ( // Spec default ntwork file copier fen

ET_MetCopyRespoπder aWetCopyRespFn, // I:Copy responder function -- server ET_NetCopyStarter • aNetCopyStartFn, // I:Copy starter function -- client -• ET~NetCopyIdler aNetCopyldleFn, // I:Coρy idler fen (or NULL) - client ET_NetCopyEπder aHetCopyEndFn // I:Copy ender fen (or NULL) -- clnt ><^• // R:TRUE for success, FALSE otherwise- void CL_GetDefaultNetCopy ( // Spec deflt netwrk file copier fen

ET_NetCopyRespαnder ^♦aNetCopyRespFn, // 10: If non-NULL, copy resp fen -srvr ET_NetCσpyStarter ^♦aNetCopyStartFn, // 10: If non-NULL, copy strtr fen -clnt ET_HetCopyIdler ^♦aNetCopyldleFn, // 10: If non-NULL, copy idler fen -clnt ET_Ne CσpyEπder *aNetCopyEndFn //^■ 10: If non-NULL, copy ender fcn-clnt ); // R:void

EngErr; CL_AddResourceToFila ( // Insert DB resource struct into file OSType aDataType, / I:The unique IP data type code void* itemDescriptor , // I:DB descr struct to add to file ' _ ' charPtr aFilePath, . // I: Source file (modify) path name charPtr anltemNa e // I.-Na efor resource/itam

>; // R:0 for success else error

EngErr CL_GetResourceFromFile ( // Insert DB resource struct into file OSType aDataType , // I -The unique IP data type code , void* itemDescriptor, . // 10: Item descr strict buffer

\ ; ^■ charPtr ■ ^' ' aFilePath _ // I:Sαurca file (extract) path name ) ; • ^■ -' // R.O for; success else error

EngErr CL_Put nItemInputFolder 1 • ^' // Create file; output DB res struct

OSType _ aDataType, // I -.The unique IP data type code void* itemDescriptor, . // I:a database descriptor structure char *sourceFile, // I: Source file (copy) path name

Boolean • . ' , ■ delateOriginal, //.I:If true, del orig file after copy charHdl . ^" newFilePath, / 10: If NON-ZERO, contains file path intS4. ^■ . optionβ ^• // I:Vaπous logical options ) . // R: 0 for success else error

EngErr CL_PutKInItemInputFolder (. // Takes data handle puts into file in . the server input foldar

OSType ^' '^• aDataType, . •// X.The unique IP data type cade

, ' void* itemDescriptor, // I:a database descriptor structure char . . _, *£ileName, // I:The created file name (not path)

- , ^' charHdl newFilaPath,. // 10: If NON-ZERO, contains file path

Handle itemDataH, // I:Hndl contains data item to send

• - . OSType filaType, ^■// I:The file type required

OSType fileCreator, // I :The file creatror required

- • intS4 options _, . // I: arious logical options ); - // R:0 for success else error

^de ine kSynchronousFileCopy 0x00000040 , // options -- synchronous file copy

#de ine kOverwriteExistingFile OxOOOOOOao .^' // overwrite exist file if dup'd

{♦define kNoServerWake 0x00000100 ^"// don't wake srvr after put the file

#de£ine kOescriptorlncluded 0x00000200 // file already has dscrptr, don't add

EngErr CL_AddItemUsingServerAlias { // Create server alias to a ile

OSType aDataType, // I:The unique IP data type coda void* - itemDescriptor, // I -.A database descriptor structure charPtr networkPath, // I : Zone/machine name containing file char - ■ *sourceFile, // I.-Source fila to copy path name •

Boolean delateOriginal^', • // I -.If true, del orig file after copy charPtr subFolderPath, ' // I:If non-NULL, alias server sb-path

" . ' charPtr username, // I:client user name (32 chars max) charPtr sPwd, . // I:usar PW for containing machine

- . ^• charPtr vPwd, // I-.user password for volume (temp) int32 options _, // I: Various logical options

- . ) ; - '^* '// R:0 for success else error

^• EngErr -!L_GatServarStatus ₍ ( ' // Check IP server status for dat type

OSType anIPdataType, , // I : Code identifying the IP datatype long ^♦optionFlags, ^" // I: mask for desired status flags

,' charHdl 'explanation - // 10 -.NULL or return explain strings

); . ^' • // R: Success 0 else error number ftde ine kNoErrorReport kErrAlready ogged // suppress err reporting/Alert dialog

Sde ine klsOiskMounted 0x08000000 // IP disks locally mounted Bit test

#define klsServerRunning 0x04000000 // IP server running on srvr Mac Bit

Sdefine kListServerPrαblems 0x02000000 // obtain list of server, problems Bit .

Sdefine kGetServerSummaryText 0x01000000 // obtain summary of server situation ftdefiine kOmitArchiveSummaryText - 0x10000000 // Omit tarchive info from summary txt

Sde ine kServerEvaluatadFlags (kListServerPrαblems+ kGetServerSummaryText+ kOmitArchiveSummaryText + OxOOFFFOOO)

- Sde ine kClientEvaluatedFlags (klsServerRunning+klsDisk ounted+OxOOOOOFFF)

Boolean CL_lsServerLocal ( ^■ //^' check if srvr can run on machine

OSType . ^■ aDataType, . // I: Data type coda for IP data type

- cHarPtr - ^' aMachinePath // I: If non-NULL, machine path checked ) ; - // R.-TS.UE if it can, FALSE if not

Boolean CL DataTypeHasServer ( ' , // check if data type has assoc server

OSType aDataType // t:The unique data type code

) ; // R-TRUE if it has a srvr, else

FALSE '

ET UniquelD CL GetNextlDFro Server ( // Get next ID in sequence for server OSType aDataType^"" . // I: ata tyoa code for srvr data type

// R:_,Unique sequence ID, zero if error • * ' « » ' - unsLong CL_GetTimeoutTick3 • { // get the timeout in ticks for given

CL_SendServerCommand options word , i_nt32 options, // I: he options word ^■ -

,' OSType , aDataType • // I:Send command-servr data type

"^' ' . ); // R:corrspending timeout in ticks . _

EngErr CL_SendServerCommand ^•", ( ,. // Send a command to a server long aCo Timand, // I: The command you wish to execute

^• - OSType . aDataType, "II I: Send command-servr data type void **aDataBuffar, ' ." // lO:Buffr pointrs send or receive long *aBufferSize, // IO:Give sand/receive data bffr size charPtr anldentifyingName,// I: IP Name (NULL for non-IP cαmmnds) intS options. 11 1 -. Various logical options

- - int32' contex , // I: Context value fr nfy

'seqNumber' -, •

// I: Additl params -internal use only

^•- )^": // R: Success 0 else error ftdefine kNoErrorRaport kErrAlready ogged // suppress err reporting/Alert dialog

Sde ine kWantDescRecs 0x00000001 . // (CL_FetchItemPreviews) rqst DB reds .^' ftdefine kDontWantData . 0x00000002 - // (CL_FatchItemPreviews) "no prvw data ftdefine kRetumSrvResponse 0x00000004 // wait/ return server response record ftdefine kAsynchSrvResponse O OQOOOOQS // dont wait -» asynchronous response . . ftdefine kResponselsIteraList 0x00000010 // expected response is an item list ftdefine klsArchiveCommand .^' .- 0x00000020 // direct cmd to servr archiving code ftdefine kuseltemlndeces 0x00000040 " " // CL_FetchIteraPreviews) item indeces ftde ine kSynchronousFileCopy 0x00000040 // (PutlnltemlnputFolder) synch fl cpy ftdefine kOverwriteExistingFile 0x00000080 // overwrite existing file if dup' d

"define kNoServerWake 0x00000100 // don't wake server after put the ls ^{" ■}-

#define kDontCleanFileCopy .0x00000200 // passed to CL_GetFilaBasedItatn(J

#define kUseAlternateNetCopy 0x00000400 //- used data type specific network cpy , ftdefine kNoSynchronousTimeout 0x00000800 // don't time out await synch rspnse ^• ftdefine kLongCommsTimaout QxOOOOlOOO / I use the extended timeout period ftdefine kShortCorπmsTimeout 0x00002000 , // use the short timeout period ftdefine kAliasToItemOnly 0x00004000 // manipulate alias to item, not origl ftdefine kNoProgrβssBar 0x00008000 // don't show a progress bar ftdefine kCustomCmdOptionl 0x00010000 // (κθ) not used by Mitopia™, custom ftdefine kCusto CmdOption2 0x00020000 // not used by Mitopia™, custom Sdefiπe kCu3tomCmd0ptioa3 0x00040000 // not used by Mitopia™, custom #dafine kCustomCmdOption* - 0x00080000 ' // not used by Mitopia™, custom Sdefine kCustomCmdOptionS 0x00100000 // not used by Mitopia™, custom ftdefine kCustomCmdOptionS-- 0x00200000 // not used by Mitopia™, custom Sdefme kCustoraCmdOption7 ^•" 0x00400000 // not used by Mitopia¹", custom ftdefine .. - kCustoraCmdOptiona 0x00800000 // not used by Mitopia™, custom

EngErr CL_RelayCommandToDrona ( // Send a command to a server drone long , aCommand, // I: The command you wish to execute

OSType • ^•aDataType, - // I : The data type for server cmd

OSType aDronaType, // I : The drone type you wish to void - **aDataBu fer, // IO:Addr. of send/receive bffr ptrs long ^♦aBuffarSize, // IO:Size of send/receive data bffers • . charPtr anldentifyingName,// I-. Name of IS

, intS4 options. // I : Various logical options

* int3 context ' 11 . 1 : Context val for nfy 'seqNumber'

); ¹ // R: Success 0 else error

EngErr CL_vCallCustomFune ( //^■ Call ClntSrvr data type fncn/cmmd long ^• aCommand, // I:The command/function to execute

^• OSType .• ■ . aDataType, . // I: The ClientServer data type int32 options. //^' I: Various logical options va_list ^' *theVaList // I: address of va_list value

) ; // R:Zero for success, else arr ft

69

SUBSTITUTE ET (RULE 2 <-τ—u~u ..JI i - --i -!-■ Ill -JI.- '-J- —■ "^■ " — " "' ""'" ""^■"

EngEri CL πallCustomFunc ( // Call-fnctn/cmmnd for ClntSrvr type

- . long aCommand, // I :Command/ unction to executa

, ' • .OSType aDataType, // I: The ClientSa^'rver data type int 2 . . options,^" // I: Various logical options ' //^'IO:Additnl fen params necessary

⁾'^• '. • // R:3ero for success, else err ft void CL_NeedServerComms " ( //^'declare intent to cα m w/ IP srvr

OSTypa dataType ' // I: ype/server to communicate with ); , . • // R:vo d void CL_DumpServerCσmms - ( // disconct from IP comma w data type

^_ . OSType dataType -'// I:data type/server to comm.. with

- ^' // R:VOld void CL_SetInitTermCall ( ^' ' -// fens to init & term Entry points

ET_IPinitTermFn anlnitFnToCall // I: Fen call address - initialisation

ET_I9inιtTermFn aTar FnToCall , //^"I: Fen cal address - termination

OSType dataType, // I:data type/srvr call applies to

. Boolean serverDe inition // I: f true, applies to srvr, not clnt

... - ' ) ; // R:void void CL_ClrInitTermCall ( ^' // functions to init &. term Entry pn s -

ET_IPinιtTermFn anlnitFnToCall // I : init fen address to clear

ET_lPiπitTermFn ' aTermFnToCali, // I -.term function to clear (or NULL)

^• OSType dataType, // I:dat type/srvr which call applies

Boolean serverDefiαition // I: If true, pplies to srvr, not clnt

, ) ; // R:void

EngErr CL_—DefinelP ( // resolve an II? reg name into IP pter

. ET_IPRecordStr ^•*ip. // IO:Pntr to^" the IP rcord created/alt , ' int32 " ' , options. .// I:Var logical optns for IP behavior , int32 events, - // I:event types mask for IP trigger

OSType aDataType, // I -.Data type/server - IP applies char urgency, _ // I: rgency level for this IP char voicelπdex // I: spec' d voice index (if speak otn) . ^■ ¹ -

^• short iconID, //,I:icon ID to identify this IP__ charPtr name„ // I: Name (hiεrarch using : ' s) of IP charHdl expression, - // I:Expressn to be eval'd by IP

' void *context // I: Context value to be passed ); • // R:Sero for success, else error code ^' . -

-ftdefine kSpeaklPname 0x00080000 // speak IP name itself on notificatn ftdefine kSpeaklPitem 0x00040000 // speak item name text for IP on notf ftdefine. kNoIPuiEffeet • 0x00000200 II IP trigger not impct view menu etc. ftdefine k ustLocalise 0x08000000 II localize/alter an existing IP rcord

. ftde ine klsServerlP 0X04000000 // IP is associated with a server ftdefine kSaveThisIP 0x00000400 // if flag set, IP auto save/restore ftdefine kMatchltemAdd 0x01000000 ll trigger when a new item matches ftdefine kMatchltemWrite 0X02000000 // trigger when match item is written ftdefine kMatchltemDelete 0x04000000 II trigger when matching item deleted ftdefine kMatchltemRead OxoaoooooQ II trigger when matching item is read ftdefine kUserConditionMask OxOOFFFFFF ll mask for event bits custom defs

ET_IPRecordPtr CL SetlPliatHdr ( - ' // Get the Interest Profile 1st header

.void _r

. - ); // R:Liat header, NULL if none void CL_UnDefineIP { • // de-alloc CL_DefineIP create IP rcrd

ΞT_IPRecordPtr ipr // I:The IP record to ba disposed of

⁾;, ^' - // R:void

Boolean CL_RegisterIP ( // register an interest profile

~^" ΞT_IPRβcordPtr alnterestProfila, //^• ;pntr to rag. interest profile charPtr aName , // • IP to be retrieved name i int32 options // I: arious logical options

) ; // R-.TRUE for- success, FALSE otherwise - ¹ •

Boolean CL_DeRegistarIP ( ¹ // deregistar an interest profile charPtr ^"- aName // I. name of IP to ba de-registered

) ; ^" // R:TRUE for success, FALSE otherwise , '

< » <

ET_IPRacordPtr CL_ResolveIP ( • // resolve IP reg name into IP pointer

^" charPtr aName, ' // I.-registry name of Interest Profile

Boolean errorOnFail // I.When TRUE, reports err on fail

^' -) ; - // .R: Interest Profile pntr or

NULL=-err - . -

EngErr CL_GatIPfiald ( -'- // get value of fild assoc w/ int

Prof

£T_IPRecordPtr ipr, // I -.The IP record short . fialdSelector, // I: Selector for the field required void ' *outBuff // 0:Address of buffer for output val . .

• ) ; . // R:Zero for success, else error ft -

' ftdafine kIPoptionsFLD 1 // alue passed in or ' options ' , ftde ine klPβventTypFLD 2 , // 32 bit event mask word ftdefine klPdataTypFLD 3 • • // ClientServer data tyoe for this IP ftde ine kIPurgencyFLD 4 .- // The urgency level for the IP ftde ine kIPvσieelndexFLD S ' // voice indx when speak IP name, etc.

,. ftde ine kIPiconldFLD s // Icon ID to identify this IP ftde ine kIPcontextFLD 7 ," * // Value passed for 'context' ftde ine kIPnameFLD a _ // The name of the IP ftdefine kIPexpressionFLD 9 // The DSL expression define IP match ftdefine kIPαiachineNamFLD 10 // The machine name (servers only) ftdefine kIPnextlPFLD 11 // Address of next IP in the IP list ftdefine kIPuserNamlPFLD . . ■ 12

-' , // The user name (servers only)

int32 CLJSetDataTypeOptions ( // Get optns when data type created

OSType aDataType // I -.Data type

) ; . ' // R-.pointer to tweak record,

NULL=err - ,

EngErr CL_FetchItemPreviaws { _' // Collect item praviewg from a ^'server

OSType aDataType, // l:Req'd data type server

OSType aDroneType, // I : Drone type to talk to, norm 0

- long* itemCount, // 10: Expect/actual srvr item count _ void* itemDescriptorList,// IO-.Database descriptor array

, . int32 options //. I: arious logical options

₍ ); - - . /./ R:aero for success, else err ft ftdefine kWantDescRecs 0x00000001 // request DB descriptor records ftdefine kDontWantData 0x00000002 // don 't actually want the prev data ftdefine kUseitemlndeces 0x00000040 // item indeces, not IDs passed ftdefine kNoProgressBar 0x00008000 // don 't show a progress bar

Boolean CL_IsFileCached ( // Determine if a file in file cache

OSType aDataType,. // I:The file's server data type charPtr aDataTypaName , // I : If ' aDataType ' ->0 , data type name

ET JniquelD _' itamID, // I: Item ID charPtr - dat ltemNama, // I:Qptnl data item name, NULL if N/A charPtr cachePath, // 10.-If non-NULL, cache file path int32Ptr dbIndex // IO: If non-NULL, DB index on exit ) ; //'RiTRUE if it is, FALSE, otherwise „

Boolean CL_AddFileToCache ( // add file path to cached^' files DB

. • OSType aDataType, // I.The data type charPtr aDataTypeName, // I: If 'aDataType' =0, data type name

ET_UnιqueID i amID, // I: Item ID

- charPtr aFilePath // 0: File/cache file path entry/ exit , , ^{' )} ; . // R:T UE for success, FALSE otherwise- , ' .' ,*

Boolean C _ _AddDataToCache _, ' ( . // add mam data bib for cached file DB •

OSType ' ; . aDataType, // I: The data type

'charPtr . ^{' '}. ' aDataTypeName, // I: If 'aDataType' =0, data type name

ETJJniquelD ^• . itemID, // I: Item ID

- - charPtr . _ , - ^'_ ^■ dataltemNa e, /'/ I:0ptnl data item name, NULL if N/A

Handle • ^" .; ^• ' data // liThe data 'blob' to be added

:^'-.^' - ' ^; - ,//^*R:TRUE for success, FALSE otherwise -. . int32 CL_ScanCacheForNewFiles _ (_ - ^• Λ // Updata file cache DB - ext files

." • void

- , _^ ),;_ ; II Recount ft of iles^" added to cache '

Boolean CL_ _PurgeCache ( // Cheek cache disk usage - double ^'.. *aPurgaRatio, . ' - // I:% of disk should be free ' ^" ^', OSType ^' , ^■ aDataType // 1:0 = purge all types, else specify

• ' ' // R:TRUE if enough dsk space obtained ,' .•

Boolean CL_ _DeleteFileFromCaehe (. ,_t -// Delete a file from the cache ^■ '

OSType ^• - ^' aDataType, // I: file server data type, 0 unknown charPtr aDat TypeName, /./ I: If 'aDataType' =0, data type name . '.

,ET_UniqueID . itemID, // I:The item ID, 0 if not known

. charPtr ,- '. aCacheFilePath . // I:The file name/path in the cache

) ; . // R:TRUE for success, FALSE

'otherwise _v - , .

Boolean CL_ _GetCacheFilePath _, ( // Get file path in cache fr data item '

OSType , . _ ' . _, aDataType, // I:The file's server or group type charPtr ^' • ^{" ' •} _, aDataTypeName, /'/ I: ame of data type or NULL

• ET_UniqueID itemID, // I: he item ID

- charPtr .•'_, -■ datal emName, // I:Optnl data item name, NULL if N/A charPtr ^• ' . ^' aCacheFilePath // 0:The file name/path in the cache

) ; . //.R:TRUE if the file is present

EngErr CL_ ΞueltemFile ' , ( II Cue ^"an item file within the server

OSType aDataType, // I:The file's server data type

ET_UniquaID . itemID, // I: Item ID

* charPtr • ' , ^' aNetworkFilePath // 0: If non-NULL, netwrk file path

^' ); // R:2ero for success, else error ^■ ft

EngErr CL FetehltemFile ( . // Fetch file-ba3ed item into loci fill

OSType aDataType , // I:The file's server data type ^•

__ ETJJniquelD . itemID, // I: Item ID charPtr anOutputFilePatb, // I: file name/path for item void* aDescriptorRecord, // IO-.If non-NULL, tem descriptor int32 options // I: arious logical options

' );^' // R:aero for success, else error ft -

Boolean CL _DefineEventType ( // define a server event type

OSType . - aDataType , // I.Data type for which defn applies int32 eventMask, // I:Mas defining the bit type positn • ' " , - charPtr \ eventName //• I: String - dialogs ref, the event

.'.^'- >^; V/ R.TRUE for success, FALSE otherwise _'

Boolean CL _MaskToEventType ( /./ Get event type mask for txt deser.

^• OSType aDataType , // I: Data type this defn applies to int32 eventMask, // I:Mask defining event type bit posn * ' - charPtr eventName II 0:Stπng in dialogs ref g the event^'

• ); // R:TRUE for success, FALSE otherwise " ^' .

72

SlfflStlTiU E SHEET (BULE 26)

73

BEET (RULE 26) ftde ine kFnArchiveRecIniter 2S " ^• II Fn. initialize an archive DB record ftdefine kFnCustomArchiveCmdHdlr 27 II Srvr call rept of custom arch cmπd ftde ine kFnCustomArchiveFuncHdlr_ 2a ^{' ■} ₍ II Client call process oust arch fns ftdefine kFnltemlnfαGetter 29 . II Function get info about an item ftdefine kFnArchivelnfoGetter ,30 .^" II Fn get in o.' about an archive chunk ftdefine kFnArchiveLUNGe 1er. 31^' ' 11 Function to get archive LUN list ftdefine kFnRe ar itionN yer , 32 - ' ^• ' 11 Fn notify repartition start/end ftde ine kFnArehiveCopier ' . 33- II Function call copy files to archive ftdefine kFnDBrecGutsClone 34 ^{■ '} II Fn clone descripr record attachmnts ftdefine kFnServerldler - ' 35 ^{" '} II Fn call when server is idle ftde ine kFnfilePutRouter . ^• 3S 11 Fn route mcom' inpt fles to dr es ftdefine kFnFileGetRouter 37 - • 11 Fn route don't cara fetehs fr drnes ftdefine kFnBusyCounter 38 II Fn current clnt load in server

_; ftdefine kFnQueryHandler ³⁹ " * // Function to handle standard - queries . ftdefine kFnClientWiper . : 40 . , // hdl clnt startup/shutdown announce

Boolean CL_DefineDataType ( // Define new srvr data type and srvr charPtr dataTypeName, // I: The name of the data type

- , OSType aDataType, // I -.data type code for srvr data type charPtr aServerMachine // I:Machine IP address for server charPtr an nputFolder, // I -.ntwrk fie path to srvr inpt fldr

¹ ,, charPtr anOutputFolder // I:ntwrk fie path to srvr outpt fldr charPtr . ^ ^' aRejectFolder, // I:ntwrk fie path to srvr rejet fldr charPtr anAliasFolder, // I:ntwrk fie path to srvr alias fldr charPtr ' aCollectionsFolder, // I : ntwrk fie pth to srvr coin fldr

' ET_IPserverTweakRecPtrtwea , // I : Server tweaking parameters

. int32 . . , . sizeo DBSdeacr , // I: size of db descriptr record type

• - _, int32 options , // I.-various logical options

_ . ' ' ); // R:TRUE for success, FALSE otherwise ' * - ftdefine klnputParamlsPref 0x00000001 // input folder param passed is a pref ftdefine kOutputParamlsPref 0x00000002 // output folder param passed is pref ftdefine kRejectParamlsPref 0x00000004 // re] set folder param passed is pref, ftdefine kAliasParamlsPref oxoooooooa // aliases fldar param passed is pref - , ftdefine kServerParamlsPref ' . 0x00000010 // srvr name param passed is a pref

- ftdefine kThisIsDroneType 0x00000020 // defn pertains to drone type ftdefine kDroneHasParentBehaviors 0x00000040 // drone implemnts parental behaviors ftdefine kOmitTypeFromChildList oxooooooao // omit type from a- db child list ftdefine kCollectionsParamlsPref 0x00000100 // collns fldr param passed- is a pref

EngErr CL_StandardHitListQuery( '

// Snd query cmmnd to srvr(s) get hits

OSType aDataType, // I -.data type for server cmd send

> " , • ET_TypeID aTypelD, // I: Server data type being queried charPtr qFuncName, // I -.MitoQuest™ fn invoked in server charPtr queryText, // I:Text string specifying the query

ETJϊitList pruneList, // I: If non-NULL, hit list to modify

ET_HitList •resultList, // 0:The resit hit list, NULL if empty

- - - -' int32" maxHits, // I: If non-zero, Max ft hits to^' return int32 minRelevancβ, // I: If non-zero,_ Min relevnca (1-

100) aaonPtr - additionallnfo // 'I:If non-NULL, heap pointer iπt32 . - options // I: arious logical options ) // R: Success 0 else error ftdefine kLimitToPruneList 0x00010000 //' limit hits to those in prune list ftdefine kExclusiveOfPruneList 0x00020000 // remove prune list from 'hits' found . ftdefine kCombineWithPruneList oxoooooααo // add the two^' hit lists together ftdefine kSortByType D 0x00100000 // sort resultant hit list by type ID ftdefine kSortByUniquelD 0x00200000 // sort resultnt hit list by unique ID ftdefine kSortayRelevan^'ce 0x00300000 // sort resultnt hit list by relevance ftdefine kSortln ncreasingOrder 0x00400000 // Sort in increasing order ftdefine kPruningOptionMask- OxOQOFQOQQ // mask optns resrvd pruning behavrs ftdefine kSortingoptionMask OxOOFOOOOO // mask for optns bits fr sort behavrs \ » . .- - APPENDIX C '. ^{' •} ftdefine kDontCloseTrayWhenDone ^• 0x40000000 ^' // dont close tray aft"er moving drive void ML_SetPlayarState _, ( // Update a player state ^r ^"" short robotNum, II I : robot -that we need a drive or . , ' , short playerNum, // I : Player number • - ' short newState // I: The new state . ^■ J ' - , . // R:vσid ^' . -

ProcPtr ML_ResolvaFune ' , ^' ' ( // resolve fn ptr by robot type/log fn ~ ' ET_CSStatePtr ' sp, ^{' '' ■} - // I: Server state pointer - , ' charPtr •^• , loaderType, // I:Autoloader type charPtr . . : ,^' . cmdNama // I : Logical command name ' ) ; . ^■// Rihandler fn, NULL if none found ' , . * . *

EngErr ML_MarkSlotBlank ( ^' . • // mark slot in robot contn blnk chnk unsLong robotNum, // I: robot ID number unsLong slot // I-.slot

. '^• ) // R:Zero for success, else error ft ' ' ^' -

EngErr ML_MarkSlotNαtBlank ^' ( // mark slot in robot as non-blank • unsLon ^• robotNum, // I: obot ID number ' unsLong ' , - ^{" '}. slot // I: slot

) ; ' ' ^{' '} // R:Zero for success, else error ft . ^' . ^*. - ^' . •_

Boolean ML_IsSlotBlank ' ( • // autoloadr slot contain blnk media? unsLong ' _λ . robotNum, // I : robot ID number unsLong slot ' // I: slot ^'• ) ; • •^{' "} // R:TRUE if blank, FALSE , otherwise

,

EngErr ML MarkSlotPresent (^' '// mark slot in autoloader as present

. unsLong robotNum, //.I: robot ID number unsLong ' .. slot ' // I: slot ) ; , // Ri Zero ^'for success, else error ft

EngErr ML_MarkSlotAbsent - . { // mark slot in autoloader as absent unsLong robotNum, // I.-robot ID number unsLong ^' slot // I:slof )-; ' // RiSero for success, else error ft . ^' - ^' . -.

Boolean ML_IsSlotPresent ^■ ( // is slot in autoloader present? ' .. . . - unsLong robotNum, ^' // I:robot ID number uns ong slot, - ^' // I-.slot Boolean forcePhysioalCheck// I: If true & supported, physicl chck ); II RiT UE if present, FALSE otherwise ..

EngErr ML_MarkSlotNotEmpty ( // mark slot^' in autoloadr as not empty . unsLong robotNum, ^'// I: robot ID number unsLong slot // I: Slot

.),- // R:Zero for success, else error

EngErr. ML_MarkSlotEmpty ( ' ^'.^' // mark slot in an autoloader as empty ' unsLong ' - robotNum, // I: robot ID. umber unsLong slot • // I: slot ^'• > '. // R: Zero for success, else error

Boolean ML_IsSlot£mpty ( // is slot in autoloader empty or not? unsLong robotNum, // I • obot ID number

, unsLong „ . *targetsize, // 10: If non-NULL, tgt max/opt btch siz ¹ - ^• ' unsLong ^■ 'blocksize // 10: If non-NULL, target block size v- ' // R:0 for sucess, else error number void ML_UseDefaultFileRouters ( ^■ // Config srvr/drone use dflt fie rtrs OSType - ' aDataType, // I . server/drone data type involved short ' algorithm // I:Algorithm to use '^• _) ; - // R:void

EngErr ML_UnloadAllMedia- . ^> (. // Rtn mounted media chunks 2 home sit short ^■ ' robotNum, // I robot being messed with! int32 shelf // I: If non-zero, unld shlf drva media ^{• •} ); //- R: noErr or error code value - - ' " ' ^' ,

Boolean ML_EhbDisRobot •_, ( //Enable/disable movement in robots int32 robotNum, 1 1 : obot number .^' - Boolean enable. 111 : True to anaJble, False to disable //Previous enabled state

Boolean ML_EnbDisAURobots ^■ // Enable/disable movement in robots Boolean enable. 111 : True to enable, False to disable

) : //Previous enabled state

Boolean ML_IsRobotEnabled //is movement for robot enabled? int32 robotNum //I : robot number

//R:TRUE if enabled, else FALSE

APPENDIX D

•^' SERVERFUNCTIONALITY API CALL] : void SV showFlunkyWindows ^' ( // Ensures all flunky windows show

* . - '* ) ; - // R void int 2 SV_WidgetModalEnter - _{ ^{' '} . // Proces ClntSrvr widget modal invocs

ET_ indowRef 'a indow , // i: Widget modal window

) ; . ^• _,-. // R-.ctxt val pass

CL_WidgetModalLeave ^• ^" void SV_ ιdgetModalLaave _, ( ' // Clean up ClntSrvr widget modal

. ET_WιndowRef . .. aWindow, _ // I: widget modal window int32 context ^{'" ■} // I:Ctxt value fr

CL_WidgetModalEntar ^• .

- ) ; . ^" // R:void

Boolean sV_SβrverInitComplete ^'( > - // mam srvr thread^' completed init?

~ ET_CSStatePtr ip ' ^■ // 10: Pointer to IP Server Status

^") ; , // R:TRUE if main srvr thread _, . init'd .. ^{" '} . ^' .

Boolean SV_AllServerInitsComplete ( •^' // Have lcl srves completed initliztπ?

) ; • -. // R:TRUE if they have void SV_SetArchiveΞuite _ ,( // spec dflt archive callback clntsrvr . ^' ' '

, OSType aDataType, > // IrData type or 0 to set deflt suite •^■

ET_Archi eGe ter anArchiveGetter, // I -.The .archive getter function

ET_Archivestarter anArchiveStarter, // I:The archive starter function

ET_ArchiveEπder .anArchiveEnder, // I:The archive ender function

ET_ArchivePoller aπArchivePoller, JI I:The archive poller. function

ET_DBSFilemover anArchiveFileMover,// I:The archive file path mover fn

ET_IPinitTermFπ anlnitFunc, // I:Optl init fn, NULL if not raq'd

ET_IPinitTermFn aTermFunc, // I:0ptl term fn, NULL if not req' d

^• ET_AliasResolver anAliasResolver, // l:0ptl output fldr alias resolver

ET_AlιasModifier anAliasModifier, // I:Optl output folder alias modifier

ET_ArchiveRecIniter anArchiveRecIniter,// I:Optl archiv DB rcrd initiaizer

ET_CustomCmdFn aCustomArchiveCmdFn,// I:Optl cstm srvr arch comnd hndlr

ET_CustoraFuncFn ' aCustomArchiveFuncFn,// I:0ptl cstm clnt arch fn handler

ET_ItemInfoGetter anArchivelnfbGetterFn, // .I:Optl get archive chunk info

. ET_ArchiveLUNGettar anArchivaLUNGetterFn,// I:Optl get arch LUN strng hπdl

ET_RepartitionNfyar aRepartitionNfyerFn, // I:Optl notify arch repartition

- ET_ArchiveCopier anArchiveCopierFn // I:Optl archive file copier fn

_, ) ; . // R:void

EngErr Sv_PutInItemRejectFolder{ ^• .Move a file to the Reject folder.

OSType ' aDataType, . 11 : Data Type we're talking about

' ^' • char ^♦sourceFile, // I: Full path name of file to move

Boolean deleteOriginal // I: If true, del orig file aftr copy

) ; // R: None

EngErr SV_GetServerFolder ( . // get full file path to srvr folder

OSType aDataType, // I: The I? data type server required charPtr aBuffer, . // 0: The buffer to hold the file

•path short aFolderType ^' // I:

^■ klnput/Output/Reject/AliasFolder ^'

) ,^r /^'/ R:Zero for success, else error

* ,^{' ' "'} ' ftdefine klnputFolder 0 - : // aFolderType - input folder ftdefine kOutputFolder ^• • 1 // aFolderType - output folder ' ftdefine kRe actFoider . 2 // aFolderType - reject folder ftdefine kAliasFolder 3 ' // aFolderType - alias folder ftdefine kCollectionsFolder " 4 ^■ // aFolderType - collections folder

ET ArchiveTweakPtr SV GetArchiveTweakRacor ( // ge pntr to arch tweak recrd type OSType aDataType // I:Data type, 0 for the default rcrd . ) ' // R:pntr to tweak record, NULL if err

ET_IPserverTweakRecPtr SV_GetTweakRecord ( // get pntr to tweak data type record

OSTyoe . aDataType ■ //_ I: Data type

^' ' ^{' '} - ^{) :} // R:pπtr to tweak record, NULL if err ^• < ^{• l} ' ι ^'■

Boolean sv_GatCurrentBatchDetails ( // Get size of current archive batch ET_CSStata?tr p, // I: Server state pointer unsLong *batchID, ^• // IO:If non-NULL, the current btch ID. unsLong ^♦currsize, // 10::If non-NULL, size of ourrnt btch unsLong *targetsize, // 10::If non-NULL, trgt opt btch size unsLong ^♦blocksize // 10::If non-NULL, block size ) ; . // R:TRUE for sucass, FALSE otherwise unsLong SV_GetBlocks ,' ( // get * blocks re 'd to hold file unsLong dataForkBytes, ^• // I: Size in bytes of data fork unsLong ^• . resForkBytes, // I: Size in bytes of resource fork . < unsLong • blocksize // I:Si2e in bytes of a block;

), // R: umber of blocks required

EngErr SV_FetchArchiveChunkID ' ( // Fetch arch chunk ID assoc w item ID OSType aDataType, // I: IP data type server required ET_UnigueID itemID, // I: Item ID within the server long ^♦aChunkID // 0:The corresponding chunk ID ); . // R:2ero for success, else error

-ft ' void SV_lndex2FolderName // convert dir indx to BASE 36 encoded ~ int32 index, // I:The index to be converted

. charPtr aBuffer // 0:Buffer to put folder name stng in >; // R:void - • ^■ int32 SV_FolderName2Index .// cnvrt base 36 fldr name to an index charPtr aBuffer // I:Buffr w fldr name of form "3XXXX" ); // R: Equiv numeric val for flder index

Boolean SV_De ineDroneType ( // Define new IP data type and server OSType . aDataType, // I:data type code for IP master type OSType ' . aDroneType, // I:data type code for drone dat type charPtr aDroneMachine, // I:Machine name for drone server charPtr aDroneFolder, ^' II I : srvr ldr parent network fie path

ET_IPservarTweakRecPtrtweak, _ • // I:Drone Server tweaking parameters ET_ArchiveTweakPtr aTweak, // I:Archive tweak record int32 . .- options // I:various logical options

⁾ // R:TRUE for success, FALSE - otherwise short SV_CountDrones // Return ft drone types for given type

OSType aDataType // I:data type code for mastr dat type // R: Count of number of drones

OSType SV SetDroneType // Return ft drone types for given type OSType aDataType , // Isdata type code for mastr dat type short index // I:Drone type index (zero based)

) // R:Tha drone type, 0 if none

Boolean SV_IsAssocιatedDroneType ( // Drne type assoc w spec'd mstr type? ^" OSType aDataType , // I:dat type code for mastr data type

OSType • aDroneType, // I:dat type code for drone data type int32 ^• * index, // IO: If non-NULL, holds drone index charPtr - addressP // 10: If non-NULL, olds drone IP addrs

); // R:TRUE if drone assoc w this master OSType SV_F ndLeastBusyDrones ( // Return least busy drone (s) int32 ' - . *a, // IO:The number of drones '

OSType ..^♦drones // 10-.array of drones to be considered // R:Th^'e least busy drone type unsshαrt sv_GetStatusWord // Retn the status word assoc w drone OSType aDataType, /-/ I:data type code for mastr dat type short index // I:Drone type index (zero based)

⁾; // ,R:The drone status,^" 0 if none

Boolean SV_DeclareStatus - ( ', // Set status word assoc with a server OSType - . _, • aDataType, // I:unique data type code for srvr unsShort ,. status // I:the new value of the status word ' // R:TRUE for success, FALSE otherwise

ET_LaunchCode SV_MakeServerView( // Create srvr view for IP data type ~ . OSType - ¹ aDataType, ^• // I:Data type to serve charPtr aHelpString,- // I -.Help string for widget (or NULL) charPtr • • aDascriptionString,// I:Description string for widget charPtr aKeyWαrdsSt ing. // I -.Keywords 3tπng (or NULL) charPtr . aHistoryStriπg, // I:Hi3tory string for widget int32 aStackSize, // I: Stack size required (or 0 = dflt) int32 ont ons // I: ogical options

>; // R:Launch Code of the created

'view

/// The following calls are ONLY valid from within the flunkies associated with a server.

Boolean SV_ArchιveThisFile ,' ( // Archive file in output folder? ET_CSStatePtr ip, ^■ .. // I :Address of params structure FSSpec .. ^♦aFile, // I:The file spec Boolean' .checkAncestorFplders// I:check ancestral folders ,^l ) ; ' . 11 R:TRUE to archive it, FALSE if not ,

Boolean SV InitiateBatchScan ( // Init. re-scan/burn of archive batch

ET_CSStatePtr ^• ^' ip, // I .-The server state record pointer unsLong batchID, . // I -.The batch Boolean ^• .' , . noUserPrompt // I:TRUE to suppress prompts to user

\ . // R:TRUE for success, FALSE otherwise

EngErr SV_PerformArchiveBurn ( // Perform an archive burn sequence ET_^CSStatePtr ip, // I:Server state pointer unsLong batchID // I:The batch ID

);. // R:Zero for success, else error ft ^" " ^"- ^'" '^" - short SV_GatStatePtr , ( // Retrn flaky indxft.get srvr gtte ptr

ETJTSStatePtr ^• *ip // 0:lf non-NULL, address of stte pntr ); // R: Flunky indx for calling widget int32 SV_GatCliantOptions // Retrn flaky indxft,get srvr stte ptr

ET_CSStatePtr ip. // I:Server IP state record short index // I : Flunky index

); // R:Command options spec'd by client :

ET_DatabasePtr SV_GetDatabasePtr ( // get db record pntr assoc w a server ET_csstatePtr iP. ' • . //• I: Pointer to IP Server Status ) ," // R: server database record address

ET_DatabasePtr SV_GetArchiveDBPtr( // get arch db record ptr assoc w srvr ET CΞStatePtr ip // I: ointer to IP Server Status .lli.ill ..mil IC a in. nil u-ii-ii"" II n n ιr r.

// R: server archive db record address . ^", ^*

-unsLong SV__AddArchiveBatchID ( // Add a new batch ID .to archive DB ^"" ET_CSStatePtr - P // I:Server^' state pointer // R:The batch ID added

EngErr SV SetFolderArchiveEnb ( // Eable/disble outpt fldr sb-pth arch ET_CSStatePtr i . // I: Pointer to IP Server Status ^• charPtr opFolderSubPat , // I: sub- fldr pth w/in srvr outpt fldr Boolean enable // I:TRUE to enable archiving .

); // R:0 for success alse error void SV_SetupMessage^' • .,. ' // Setup new msg for display by master

ET_CSStatePtr ip. // 10: Pointer to IP Server Status , int32 index, // I : Index of message sender int32 options , // I: If true, msg log to time ^'hist fid charPtr aFormatString, // I: ormat strng for msg to display // I:Additional parameters as req'd // R:Void ftdefine kLogToTimeHistory oxαoooooαi // log massage to time history field ftdefine kSendToMaintPage 0x00000002 // msg sent to maintenance page

Boolean SV_PingClient _γ ( // Ping a client to see if still there ET_CSStatePtr sp, // I:Server state record pointer short flunkyIndex // I: Flunky index

// R:TRUE if client was found void SV_AutoPingClient ^• ( // Pmg a client to see if still there V . ET_CSStatePtr sp, // I:Server state record pointer short flunkyIndex, // I: lunky index unsLong pinglnterval, // I: Interval between pings (in ticks) . long pingContext , // I: Context pass to 'aDeathHandler' ET_ClientDiedFunc aDeathHandler // I:Fn invoke if client dissapears

.) ^• // R-.void ,. void SV_SetPicon ^• ' ( // Attch/Detch picon to/fr srvr cntrl

ET_CSStatePtr ⁱp. // IO:Address of cmd param structure PicHandle newPicon // Iipicon hndl attachd to dsply cntrl ); // R: Void

EngErr Sv_SendToClient // Send msg to 'client' of srvr flunky

~ ET_CSStatePtr sp, // I: Server state record pointer short flunkyIndex, // I: Flunky index void* aMessageBu f, // I -.buffer to be sent to the client Boolean isFinalResponse // I:TRUE if last message to' be sent

); // R:2ero for success, else error ft

EngErr SV_ForwardCommand ( // Forward command to srvr from a srvr ^■ ~ ET_CSStatePtr IP. // 10: Pointer to IP .Server Status ^" int32 flunkyIndex, // I: Index for the flunky being used OSType aDataType // I:data type for srvr to send cmd to ' // R: Success 0 else error

EngErr SV_PrepareFileForXfer ( // Prepare- file for transfer" o client " - ET_CSStatePtr 3-Pr // IO: Pointer to IP Server status . int32 flunkyIndex, // I: Index for items flunky being used '. . ^{• '}

ET_UniqueID itemID, // I: Item ID of item file being sent charPtr aSou^'rceFilePath, // 0:full source file path in server int32 . options // I: Various logical options );' . • // R:0= success, -1= cmnd forwarded, err

EngErr SV_ReplacaCommand ' ( // Replace flunky command by a new one ET CSStatePtr ip. // 10/ Pointer to IP Server Status int32 •flunkyIndex, '// I -.Index for the flunky being used

" long ^• - -' ' aCommand, // I: The command you wish to execute void **aDataBuffer, . // lθ:Addr. of send/receive bffar ptrs long *aBufferSize, .// 10: size! of send/receive data _, . bffers . ^• • . _ charPtr .. anldentifyingName, // I: Name of IP int32 options, // I: Various logical options

^{• '} int 2 ' context ' // I: Context value fr nfy

' seqNu ber '

;. ^{' "}) ; // R: Success 0 else error int32 SV SetOptions ι // get optns val assoc with a server

^_ ET_CSStataPtr ip ^" // I: Server IP state record

. ⁾; // R:valua of the options field void SV_^SetOptiong ( ^'. ,

^'// set optns val assoc with a server :

~ ET_CSStataPtr ip. , // I: Server IP state record int32 options . ' // l:The valua of the options field

>; // R-.void void* SVJάetContext . { - // gat cntxt val prev CS_SetContext cl

^•' „ ' ET_CSSt^'atePtr _{t ,} " ip. ^'' // I: Server IP state record . ^' - int32 •usageCount // I: If non-NULL, ft flunkies alive

. ^' ><' . ^', // R:valus of the context field void 3V_SetCσntext . ( // cntxt val assigned w srvr state red

, , ~ ET_CSStatePtr ip. // I: Server IP state record

• . void ' 'context // I -.The value of the context field

, , );

' . ' ^" ' ;. // R:void -. _; voi^'d* SV_GetArchiveCOπtext ( ^• ; ; // get ^'archive contxt record for srvr

ET_CSStatePtr ip - - // I: Server IP state record

- , . >; // R:Archive context storage recrd ptr - void 3V_SetArchiveContext ( ' ' // set arch cntxt val of srvr stte red

- ET_CSStatePtr ^' ip. , // I: Server IP state record void ' ' , •context // I :value of archive context field -

); .^" // R:void ; .■ ' • '

EngErr SV_SetReply ^' - ( // val of reply block fo 'srvr cllbck

ET_CSΞtatePtr ip. , // I: Server IP state record int 2 index, // I.-Index of message sender ■ void *aBuffar, // I:Pner to bffr contain data to rtrn int32 . ; aBuffarSize // I: size of the buffer to be attached ' -

^' '); // R:2ero for. success, else Error

#

EngErr s ]_SetIteraListReply ( // Set reply record for command as 1st

^• _ ET_CSStatePtr . ip. // 10: ointer to IP Server Status

, .^" . int32 flunkyIndex, - // I : Index for the lunky being used

- • int32 , αltems, // I:The number of items to send

ET Jniσue DHdl ' idListH, ^• i // I -.Handle to item list IDs to send

ET_ItemMarkupFn aMarkupFn, // I:Markup function, or NULL

* long aMarkupContext // I:Context param pass to markup fen

- ft - ⁾; // R:Zero for success, else error

^' void Sv_DontWai FαrMe ( > - ^{' '} -

// Prevent fav flunky from waiting ^•

ET_CSStatePtr ^'. ^£P^« // 10: Pointer to IP Server Status

-int32 - i index, // I: Index of message sender

Boolean ' - dontWait // I:TRUE to prevent waiting

. )•; ^•. // R:Void' ' ^' '

Boolean SVjSatClientLocation ( ^■ // Get clnt path when w/in srvr flunky

ET CSStatePtr ip. // 0: Pointer to IP Server Status

82

_λ i; , '.'. .,'—ih.ti sir π.-i - int32

\ index, // I: Index of flunky (must be > 0)

"-charPtr ^'. ^': cliantMachine, II 10: If non-NULL, Holds clnt ach pth

- charPtr _, ' ^"• clientUser, II 10: If non-NULL, Holds clnt user name

• Boolean ^• , fromClientMessage II I: true use val sent w/in clnt msg

); // R:TRUE for success, FALSE otherwise

^' EngErr ΞV_ 3etOutputFilePath ( ^"' II get item output partial file path

ET_CSStatePtr ' _, ' ip, ' ' II I: Pointer to IP Server Status

- ΞT_UniqueID itemID, II I:The item ID required charPtr aFilePath, II 0: etrieved partial file path bffr

• ^" ιnt32 maxChars // I: size of 'FilePath' buffer in chrs • ..

¹ -. ); " - // R:Zero for success, else error ft

Boolean SV _De ineMainteriancePage ( • // def srvr aint view/page and hndler

ET_jSStatePtr ip, // I: Pointer to IP Server Status

. charPtr pageName, // IiStrng used in pge slct popup menu

ET_BookPageHandler aHandler, // I:Address of hndler fn for this pge

ET~FCTLresHdl aViewControl // I: iew control/FVEW resource

); • // R:TRUE for success, FALSE ^• otherwise • • , . . .

ET_ContrαlRef SV_GetActiveMaintPage ( // get details for button assoc w page

ET_CSStatePtr ip. // I: Pointer to IP Server Status charPtr , pageName // 0:The name of the current page ); // R:book control handle for success , , ,

ET^StringList SV_GetPageRegistry( , .- // ET StringList registry fr maint pge'

, ET_CSStatePtr p ; // I: Pointer to IP Server Status charPtr pageName // I Maintenance page name

^{' "} • - ' ); // R:Registry for the page

Boolean SV _DefineltamPageButton( -' ' • ^• // define/reg button assoc w a page

ET_StringList . , pageRegistry, // IO.-ET_StringLιst registry for page

- ' charPtr buttoπName, // I:The name of the button charPtr buttonHelp, .// I:Help string for the button

• OSType bu tpnType, // I:Bu ton type

Handle buttonData, // I.-handle to the data for the button

ET_MaintButtHandler aButtonHandler // I:Address of button handler fn ); // R.-TRUE for success, FALSE

^'otherwise '

Boolean SV _GetItemPageButton ( - - // get details for button assoc w page

ET_StringList pageRegistry, // IO:ET_StringLιst rag for desr'd pge charPtr buttoπName, // 10:Buffer to hold button name charPtr ' . buttσnHelp, // 10: Buffer to hold button Help strng

- _ OSType ^♦buttonType, // 0:Button type ('cicn' or 'PICT')

Handle ♦buttonData, // 0:The handle to the data for buttn

ET_MaintButtHandler •aButtonHandler // 0:Addres3 of button handler fn

); // R.-TRUE for success, FALSE

Otherwise int32 SV_GetFavoriteFlunkyID ( // get avorite- srvr flunky widget ID

_, ET_CSStatePtr ip // 10: Pointer to IP Server Status ); // R:Favorite flunky ID, 0 if error • * . •

Boolean SV 2ustomCommandFavFlunk ( -

// Issue cust comnd to srvr fav flunky

ET_CSStatePtr - . iP, // I:Server state ptr

- long con ex , /^"/ I:Cntxt val ss to flunky

. cllbcks

, long ^♦result, // I:Addrss to rtn 'aCommandProc' rslt -

ET_FlunkyCmdProc^' aCommandProc // I-procdre Addrss execute w/in Flnky '

_ , ) ; . // R:TRUE for success, FALSE otherwise

-

83'

SUBSTITUTE SBEET (RULE 26

[NEEDS DESCRIPTION] static EngErr ND_Move // InfiniDISC move command

" unsLong robotNum, // I: robot ID number unsLong src, 11 1 : source location of desired CD unsLong dest, ■ // I: destination location for CD int 2 •options // I: various logical options ' ■^• ■ // R: aero for success, else error code { if ( ML_IsRobotEnabled (robotNum) ) - •^'• ^• return kRobotNotAvailable ; . , . if ( src is a slot ! ^' • ^' - // source is a robot CD slot

{ ' . ^{" •'}

IND_SlotToXY(con Ptr, src,'robotNum, &Xs, &Ys) if ( dest is a slot ) // destination is a robot CD slot

IND_SlαtTσX (contPt , est,robotNum, δ-Xd, &Yd) ; err = IND_Move (robotNum, src, kMediaTransportElement, options) ,- if ( !err ) , 11 map logical to physical moves

{

. pickerFull =» YES; - ^{' '}

.err 3 IND_Move(robotNum,kMediaTransportElement,dast, options) , if ( !err ) pickerFull = NO;

}

} else' if ( dest is a drive )

{ • ' ' ^{• '}. ^' , ll destination is a drive err = IND_Move (robotNum, src, kMediaTransportElement, options) ; if ( !err ) -

{ '■ pickerFull = YES; - ' ' err = IND_Move (robotNum, kMediaTransportElement,dest, options) if ( ierr ) pickerFull « NO;

}^'

} else if ( dest == kMediaTransportElement )

{ ^' // destination is transport eject the disk from the src carousel - . , - pick it with arm , . ■ pickerFull = YES; ^" , "

} else if ( dest « ^'kMedialmportExportElement^' ) '^{^} ■

C ^'• ' ' ^' err β IND_Move (robotNum, src, kMediaTransportElement, options) ; if ( !err ) •• // move from transport to mailbox

^' - {

' pickerFull = YES; // The picker/arm is our import/export

.err = IND_Move (robotNum,kMediaTransportElement, kMedialmportExportElement, options) ; . ' if ( !err ) pickerFull = NO;

} . ^' - "^■

" } -else^" err = kBadPara eter; ^• - . } else if (src is a drive ) ^' - ^' - ' ■•

{ . ^{• •} - - ^■ - . ,

... etc. . ' ,

} else if ( src == kMediaTransportElement ) . . . etc .

60092143- 2.DOC A SYSTEM FOR EXCHANGING BINARY DATA Inventor: John Fairweather

BACKGROUND OF THE INVENTION ^'

In most modern computer environments, such as programming languages, and applications, the programming language compiler itself performs the job of defining data, structures and the types and the fields that make them up. That type information is compile- tifne determined This approach has the advantage of allowing the compiler itself to detect many common programmer errors in accessing compound data structures rather than v allowing such errors to occur at run-time where they are much harder to find. However, this approach is completely inadequate to the needs of a distributed and evolving system since it is impossible to ensure that the code for all nodes on the system has been compiled with a « compatible set of type definitions and will therefore operate correctly. The problem is aggravated when systems from different vendors wish to exchange data and information since their type definitions are bound to be different and thus the compiler can give no help in the exchange. In recent years, technologies such as B2B suites and XML have emerged to try to facilitate the exchange of information between disparate knowledge representation systems by use of common tags, which may be used by the receiving end to identify the content of specific fields. If the receiving system does not understand the tag involved, the corresponding data may be discarded. These systems simply address the problem of converting from one 'normalized' representation to another, (i.e., how do I get it from my relational database into yours?) by use of a tagged, textual, intermediate form (e.g. XML). Such text-based markup-language approaches, while they work well for simple data objects, have major shortcomings when it comes to the interchange of complex multimedia and non- flat (i.e., having multiple cross-referenced-allocations) binary data. Despite the 'buzz' associated with the latest data-interchange techniques, such systems and approaches are totally inadequate for addressing the kinds of problems faced by a system, such as an intelligence system, which attempt to monitor and capture ever-changing streams of unstructured or semi-structured inputs, from the outside world and derive knowledge, computability, and understanding from the data so gathered. The conversion of information, especially complex and multimedia information to/from a textual form such as XML ' becomes an unacceptable burden on complex information systems and is inadequate for descπbmg many complex data interrelationships. This approach is the current state of the art. At a minimum, what is needed is an interchange language designed to describe and

86

APPENDIX 1 manipulate typed binary data at run-time. Ideally, this type information will be held in a 'flat' (i.e., easily transmitted) form and ideally is capable of being embedded in the data itself without impact on data integrity. The system would also ideally make use of the power of compiled strongly typed programming languages (such as C) to define arbitrarily interrelated and complex structures, while preserving the ability to use this descriptive power at run-time to interpret and create new types.

SUMMARY OF INVENTION

The present invention provides a strongly-typed, distributed, run-time system capable of describing and manipulating arbitrarily complex, non-flat, binary data derived from type descriptions in a standard (or slightly extended) programming language, including handling of type inheritance. The invention comprises four main components. First, a plurality of databases having binary type and field descriptions. The flat data-model technology (hereinafter "Claimed Database") described in Appendix 1 is the preferred model for storing such information because it is capable of providing a 'flat' (i.e., single memory allocation) representation of an inherently complex and hierarchical (i.e., including type inheritance) type and field set. Second, a run-time modifiable type compiler that is capable of generating type databases either via explicit API calls or by compilation of unmodified header files or individual type definitions in a standard programming language. This function is preferably provided by the parsing technology disclosed in Appendix 2 (hereinafter "Claimed Parser"). Third, a complete API suite for access to type information as well as full support for reading and writing types, type relationships and inheritance, and type fields, given knowledge of the unique numeric type ID and the field name/path. A sample API suite is provided below. Finally, a hashing process for converting type names to unique type IDs (which may also incorporate a number of logical flags relating to the nature of the type). A sample hashing scheme is further described below.

The system of the present invention is a pre-requisite for efficient, flexible, and adaptive distributed information systems. ^'.^'•^" BRIEF DESCRIPTION OF THE DRAWINGS ^'.

Figure 1 provides a sample implementation of the data structure ET_Field;

Figure 2 provides a sample code implementation of the data structure ETJType; .,

Figure 3 is a block diagram illustrating a sample type definition tree relating ETJType and ET_Field^' data structures; and ^', J _. ." .

Figure 4 provides a sample embodiment of the logical flags that may be used to describe the typelD. ^{' •} .^' " . - -_ _, ^{' • '} ..^'. . . ^{• '}.

^' DETAILED DESCRIPTION OF THE INVENTION ,^'

The following description provides an overview of one embodiment of the mvention. Please refer to the patent application incorporated herein for a more complete understanding of the Claimed Parser and Claimed Database. .

All type information can be encoded by using just two structure variants, these are the 'ET_Field' structure, which is used to describe the fields of a given type, and the ΕTJType' structure, which is used to described the type itself. .Referring now to Figure 1, a sample implementation of the ET_Field structure 100 is provided. The fields in the ET_Field structure are defined and used as follows:

"hdr" 102 - This is a standard header record of type ET_ Hdr as defined in the Claimed Database patent application.

"typelD" 104 - This field, and the union that surrounds it, contain a unique 64-bit ^• ' type DD that will be utilized to rapidly identify the type of any data item. The method ^'' for computing this type ID is discussed in detail below.

"fName"106 - This field contains a relative reference to an ET_String structure specifying the name of the field. ; • . ^" '

"fDesc" 108 . - This field may contain a relative reference to an ET_String structure containing any descriptive text associated with the field (for example the , contents of the line comments in the type definitions above).

"fieldLink" 110 - This field contains a relative reference to the next field of the current type. Fields are thus organized into a link list that starts from the "fieldHDR" 220 field 220 of the type and passes through successive "fieldLink" 110 links 110 until there are no more fields. •

"offset" 112 - This field contains the byte offset from the start of the parent type at which the field starts. This offset provides rapid access to field values at run-time. .' "unitiD" 114 - This field contains the unique unit ID of the field. Many fields have units (e g., rmles-per-hour) and knowledge of the units for a given field is essential when using or compaπng field values.

"bounds" 116 - For fields having aπay bounds (e.g , and array of char[80]), this field contains the first array dimension.

"bounds2" 118 - For two dimensional aπays, this field contains the second dimension. This invention is particularly well-adapted for structures of a higher dimensionality than two, or where the connections between elements of a structure is more complex that simple array indexing.

"fScript" 120 - Arbitrary and pre-defined actions, functions, and scripts may be associated with any field of a type. These 'scripts' are held in a formatted character string referenced via a relative reference from this field

"fAπnotation" 122 - In a manner similar to scripts, the text field referenced from this field can contain arbitrary annotations associated w th the field. The use of these annotations will be discussed in later patents. " ,

"flaglndex" 124 - It is often convenient to refer to a field via a single number rather than carrying around the field name. The field index is basically a count of the field occuπence index within the parent type and serves this purpose

"fEchoField" 126- This field is associated with forms of reference that are not relevant to this patent and is not discussed herein.

"flaglndexTypelD" 128- In cases where a field is embedded within multiple enclosing parent types, the 'flaglndex' value stored in the field must be tagged in this manner to identify which ancestral enclosing type the index refers to.

Referring now to Figure 2, a sample embodiment of the ETJType structure 200 is provided. The fields of the ETJType structure 200 are defined and used as follows:

"hdr" 202 - This is a standard header record of type ET_Hdr as defined m the Claimed Database patent application. "typelD" 204 - This field, and the union that suπounds it, contain a unique 64-bit type ID that will be utilized to rapidly identify the type of any data item. The method for computing this type ED is discussed in detail below

"name" 206 - This is a relative reference to a string giving the name of the type

"edιt","dιsplay" 208 - These are relative references to stπngs identifying the "process" to be used to display/edit this type (if other than the default). For example the specialized process to display/edit a color might be a color-wheel dialog rather than a simple dialog allowing entry of the fields of a color (red,green,blue).

"description" 210- This is a relative reference to a stπng describing the type.

"ChildLink" 212 - For an ancestral type from which descendant types inherit, this field gives the relative reference to the next descendant type derived from the same ancestor Type hierarchies are defined by creating trees of derived types. The header to the list of child types at any level is the "childHdr" field 218, the link between child types is the "ChildLink" field 212. Because types are organized into multiple type databases (as discussed later), there are two forms of such links: the local form and non-local form. The'non-local form is mediated by type ID references, not relative references (as for the local form), and involves the fields "childlDLink" 236, "childlDHdr" 238, and "parentID" 240 (which hold the reference from the child type to its parent). The parent reference for the local form is held in the "parent" field of "hdr" 202.

"cTypedef ' 216- - This field may optionally contain a relative reference to a string giving the C language type definition from which the type was created.

"childHdr" 218 - This field contains the header to the list of child types at any level .

"fieldHDR" 220 - Fields are organized mto a link list that starts from the this field.

"keywords" 222 -This field contains a relative reference to a string contain key words by which the type can be looked up

"bounds" 224, "bounds2" 226 - aπay dimensions as for ETJield

"size" 228 - Total size of the type in bytes. r "color" 230- To facilitate type identification in various situations, types may be assigned inheritable colors. , . .

"filelndex" 232 - used to identify the source file from which the ^"type was created.

"keyTypeED" 234 - This field is used to indicate whether this type is designated a "key" type. In a full data-flow based system, certain types are designated 'key' types • and may have servers associated with them. .

"nextKeyType" 246 - This field is used to link key types into a list.

"tScript" 242,"tAnnotation" 244 - These fields reference type scripts and annotations ; as for ET ield 100.

"maxFieldlndex" 248 - This field contains the maximum field index value (see ET_Field 100) contained within the cuπent type.

"numFields" 250 - This gives the total number of fields within the cuπent type. - -

,To illustrate the application of these structures 100, 200 to the respresentation of types and the fields within them, consider the type definitions below whereby the types "Cat" and "Dog" are both descendant from the higher level type "Mammal" (denoted by the "::" symbol similar to C++ syntax). ■

typedef struct Mammal . . .^' - i 1 ^■ ^' RGBColor hairColor; int32 ^• gestation; ' // in days ,

} Mammal; , ^v

typedef struct Dog: :Mammal

{ ^{• ■ • ■ " '} . •

. int32 barkVol; /.in decibels

} Dog; . . . ^•

typedef struct Cat: :Mammal s . . -

{ I int32 . - • puπVol; ^■•// in decibels

} Cat; ^{■ •} ;. ^' . . • ^'' ' .'^•' . \ ^'

^' Because they are mammals, both Cat and Dog inherit the fields "hairColor" and "gestationPeriod" which means the additional field(s) defined for each start immediately after the total of all inherited fields (from each successive ancestor). Referring now to Figure 3 , this portion of the type definition tree when viewed as a tree of related ETJType 200 and ETJField 100 structures is shown. In this diagram, the vertical lines 305 linking the types 315, 320 are mediated via the "childHdr" 218 and "parent" 240 links. The horizontal line 310 linking Dog 320 and Cat 325 is mediated via "ChildLink" 242. Similarly for the field links 330, 335, 340, 345 within any given type, the fields^' involved are "parentTD" 240, ^{' *} "fieldHDR" 220, and "fieldLink" 110. It is thus very' obvious how one would navigate through the hierarchy in order to discover say all the fields of a given type. For example, the following sample pseudo code illustrates use of recursion to first process all inherited fields before processing those unique to the type itself. - . . . ^' , ^"

void LoopOverFields (ETJType *aType)

if ( aType->hdr.parent ) , ^'

LoopOverFields(aType->hdr.parent) ^" . for ( fieldPtr = aType->fieldHdr ; fieldPtr ; fieldPtr = fιeldPtr->fieldLink )

— do something with the field

} ^'

- Given this simple-tree-structure in which type information is-stored and accessed, it - should be clear to any capable software engineer how to implement the algorithms setr forth in the Applications Programming Interface (API), given below. This API illustrates the nature and scope of one set of routines that provide full control over the run-time type system of this invention. This API is intended to be illustrative of the types of capabilities provided by the system of this invention and is not intended to be exhaustive. Sample code implementing the following defined API is provided in the attached Appendix A. -

The routine TM_CruiseTypeHierarchy() recursively iterates through all the subtypes contained in a root type, call out to the provided callback for each type in the ! hierarchy. In the prefened embodiment, if the function 'callbackFunc' returns -1, this routine omits calling for any of that types sub-types. ^»

The routine TM_Code2TypeDB() takes a type DB code (or TypelD value) and converts it to a handle to the types database to which it coπesponds (if any). The type system of this mvention allows for multiple related type databases (as described below) and this routine determines which database a given type is defined in.

TMJtaitATypeDBO and TMJIermATypeDBO initialize and terminate a types database respectively. Each type DB is simply a single memory allocation utilizing a 'fiat' memory model (such as the system disclosed in the Claimed Database patent application) containing primarily records of ETJType 100 and ET_Field 200 defining a set of types and their inter-relationships ^l ,

TM_SaveATypeDB0 saves a types database to a file from which it can be re-loaded for later use.

TM_AlignedCopyO copies data from a packed structure in which no alignment rules are applied to a normal output structure of the same type for which the alignment rules do apply. These non-aligned structures may occur when reading from files using the type manager. Different machine architectures and compilers pack data into structures with different rules regarding the 'padding' inserted between fields. As a result, these data structures may not align on convenient boundaries for the underlying processor. For this reason, this function is used to handle these differences when passing data between dissimilar machine architecture.

TMJFixByteOrderingO coπects the byte ordering of a given type from the byte ordering of a 'source' machine to that of a 'target' machine (normally 0 for the cuπent machine architecture). This capability is often necessary when reading or writing data from/to files originating from another computer system Common byte orderings supported are as follows:

^• kBigEndian - e.g., the Macintosh PowerPC

^• kLittleEndian - e.g., the Intel χ86 architecture

^• kCurrentByteOrdering - cuπent machine architecture

94

UBSTITUTE SHEET TMJF^'mdTypeDBO can be used to find the TypeDB handle that contains the definition of the type name specified (if any). There are multiple type DBs>in the system ' which are accessed such that user typeDBs are consulted first, followed by system type DBs. The type DBs are accessed in the reverse order to that in which they were defined. This means that it is possible to override the definition of an existing type by defining a new one in a later types DB. Normally the containing typeDB can be deduced from the type ID alone (which contains an embedded DB index), however, in cases where only the name is known, this function deduces the conesponding DB. This routine returns the handle to containing type DB or NULL if not found. This invention allows for a number of distinct type DBs to co-exist so that types coming from different sources or relating to different functional areas may be self contained. In the prefened embodiment, these type DBs are identified by the letters of the alphabet ('A' to 'Z') yielding a maximum of 26 fixed type databases. In addition, temporary type databases (any number) can be defined and accessed from within a given process context and used to hold local or temporary types that are unique to that context. All type DBs are connected together via a linked list and types from any later database may reference or derive from types in an earlier database (the converse is not true). Certain of these type DBs maybe pre-defined to have specialized meanings. A prefened list of type DBs that have specialized meanings as follows:

'A' - built-in types and platform Toolbox header files

'B' - GUI framework and environment header files

'C - Project specific header files J

'D' — Flat data-model structure old-versions DB (allows automatic adaption to type changes) ' .

Ε' - Reserved for 'proxy' types

'F' - Reserved for internal dynamic use by the environment

T - Project specific ontology types

TM_^GetTypeJDO retrieves a type's ID Number when given its name. If aTypeName is valid, the type ID is returned, otherwise 0 is returned and an eπor is reported. ^'»' TMJfcKnownTypeNameO is almost identical but does not report an eπor if the specified type name cannot be found. . ' ^;

TM_ComputeTypeBaseIDO computes the 32-bit unique type base ID for a given type name, returning it in the most significant 32-bit word of a 64-bit ET_TypeID 104. The base ID is calculated by hashing the type name and should thus be unique to all practical purposes. The full typelD is a 64-bit quantity where the base ID as calculated by this routine forms the most significant 32 bits while a variety of logical flags descnbing the type occupy the least significant 32-bits. In order to ensure that there is a minimal probability of two different names mapping onto the same type ED, the hash function chosen in the prefened embodiment is the 32-bit CRC used as the frame check sequence in ADCCP (ANSI X3.66, also known as FTPS PUB 71 and FED-STD-1003, the U.S. versions of CCITT's X.25 link- level protocol) but with the bit order reversed. The FTPS PUB 78 states that the 32-bit FCS reduces hash collisions by a factor of 10^Λ-5 over the 16-bit FCS. Any other suitable hashing scheme, however, could be used. The approach allows type names to be rapidly and uniquely converted to the coπesponding type ID by the system. This is an important feature if type ' information is to be reliably shared across a network by different machines. The key point is that by knowledge of the type name alone, a unique numeric type ID can be formed which can then be efficiently used to access information about the type, its fields, and its ancestry. ^* The other 32 bits of a complete 64-bit type ID are utilized to contain logical flags concerning the exact nature of the type and are provided in Appendix A.

Given these type flag definitions and knowledge of the hashing algorithm involved, it is possible to define constants for the various built-in types (i.e., those directly supported by the underlying platform from which all other compound types can be defined by accumulation). A sample list of constants for the various built in types is provided in Appendix A.

. Assuming that the constant definitions set forth in Appendix A are used, it is clear that the very top of the type hierarchy, the built-in types (from which all other types eventually derive), are similar to that exposed by the C language.-

Referπng now to Figure 4, a diagrammatic representation of a built-in type is shown (where indentation implies a descendant type). Withm the kUniversalType 405, the set of direct descendants includes kVoidType 410, kScalarType 415, kStructType 420, kUnionType

96

SUBSTITUTE S^EE RULE ,' 425, and JcFunctionlype 43U. kScalarType also includes descendants for handling integers 435, descendants for handling real numbers 440 and descendants for handling special case ^' scalar values 445. Again, this illustrates only one embodiment of built-in types that may be ■ utilized by the present system. , • , ^{' •}

The following description provides a detailed summary of some of the functions .that may be used in conjunction with the present invention. This list is not meant to be exhaustive ^"nor or many of these functions required (depending upon the functionality required for a given implementation). The pseudo code associated with these functions is further illustrated in attached Appendix A. It will be obvious to those skilled in the art how these functions could be implemented in code. _t - . - ^■•

- Returning now to Appendix A, a function TM_CIeanFieldName() is defined which provides a standardized way of converting field names within a type into-human readable labels that can be displayed in a UI. By choosing suitable field names for types,^"the system can create "human readable" labels in the coπesponding UI. The conversion algorithm can ^" ' be implemented as follows: • ^' . ' • ^■ . _ • • ' . .^' >' ^'.

• 1) Convert underscores to spaces, capitalizing any letter that immediately follows the underscore ,. • '

^' . ■ , 2) Capitalize the first letter . ^•• •

' 3) Insert a space in front of every capitalized letter that immediately follows a lower . case letter ' ^"

- 4) Capitalize any letter that immediately follows a '.'-character (field path delimiter)

_^' 5) De-capitalize the first letter of any of the following filler words (unless they start ^" ~ the sentence): ,

- ' "an"_!"and","of',"the^,*,"or",."to","is","as","a" ] - -

* ' .' , ^* - * - '

- . So for example: " . . •

• ^{■ '} "aFieidName" would become "A Field Name" as would "a_field_name"

"timeOfDay" would become "Time of Day" as would "time_of_day" ^{' •}• ; . ^• A function, such as TM_AbbreveFieldName()j could be used to provide a standardized way of converting field names within a type into abbreviated forms that are still (mostly) recognizable. Again, choosing suitable field names for types ensures both human readable labels in the conesponding UI as well as readable abbreviations for other purposes (such as generating database table names in an external relational database system). The „ conversion algorithm is as follows: ^' ' - ■ ^{' "} -

1) The first letter is copied over and capitalized. - - , -

^{■ ■} 2) For all subsequent letters:

a) If the letter is a capital, copy it over and any 'numLowerCase' lower case letters that immediately follow it. - " .^' .^{" '}

^•' b) If the letter follows a space or an underscore, copy it over and capitalize it

c) If the letter is '.', '[', or ']', convert it (and any immediately subsequent letters in , this set) to a single '_' character, capitalize the next letter (if any). This behavior allows this function to handle field paths.

d) otherwise disgard it ^;" - ■ ■ ^{■ ' '}

So for example: • -

' "aFieldName" would become "AFiNa" as would "a_field_name" if 'numLowerCase' was 1 , it would be 'AFieNam' if it were 2

. .. ^' "timeOfDay" would.become "TiOfDa" as would "time of day" if 'numLowerCase' was 1 , it would be 'TimOfDay¹ if it were 2 - ^•

For a field path example:

"geog.city[3].population" would become "Ge_Ci_3_Po" if 'numLowerCase' was 1

'. Wrapper functions, such as TM_SetTypeEditO, TM_SetTypeDisplayO, TM_SetTypeCouverterO, TM_SetTypeCtypedefO, ' '

TM_SetTypeKeyWordsO,TM_SetTypeDescriptionO , and TM_SetTypeCotor(), may be used set the coπesponding field of the ETJType structure 200. The corresponding 'get' functions are simply wrapper functions to get the same field. ^" ■ ^' A function, TM_SetTypeIc n(), may be provided^' that sets the color icon LD associated with the type (if specified). It is often useful for UI purposes tec associate an identifiable icon with particular types (e.g., a type of occupation), this icon can be specified using TM_SetTypeIconQ or as part of the normal acquisition process. Auto-generated UI (and many other UI context) may use such icons to aid in UI clarity. Icons can also be inherited from ancestral types so that it is only necessary to specify an icon if the derived type has a sufficiently different meaning semantically in a UI context. The function TM_GetTypeIcoπO returns the icons associated with a type (if any). ' . ^' , '.

A function, such as TM_SetTypeKeyType(), may be used to associate a key data type (see TM_GetT peKeyType) with a type manager type. By making this association, it is possible to utilize the full suite of behaviors supported for external APIs such as Database and Client-Server APIs, including creation and communication with servers) of; that type, symbolic invocation, etc. For integration with external APIs, another routine, such as TMJKeyTypeToTypelDO' ^mav b^{e use(}* to obtain the type manager type ED coπesponding to a given key data type. If there is no coπesponding type ID, this routine returns zero.

Another function, TM_GefTypeName(), may be used to get a type's name given the type ID number. In the prefened embodiment, this function returns using the 'aTypeName' parameter, the name of the type. . ' . ^■ . - • - ^•

^' A function, such as TM_.FindTypesByKeywordO, may be used to search for all type DBs (available from the context in which it is called) to find types that contain the keywords specified in the 'aKeywordList' parameter. If matches are found, the function can allocate and return a handle to an array of type IDs in the 'thelDList' parameter and a count of the number of elements in this anay as it's result. If the function result is zero, 'theEDList' is not allocated.

The function TM_GetTypeFiieName() gets the name of the header file in which a - type was defined (if any). ^' - . • . • -

Given a type ID, a function, such as TM_GetParentTypeEDO, can be used to get the ID of the parent type.. If the given LD'has na parent, an ED of 0 will be returned. If an enor occuπs, a value of -1 - ill be returned. , ^' , Another function, such as TM_IsTypeDescendantO, may be used to determine if one type is the same as or a descendant of another. The TM_IsTypeDescendant() call could be used to check only direct lineage whereas TM_AreTypesCompatible() checks lineage and other factors in determining compatibility. If the source is a descendant of, or the same as, , -the target, TRUE is returned, otherwise FALSE is returned. - --

^• Another set of functions, hereinafter refened to as TM_TypeIsPointer(), TMjTypelsHandleO, TMJTypeIsRelRef(), TMJTypelsCoIIectioπRefO, TMJTypelsPersistentRefO, may be used to determine if a typelD represents a _, pointer/handle/relative etc. reference to memory or the memory contents itself (see typeED flag definitions). The routines optionally return the typeLD of the base type that is referenced if the type ID does represent a pointer handle/ref. In the prefened embodiment, when calling TM_TypeIsPtrO_> tyP^e TD that is a handle will return FALSE so the determination of whether the type is a handle, using a function such as TMjTypelsHandleO, could be checked

^• first where both-possibilities may occur. The function TM_TypeIsReference() will return true if the type is any kind of reference. This function could also return the particular

^• reference type via a paramter, such as the 'refType' parameter. -

Another function, such as TMJTypesAreCompatibleOj may be used to chec ,if the source type is the same as, or a descendant of, the target type. In the prefened embodiment, this routine returns: ^{' '} • ' ^;

+1 If the source type is a descendant of the target type (a legal connection)

-1 If the source type is a group type (no size) and the target is descended from it (also a legal connection) ' - .'

0 Otherwise (an illegal connection) , . "

If the source type is a 'grouping' type (e.g., Scalar), i.e., it has no size then this routine will return compatible if either the source is ancestral to the target or vice-versa. This allows for data flow connections that are typed using a group to be connected to flows that are more restricted. > ,' ' .^' - . ^'' , ^' ;

Additional functions,^' such as TM_GetTypeSize() and TM_SizeOf0, could be ^' applied in order to return the size of the specified data.type. For example, TM_GetTypeSize() could be provided with an optional data handle which may be used to » determine the size of variable sized types (e.g.; strings). Either the size of the type could be returned or, alternatively, a 0 could be returned for an eπor. TM_SizeOfO^:could be provided with a similar optional data pointer. It also could return the size of the type or 0 for an eπor. ,

A function, such as TM_GetTypeBoundsO, could be programmed to return the aπay bounds of an aπay type. If the type is not an aπay type, this function could return a FALSE indicator instead. , ^• ,

¹ The function TM_GetArrayTypeElementOffset0 can be used to access_^ the . individual elements of an aπay type. Note that this is distinct from accessing the elements an anay field. If a type is an aπay type, the parent type is the type of the element of that anay. This knowledge can be used to allow assignment or access to the aπay elements through the

^• type manager API. ' . ' - ^' ' ^'-

The function TM_InitMem() mitializes an existing block of memory for a type. The memory will be. set to zero except for any fields which have values which will be initialized ^' to the appropriate default (either via annotation or script calls - not discussed herein). The function TM_NewPtrfJ allocates and initializes a heap data pointer. If you wish to allocate a larger amount of memory than the type would imply, you may specify a non^zero value for the 'size' parameter. The value passed should be TM_GetTypeSize(...) + the extra memory required. If a type ends in a variable sized anay parameter, this will be necessary in order to ensure the coπect allocation. The function TM_NewHdl() performs a similar function for a heap data handle. The functions TMJDisposePtrO and TM_DisposeHdlO may be used to

• de-allocate memory allocated in this manner.

The function TMJLocaLFieldPathO can be used to truncate a field path to that . . portion that lies within the specified enclosing type. Normally field paths would inherently satisfy this condition, however, there are situations where a field path implicitly follows a reference. This path truncation behavior is performed internally for most field related calls. ■ This function should be used prior to such calls if the possibility of a non-local field path exists in order to avoid confusion. For example: . ' . -

typedef struct, tl

{. ^• - char x[16];

} ti ; - .

Given a type ID, and a field within that type, TM_GetFieldTypeEDO will return the type ED of the aforementioned field or 0 in the case of an eπor.

The function TM_GetBuiItIrιAncestorO returns the first built-in direct (i.e., not via a reference) ancestor of the type ID given. ^'

Two functions, hereinafter called TM_GetIntegerValueO and TM_GetRealValue(), could be used to obtain integer and real values in a standardized form. In the prefened embodiment, if the specified type is, or can be converted to, an integer value, the TM_GetIntegerValueO would return that value as the largest integer type (i.e., int64). If the specified type is, or can be converted to, a real value, TM_GefRealVahιeO would return that value the largest τeal type (i.e., long double). This is useful when code does not want to be concerned with the actual integer or real variant used by the type or field. Additional functions, such as TM_SetIntegerVaIueO and TMjSetRealVaiueQ, could perform the same function in the opposite direction. ,

Given a type ID, and a field within that type, a function, hereinafter called TM_GetFieldContaiπerTypeEDOj could be use to return the container typelD of the aforementioned field or 0 in the case of an enor. Normally the container type ID of a field is identical to 'aTypelD', however, in the case where a type inherits fields from other ancestral types, the field specified may actually be contributed by'one of those ancestors and in this case, die type ID returned will be some ancestor of 'aTypelD' : In the prefened embodiment, if a field path is specified via 'aFieldName' (e.g., fieldl .field2) then the container type ED returned would coπespond to the immediate ancestor of 'field2', that is 'fieldl': Often these inner structures are anonymous types that the type manager creates during the types . acquisition process. ^' . ^• " ' ' . O 03/065180

A function, hereinafter called TM_GetFieidSize^'θ, returns the size, in bytes, of a field, given the field name and the field's enclosing type; 0 is returned if unsuccessful.

A function, hereinafter called TM_IsLegalFieldPathO, determines if a string could be a legal field path, i.e., does not contain any characters that could not be part of a field path. This check does not mean that the path actually is valid for a given type, simply that it could be. This function operates by rejecting any string that contains characters that are not either alphanumeric or in the set '[',']',_', or '.'. Spaces are allowed only between '[' and ']'.

Given an enclosing type ED, a field name, and a handle to the data, a function, hereinafter known as TM_GetFieldVaIueH(), could be used to copy the field data referenced by the handle into a new handle. In the prefened embodiment, it will return the handle storing the copy of the field data. If the field is an aπay of 'char¹, this call would append a terminating null byte. That is if a field is "char[4]" then at least a 5 byte buffer must be allocated in order to hold the result. This approach greatly simplifies C string handling since returned strings are guaranteed to be properly terminated. A function, such as TM_GetFieIdVaIueP , could serve as the pointer based equivalent. Additionally, a function such as TM_SetFieldValue() could be used to set a field value given a type ID, a field name and a binary object. It would also return an enor code in an eπor.

A function, such as TM_SetCStringFieldValueO, could be used to set the C string field of a field withm the specified type. This function could transparently handle logic for the various allowable C-string fields as follows:

1) if the field is a charHdl then:

a) if the field already contains a value, update/grow the existing handle to hold the hew value <

^*? b) otherwise allocate a handle and assign it to the field

2) if the field is a charPtr then:

a) if the field already contains a value-

i) if the previous string is equal to or longer than the new one, copy new string into existing pointer ii) otherwise dispose of previous pointer, allocate a new one and assign it

b) otherwise allocate a pointer and assign it to the field

3) if the field is a relative reference then.

a) this should be considered an eπor. A pointer value could be assigned to such a, field pπor to moving the data into a collection in which case you should use a function similar to the TM_SetFieIdValue() function descπbed above.

4) if the field is an aπay of char then:

a) if the new value does not fit, report anay bounds enor

b) otherwise copy the value into the aπay

A function, such as TM_AssignToField(), could be used to assign a simple field to a value expressed as a C string. For example, the target field could be:

a) Any form of stπng field or string reference,

b) A persistent or collection reference to another type, or

c) Any other direct simple or structure field type. In this case the format of the C string given should be compatible with a call to TM_StrmgToBinaryQ (descπbed above) for the field type involved. The delimiter for TM_StringToBinaryO is taken to be "," and the "kCharAnayAsString' option (see TMJBinaryToString) is assumed.

. -in the prefened embodiment, the assignment logic used by this-routine (when the - - 'kAppendStringValue' is present) would result m existing stπng fields having new values appended to the end of them rather than being overwritten. This is in contrast to the behavior of TM_SetCStπngFιeldValueO descπbed above. For non-stπng fields, any values specified overwrite the previous field content with the exception of assignment to the 'aStrmgH' field of a collection or persistent reference with is appended if the 'kAppendStringValue' option is present. If the field being assigned is a collection reference and the T AppendStπng Value' option is set, the contents of 'aStringPtr' could be appended to the contents of a stπng field. If the field being assigned is a persistent reference, the 'kAssignToRefTypeVkAssignToUmquelD' or 'kAssignToStπngH' would be used to .determine if the typelD, unique ID,, or 'aStringH' field of the reference is assigned. Otherwise the assignment is to the name field. In the case of 'kAssignToReffype', the: string could be assumed to be a valid type name which is first converted to a type TD. If the field is a _, relative reference (assumed to be to a string), the contents of 'aStringPtr¹ could be assigned ^'to - ' it as a (internally allocated) heap pointer. , - ^'

Given an enclosing type ID, a field name, and a pointer to the data, a function such as . TM_SetArrFieldValue 0 could be used to copy the data referenced by the pointer into an element of an aπay field element into the buffer supplied. Array fields may have one, or two dimensions. ' ' - , '- . - ^'

Functions, hereinafter named TM_GetCStringFieldValueBO, TM_GetCStringFieldValuePO and TM_GetCStringFieldValueHO, could be' used to get a C string field from a type into a buffer/pointer/handle. In the case of a buffer, the buffer supplied must be large enough to contain the field contents returned. In other cases the function or program making the call must dispose of the memory returned when no longer required. In the prefened embodiment, this function will return any string field contents regardless of how is actually stored in the type structure, that is the field value may be in an array, via a pointer, or via a handle, it will be returned in the memory supplied. If the field type is not appropriate for a C string, this function could optionally return FALSE and provide an empty output buffer. . ^l .

Given an enclosing type ID, a field name, and a pointer to the data, the system should also include a function, hereinafter name TM_GetArrFieldValueP 0_» that will copy an element of an aπay field element's data referenced by the pointer into the buffer supplied. . Array fields may have one, or two dimensions.

Simple wrapper functions,^' hereinafter named TM_GetFieldBoundsO, TM_GetFieldOffset0, TM_GetFieldUuitsO, and TM_GetFieldDescriptionO, could be provided in order to access the coπesponding field in ETJTield 100. Coπesponding 'set' functions (which are similar) could also be provided.

The function TMJForAIIFieldsLoopO is also provided that will iterate through all^" fields (and sub-fields) of a type invoking the specified procedure. This behavior is commonplace in a number of situations involving scanning the fields of a type. In the • prefened embodiment, the scanning process should adhere to a common approach and as a O 03/06518

' result a function, such as this one, should be used for that purpose. A field action function takes the following form:

Boolean myActionFn ( // my field action function

ET TypeDBHdl aTypeDBHdl, // 1: Type DB (NULL to default)

ETJTypelD 104 aTypeTD, // 1: The type ED

- ET TypeTD 104 aContainingTypeED, // 1: containing Type ED of field anoπPtr aDataPfr, // 1: The type data pointer anonPtr context, // IO:Use to pass custom context charPtr fieldPath, // Field path for field

ET TypeTD 104 aFieldTypeTD, // I:Type ID for field int32 dimensionl, // LField aπay bounds 1 (0 if

N/A) int32 dimension, // LField aπay bounds 2 (0 if

N/A) int32 fieldOffset, // LOffset of start of field int32 options, // Options flags anonPtr intemalUseOnly // I.For internal use only

) - // R-TRTJE for success

In this example, fields are processed in the order they occur, sub-field calls (if appropriate) occur after the containing field call. If this function encounters an anay field (1 or 2 dimensional), it behaves as follows:

a) The action function is first called once for the entire field with no field indexing specified in the path.

b) If the element type of the anay is a structure (not a union), the action function will be invoked recursively for each element with the appropriate element index(es) reflected in the 'fieldPath' parameter, the appropπate element specific value in 'fieldOffset', and 0 for both dimensionl and dimension2.

This choice of behavior for aπay fields offers the simplest functional interface to the action function. Options are:

kRecursiveLoop - If set, recurses through sub-fields, otherwise one-level only kDataPtrlsViewRef - The 'aDataPtr¹ is the address of an ET_ViewRef designating a collection element

A function, hereinafter refeπed to as TMJPieldNameExistsO, could be used to determine if a field with the given name is in the given type, or any of the type's ancestral types. If the field is found return it returns TRUE, otherwise it returns FALSE.

A function, hereinafter refened to as TM_GetNumberOfFields0, may be used to return the number of fields m a given structured type or a -1 m the case of an eπor. In the prefened embodiment, this number is the number of direct fields within the type, if the type contains sub-structures, the fields of these sub-structures are not counted towards the total returned by this function. One could use another function, such as

TMJForAllFιeldsLoopO,to count fields regardless of level with 'kRecursiveLoop' set true and a counting function passed for 'aFieldFn' (see TM_GetTypeMaxFlagIndex).

Another function, refeπed to as TM_GetFieldFlagIndexO, can provide the 'flag index' for a given field within a type. The flag index of a field is defined to be that field's index in the series of calls that are made by the function TMJForAlTFieldsLoopO (descπbed above) before it encounters the exact path specified. This index can be utilized as an index into some means of storing information or flags specific to that field within the type. In the prefened embodiment, these indeces include any field or type aπays that may be within the type. This function may also be used internally by a number of collection flag based APIs but may also be used by external code for similar purposes. In the event that TM_ForAllFieldsLoopO calls back for the enclosing structure field before it calls back for the fields withm this enclosing structure, the index may be somewhat larger than the count of the 'elementary" fields within the type. Additionally, because field flag indeces can be easily converted to/from the conesponding field path (see TM_FlagIndexToFieldPath), they may be a useful way of referπng to a specific field in a variety of circumstances that would make maintaining the field path more cumbersome. Supporting functions include the following- TMJFieldOffsetToFlaglndexO is a function that converts a field offset to the coπesponding flag index withm a type; TMJFlaglndexToFieldPathO is a function that converts a flag index to the coπesponding field path withm a type, and the function

TM_GetTypeMaxFlagIndexO returns the maximum possible value that will be returned by TM_GetFieldFlagIndexO for a given type. This can be used for example to allocate memory for flag storage. Another function, refeπed to as TMJFieldNamesToIndecesO, converts a comma seperated list of field names/paths to the coπesponding zero terminated list of field indeces. It is often the case that the 'fieldNames' list references fields within the structure that is actually referenced from a field within the structure identified by 'aTypeED'. In this case, the index recorded in the index list will be of the referencing field, the remainer of the path is ignored. For this reason, it is possible that duplicate field indeces might be implied by the hst of 'fieldNames' and as a result, this routine can also be programmed to automatically eliminate duplicates.

A function, hereinafter name TM_GetTypeProxyO, could be used to obtain a proxy type that can be used within collections in place of the full persistent type record and which contains a limited subset of the fields of the original type. While TM_GetTypeProxyO could take a list of field indeces, the function TMJYlakeTypeProxyFromFieldsO could be used to take a comma separated field list. Otherwise, both functions would be identical. Proxy types are all descendant of the type ET_Hit and thus the first few fields are identical to those of ET_Hιt. By using these fields, it is possible to determine the original persistent value to which the proxy refers. The use of proxys enables large collections and lists to be built up and fetched from servers without the need to fetch all the coπesponding data, and without the memory requirements implied by use of the referenced type(s). In the prefened embodiment, proxy types are formed and used dynamically. This approach provides a key advantage of the type system of this invention and is crucial to efficient operation of complex distributed systems. Proxy types are temporary, that is, although they become known throughout the application as soon as they are defined using this function, they exist only for the duration of a given run of the application. Preferably, proxy types are actually created into type database 'E' which is reserved for that purpose (see above). Multiple proxys may also be defined for the same type having different index lists. In such a case, if a matching proxy already exists in Ε, it is used. A proxy type can also be used in place of the actual type in almost all - situations, and can be rapidly resolved to obtain any additional fields of the original type. In one embodiment, proxy type names are of the form:

typeName_Proxy_n

Where the (hex) value of 'n' is a computed function of the field index list. Another function that may be provided as part of the API, hereinafter called TMJVlakeTypeProxyFrornFilterO, can be used to make a proxy type that can be used within collections in place of the full persistent type record and which contains a limited subset of the fields of the original type. Preferably, the fields contained in the proxy are those allowed by the filter function, which examines ALL fields of the full type and-returns TRUE to include the field in the proxy or FALSE to exclude the field. For more information concerning proxy types, see the discussion for the function

TM_MakeTypeProxyFromFieldsO.The only difference between this function and the function TM_MakeTypeProxyFrornFieIds is that TM_MakeTypeProxyFromFieldsO expects a commma separated field list as a parameter instead of a filter function. Another function, TM_IsTypeProxyO_> could be used to determine if a given type is a proxy type and if so, what original persistent type it is a proxy for. Note that proxy type values start with the fields of ET_Hιt and so both the unique ID and the type ID being referenced may be obtained more accurately from the value. The type ID returned by this function may be ancestral to the actual type ID contained within the proxy value itself. The type ET_Hit may be used to return data item lists from servers in a form that allows them to be uniquely identified (via the _system and _id fields) so that the full (or proxy) value can be obtained from the server later. ET Hit is defined as follows:

typedef struct ET_Hit // list of query hits returned by a server

{

OSType _system; // system tag unslnt64 J // local unique item ID

ET TypeTD 104 Jype; // type ID int32 _relevance; - // relevance value 0 .100

} ETJ Hit;

-

The function TM_GetNthFieldType() gets the type of the Nth field in a structure. TM_GetNthFieldNameO obtains the conesponding field name and TM_GetNthFieldOffset0 the coπesponding field offset.

Another function that may be included with the API toolset is a function called TM_GetTypeChildren(). This function produces a list of type EDs of the children of the given type This function allocates a zero terminated aπay of ETJTypelD 104's and returns ',the address of the aπay in 'aChildlDList'; the type ED's are written into this aπay. If 'aChildlDList' is specified as NULL then this array is not allocated and the function merely counts the number of children; otherwise 'aChildlDList' must be the address of a pointer that will point at the typelD aπay on exit. A negative number is returned in the case of an eπor. In the prefened embodiment, various specialized options for omitting certain classes of child types are supported. ^■ J . ,

A function, hereinafter refeπed to as TM_GetTypeAncestorsO- may also be provided that produces a list of type EDs of ancestors of the given type. This function allocates a zero terminated anay of ETJTypelD 104 and returns the address of the aπay in 'ancestrallDs'; the type EO's are wπtten into this aπay. If 'ancestralTDs' is specified as NULL then this anay is not allocated and the function merely counts the number of ancestors; . otherwise 'ancestrallDs' must be the address of a pointer that will point at the typelD aπay on exit. The last item in the list is a 0, the penultimate item is-the primal ancestor of the given type, and the first item in the list is the immediate predecessor, or parent, of the given type. The function TM_GetTypeAπcestorPathO produces a ':' seperated type path from a given ancestor to a descendant type. The path returned is exclusive of the type name but inclusive of the descendant, empty if the two are the same or 'ancestorTD' is not an ancestor or 'aTypelD'. The function TM_GetInheritanceChainO is very similiar to TM_GetTypeAncestorsO with the following exceptions: - . ^' -.^,

(1) the anay of ancestor type ids returned is in reverse order with the primal ancestor being in element 0

(2) the base type from which the list of ancestors is determined is included in the aπay and is the next to last element (anay is 0 terminated)

(3) the count of the number of ancestors includes the base type

In the prefened embodiment, this function allocates a zero terminated array of ETJTypelD 104's and returns the address of the anay in 'inheritanceChainlDs'; the type ID's are written into this aπay. If 'inheritanceChainlDs' is specified as NULL then this array is not ' allocated and the function merely counts the number of types in the inheritance chain; otherwise 'inheritanceChainlDs'^' must be the address of a pointer that will point at the typeED anay on exit. The last item in the list is 0, element 0 is the primal ancestor of the base type, and the next to last item, in the list is the base type. • The API could also include a function, hereinafter called

TM_GetTypeDescendantsO, that is able to create a tree collection whose oo node is the type specified and whose branch and leaf nodes are the descendant types of the root. Each node in the tree is named by the type name and none of the nodes contain any data. . Collections of derived types can serve as useful frameworks onto which various instances of that type can be 'hung' or alternatively as a navigation and/or browsing framework. The resultant collection can be walked using the collections API (discussed in a later patent). The function TM_GetTypeSiblingsO produces a list of type IDs of sibling .types of the given , , type. This function allocates a zero terminated anay of ET_TypeED 104's and returns the address of the array in 'aListOSibs', the type ID's are written into this aπay. If 'aListOSibs' is specified as NULL then this aπay is not allocated and the function merely counts the number . of siblings; otherwise 'ancestraUDs' must be the address of a pointer that will point at the typeED anay on exit. The type whose siblings we wish to find is NOT included in the

. returned list. The function TM_GetNthChildTypeIDO gets the n'th child Type ID for the passed in parent. The function returns 0 if successful, otherwise it returns an enor code.

The function TMJBinaryToStringO converts the contents of a typed binary value into a C string containing one field per delimited section. During conversion, each field in turn is converted to the equivalent ASCH string and appended to the entire string with the specified delimiter sequence. , If no delimiter is specified, a new-line character is used. The - handle, 'aStringHdl', need not be empty on entry to this routine in which case the output of this routine is appended to whatever is already in the handle. If the type contains a variable

^" sized aπay as its last field (i.e., stuffTJ), it is important that 'aDataPtr' be a true heap allocated pointer since the pointer size itself will be used to determine the actual dimensions of the

- anay: -In the prefened embodiment, the following specialized options are also available:

kUnsignedAsHex r- display unsigned numbers as hex

kCharAπayAsS ring - display char aπays as C strings

kShowFieldNam.es ~ prefix all values by fieldName: , - ^" .

kOneLevelDeepOnly — Do Not go down to evaluate sub-structures:

111

SUδSTTTTJ TE SHEET (RULE26 An additional function, hereinafter refened to as TM_StringToBinaryO, may also be provided in order to convert the contents of a C string of the format created by TM_BιnaryToStπng() into the equivalent binary value in memory.

The API may also support calls to a function, hereinafter refeπed to as TMJ owestCommottAncestorO, which obtains the lowest common ancestor type ID for the two type IDs specified. If either type ID is zero, the other type ID is returned. In the event that one type is ancestral to the other, it is most efficient to pass it as the 'typeED2' parameter.

Finally, a function, refened to as TMJDefϊneNewTypeO, is disclosed that may be used to define a new type to be added to the specified types database by parsing the C type definition supplied m the string parameter. In the prefened embodiment/the C syntax typedef stπng is preserved in its entirety and attached to the type definition created so that it may be subsequently recalled. If no parent type ED is supplied, the newly created type is descended directly from the appropriate group type (e.g , structure, integer, real, union etc ) the typedef supplied must specify the entire structure of the type (i.e., all fields). If a parent type ED is supplied, the new type is created as a descendant of that type and the typedef supplied specifies only those fields that are additional to the parental type, NOT the entire type. This function is the key to how new types can be defined and incorporated into the type system at run time and for that reason is a critical algorithm to the present invention. The implementation is based on the parser technology descπbed in Claimed Parser patent application and the lexical analyzer technology (the"Claimed Lexical Analyzer") as provided in Appendix 3 As set forth above, those pending applications are fully incorporated herein. The reader is refeπed to those patents (as well as the Claimed Database patent application) for additional details. The BNF specification to create the necessary types parser (which interprets an extended form of the C language declaration syntax) is provided in Appendix A. The coπesponding lexical analyzer specification is also provided in Appendix A.

As can be seen from the specifications Appendix A, the types acquisition parser is designed to be able to interpret any construct expressible in the C programming language but has been extended to support additional features. The language symbols associated with these extensions to to C are as follows:

script - used to associate a scπpt with a type or field

annotation - used to associate an annotation with a type or field ^■ ; ' @ - relative reference designator (like '*' for a pointer)

- ' _, @@ ^_ collection reference designator , , :

# — persistent reference designator . ^{' '}

<on> - script and annotation block start delimiter ' ^'

^, <no> - script and annotation block end delimiter

>< - echo field specification operator ' . ^' . •

In order to complete the types acquisition process, a 'resolver' function and at least • one plug-in are provided. A pseudo code embodiment of one possible resolver is set forth in Appendix A. Since most of the necessary C language operations are already provided by the built-in parser plug-in zero, the only extention of this solution necessary for this application is - the plug-in functionality unique to the type parsing problem itself. This will be refeπed to as ^' plug-in one and the pseudo code for such a plug in is also provided in Appendix A.

The foregoing description of the prefened embodiments of the invention has been presented for the purposes of illustration and description. For example, although described with respect to the C programming language, any programming language could be used to implement this invention. Additionally, the claimed system and method should not be limited to the particular API disclosed. The descriptions of the header structures should also not be limited to the embodiments described. .While the sample pseudo code provides examples of the code that may be used, the plurality of implementations that could in fact be developed is nearly limitless. For these reasons, this description is not intended to be exhaustive or to limit • the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

SYSTEM AND METHOD FOR MANAGING COLLECTIONS OF DATA ON A

NETWORK Inventor: John Fairweather

BACKGROUND OF THE INVENTION

There are several problems associated with sharing aggregated data in a distπbuted environment. The primary problems involve: (I) enabling systems to share their "knowledge" of data; (2) enabling storage of data for distribution across the computing environment; and (3) a framework for efficiently creating, persisting, and shaπng data across the network. The problem of defining a run-time type system capable of manipulating strongly typed binary information in a distributed environment has been addressed in a previous patent, attached hereto as Appendix 1, hereinafter refeπed to as the "Types Patent". The second problem associated with sharing data in a distπbuted environment is the need for a method for creating and sharing aggregate collections of these typed data objects and the relationships between them. A system and method for achieving this is a 'flat', i.e., single contiguous allocation memory model, attached hereto as Appendix 2. This flat model, containing only 'relative' references, permits the data to be shared across the network while maintaining the validity of all data cross-references which are now completely independent of the actual data address in computer memory. The final problem that would preferably be addressed by such a system is a framework within which collections of such data can be efficiently created, persisted, and shared across the network. The goal of any system designed to address this problem should be to provide a means for manipulating arbitrary collections of iπteπelated typed data such that the physical location where the data is 'stored' is hidden from the calling code (it may in fact be held in external databases), and whereby collections of such data can be transparently and automatically shared by multiple machines on the network thus inherently supporting data 'collaboration' between the various users and processes on the network. Additionally, it should be a primary goal of such a framework that data 'storage' be transparently distributed, that is the physical storage of any given collection may be within multiple different containers and may be distπbuted across many machines on the network while providing the appearance to the user of the access APL of a single logical collection whose size can far exceed available computer memory.

Any system that addresses this problem would preferably support at least three different 'container' types withm which the collection of data can transparently reside

114

APPENDIX 2

SUBSTITUTE SI€ET R (meaning the caller of the API does not need to know how or where the data is actually stored). The first and most obvious is the simple case where the data resides in computer memory as supported by the 'flat' memory model. This container provides maximum efficiency but has the limitation that the collection size cannot exceed the RAM (or virtual) memory available to the process accessing it. Typically on modern computers with 32-bit architectures this puts a limit of around 2-4GB on the size of a collection. While this is large for many applications, it is woefully inadequate for applications involving massive amounts of data in the tenabye or petabyte range. For this reason, a file-based storage container would preferably be implemented (involving one or more files) such that the user of a collection has only a small stub allocation m memory while all accesses to the bulk of the data in the collection are actually to/from file (possibly memory-cached for efficiency). Because the information in the flat memory model contains only 'relative' references, it is equally valid when stored and retrieved from file, and this is an essential feature when implementing 'shadow' containers. The file-based approach minimizes the memory footprint necessary for a collection thus allowing a single application to access collections whose total size far exceeds that of physical memory. There is essentially no limit to the size of data that can be manipulated in this manner, however, it generally becomes the case that with such huge data sets, one wants access to, and search of, the data to be a distπbuted problem, i.e , accomplished via multiple machines in parallel. For this reason, and for reasons of data- sharing and collaboration, a third kind of container, a 'server-based' collection would preferably be supported. Other machines on the network may 'subscribe' to any previously 'published' server-based collection and manipulate it through the identical API, without having to be aware of its possibly distributed server-based nature.

SUMMARY OF INVENTION

The present invention provides an architecture for supporting all three container types. The present invention uses the following components: (I) a 'flat' data model wherein arbitrarily complex structures can be instantiated within a single memory allocation (including both the aggregation aπangements and the data itself, as well as any cross references between them via 'relative' references); (2) a run-time type system capable of defining and accessing binary strongly-typed data; (3) a set of 'containers' within which information encoded according to the system can be physically stored and preferably include a memory resident form, a file-based form, and a server-based form; (4) a client-server environment that is tied to ' , the types system and capable of interpreting and executing all necessary collection . .. .' - manipulations remotely; (5) a basic aggregation structure providing as_;a mimimum a 'parent', 'nextChild', 'previousChild', 'firstChild', and 'lastChild' links or equivalents; and (6) a data . attachment structure (whose size may vary) to which strongly typed data can be attached and which is associated in some manner with (and possibly identical to) a containing aggregation node in the collection. The invention enables the creation, management, retrieval, distribution of massively targe collections of information that can be shared across a distributed network without building absolute references or even pre-existing knowlecge of the data and data ^' structures being stored in such an environment. .^' ' ." " ■ , ."

The present invention also provides a number of additional features that extend this ^' : functionality in a number of important ways. For example, the aggregation models supported by the system and associated API include support for stacks, rings, arrays (multi-dimensional), queues, sets, N-trees, B-trees, and lists and arbitrary mixtures of these types within the same organizing framework including the provision of all the basic operations (via APT) associated with the data structure type involved in addition to searching and sorting. The present invention further. includes the ability to 'internalize' a non-memory based storage container to memory and thereafter automatically echoing all write actions to the actual container thereby ^' gaining the performance of memory based reads with the assurance of persistence via automated echoing of writes to the external storage container. The present incention also supports server-based publishing of collections contents and client subscription thereto such that the client is transparently and automatically notified of all changes occurring to the server- based collection and is also able to transparently affect changes to that collection thereby „ facilitating automatic data collaborations between disparate nodes on the network. This invention and other improvements to such invention will be further explained below.

BRIEF DESCRIPTION OF THE FIGURES

"Figure 1 illustrates a sample όne-dimensional structure.

Figure 2 illustrates a generalized N-Tree. • ' . .

Figure 3 illustrates a 2*3 two-dimensional anay.

Figure 4 illustrates a sample memory structure of a collection containing 3 'value' nodes.

, Figure 5 illustratse a sample memory structure having various fields including references to other nodes in the collection. ^• -

- Figure 6 illustrates a diagrammatic representation of the null and dirty flags of the present invention. . , ^'

/ . DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

. For the purposes of this description, the existence of a client-server architecture tied to types via the 'key data type' concept, as disclosed in the Types Patent, such that the location of the server from which a given collection can be obtained will be assumed. The actual ^• - 'physical manifestation of a server-based collection may use any of the three container types described above (i.e., memory, file and server) thus it is possible to construct trees of server- based collections whose final physical form may be file or memory based.

' To manipulate any arbitrary collection of related data in a distributed environment, some form of representation of an inherently complex and hierarchical collection of information is required. In the prefened embodiment, a 'flat' (i.e., single memory allocation) form of representation is used. The flat data-model technology attached hereto in Appendix 2 (hereinafter the "Memory Patent") provides the ideal environment for achieving this. In order to understand many of the descriptions below, the reader is refeπed to the Memory Patent, which is incoφorated by reference herein. Just two structure variants based on this model are needed to encode collection and data information, these are the 'ET_Simplex' structure (which is used to hold and access the typed data described via the 'typelD' using the run-time type system described in Appendix 1 attached hereto (hereinafter the "Types Patent")) and the - 'ET_CompIex' structure (used to describe collections of data elements and the parent/child relationships between them). These two structures are set forth in pseudo code and defined below (in addition to the Memory Patent).

typedef struct ET Simplex // Simplex Type record

{ ,, . //

_„ ET_Hdr.. , ^{" •} hdr ; // Standard header int32 size ; // size of simplex value

( in bytes) _r

ET_Offset /* ET_Simplex */ null Flags ; // ! ! ! ref . to null flags simplex . -

ET_0££set /* ET_Simplex */ dirtyFlags ; // ! ! ! ref . to dirty flags simplex - long nσtUsed [2] _; // spare char value [NULL VRR] ,- // value (actual s^'i2e varies) ' ^{■ '}

} ET _Simplex,- . // , typedef struct ET Complex ^" , ^{' "}// Complex Type record^'

{ ^{" •}: / / _^ •

- ^' ET_Hdr hdr ; - // Standard header

ETJtiexHdl - ^' >.' recognizer ,- // Name recognizer DB

( if applicable) Handle valueH; // handle to value of element ;

ET_Offset /* ET_SimplexPtr */ valueR; // ref_, to value simplex union ' -' '

{

ETJTypelD . .. • _ , typeID; // ID of this type

• • struct , ' '

{ . ^'•' ;.,-'^" - unslnt32 crc; // ID viewed as_, a pair of integers unslnt32 ^' :^{' ' '} flags,- } a; . ' _v .-

} u,

ET_0ffset /* ET_ComplexPtr */ nextElem; II ! ! !. link to next element

ET_0ffset /*^' ET_ComplexPtr */ prevElem; II ! I 1 ^' link to previous element - .

ET_0ffset /* ET_ComplexPtr */ childHdr; ^• .// ! ! I link to first child element

• ET_0ffset /* ET_ComplexPtr */ childTail; " " ll >. ! ! link to last child element ' - '' long fromWhich; ll collection type int32 ; ^; dimension; ' II current # of nodes children - , char . ^■ name [kNodeNameS LZβ] ; // element name long tag; // tag value (if used)

ET_Offset /*,ET_StringPtr */_ description; // Description (if relevant) , •

ET_0ffset /* ET_StringPtr ^J l tags ,- _ . // ! ! ! ref. to tags string -

ET_ElementDestructor '._^' ' destructorFπ; // Custom destructor function •> - ' unslnt32 • , „ shortcut; ' // Shortcut sequence (if any) ' .- - ^• .

ET_ProcreatorFn , - ^■• procreator; // Procreator function long notUsed[3]_; , -// not used

}' ET_Com lex; • // *

In the preferred embodiment, the various fields within the ET_Simplex stracture are defined and used as follows: '

"hdr" - This is a standard header record of type ET_Hdr

"size" - This field holds the size of the 'value' aπay (which contains the actual typed data) in bytes. ^' _, '

"nullFlags" - This is a relative reference to another ET_Simplex structure containing the null flags anay. - . '

"dirtyFlags" - This is a relative-reference to another ET_Simplex structure containing the ' dirty flags aπay. . ^• . ' , ' .^" . ,

"value" - This variable sized field contains the actual typed data value as.deteπnined by the 'typeED' field of the parent complex record. , The vaπous fields with the ET_Complex structure are defined and used as follows: "hdr" - This is a standard header record of type ET_Hdr as

"recognizer' - This field may optionally hold a reference to a lexical analyzer based lookup table used for rapid lookup of a node's descendants in certain types of complex structure aπangements (e.g., a 'set'). The use of such a recognizer is an optimization only. "valueH" - Through the API described below, it is possible to associate a typed value with a node either by incorporating the value into the collection as a simplex record (referenced via the 'valueR' field), or by keeping the value as a separate heap-allocated value referenced directly from the 'valueH' field. The use of intemal values via the 'valueR' field is the default and is prefened, however, some situations may require non-flat reference to external memory, and by use of the 'valueH' field, this is possible.

"valueR" - This field contains a relative reference to the ET_Sιmplex record containing the value of the node (if any).

"typelD" - This field (if non-zero) gives the type ID of the data held in the associated value record.

"prevElem" - This field holds a relative reference to the previous sibling record for this node (if any).

"nextElem" - This field holds a relative reference to the next sibling record for this node (if any).

"childHdr" - This field holds a relative reference to the first child record for the node (if any). "childTail" - This field holds a relative reference to the last child record for the node (if any). "fromWhich" - For a root node, this field holds the complex structure variant by which the descendants of the node are organized. The minimum supported set of such values (which supports most of the basic data aggregation metaphors in common use) is as follows (others are possible):

kFromArray - one dimensional aπay structure kFromList - one directional List Structure kFromStack - Stack structure kFromQueue - Queue structure kFromSet - Set Type kFromBTree - Binary tree kFrornNTree - Generalized Tree with variable branches/node kFromAπayN - N dimensional aπay structure "dimension" -Although it is possible to find the number of children of a given node by walking the tree, the dimension field also holds this information. In the case of multidimensional array accesses, the use of the dimension field is important for enabling efficient access. . -

"name" - Each complex node in a collection may optionally be named. A node's name is held in the "name" field. By concatenating names of a node and its ancestors, one can construct a unique path from any ancestral node to any descendant node, "tag" - This field is not utilized internally by this API and is provided to allow easy tagging and searching of nodes with arbitratry integer values.

"description" - Arbitrary textual descriptions may be attached to any node using this field via the API provided.

"tags" - This string field supports the element tags portion of the API (see below). "destructorFn" - If a node requires custom cleanup operations when it is destroyed, this can be accomplished by registering a destructor function whose calling address is held in this field and which is guaranteed to be called when the node is destroyed, "shortcut" - This field holds an encoded version of a keyboard shortcut which can be translated mto a node reference via the API. This kind of capability is useful in UI related applications of collections as for example the use of a tree to represent arbitrary hierarchical menus.

"procreator" - This field holds the address of a custom child node procreator function registered via the API. Whenever an attempt is made to obtain the first child of a given node, if a procreator is present, it will first be called and given an opportunity to create or alter the child nodes. This allows "lazy evaluation" of large and complex trees (e.g., a disk directory) to occur only when the user actions actually require the inner structure of a given node to be displayed.

Given the structures described above, it is clear that implementation of a one- dimensional structure is simply a matter of connecting the 'next' and 'prev' links of ET_Complex records and then providing the appropriate operations for the logical type (e.g., push pop for a stack, queue/dequeue for a queue etc.). One familiar with data stractures can readily deduce the actual algorithms involved in implementing all such operations given knowledge of the representation above. ' . ^' • Referring now to Figure I, a graphical representation of a sample one-dimensional - structure is provided. In this figure, 'root' node 100 contains three child elements 120, 130,

140, all of which have the root node 110 as their direct parent but which are linked 125, 135 ^' . as siblings through the 'next' and 'prev' fields. - -. .

Referring now to Figure 2, a graphical representation of a generalized N-Tree is shown. In this figure, the root node 205 has three child nodes 210, 215, 220 and child node 215 in turn has two children 225, 230 with node 230 itself having a single child node 235. It ^" should be readily apparent how this approach can be extended to trees of arbitrary depth and complexity. To handle the representation of multi-dimensional aπays, we would preferably. introduce additional 'dimension' nodes that serve to organize the 'leaf or data-bearing . nodes in a manner that can be efficiently accessed via aπay indeces.

Referring now to Figure 3, a graphical representation of a 2*3 two-dimensional aπay is shown. In this figure, the six nodes 320, 325, 330, 335, 340, 345 aTe the actual data-bearing nodes of the array. The nodes 310, 315 are introduced by the API in order to provide access to each 'row' of 3 elements in the anay. In fact a unique feature of the aπay implementation - in this model is that these grouping nodes can be addressed by supplying an incomplete set of indeces to the API (i.e., instead of [n,m] for a 2-D anay, specify [n]) which allows operations to be trivially performed on aπays that are not commonly available (e.g., changing row order). It is clear that this approach can be extended to any number of dimensions, thus for a 3-dimensional aπay [2*3*4], each of the nodes 320, 325, 330, 335, 340, 345 would become a parent grouping node to a list of four child data-bearing nodes. In order to make array accesses as efficiently as possible, an additional optimization in the case of aπays whose dimensions are known at the time the collection is constructed by taking advantage of knowledge of how the allocation of contiguous node records occurs in the flat memory model. That is the offset of a desired child node for a given dimension can be calculated by "off = index * m * sizeof(ET_Complex)", thus the offset to any node in a multi-dimensional array can be efficiently obtained by recursively applying this calculation for each dimension and summing the results. _

In the prefened embodiment, any node in a collection can be designated to be a new root whose 'from Which' may vary from that of its parent node (see TCJVIakeRoot). This means for example that one can create a tree of aπays of stacks etc. Besause this model permits changes to the aggregation model at any root node while maintaining the- ability to directly navigate from one aggregation to the next, complex group manipulations are also supported and are capable of being performed very simply. ^'" \ ' •^' .' , ,

In order to handle the various types, of non-memory storage containers associated with collections in a transparent manner, the present invention preferably includes a minimum - memory 'stub' that contains sufficient information to allow access to the actual container. In the prefened embodiment, this 'stub' is comprised of a standard 'ET TextDB* header record (see the Memory Patent) augmented by additional collection container fields. An example of such a-header record in pseudo code follows: .

typedef struct ET TextDB ^• // Standard allocation header record

{ ^{' ;}, ^■ - .

ET_Hdr . ^{■ '} , - hdr; // Standard heap data reference fields

ET_Offset /* ET_StringPtr */. name,- // ref.^' to name of database // other fields not discussed herein

ET_TextDBvariant u; // variant types

} ET_TextDB; -. - .

By examining the 'options' field of such a complex object variant (internally to the APT), it is possible to identify if a given collection is memory, file, or server-based and, by using the additional fields defined above, it is also possible to determine where the collection resides. One embodiment of a basic code structure which supports implementation

In the memory based case, the code checks to see if the collection is actually an 'internalized' file-based collection (see option 'klntemalizelfPossible' as defined below) and if so, echoes all operations to the file. This allows for an intermediate state in terms of efficiency between the pure memory-based and the file-based containers in that all read operations on such an internalized collection occur with the speed of memory access while only write operations incur the overhead of file I/O, and this can be buffered/batched as can ^' be seen from the type definitions above. Note also that in both the file and memory based cases, the collection may have been 'published' and thus it may be necessary to notify the

124

SUBSTITUTE a * ET R \ subscribers of any changes in the collection. This is also the situation inside the server ^'" "^• associated with a server-based collection-. Within the server, the collection appears to be file/memory based (with subscribers), whereas to the subscribers themselves, the collection '

(according to the memory stub) appears to be server-based.

' ^' Server-based collections may also be cached at the subscriber end for efficiency ' purposes. In such a case, it may be necessary to notify the subscribers of the exact changes made to the collection. This enables collaboration between multiple subscribers to a given ^' collection and this collaboration at the data representation level is essential in any complex distributed system. The type of collaboration supported by such a system is far more powerful that the Ul-level collaboration in the prior art because it leaves the UI of each user free to display the data in whatever manner that user has selected while ensuring that the underlying data (that the UI is actually visualizing) remains consistent across all clients. This - automation and hiding of collaboration is a key feature of this invention. In the prefened . . embodiment, the UI itself can also be represented by a collection, and thus TJI-level collaboration (i.e., when two users screens are synchronized to display the same thing) is also • available as a transparent by-product of this approach simply by having one user 'subscribe' to the UI collection of the other.. ' -

Referring now to Figure 4, a sample^'memory structure of a collection containing 3 'value' nodes is shown. As explained above, the job of representing aggregates or collections of data is handled primarily by the ET_Complex records 405, 410, 415, 420, while that of

, holding the actual data associated with a given node is handled by the ET_SimpIex records 425, 430, 435. One advantage of utilizing two separate records to handle the two aspects is that the ET Simplex records 425, 430, 435 can be variably sized depending on the typelD of the data within them, whereas the ET_Complex records 405, 410, 415, 420 are of a fixed size. By separating the two records, the navigation of the complex records 405, 410, 415, 420 is optimized. In the prefened embodiment, the various fields of a given type may also ' include references to other nodes in the collection either via relative references (denoted by the '@' symbol), collection references (denoted by the '@@' symbol) or persistent ^' , references (denoted by the '#' symbol). This means, for example/that one of the fields of a

^' simplex record 425,430, 435 may in-fact refer to a new collection with a new root node embedded within the same memory allocation as the parent collection that contains it. * •' Refemng now to Figure 5, a sample memory structure having various fields including references to other nodes in the collection is shown. In this figure, the 'value' of a node 425 represents an organization. In this case, one of the fields is the employees of the organization. This figure illustrates the three basic types of references that may occur between the vaπous ET_Simplex records 425, 430, 435, 525, 530, 535, 540 and ET _Complex records 405, 410, 415, 420, 505, 510, 515, 520 in a collection. The relative reference '@' occurs between two simplex nodes 525, 540 in the collection, so that if the 'notes' field of a node 525 were an arbitrary length character string, it would be implemented as a relative reference (char @notes) to another simplex record 540 containing a single variable sized character aπay This permits the original "Person" record 525 to have fixed size and an efficient memory footprint, while still being able to contain fields of arbitrary complexity within it by relative reference to another node 540. Another use of such a reference might be to a record containing a picture of the individual. This would be implemented in an identical manner (Picture @picture) but the referenced type would be a Picture type rather than a character aπay.

The collection reference '@@' in record 425 indicates that a given field refers to a collection 500 (possibly hierarchical) of values of one or more types and is mediated by a relative reference between the collection field of record 425 and the root node 505 of an embedded collection 500 containing the referenced items. In the prefened embodiment, this embedded collection 500 is in all ways identical to the outer containing collection 400, but may only be navigated to via the field that references it. It is thus logically isolated from the outermost collection 400. Thus the field declaration "Person @@employees" in record 425 implies a reference to a collection 500 of Person elements. Obviously collections can be nested within each other to an arbitrary level via this approach and this gives incredible expressive power while still maintaining the flat memory model. Thus for example one might reference a 'car', which internally might reference all the main components (engine, electrical system, wheels) that make up the car, which may in turn be built up from collections of smaller components (engine parts, electrical components, etc).

The persistent reference '#', illustrated as a field in record 525, is a singular reference from a field of an ET_Simplex record to an ET_Complex node contaimng a value of the same or a different type. The reference node can be in an embedded collection 500 or more commonly in an outer collection 400. In this case the 'employer' field of each employee of a given organization (^employer) would be a persistent reference to the employing organization as shown in the diagram. Additional details of handling and resolving collection and persistent references is provided in Appendix 2.

In order to make efficient use of any space freed up by deleted nodes, the collections mechanism can also maintain a garbage list, headed by a field in the collection variant of the base ET TextDB record. Whenever any record is deleted, it could added, into a linked list headed by this field and whenever a new record is allocated the code would first examine the garbage list to find any unused space that most closely fits the needs of the record being added. This would ensure that the collection did not become overly large or fragmented, and to the extent that the ET_Complex nodes and many of the ET_S implex nodes have fixed sizes, this reclamation of space is almost perfect. . ,

Another key feature of this invention is the concept of 'dirty' and 'null' flags, and various API calls are provided for this purpose (as described below). The need for 'null', flags is driven by the fact that in real world situations there is a difference between a field having an undefined or NULL value and that field having the value zero. In database situations, an undefined value is distinguished from a zero value because semantically they are very different, and zero may be a valid defined value. Similarly, the present invention ^• may use null and dirty flags to distinguish such situations. Referring now to Figure 6, a diagrammatic representation of the null and dirty flags of the present invention are shown. In this figure, the null and dirty flags are implemented by associating child simplex record.610 with any given simplex for which empty/dirty tracking is required as depicted below. Each flags array is simply a bit-field containing as many, bits as there are fields in the associated ^' type and whose dimensions are given by the value of TM_GetTypeMaxFlagIndex() (see Types Patent). If a field 610 has a null value, the conesponding bit in the 'nullFlags' record 611 is set to one, otherwise it is zero. Similarly, if a field 610 is 'dirty', the coπesponding bit in the 'dirtyFlags' record 612 is set to one, otherwise it is zero.^' The requirement for the 'dirty' flag is driven by the need to track, what has changed within a given record since it was first instantiated; This comes up particularly when the record is being edited by an associated UT. By examining the dirty flags after such an editing, session it is possible to "determine exactly which fields need to be updated to external storage such as an associated relational database. ' .^■ . ^{' '}. . • In certain situations, especially those encountered when implementing high performance servers for data held in the collection model, it is necessary to add additional binary descπptive and reference fields to the the collection to facilitate efficient navigation (e.g., in an inverted file implementation). The present invention supports this functionality by allowing the ET_Complex record to be extended by an arbitrary number of bytes, hereinafter termed 'extra bytes', within which information and references can be contained that are known only to the server (and which are not shared with clients/subscribers). This is especially useful for security tags and similar information that would preferably be maintained in ajnanner that is not accessible from the clients of a given collection. This capability would generally need to be customized for any particular server-based implementation.

' Another requirement for effective sharing of information across the network is to ensure that all clients to a given collection have a complete knowledge of any types that may be utilized within the collection. Normally subscπbers would share a common types hierarchy mediated via the types system (such as that descπbed in the Types Patent. Such a types system, however, could also include the ability to define temporary and proxy types. In the case of a shared collection, this could lead to problems in client machines that are unaware of the temporary type. For this reason, the collections API (as described below) provides calls that automatically embed any such type definitions m their source (C-like) form within the collection. The specialized types contained within a collection could then be referenced from a field of the ETJTextDB header record and simply held in a C format text string containing the set of type definition sources. Whenever code subscπbes to a collection, the API automatically examines this field and instantiates/defines all types found in the local context (see TM_DefineNewType descπbed below). Similarly when new types are-added to the collection, the updates to this type definition are propagated (as for all other changes except extra-bytes withm the collection) and thus the clients of a given collection are kept up to date with the necessary type information' for its interpretation.

When shaπng and manipulating large amounts of data, it is also often necessary to associate arbitrary textual and typed binary tags with the data held within a collection. Examples of this might be tags associated with UI appearance, user annotations on the data, etc. This invention fully supports this capability via the "element Tag" API calls provided to access them. In the prefened embodiment, the element tags associated with a given node in the collection are referenced via the 'tags' field of the ET_Complex record which contains a relative reference to a variable sized ET_Stπng record containing the text for the tags. In a manner identical to that used in annotations and scripts (descnbed below), tags could consist of named blocks of arbitrary text delimited by the "<on>" and "<no>" delimiter sequences occuπing at the start of a line. The "<on>" delimiter is followed by a string on the same line which gives the name of the tag involved. By convention, all tag names start with the '$' character in order to distinguish them from field names which do not. Some of the API calls below support access to tags as well as fields via dual use of the 'fieldName' parameter. For example, it is possible to sort the elements of a collection based on the associated tags rather than the data within. This can be very useful in some applications involving the manipulation and grouping of information via attributes that are not held directly within the data itself. In an implementation in which the tags are associated with the ET_Complex record, not the ET_Simplex, the collections can be created and can contain and display information without the need to actually define typed values. This is useful in many situations because tags are not held directly in a binary encoding. While this technique has the same undesirable performance penalties of other text-based data tagging techniques such as XML, it also provides all the abilities of XML tagging over and above the binary types mechanism ^v described previously, and indeed the use of standardized delimiters is similar to that found in XML and other text markup languages. In such an implemtation, when accessing tag information, the string referenced by the 'tags' field is searched for the named tag and the text between the start and end delimiters stripped out to form the actual value of the tag. By use of a standardized mechanism for converting binary typed values to/from the conesponding text string, tags themselves may be strongly typed (as further illustrated by the API calls below) and this capability could be used extensively for specialized typed tags associated with the data. Tags may also be associated either with the node itself, or with individual fields of the data record the node contains. This is also handled transparently via the API by concatenating the field path with the tag name to create unique field-specific tags where necessary. As will be understood by those skilled in the art, the ability to associate arbitrary additional textual and typed tags with any field of a given data value withm the collection allows a wide range of powerful capabilities to be implemented on top of this model.

Appendix A provides a listing of a basic API suite that may be used in conjunction with the collection capabdity of this invention. This API is not intended to be exhaustive, but is indicative of the kinds of API calls that are necessary to manipulate information held in this model. The following is a brief description of the function and operation of each function listed, from which, given the descriptions above, one skilled in the art would be able to implement the system of this invention.

A function that may be included in the API, hereinafter refeπed to as . TCJ-etCoIIectionNameO, sets the name of a collection (as returned by TC_GetCollectioπName) to the string specified. A function that may also be included in the API, hereinafter refeπed to as TC_GetCollectionName(), that obtains the name of a collection. :' • . ^{' •} - '

A function that may also be included in the API, hereinafter refeπed to as TC_FindEOFhandIe0₅ that finds the offset of the final null record in a container based collection.^{' "} , - ^' . ^" •

A function that may also be included in the API, hereinafter refeπed to as TC_SetCollectionTagO and TC_GetColIectionTagO, that allow access to and modification of the eight 64-bit tag values associated with every collection. In the prefened embodiment, these tag values are not used internally and are available for custom purposes.

A function that may also be included in the API, hereinafter refeπed to as TC_SetCollectionFlagsO, TC_ClrCoUectionFIags()₅ and TC_GetCoUectioαFlags(), that would allow access to and modification of the flags associated with a collection. -

^~ A function that may also be included in the API, hereinafter refened to as - TC_StripRecognizersO, which strips the recognizers associated -with finding paths in a collection/ The only effect of this would be to slow down symbolic lookup but would also save a considerable amount of memory.

A function that may also be included in the API, hereinafter refeπed to as TC_StripCollectionO_> strips off any invalid memory references that may have been left over from the source context. - ^•

A function that may also be included in the API, hereinafter refeπed to as TC_OpeaContamεr0j opens the container associated with a collection (ifany). In the - prefened embodiment, once a collection container has been closed using TC_CloseContaιnerQ, the collection API functions on the collection itself would not be usable until the container has been re-opened The collection container is automatically created/opened during a call to TC_CreateCollectιonO so no initial TC_OpenContainerO call is required.

A function that may also be included in the API, hereinafter refeπed to as TC_CloseContainerOj closes the container associated with a collection (if any). In the prefened embodiment, once a collection container has been closed using TC_CloseContainerO, the collection API functions on the collection itself would not be usable until the container had been re-opened

A function that may also be included in the API, hereinafter refened to as TC_GetContainerSpecO_» may be used to obtain details of the container for a collection. In the prefened embodiment, if the collection is not container based, this function would return 0 If the container is file-based, the 'specStrmg' variable would be the full file path. If the container is server-based, 'serverSpec' would contain the server concerned and 'specString' would contain the unique string that identifies a given collection of those supported by a particular server.

A function that may also be included in the API, hereinafter refened to as TC_GetDataOffset0, may be used to obtain the offset (in bytes) to the data associated with a given node in a collection. For example, this offset may be used to read and write the data value after initial creation via TC_ReadData() and TC_WriteDataQ.

A function that may also be included in the API, hereinafter refeπed to as TC_GetRecqrdOffset0, may be used to obtain the record offset (scaled) to the record containing the data associated with a given node in a collection. This offset may be used in calculating the offset of other data within the collection that is referenced from within a field of the data itself (via a relative, persistent, or collection offset - @, #, or @@). For example if you have a persistent reference field (ETJPersistentRef) from collection element 'sourceElem' within which the 'elementRef field is non-zero, the element designation for the target element ('targetElem', i.e., a scaled offset from the start of the collection for the target collection node) can be computed as:

targetElem = perfP elementRef + TC_GetRecordOffset(aCollectιon,0,0,sourceElem,NO); , ^' The coπesponding data offset for the target element would then be:

targetDataOff= TC_GetDataOffset(aCollectιon,0,0₎targetElem);

Functions that may also be included in the API, hereinafter refeπed to as TC_RelRefToDataOffsetO, TC_DataOffsetToRelRefO, TC_RelRefToRecordOffsetO, TCJDataToRecordOffsetO, TC_RecordToDataOffset(), TC_ByteToScaledOffset(), and TC_ScaledToByteOffset(), could be used to convert between the "data offset" values used in this API (see TC_GetDataOffset, TC_ReadData, TC _WriteData, and TC_CreateData), and the ET_Offset values used internally to store relative references (i.e , '@' fields). In the prefened embodiment, the^' routine TC_RefToRecordOffset() would be used in cases where the reference is to an actual record rather than the data it contains (e.g., collection element references). Note that because values held in simplex records may grow, it may be the case that the "data offset" and the coπesponding "record offset" are actually in two very different simplex records. In on embodiment, the "record offset" always refers to the Tsase' record of the simplex, whereas the "data offset" will be in the 'moved' record of the simplex if applicable. For this reason, it is essential that these (or similar) functions are used when accessing collections rather than attempting more simplistic calculations based on knowledge - of the structures, as such calculations would almost certainly be eπoneous.

A function that may also be included in the API, hereinafter refened to as TC_RelRefToElementDesignatorO» which could be used to return the element designator for the referenced element, given a relative reference from one element in a collection to another.

A function that may also be included in the API, hereinafter refened to as C_PersRefToElemeαtDesignatorO, which could be used to return the element designator for the referenced element, given a persistent or collection reference (e.g., the elementRef field of either) from the value of one element in a collection to the node element of another

A function that may also be included in the API, hereinafter refened to as TCJElementDesignatorToPersRefO, which, if given an element designator, could return the relative reference for a persistent or collection reference (e.g., the elementRef field of either) from the value of one element in a collection to the node element of another. ^" A function that may also be included in the API, hereinafter refeπed to as TC_ValueToElementDesignatorO, given the absolute ET_Offset to a value record • (ET_Simplex) within a collection, could be used to return the element designator for the _ corresponding collection node (element designator). This might be needed, for example, with ^' the result of a call to TC JjetFieldPersistentElementf).

A function that may also be included in the API, hereinafter refeπed to as TC__LocalizeRelRefs()j can be called to acheive the following effect for an element just added to the collection: It is often, convenient for relative references (i.e., @fieldName) to be held as pointer values until the time the record is actually added to the collection. At this time the pointer values held in any relative reference fields would preferably be converted to the appropriate relative reference and the original (heap allocated) pointers disposed.

A function that may also be included in the API; hereinafter refeπed to as TCJReadDataO, can be used to read the value of a collection node (if any) into a memory , buffer. 'In the prefened embodiment, this routine would primarily be used within a sort function as part of a-'kFindCPX' (TC_Find) or kSortCPX (TC_Sort) call. The purpose for supplying this call is to allow sort functions to optimize their container access or possibly cache results (using the custom field in the sort record). The collection handle can be • obtained from "elementRef.theView" for one of the comparison records, the 'size' parameter is the 'size' field of the record (or less) and the 'offset' parameter is the "u.simplexOff ' field. In such a case, the caller would be responsible for ensuring that the 'aBuffer' buffer is large enough to hold 'size' bytes of data. -

A function that may also be included in the API, hereinafter refeπed to as TO WriteDataO, which could be used to write a^" new value into^'an existing node within a collection handle. . ^' - - •

A function that may also be included in the API, hereinafter refeπed to as TC_WriteFϊeIdDataO, which could be used to write a new value into a field of an existing node within a collection handle.

' . r' .

A function that may also be included in the API, hereinafter refened to as TC_CreateDataQ, could be used to create and write a new unattached data value into a collection. The preferred way of adding data to a collection is to use TC_SetValueO. In the' case where data within a collection makes a relative reference (i.e, via a '@' field) to other data within the collection, however, the other data may be created using this (pr a similar) function. - - , - ^' ' ,- ', ' •

. - A function that may also be included in the API, hereinafter refeπed to as TC_CreateRootNode(), could be used to create and write a new unattached root node into a collection handle. In, the case where data within a collection makes a collection reference (i.e, via a '@@' field) to other data that is to be internalized into the same collection handle, it • is preferable to create an entirely seperate root node that is not directly part of the parent collection yet lies within the same handle. _, ^{• •} . ^■- -

^' A function that may also be included in the API, hereinafter refened to as ^'

TC_CreateRecord()j could be used to create specified structures within a collection, -' including all necessary structures to handle container based objects and persistent storage. In the prefened embodiment, the primary purpose for using this routine would be to create additional structures within the collection (usually of kSimplexRecord type) that can be referenced from the fields of other collection elements. Preferably, this type of function would only be used to create the following stmcture types: kSimplexRecord, kStringRecord, kComplexRecord. ^' -, " ^{: '} • . ^{" •} > , ^' ' .

A function that may also be included in the API, hereinafter refened to as TC_CreateCoUectionO, could be used to create (initialize) a collection, i.e. a container object — such as an anay, or a tree, or a queue or stack, or a set — to hold objects of any type which may appear in the Type Manager database. For example, if the collection object is an aπay, then a size, or a list of sizes, would preferably be supplied. If the collection is of ' unspecified size, no sizing parameter'need be specified. Possible collection types and the additional parameters that would preferably be supplied to create them are as follows: '^• • ^!

. ^" kFro List - List Structure ; kFromStack - Stack structure ^' . kFromQueue - Queue structure , - , kFromSet - Set . ^> - ^' . , ^' ^' kFromBTree - Binary tree ^■ kFro NTree - Generalized Tree ' no additional parameters kFromAπay - one dimensional aπay structure _, ' . • • ^'' dimensionl (int32) - aπay dimension (as in C) kFromAπayN - N dimensional aπay structure N (int32) - number of dimensions dimensionl (int32) - anay dimension I (as in C)

dimensionN (int32) - array dimension N (as in C)

A function that may also be included in the API, hereinafter refeπed to as TC_KillReferencedMemoryO, which could be provided m order to clean up all memory - associated with the set of data records within a collection. This does not include any memory associated with the storage of the records themselves, but simply any memory that the fields within the records reference either via pointers or handles. Because a collection may contain nested collections to any level, this routine would preferably recursively walk the entire collection hierarchy, regardless of topology, looking for simplex records and for each such record found, would preferably de-allocate any referenced memory. It is assumed that all memory referenced via a pointer or a handle from any field within any structure represents a heap allocation that can be disposed by making the appropriate memory manager call. It is still necessary to call TCJDisposeCollectionO after making this call in order to clean up memory associated with the collection itself and the records it contains.

A function that may also be included in the APL hereinafter refened to as TCJDisposeCollectionO, which could be provided in order to delete a collection. If the collection is container based, then this call will dispose of the collection in memory but has no effect on the contents of the collection in the container. The contents of containers can only be destroyed by deleting the container itself (e.g., if the container is a file then the file would preferably be deleted).

A function that may also be included in the API, hereinafter refened to as TC_PurgeColIectionO, which could be provided ra order to compact a collection by elunmatmg all unused records. After a long sequence of adds and deletes from a collection, a 'garbage' list of records may build up containing records that are not cunently used but which are available for recycling, these records are eliminated by this call. Following a purge, all references to intemal elements of the collection may be invalidated since the coπesponding record could have moved. It is essential that you re-compute all such internal references after a purge.

I • ■ A function that may also be included in the API, hereinafter refeπed to as TC_CloneRecordO, which could be provided in order to clone an existing record from one node of a collection to another node, possibly in a different collection. Various options allow -the cloning of other records referenced by the record being cloned. Resolved persistent and ^' collection references within the record are not cloned and would preferably be re-resolved in the target. If me structure contains memory references and you do not specify "kCloneMemRefs¹, then memory references (pointers and handles found in the source are NULL in the target), otherwise the memory itself is cloned before inserting the coπesponding reference in the target node. If the TcCloneRelRefs' option is set, relative references,. such as those to strings are cloned (the cloned references are to new copies iii the target collection),^' otherwise the coπesponding field is set to zero. _, . ^{• ■}

A function that may also be included in the API, hereinafter refened to as • ..• ^" - ' . TC_CIoneCollectionfj, which could be provided in order to clone all memory associated with a type manager collection, including all memory referenced from fields within the collection (if 'recursive' is true). ^" • . . . .

' : A function that may also be included in the API, hereinafter refeπed to as - TC_AppendCoHection(), which could be provided in order to append a copy of one , collection in its entirety to the designated node of another collection. In this manner multiple existing collections could be merged into a single, larger collection. In the prefened embodiment, when merging the collections, the root node of the collection being appended and all nodes below it, are transfened to the target collection with the transfeπed root node becoming the first child node of non-leaf 'tgtNode' in the target collection.

. ^'. ^■ A function that may also be included in the API, hereinafter refeπed to as TCJPossessDisPossessCollectionO, which could be provided in order to can be used to possess/dispossess all memory associated with a type manager collection, including all memory referenced from fields within the collection. ■ - • • '

A function that may also be included in the API, hereinafter refeπed to as TC_LowestCommooAncestorO,, which could be provided in order to search the collection from the parental point designated and determine the lowest common ancestral type ID for all elements' within. ^{' l} . • . . ^' . ^': ^{' •} ■A function that may also be included in the API, hereinafter refened to as TCJFIndFirstDescεndantO, which could be provided in order to search the collection from the parental point designated and find the first valued node whose type is equal to or descendant from the specified type. . :^' .

A function that may also be included in the API, hereinafter refened to as TCJ-sValidOperationO, which could be provided in order to determine if a given operation is valid for the specified collection. ^' '.

A function that may also be included in the API; hereinafter refened to as TC_vComplexOperation0_> which is identical to TC_ComplexOperation() but could instead take a variable argument list parameter which would preferably be set up in the caller as in the following example: ■ ', ^' . . -^' _, • • .'.

va_list ^' ap; - .. ' "■

Boolean , res; '• . . , " . - • ^''"

va_start (ap, aParameterName); . . - >. ' ^" res = TC_yComplexOperation(aCollection,theParentRef,anOperation,options,&ap); va_end(ap); ■ . - . ^'

A function that may also be included in the API, hereinafter refened to as TC__CompIexOperationQ, which could be provided in order to perform a specified operation on a collection. In- the prefened embodiment, the appropriate specific wrapper functions define the operations that are possible, the collection types for which they are supported, and the additional parameters that would preferably be specified to accomplish the operation. Because of the common approach used to implement the various data structures, it is possible to apply certain operations to collection types for which those operations would not normally be supported. These additional operations could be very useful in manipulating collections in ways that the basic collection type would make difficult.

, A function that may also be included in the API, hereinafter refened to as TCJPopO, which could be provided in order to pop a stack. When applied to a Queue, TC_PopQ would remove the last element added, when applied to a List or set, it would remove the last entry in ,^'. the list or set When applied to a tree, the tail child node (and any children) is removed. For a

137

SUBSTITUTE SH! RULE 2 stack, the pop action follows normal stack behavior. This function may also be refered to as^' , TC_RemoveRightO when applied to a binary tree. , ^• , ^' ' ^{• '}

A function that may also be included in the API, hereinafter refeπed to as CJPushO} which could be provided in order to push a stack. When applied to a List or Set, this function would add an element to the end of the list set When applied to a tree, a new tail child node would be added. For a stack, the push action follows normal stack behavior. This function may also be refered to as TCJEnQueuefJ when applied to a queue, or TC_AddRightO when applied to a binary tree.

A function that may also be included in the API, hereinafter refeπed to as TC_InsertO, could be provided in order to insert an element into a complex collection list. ,

A function that may also be included in the API, hereinafter refened to as , ^'. ^■

TCjSetExtraBytesO, could allow the value of the extra bytes associated with a collection element^'node record (if any) to be set. In the prefened embodiment; the use of this facility is strongly discouraged except in cases where optimization of collection size is paramount ^' Enlarged collection nodes can be allocated by passing a non-zero value for the 'extraBytes' ^{■ •} . parameter to TC_Insert(). This call would create additional empty space after the node record that can be used to store an un-typed fixed sized record which can be retreived and ^' updated using calls such as TC_GetExtraBytes() and TC_SetExtraBytes() respectively. This approach is primarily justified because the additional bytes do not incur the overhead of the ET_Simplex record that normally contains the value of a collection element's node and which is accessed by all other TC _ API calls. If data is associated with a node in this manner, a destructure function would preferably be associated with a node to be disposed when the • • . collection is killed, such as making a call to- a function such as TC_SetElementDestructor().

^' - A function that may also be included in the API, hereinafter refeπed to as TC_GetExtraBytes()₅ which could be provided in order to get the value of the extra bytes associated with a collection element node record (if any). See TC_SetExtraBytes() for ^' details. ^" . -^". . , ^' ' , , .. .

A function that may also be included in the- API, hereinafter refened to as- ^' TC_RemoveO_s could be provided in order to remove the value (if any) from a^'collection ^' node. - ^{' "} - ■ ^• "- , - . . . ^' ' A function that may also be included in the API, hereinafter refened to as TCJhidexRefO, could be provided in order to obtain a reference ΕT_Offset' to a specified indexed element (mdeces start from 0). This reference can be used for many other operations on collections. When used to access data in a multi-dimensional anay, it is essential that all array indeces are specified. However, each 'dimension' of a multi-dimensional anay can be seperately manipulated using a number of operations (e.g., sort) and thus a partial set of indeces may be used to obtain a reference to the elements of such a dimension (which do not normally contain data themselves, though they could do) in order to manipulate the elements of that dimension. In this manner, a multi-dimensional aπay can be regarded as a specialized case of a tree. When multiple indeces are used to refer to a tree, later indeces in the list refer to deeper elements of the tree. In such a case, a subset of the indeces should be specified in order to access a given parental node in the tree. Note that in the tree case, the dimensionality of each tree node may vary and thus using such an indexed reference would only make sense if a conesponding element exists.

A function that may also be included in the API, hereinafter refeπed to as TCJVIakeRoot()j could be provided in order to convert a collection element to the root of a new subordinate collection. This operation can be used to convert a leaf node of an existing collection into the root node of a new subordinate collection. This is the mechanism used to create collections within collections. Non-leaf nodes cannot be converted.

A function that may also be included in the API, hereinafter refeπed to as TC_Sort()» could be provided in order to sort the children of the specified parent node according to a sorting function specified in the 'cmpFun' parameter. Sorting may be applied to any collection type, including aπays. Note that the comparision function is passed two references to a record of type ΕT_ComplexSorf . Within these records is a reference to the original complex element, as well as any associated data and the type ID. The 'fromWhich' field of the record will be non-zero if the call relates to a non-leaf node (for example in a tree). The 'kRecursiveOperation' option applies for hierarchical collections.

A function that may also be included in the API, hereinafter refened to as TC_TJnSortO, which could be provided in order to un-sort the children of the specified parent node back into increasing memory order. For arrays, this is guaranteed to be the original element order, however, for other collection types where elements can be added and removed, it does not necessaπly coπespond since elements that have been removed may be > re-cycled later thus violating the memory order property. The 'kRecursiveOperation' option- applies for hierarchical collections. ' ,_ ^* . ' ^• • ,

. , A function that may also be included in the API, hereinafter refeπed to as • • TC_SortByField0₅ which could be provided in order to sort the children of the specified parent node usmg a built-in sorting function which sorts based on specified field path which would preferably refer to a field whose type is built-in (e.g., integers, strings, reals, struct etc.) or some descendant of one of these types. Sorting may be applied to any collection type, including aπays. The 'kRecursiveOperation' option applies for hierarchical collections. In the prefened embodiment, if more complex sorts are desired, TC_Sort() short should be used and and.'cmpFun' supplied. This function also could also be used to support sorting by element tags (field name starts with '$'). _ * , '- - , . ^• •

A function that may also be included in the API, hereinafter refeπed to as ^■

. TC_DeQueue(), could be provided in order to de-queue an element from the front of a queue.^" The operation is similar to popping a stack except that the element comes from the opposite end of the collection. In the prefened embodiment, when applied to any of the other " collection types, this operation would return the first element in the collection. This function may also be refered to as TC_RemoveLeft() when^' applied to a binary tree.

A function that may also be included in. the API, hereinafter refeπed to as TCJNextfJ, which could be provided in order to return a reference to the next element in a collection given a reference to an element of the collection. If there is no next elemenζ the function would return FALSE. , ^"

A function that may also be included in the API, hereinafter refeπed to as TC_Prev(), which could be provided in order to return a reference to the previous element in a collection given a reference to an element of the collection. If there is no previous element, the function returns FALSE.

. A function that may also be included in the API, hereinafter refeπed to as TC_ParentO_> which could be provided in order to return a reference to the parent element of - a collection given a reference to an element of the collection. In the prefened embodiment, the value passed in the 'theParentRef parameter is ignored and should thus be set to zero. A function that may also be included in the API, hereinafter refeπed to as TC_RootRef(), could be provided in order to return a reference to the root node of a collection. This (or a similar) call would only be needed if direct root node manipulation is desired which could be done by specifying the value returned by this function as the 'anElem' parameter to another call. Note that root records may themselves be directly part of a higher level collection. The check for this case can be performed by using TCJParentO which will return 0 if this is not true.

A function that may also be included in the API, hereinafter refeπed to as TC_RootOwner(), could be provided in order to return a reference to the simplex structure that references the collection containing the element given. In the prefened embodiment, if the element is part of the outermost colllection, it is by definition not owned arid this function returns false. If the root node is not owned/referenced by a simplex record, this function , returns false, otherwise true. If the collection containing 'anElem' contains directly nested collections, this routine will climb the tree of collections until it finds the owning structure (or fails).

A function that may also be included in the API, hereinafter refeπed to as TC_HeadO, could be provided in order to return a reference to the head element in a collection of a given parent reference. If there is no head element, the function would return FALSE. For a binary tree, TC_LeftChildO would preferably be used

A function that may also be included in the API, hereinafter refeπed to as TCJTailOj could be provided in order to return a reference to the tail element in a collection of a given parent reference. If there is no tail element, the function would return FALSE. For a binary -tree, TC_RightChildO would preferably be used. - -

A function that may also be included in the API, hereinafter refeπed to as TCJExchangefJ, could be provided in order to exchange two designated elements of a collection.

A mnction that may also be included in the API, hereinafter refened to as TC_Count(), could be provided in order to return the number of child elements for a given parent. In the prefened embodiment, for non-hierarchical collections, this call would return the number of entries in the collection. . -" A function that may also be included in the APL hereinafter refened to as ^• TC_SetVaIue(), could be provided in order to set the value of a designated collection element to the value and type ED specified. , ' ' - .

• A function that may also be included in the API, hereinafter refeπed to as TC_SetFϊeldValue(), which could be provided in order to set the value of a field within the specified collection element.

A function that may also be included in the API, hereinafter refeπed to as - •

TC_GetAnonRefFieldPtrO, which could be provided in order to obtain a heap pointer coπesponding to a reference field (either pointer, handle, or relative). The field value would preferably already have been retreived into an ETJDataRef buffer. In the case of a pointer or handle reference, this function is trivial, in the 'ease of a relative reference, the function would perform the following: ^'

doff= TC_RefToDataOffset(aDat^'aRef-> ^• -

^' relativeRef,TC_GetDataOffset(aCollection,0,0,anElem)); _J ■ • ' _,

TC_ReadData(aColIection,0,doff,0,&cp,0); ^{' ' ' '} _, . - _^"' . '

'_, return cp; . ■ ' ,

A function that may also be included in the API, hereinafter refeπed to as TC_GetCStringRefFieldPtrO, which could be provided in order to obtain the C string coπesponding^' to a reference field (either pointer, handle, or relative). The field value would preferably already have been retreived into an ETJDataRef buffer. In the case of a pointer or handle reference, this function is trivial, in the case of a relative reference, the function would perform the following: ^{" '} , ^' . ' .- ^' . .

• doff=TC_RefToDataOffset(aDataRef-> relativeRef,TC_.GetDataOffset(aCollection,0,0,anElem)); TC_ReadData(aCollection,0,doff,0,&cp₁0); . ^" . ^{■ "} . return cp; >

A function that^' may also be included in the API, hereinafter refeπed to as ^' ' TCjSetCStringFieldValueO, which could be provided in order to set the C string field of a field within the specified collection element- Ideally, this function' would also transparently handle all logic for the various allowable C-string fields as follows: ' ^" 1) if the field is a charHdl then: - ^' - ^" - ^' • '^' .. a) if the field already contains a value, update/grow the existing handle to hold the new- value .- ', ; - . '

. b) otherwise allocate a handle and assign it to the field

2) if the field is a charPtr then: . • ; - , ^' a) if the field already contains a value:

^■ . . - ' i) if the previous string is equal to or longer than the new one, copy new string into existing pointer ^' . . ii) otherwise dispose of previous pointer, allocate a new one and assign it b) otherwise allocate a pointer and assign it to the field

3) if the field is a relative reference then: a) if the reference already exists, update its contents to hold the new string . ' b) otherwise create a new copy of the string in the collection and reference the field to it - ' . ,^'

4) if the field is an anay of char then: . . , a) if the new value does not fit, report^' anay bounds enor ^' ' b) otherwise copy the value into the aπay ^"

' A function that may also be included in the API, hereinafter refeπed to as TC_AssignToFieid0₅ could be provided in order to assign an arbitrary field within a collection element to a value expressed as a C string. If the target field is a C string of some type, this function behaves similarly to TC_SetCStringFieIdValue() except that if the ^•kAppendStringValue' option is set, the new string is appended to the existing field contents. In all other cases, the field value would preferably be expressed in a format compatible with TM_StringToBinaryO,for the field type concerned and is assigned. ^" .^"

A function that may also be included in the API, hereinafter refeπed to as ^/ϊ^,C_GetValueO, which could be provided in order to get the value and type ID of a designated collection element . ' , '• ' .

A function that may also be included in the API, hereinafter refeπed to as TC_GetTypeID0, could be provided in order to return the type ID of a designated collection element This function is only a convenience over TC_GetValue() in that the type is returned as a function return value (0 is returned if an eπor occurs) . A function that may also be included m the API, hereinafter refeπed to as TCJHasValueO, could be provided in order to determine if a given node in a collection has a value or not. Again, the function would return either a positive or negative indicator in response to such a request.

A function that may also be included in the API, hereinafter refened to as TC_RemoveValueO, could be provided in order to remove the value (if any) from a collection node

A function that may also be included in the API, hereinafter refeπed to as TC_GetFieIdValueO, could be provided in order to get the value of a field within the specified collection element.

A function that may also be included in the API, hereinafter refeπed to as TCJGetCStringFieldValueO, could be provided in order to get a C string field from a " collection element into an existing buffer. In the prefened embodiment, if the field type is not appropriate for a C string, this function returns FALSE and the output buffer is empty. Preferably, if the field specified is actually some kind of reference to a C string, this function will automatically resolve the reference and return the reesolved string. In the case of a persistent (#) reference, this function would preferably return the name field or the contents of the string handle field if non-NULL. In the case of a collection (@@) reference, this function will preferably return the contents of the string handle field if non-NULL.

A function that may also be included in the API, hereinafter refened to as TC_GetFieldPersistentElementO, could be provided in order to obtain the element designator conesponding to a persistent reference field. In the prefened embodiment of this function, if the field value has not yet been obtained, this function will invoke a script which causes the referenced value to be fetched from storage and inserted into the collection at the end of a list whose parent is named by the referenced type and is immediately below the root of the collection (treated as a set). Thus, if the referenced type is "Person", then the value will be inserted below "Person" in the collection.

A function that may also be included in the APL hereinafter refeπed to as - TC_GetFieldColIectionO, could be provided in order to obtain the collection offset conesponding to the root node of a collection reference. In the prefened embodiment, if the field collection value has not yet been obtained, this function will invoke a script for the field which causes the referenced values to be fetched from storage an inserted into the referencing collection as a seperate and distinct collection within the same collection handle. The collection and element reference of the root node of this collection .is returned via the 'collectionRef parameter. ^• ' ^{" '} .

A function that may also be included in the API, hereinafter refeπed to as ^' TC_GetPersistentFieldDomain(), could be provided in order to obtain the collection offset coπesponding to the root node of a domain collection for a persistent reference field. If the field domain collection value has not yet been obtained, this function will invoke a script, ^" such as the "SGetPersistentCollection" script, for the field which causes the referenced values to be fetched from storage and inserted into the referencing collection as a seperate and distinct collection within the same collection handle. The collection and element reference of , the root node of this domain collection is returned via the 'collectionRef parameter.

■ A function that may also be included in the API, hereinafter refened to as TCJSetFieldDirtyQ, could be provided in order to mark the designated field of the collection element as either 'dirty¹ (i.e., changed) or clean. By default, all fields start out as being , 'clean'. In the prefened ambodiment, this function has no effect if a previous call to TC Jj-itDirtyFlagsO has not been made in order to enable tracking of clean dirty for the collection element concerned. Preferably, once a call to TC JnitDirtyFlagsO has been made, subsequent calls to set the field value (e.g., TC_SetFieldValue) will automatically update the 'dirty' bit so that it in not necessary to call TC_SetFieldDirtyQ explicitly.

A function that may also be included in the API, hereinafter refeπed to as TCJCsFieldDirtyO_* which could be provided in order to return the dirty/clean status of the specified field of a collection element. If dirty/clean tracking of the element has not been enabled using TCJtnitDirtyFlagsO, this function returns FALSE.

A function that may also be included in the API, hereinafter refened to as TC_InitDirryFlags _> which could be provided in order to set up a designated collection element to track dirty/clean status of the fields within the element: By default, dirty/clean . tracking of collection elements is turned off and a call to TC JsFieldDirtyO will return FALSE. ^' ' _ ^•■ . ^' - ^{• •}

A function that may also^' be included in the API, hereinafter refeπed to as TCJSetFieldEmptyO, which could be provided in order to mark the designated field of the collection element as either 'empty' (i.e., value undefined) or non-empty (i.e., value defined). By default all fields start out as being non-empty. In the prefened embodiment, this function has no effect if a previous call to TCJhiitErnptyFlagsQ has not been made m order to enable tracking of defined/undefined values for the collection element concerned. Once a call to TC JnitEmptyFlagsO has been made, subsequent calls to set the field value (e.g., TC_SetFieldValue) will automatically update the 'empty' bit so that it in not necessary to call TC_SetFieldEmptyO explicitly

A function that may also be included in the API, hereinafter refened to as TCJEstablishEmptyDirtyStateO, which could be provided in order to calculate a valid initial empty/dirty settings for the fields of an element. In the prefened ambodiment, the calculation would be performed based on a comparison of the binary value of each field with 0. If the field's binary value is 0, then it is assummed the field is empty and not dirty. Otherwise, the field is assummed to be not empty and dirty. If the element already has empty/dirty tracking set up, this function simply returns without modifying anything.

A function that may also be included in the API, hereinafter refened to as TCJsFieldEmptyO, which could be provided in order to return the empty/full status of the specified field of a collection element. If empty/full tracking of the element has not been enabled using TCJ tEmptyFlagsO, this function will return FALSE.

A function that may also be included in the API, hereinafter refeπed to as TCjSetEIementTagO, could be provided in order to add, remove, or replace the existing tag associated with a field of a 'valued' element within a collection, or alternatively if 'aFieldName' is NULL, the tag is associated with the element itself. Unlike annotations and scripts (see the TypeScripts package) that are applied to the definitions of the type or field, tags are associated with node a collection, normally (but not necessarily) a valued node. Tags consist of arbitrary strings, much like annotations. There may be any number of different tags associated with a given record field. In the prefened ambodiment, if the collection concerned is file or server-based, tags will persist from one run to the next and thus form a convenient method of arbitrarily annotating data stored in a collection without formally changing its stmcture. Tags may also be used extensively to store temporary data/state information associated with collections. A function that may also be included in the API, hereinafter refeπed to as TC_GetEIementTagOj which could be provided in order to obtain the tag text associated with a given field within a 'valued' collection element. If the tag name cannot be matched, NULL is returned.

A function that may also be included in the API, hereinafter refeπed to as TCjSetElemerttNumericTagO, which could be provided ra order to add, remove, or replace the existing numeric tag associated with a field of a 'valued' element withm a collection, or alternatively if 'aFieldName' is NULL, the tag is associated with the element itself (which may have no value). This would provide a shorthand method for accessing numeric tags and uses TC_SetElementTag(). The 'tagFormat' value would preferably be one of the following predefined tag formats: TcTaglsInteger'.'kTaglsIntegerList'.'kTaglsRea , or TcTaglsRealLisf In the case of integer tags, the elipses parameter(s) should be a series 'valueCounf 64-bit integers. In the case of real tags, the elipses parameters) should be a series of 'valueCounf doubles.

A function that may also be included in the API, hereinafter refened to as TC_SetElementTypedTagO, which could be provided in order to add, remove, or replace the existing typed tag associated with^' a field of a 'valued' element within a collection, or alternatively if 'aFieldName' is NULL, the tag is associated with the element itself (which may have no value). This function provides a shorthand method for accessing typed tags and uses TC_SetElementTagQ. The tag format is set to TcTaglsTyped'. Preferably, the tag string itself consists of a line containing the type name followed by the type value expressed as a string using TM_BinaryToString (..., kUnsignedAsHex + kCharAπayAsString).

A function that may'also be included in the API, hereinafter refeπed to as TC_GetElementNumericTagO, which could be provided in order to obtain the existing numeric tag associated with a field of a 'valued' element within a collection, or alternatively if 'aFieldName' is NULL, the tag is associated with the element itself (which may have no value). This provides a shorthand method for accessing numeric tags and uses TCjjetElemenfTagO. The 'tagFormat' value would preferably be one of the following predefined tag formats: ^,kTagIsInteger',^,kTagIslntegerList^,,'kTagIsRear, or TcTaglsRealLisf. In the case of integer tags, the elipses parameters) would be a series 'valueCount' 64-bit integer addresses. In the case of real tags, the elispes parameters) would be a series of 'valueCounf double addresses. . A function that may also be included in the APL hereinafter refeπed to as TC_GetEIementTypedTagO, which could be provided in order to obtain ^"the existing typed tag associated with a field of a 'valued' element within a collection, or alternatively if 'aFieldName' is NULL, the tag is associated with the element itself (which may have no value). This provides a shorthand method for accessing numeric tags and uses TCJ etElementTagO- ^■ -

A function that may also be included in the API, hereinafter refeπed to as TC_GetElementTagListO, which could be provided in order to obtain a string handle containing an alphabetized list (one per line) of all element tags appearing in or below a given node .within a collection. . . ^'

A function that may also be included in the API, hereinafter refened to as TC_GetAHElemeπtTagsO, which could be provided in order to obtain a character handle containing all element tags associated with a specified element [and field] of a collection. This function may be used to optimize a series of calls to TC_GetElemeπfTagO by passing 'aCollection' is NULL to TC_GetElementTag() and passing an additional 'charHdl' parameter that is the result of the TC_GetAllElementTagsO call. This can make a significant difference in cases where a series of different tags need to be examined in succession.

• A function that may also be included in the API, hereinafter refened to as TCJiπtEmptyFlagsOj which could be provided in order to set up a designated collection ^" element to track empty/full status of the fields within the element By default, empty/full tracking of collection elements is turned off and a call to TC_IsFieldEmptyO will return FALSE if the field value is non-zero, the function will return TRUE otherwise.

A function that may also be included in the API, hereinafter refened to as ^• TC_ShiftTaiI0, which could be provided in order to make the designated element the new tail element of the collection and preferably disgards all elements that were after the designated element.

A function that may also be included in the API, hereinafter refened to as TC_SαiftHeadO, which could be provided in order to make the designated element the new head element of the collection and preferably disgards all elements that were before the designated element. ^' - A function that may also be included in the API, hereinafter refened to as TC_RotTailOj which could be provided in order to make the designated element the new tail element of the collection by rotating the collection without disgarding any other elements. The rotation operation is usually applied to 'Ring' structures.

A function that may also be included in the API, hereinafter refeπed to as TCJRotHeadOj which could be provided in order to make the designated element the new head element of the collection by rotating the collection without disgarding any other elements.

A function that may also be included in the API, hereinafter refeπed to as TCjSetNameO, which could be provided in order to assign a name to any member element of a collection. In the prefened embodiment, the element may subsequently be accessed using its name (which would preferably be unique). Ixi essence, this is the basic operation of the TcFromSe collection, however, it can be applied and used for any of the other collection types. In the case of a tree element, the name specified would be the name of that node, however, to use the name to access the element using TC_SymbolicRef(), it is preferable to specify the entire 'path' from the root node where each ancestor is seperated from the next by ':'. Alternatively, the TcPathRelativeToParenf option can be used to allow the use of partial relative paths. Preferably, names would consist of alphanumeric characters or the 'J character only, and would be less than 31 characters long.

A function that may also be included in the API, hereinafter refened to as TC_GetName()₅ which could be provided in order to return the name (if any) of the specified element of a collection. Note that in the case of a tree, the name would refer just to the local node. Preferably, to access the element symbolically, the path which can be obtained using TC JjetPathO would be used. The 'aName' buffer should be at least 32 characters long.

A function that may also be included in the API, hereinafter refeπed to as TC_GetPathO, which could be provided in order to apply return the full symbolic path (if defined) from the root node to the specified element of a collection in a tree. Preferably, the 'aPath' buffer should be large enough to hold the entire path. The worst case can be calculated using TC_GetDepthO and multiplying by 32.

A function that may also be included in the API, hereinafter refeπed to as TC_SymbolϊcRefO, which could be provided in order to obtain a reference to a given < element of a collection given its name (see TC_SetName) or in the case of a tree, its full path. Sometimes for certain collections it is more convenient (and often faster) to refer to elements by name rather than any inherent order that they might have. This is the central concept behind the TcFromSef collection, however, it may also be applied to any other collection type. An element could also be found via its relative path from some other non-root node in the collection using this call simply by specifying the 'kPathRelativeToParenf which causes 'theParentRef , not the collection root, to be treated as the starting point for the relative path 'aName'. ^' - ._ . ' , ^" • , . •

. A function that may also be included in the API, hereinafter refeπed to .as TC_Frad0s which could be provided in order to scan the collection in order by calling the searching r function specified in the comparison function parameter. In the prefened embodiment, the comparision function is passed two references, the second is to a record of type ΕT_ComρlexSorf which is identical to that used during the TC_Sort() call. The first reference would be to a 'srchSpec' parameter. The 'srchSpec' parameter may be the address of any arbitrary structure necessary, to specify to the search function how it is to do its search. The 'fromWhich' field of the 'ET_ComplexSorf record will be non-zero if the call relates to a

" non-leaf node (for example in a tree). The "kRecursiveOperation' applies for hierarchical collections. The role of the search function is similar to that of the sort function used for TCJSortO calls, that is it returns a result that is above,below, or equal to zero based on comparing the information specified in the 'srchSpec' parameter with that in the

. 'ET_ComplexSorf parameter. By repeatedly calling this function, one can find all elements in the collection that match a specific condition. In the prefened embodiment, when the TcRecursiveOperation' option is set, the hits will be returned for the entire tree below the parent node specified according to the the search order used internally by this function. Alternatively, the relevant node could be specified as the parent (not the root node) in order to restrict the search to some portion of a tree. . ' •

A function that may also be included in the API, hereinafter refeπed to as TCJF dBylDO, which could be provided in order to use the TC_Find() to locate a record within the designated portion of a collection having data whose unique ID field matches the ^' value specified.. This function could form the basis of database-like behavior for collections. .

A function that may also be ncluded in the API, hereinafter refeπed to as TCJFindByTagO, which could be provided in order to make use of TCJVisitO to locate a . record within (i.e., excluding the parent node) the designated portion of.a collection whose tag matches the value specified. - • ^' , . . _, . _■ ■ ■

' A function that may also be included in the API, hereinafter refeπed to as TCJFindNextMatchingFlagsOj which could be provided in order to make use of TC_Visit() . to locate a record within (i.e., excluding the parent/root node) the designated portion of a collection whose flags values match the flag values specified. , ; ,

A function that may also be included in the API, hereinafter refened to as ^•

TCJFindByTypeAndFieldMatchO, which could be provided in order to make use of TC J ndQ to locate a record(s) within the designated portion of a collection having data whose type ID matches 'aTypelD' and for which the 'aFieldName' value matches that - referenced by 'matchValue'. This is an optimized and specialized form of the general

• capability provided by TC_SearchO- Preferably, in the case of string fields, a "strcmpO". comparision is used rather than the full binary equality comparison "memcmpO" utilized for all other field types. For any more complex search purpose it is preferable to use TC_Search() directly. Persistent reference fields may also be compared by ID if possible or

• name otherwise. For Pomter, Handle, and Relative reference fields, the comparison is performed on the referenced value, not on the field itself. This approach makes it very easy to compare any single field type for an arbitrary condition without having to resort to more sophisticated use of TC_Find(). In cases where more than one field of a type would preferably be examined to determine a match, particularly when the algorithm required may vary depending on the ontological type involved, the routine TC_FindByTypeAndRecordMatchO could be used.

- - - - -A function that may also be included in the API, hereinafter refeπed to as ' " ^"' " ^{r *} TCJFindMatchingEIementsO, which could^'be provided in order to make use of TC_FindO to locate a record(s) within the designated portion of a collection having data for which the various fields of the record can be used in a custom manner to determine if the two records refer to the same thing. This routine operates by invoking the script SElementMatch when it finds potentially matching records, this script can be registered with the ontology and the algorithms involved may thus vary from-one type to the next. This function may be used when trying to determine if two records relate to the same item, for example when comparing people one might take account of where they live, their age or, any other field that can be used to discriminate including photographs if available. In the prefened embodiment, the operation of the system is predicated on the application code registering comparison scripts that can be invoked via this function. The comparison scripts for other types would necessarily be different.

A function that may also be included in the API, hereinafter^" refeπed to as TC_GetUniquelDO, which could be provided in order to get the unique persistent LD value associated with the data of an element of a collection.

_, A function that may also be included in the API, hereinafter refeπed to as TC SetUniquelDO, which could be provided in order to set the unique persistent ID value associated with the data of an element of a collection. ' -

A function that may also be included in the- API, hereinafter refened to as TC_SetElementDestructor(), which could be provided in order to .set a destructor function ' to be called during collection tear-down for a given element in a collection. This function would preferably only be used if disposal of the element cannot be handled automatically via the type manager facilities. The destructor function is called before and built-in destructor actions, so if it disposes of memory associated with the element, it would preferably ensure that it alters the element value to reflect this fact so that the built-in destruction process does not duplicate its actions. ^' . . ^•

A function that may also be included in the API, hereinafter refened to as TC_GetEIementDestructorO, which could be provided in order to get an element's . destructor function (if any). • ^• ,

• A function that may also be included in the APL hereinafter refened to as TC_GetDepthO_> which could be provided in order to return the relative ancestry depth of two elements of a collection. That is if the specified element is an immediate child of the parent, its depth is 1, a grandchild (for trees) is 2 etc. If the element is not a child of the parent, zero is returned. . - • ^{■ •}

A function that may also be included in the API, hereinafter refeπed to as . r '

TC_Pruπe(), which could be provided in order to remove all children from a collection. Any handle storage associated with elements being removed would preferably be disposed. A function that may also be included in the API, hereinafter refeπed to as TC_AddPathO, which could be provided in order to add the specified path to a tree. In the prefened embodiment, a path is a series of':' seperated alphanumeric (plus 'J) names representing the nodes between the designated parent and the terminal node given. If the path ends in a '^■', the terminal node is a non-leaf node, otherwise it is assumed to be a leaf. For example the path "animaIs:mammals:dogs:fido" would create whatever tree structure was necessary to insert the non-leaf nodes for "animals", "mammals" and "dogs" below the designated parent, and then insert the leaf node "fido" into "dogs". Note that whde the parent is normally the root of the tree, another existing non-leaf node of the tree may be specified along with a path relative to that node for the add.

A function that may also be included in the API, hereinafter refened to as ,

TC_8hove(), which-could be provided in order to add a new element at the start of the collection. When applied to a tree, a new head child node is added. When applied to a binary tree, it is preferablt to use TC_AddLeft().

A function that may also be included in the API, hereinafter refened to as TCJFIipO, which could be provided in order to reverse the order of all children of the specified parent. The TcRecursiveOperation' option may also apply.

A function that may also be included in the API, hereinafter refened to as . .

TC_SetFlagsO_s which could be provided in order to set or clear one or more of the 16 custom flag values associated with each element of a collection. These flags are often useful for indicating logical conditions or states associated with the element

A function that may also be included in the API, hereinafter refened to as TC_GetFlagsO_? which could be provided in order to get one or more custom flag values associated with each element of a collection.

A function that may also be included in the API, hereinafter refened to as TC_SetReadOnly0, which could be provided in order to alter the read-only state of a given element of a collection. If an element is read-only, any subsequent attempt to alter its value will fail.

A function that may also be included in the API, hereinafter refeπed to as TCJLsReadOnlyO, which could be provided in order to determine if a given element of a

'153

SUBSTITUTE SHI (RULE 26 collection is marked as read-only or not. If an element is read-only, any attempt to alter its value will fail.

A function that may also be included in the API, hereinafter refeπed to as TC_SetTagO, which could be provided in order to set the tag value associated with a given element. The tag value (which is a long value) may also be used to store any arbitrary information, including a reference to other storage. In the prefened embodiment, if the tag value represented other storage, it is important to define a cleanup routine for the collection that will be called as the element is destroyed in order to clean up the storage.

A function that may also be included in the API, hereinafter refened to as TC_GetTagO, which could be provided in order to get the tag value associated with an element of a collection.

A function that may also be included in the API, hereinafter refeπed to as TCjSetShortCutOj which could be provided in order to set the shortcut value associated with a given element

A function that may also be included in the API, hereinafter refened to as TC_SetDescriptionO, which could be provided in order to set the description string associated with a given element. The description may also be used to store any arbitrary text information.

A function that may also be included in the API, hereinafter refeπed to as TC_GetDescriptionOj which could be provided in order to get the tag value associated with an element of a collection.

A function that may also be included in the APL hereinafter refeπed to as TC_CoIITypeO, which could be provided in order to obtain the collection type (e.g., kFromAπay etc.) for a collection

A function that may also be included in the API, hereinafter refeπed to as TCJVisitO, which could be provided in order to visit each element of a collection in turn. For non- hierarchical collections, this function would be a relatively simple operation. For trees, however, the sequence of nodes visited would need to be set using a variable, such as 'postOrder¹. In the prefened embodiment, if 'postOrder¹ is false, the tree is searched in pre- ! order sequence (visit the parent, then the children). If it is true, the search would be - conducted in post-order sequence (visit the children, .then the parent). At each stage in the 'walk', the previous value of 'anElem' could be used by the search to pick up where it left off. To start the 'walk', the variable 'anElem' could be set to zero. The 'walk' would terminate when this function returns FALSE and the value of anElem on output becomes zero^'. The advantage of using TC_Visit() for all collection scans, regardless of hierarchy, is that the " same loop will work with hierarchical or non-hierarchical collections. Loops involving . operations like TCJMextfJ do not in general exhibit this flexibility. If the TcRecursiveOperation' option is not set, the specified layer of any tree collection will be ^"" traversed as if it was not hierarchical. This algorithm is fundamental to almost all other collection manipulations, and because it is non-trivial, it is further detailed below: ^■

Boolean TC_Visit // Visit each element of a collection

ET_CollectionHdl aCollection, // IO:The collection' int32 options, // I: arious logical options

ET_Offset theParentRef, // I:Parent element reference

ET_Offset* ,' anElem, , // 10:Previous element (or 0) .next

Boolean postOrder // I:TRUE/FALSE = postOrder/preOrder

) // R:TRUE for success, else FALSE

{ off = *anElem; ^' . • prt » resolve parent reference objT = root node 'fromWhich' - ^' _, , if ( l off ) • ^■ • ■

{ ^' , ^{' "} -' ^' off a (prtP->ch.ildHdr) ? theParentRef + prtP->childHdr : 0; • if ( off ) - . •

^• { - : ' ^" . ^■ ; - ^{■ •} cpxP = resolve off reference ι , if ( post && (options & kRecursiveOperation) ) while ( off && cpxP->childHdr ) // now dive down to any children

( off =» off + cpxP- >childHdr,-- cpxP = resolve off reference

}

} ^'

} else ' • c xP = resolve off reference noskip = NO; , if ( post ) // pos -order traversal

{ //^•'at -the EOF so only if we ' e in if ( ! cpxP->nextEl^'e i && (options & kRecursiveOperation) )

155

SUBSTITUTE SHi . £ // a hierarchy may there be more ^■ - - ^' ; • if ( objτ =.= kFromBTree [ | objT -= kFromNTree | | objT == kFromArrayN ) .

, -^• .{ ^{' ■ ■ ' '} . .. ' . : ^'

• if ( cpχP->hd .parent ) .. -

^■- ^{•' ' ' ' •}{-, ^' . - ^■ ' ^' . " ' off =» off +• cpxP - >hdr . arent ; // climb up next parent

' cpxP =■ resolve off reference

'if ( (cpxP '." kComplexRecord || off == theParentRef ) ) . ^: • . o f = 0 ,- _ . •

, } else - - . •

_,._,- .•• off = 0; . ^' ! ' .^'. . ' noskip = YES;. ._, .. // parents examined after children ^' •

} else ' • '

^• , ^' ' off = 0,- ' ^•• ,

' 'if ( off && ! noskip ). '■ ^• - . . .

.. { . ^{• ' ' '} ' ^{' ■ ' '} ' ^{' '} off = ( cpxP->nexcElem ) ? off + cpxP->nextElem : 0,-.

-^' if (-off ) "

^" . - ^• .{ ' ^{• • •} , ^' : ^{• • •••} > ^{• ■ "} . . cpxP =» .resolve off reference ^•' ^• , ^{* ■} ^'' . ' if ( options & kRecursiveOperation ) ■'

'^' ' while { off &.& cpxP->childHdr ) // depth 1st dive to children • .^' . . ' - ... , ' off = off + cpxP->childHdr; ' cpxP = resolve of reference ' ^• ^{' ■ '}- ^• - } . ^•. . : : ^{■ ■} - ^{■ ■}: ^■

} " - ^{• " ' •} - ^{' ■} . . - ^■' .

• } else ' ' ''.. ' // pre-order traversal

( ^{' ' '} ' ' ^'

' if ( cpxP->childHdr && (options & kRecursiveOperation) ) •

{ ^{' '} '. ' off = off + cpxP->childHdr;

'^'cpxP = resolve off reference .

} else if ( cpxP->nextElem )

. ^• , . off = off + cpxP->nextElem; - .

' - cpxP a resolve off reference

. ^{' •} }' ^{' ' "}' .^{" '}

- • • ' else if ( options & kRecursiveOperation ) •

{ - . '. . ' ' .

^• if ( objT == kFromBTree | | objT == kFromNTree | | objT == kFromArrayN ) _, -^■ . ■ ^' for ( ; o f a& !cpxP->nextElem ; ) ^'

-. ( - " ... ',. -

'if ( cρxP->hdr. parent )

^' ' ^{' ' '} { ■■ ' ' . ^{• '}. ^•. ^■

• ^{" ' ' '} _ off =» off + cpxP ->hdr. aren ; cpxP = resolve off reference ] else ^{•" '} • ^{' ■}• '

^'' ' ^' . • - .. o f = 0 ,- • ^' if ( off && (record != kComplexRecord | | off ==- theParentRef) ■) off = 0;

A function that may also be included in the API, hereinafter referred to as TC RandomO, could be provided in order to randomize the order of all children of the specified^' parent. The "kRecursiveOperation' option applies. ^'- . '. ^• ■

A function that may also be included in the API, hereinafter referred to as TCJHasEmptyFIagsO_? could be provided in order to check to see if a designated collection element has tracking set up for empty/non-empty status of the fields within the element

A function that may also be included in the API, hereinafter referred to as TC_HasDirtyFlags0jCOuld be provided in order to check to see if a designated collection element has tracking set up for dirty/clean status of the fields within the element.

A function that may also be included in the API, hereinafter referred to as TC_GetSetDirtyFlagsQ, could be provided in order to getfset the dirty flags for a given record. This copy might also be used to initialize the flags for another record known to have a similar value. To prevent automatic re-computation of the flags when cloning is intended (since this computation is expensive), it is preferable to use the TcNoEstablishFlags' option when creating the' new record to which the flags will be copied. The buffer supplied in 'aF.lagsBuffer' would preferably be large enough to hold all the resulting flags. The size in bytes necessary can be computed as: -

bytes = (((TM_GetTypeMaxFlagJ-αdexQ - 1) | 0x07) + 1) » 3;

A function^, that may also be included in the API, hereinafter referred to as TC_GetSetEmptyFIagsO, could be provided in order to get/set the empty flags for a given record. For example, this copy might be used to initialize the flags for another record known to have a similar value. To prevent automatic re-computation of the flags in cases where ' ■ such cloning is intended (since this computation Is expensive), it is preferably to use the 'kNoEstablishFlags' option when creating the new record to which the flags will be copied. The buffer supplied in -'aFlagsBuffer¹ would preferably be large enough .to hold all the resulting flags. The size in bytes necessary can be computed as: ^'

^" bytes = (((TM_GetTypeMaxFlagInde^χO - 1) | 0x07) + 1) » 3; ^{' ■} .

A function that may also be included in the API, hereinafter referred to as TC_GetServerCoHectioπsOj could be provided in order to obtain a string handle containing an alphabetized series of lines, wherein each line gives the name of a 'named' collection associated with the server specified. These names could be used to open a server-based collection at the client that is tied to a particular named collection in the list (see, for example, TC_OpenContainer). ' . ^'

A function that may also be included in the API, hereinafter referred to as TC PublishO, could be provided in order to publish all collections (wake function). -

A function that may also be included in the API, hereinafter referred to as ' TC UnPublisuO* could be provided in order to un-publish a previously published collection at a specified server thus making it no-longer available for client access. In the preferred embodiment, un-publishing first causes all current subscribers to be un-subscribed. If this , process fails, the un-publish process itself is aborted. Once un-published, the collection is . removed from the server and any subsequent (erroneous) attempt to access it will fail.

A function that may also be included in the API, hereinafter referred to as TC_SιιbscribeO, could be provided in order to subscribe to a published collection at a specified^" server thus making accessible in the client. A similar effect could be achieved by . using TC_CreateCoIlectionO combined with the TcServerBasedCollection' option.

A function that may also be included in the API, hereinafter referred to as TC_UπsubscribeO} could be provided in order to un-subscribe from a published collection at a specified server. In the preferred embodiment, the collection itself does not go away in the . server, un-subscribing merely removes the connection with the client

A function that may also be included in the API, hereinafter referred to as TC _ContainsTypedefO, could be provided in order to determine if a typedef for type name given is embedded in the collection. Because collections may be shared, and may contain types that are not known in other machines sharing the collection, such as'proxy types that may have been created on the local machine, it is essential that the collection itself contain the necessary type definitions within it. In the preferred embodiment, this logic would be enforced automatically for any proxy type that is added into a collection. If a collection contains other dynamic types and may be shared, however, it is preferable to include the type definition in the collection. ^' . ^'

A function that may also be included in the API, hereinafter referred to as TC_AddTypedefO, could be provided in order to add/embed a typedef for type name in a collection: Because collections may be shared, and may contain types that are not known in other machines sharing the collection, such as proxy types that may have been created on the local machine, it is preferable for the collection itself to store the necessary type definitions ^'within it. In the prefened embodiment, this logic would be enforced automatically for any proxy type that is added into a collection. If a collection contains other dynamic types and may be shared, however, is is preferably to ensure that the type definition is included in the collection by calling this function. .^{■ "} - . ' -

A function that may also be included in the API, hereinafter referred to as ' .

TCJBuϋdTreeFromStringsO, could be provided in order to create a tree collection and a set of hierarchical non-valued named nodes from a series of strings formatted as for TC_AddPathO, one per line of input text. The root node itself may not be named. If a . collection is passed in, the new collection could attached to the specified node. Alternatively, ' an entirely new collection could be created and returned with the specified tree starting at the root. ^{' ■ '}

A function that may also be included in the API, hereinafter referred to as TC_RegisterServerCollectioπO,could be provided in order to register a collection by name within a server for subsequent non-local access via a server using server-based collections in the clients. • '

A function that may also be included in the API, hereinafter referred to as TC_DeRegisterServerCoUection(),could be provided in order to deregister a collection byname to prevent subsequent accesses via TC_ResolveServerCollectionO- ' . ' . One feature that is important in any complete data model is the ability to associate and execute arbitrary code or interpreted script routines whenever certain logical actions are

"^' performed on the data of one of its fields. In the system of this invention, this capability is provided by the 'scripts' API (prefix TS_) a portion of which could be implemented as set forth below: ^' . ' . • , '

Boolean TS_SetTypeAnnotation( // Modify annotation for a given type - •

. ETJTypeDBHdl ' aTypeDBHdl, // I:Type DB handle (NULL to default)

₎ ^' ETJTypelD aTypelD, // I:Type ID charPtr name. // I.-Annotation name •^■ ,

"SanAnnotation" ,

■ charPtr " annotation -. // I:Annotation, NULL to remove - ^' ^{' )}; ^: // R:TRUE for success,

FALSE otherwise - ^• ' -

Boolean TS_SetFieldAnnotation( // Set field annotation text _'

ETJTypeDBHdl ' aTypeDBHdl, // I:Type.DB handle (NULL to default)

' ETJTypelD aTypelD , // IsType ID charPtr aFieldName, // -.Name of the field/field path . ' - charPtr ^{• •} name , // I:Annotation name as -^'in

"<on> $name" • charPtr anAnnota io , // IsText of annotation,/ ^*

NULL to remove

... // I: 'fieldName' could be sprintf () - ^'

; ); _t // R:TRUE for success,

FALSE otherwise *

•^' charHdl TS_GetTypeAnπotation ( // Obtain annotation for a given type • -

.ETJTypeDBHdl ^' aTypeDBHdl, // I:Type DB handle (NULL to default) .

ETJTypelD ^• aTypelD, _, ^•' // I:Type ID

'" " charPtr . - . .- " name , ' //^• I:Annotation -name as' in

"<on>. $name" - int32 options. // IrVarious logical options (see notes)

ETJViewRef ^♦collectioriNode,/./ I: If non-NULL, collection node > ■

ETJTypelD *fromWho // 10:holds registering

-type.ID

^• . ); '"// RtAnnotation text, NULL if none *

, charHdl TS GetFieldAnnotatiαn( // Get annotation fo a ^• field

ETJTypeDBHdl aTypeDBHdl , II I:Type DB handle (NULL to default) -• ' .

. ETJTypelD aTypelD, // I:Type ID charPtr aFieldName , // I: ame of the ■ field/field oath , ιnt32 options, // I:Various logical options (see notes) '

ETJViewRef , *collectionNode, // I: If_. non-NULL, collection node

ETJTypelD ^♦fromWho, ^' // I0:holds registering type ID charPtr - name, // IrAimotation name as in " <on> Sname" , // I : ' fieldName' may be sprintf 0. - // R.-Annotation text, NULL if none ftdefine' NoInheritance OxOlOOOQOQ // options - I inherit from 'ances . types ftdefine NoReflnherit '0x02000000 // options - ! inherit for ref. fields ftdefine kNoNodeInherit oxoaoooooo // options - ! inherit f ances . nodes charHdl TS_GetFieldScript // Get script for action 4 field

ETJTypeDBHdl aTypeDBHdl, // I rType DB handle (NULL to default)

ETJTypelD aTypelD, // -I:Type ID charPtr aFieldName , // I:Name of the field/field path charPtr anAction, // I:Action -name as in "<on> anAction" int3 options. // I: arious logical options (see notes) ^*

ET_ViewRef *collectionNode,// I: If non-NULL, ■collection node

ETJTypelD ^♦fromWho, // 10: registering type ID

Boolean *isLocal, // 10:TRUE if local script, else false

// I :' aFieldName ' may be sprint ()

); // R:Action script, ULL if

ftdefine kGlobalDefnOnly 0x04000000 // options - only obtain global def . _,

Boolean TS_SetTypeScriρt ( // Set script for action & type

ETJTypeDBHdl aTypeDBHdl , //^' I:Type DB handle (NULL to default)

ETJTypelD aTypelD, // I rType ID charPtr anAction, // I:Action name as in "<on> anAction" charPtr ^• aScript, // IrType script/proc, NULL to remove

. int32 options // _ I : arious logical options (see notes)

); // RrTRUE for success,

FALSE otherwise ftdefine kLocalDefnOnly 0x00000001 // options - local script override ftdefine kProcNotScript , _• 0x00000002 // options - 'aScript' is a fn. address

Boolean TS_SetFieldScript (_^ // Set field action script

ETJTypeDBHdl aTypeDBHdl , // I:Type. DB handle (NULL to default) ^{• "}

ETJTypelD aTypelD, . // I rType" ID , .

' - charPtr '. aFieldName, // I:Name of the field/ ield path ^{• '} .. charPtr anAction, ■~ // I: Selector name as in

" on> anAction" '.;■ - •

.. charPtr . aScript, // I: Field script/proc,

NULL to remove int32. options, // I: arious logical options (see notes) ^■ •

.. - - // I: 'aFieldName' may be sprintf 0 ' ' '- ^' -,^' ^{• '} . >' // R:TRUE for success,

FALSE otherwise

. . • charHdl TSJGetTypeScript ( , , // Get type script for action •

^" ETJTypeDBHdl ' aTypeDBHdl , // I:Type DB handle (NULL to default)

ETJTypelD aTypelD, // I:Type ID charPtr anAction, '// I: ction name as in

"<on> anAction"

. int32 ^{• ■}' , '^• options. // I:Various logical options (see notes) . _ι ,

ETJViewRef • ^♦collectionNode,// I: If non-NULL, - , collection node - , •

ETJTypelD , *fromWho , ' '// 10: egistering type ID

- " Boolean ^• -*isLocal // 10: If non-NULL, .set TRUE if local -

• _,// R:Actiόn script, NULL if none -

EngErr TS_InvokeScript ( - // Invoke a type or field • action script •

ETJTypeDBHdl aTypeDBHdl , // I rType DB handle (NULL to default) . ^

ETJTypelD aTypelD, // I -.Type ID charPtr aFieldName, // I:Name of the field/field path . - ... . . . charPtr anAction, ~" // I:Action name as in ^*

"<αn> anAction" - charPtr aScript, // I: type/field script, ULL to default

ETJTypelD fromWho, // I.-Registering Type id. or 0 , - anonPtr aDataPtr, // -I rType data buffer, or

NULL '.' - ' .

ET_CollectionHdl aCollection, // IrT e collection handle, or NULL

. ET_Offset offset, I I- IrCollection element

' reference _,, ^* ' _, i^'nt32 options, // I:Vari^'ous logical options - ^• ■ , -

//- IO:Additional 'anAction' parameters

) ; . // R: Zero for success ,

FALSE otherwise ^*ι _, ftdefine kSpecializedOptiαnsMask QxOOOOFFFF // other' bits are predefined ¹ . ftdefine klntarnal-iϊeResults QxOQ OlQOOO // options - value should be embedded .- . • • . . '

Boolean TS_RegisterScriptFα ( // register a script function

ETJTypeScriptFn •■ aScriptFunct ion , // I r address of script function charPtr , ^• aName // I : name of script function ■' '

^{' '} . ) ^; . // R r TRUE for success .

FALSE otherwise _, - .'

^: '

Every type or type field may also have 'action' scripts (or procedures) associated with it. For example, certain actions could be predefined to equate to standard events in the environment Actions may also be arbitrarily extended and used as subroutines within other scripts, however, in order to provide a rich environment for describing all aspects of the behavior of a type or any UI associated with it. Such an approach would allow the contents of the type to be manipulated without needing any prior knowledge of the type itself. Type and Field script procedures could have the following calling API, for example ■^', •^• - • (ETJTypeScriptFn): . , . . . .^" - ^{' '}. . . "

EngErr myScript ( ' - - // my script procedure

• ETJTypeDBHdl aTypeDBHdl, ..// LType DB handle (NULL to . . default) 1 . ^' -

ETJTypelD typelD, ^" ' .. // LType ED - ' charPtr fieldName, // LField name/path, NULL for type

. charPtr -.. .. action, // LThe script action being invoked charPtr script, // LThe script text . anonPtr dataPtr, // LType data pointer or NULL •

- ETjCollectionHdl aCollection, // LThe collection handle, or NULL

ETjOffset ^{' '} offset, // LCollection element reference va_list ap // Lva_list to additional params.

- ^' ) // R:0 for success, else Error number

In the case of a script, these parameters can be referred to using Saction, ^" - SaTypeDBHdl, StypelD, SfieldName and SdataPtr, any additional parameters are referred to by their names as defined in the script itself (the 'ap' parameter is not accessible from a script). Preferably, Scripts or script functions would return zero if sucessful, an error number otherwise. In the case-of a C function implementing the script, the "ap" parameter can be used to obtain additional parameter values using va_arg0- A number of script actions may also be predefined by the environment to allow registration of behaviors for commonly occunng actions. A sample set of predefined action scripts are listed below (only additional parameters are shown), but many other more specialized scripts may also be used.

SGetPersistentRef(ETJPersistentRef *persistentRef) Resolve a persistent reference, once the required data has been loaded (e. , from a database), the 'memoryRef or 'elementRef field should be set to reference the element designator obtained. This corresponds to resolving the 'typeName #id' persistent reference language construct. Note that if the 'id' field of the ETJPersistentRef is zero, the 'name' field will contain a string giving the name of the item required (presumably unique) which the function should then resolve to obtain and fill out the 'id' field, as well as the 'memory/element Ref field. The contents of the 'stringH' field of 'persistentRef may contain text extracted during data mining (or from other sources) and this may be useful in resolving the reference. The following options are defined for this script:

klntemalizeResults — the resultant value should be created within the referencing collection kGetNameOnly — Just fetch the name of the reference NOT the actual value

$GetCoIIection(charPtr SfilterSpec, charPtr fieldList, ETjCollectionRef *collectionRef) This script builds a type manager collection containing the appropriate elements 'given the parent type and field name. Once the collection has been built, the 'collection' field value of 'collectionRef should be set equal to the collection handle (NULL if empty or problem creating it). This normally corresponds to resolving the 'typeName @@collectionName' collection reference language construct. The value of SfilterSpec is obtained from the "SFilterSpec" annotation associated with the field (if any). Note also that the contents of the 'stringH' field of 'collectionRef may also contain text extracted during data mining (or from other sources) and this may be useful in determining how to construct the collection. The value of the 'fieldList' parameter may be set to NULL in order to retrieve all fields of the elements fetched, otherwise it would preferably be a comma seperated list of ^' field names required in which case the resulting collection will be comprised of proxy types containing just the fields specified. The 'klnternalizeResults' option may apply to this script.

SGetPersisteatCollection(charPtr SfilterSpec, charPtr fieldList, ETJPersistentRef ^persistentRef) This script function is similar to "SGetCollection" but would be called only for persistent reference fields. The purpose of this script is to obtain a collection (into the 'members' field of the ET_PersistentRef) of the possible choices for the persistent reference. This can be seen in the UI when the field has a list selection menu next to it to allow setting - of new values, clicking on this list selection .will result in a call to this script in order to . populate- the resulting menu. "SfilterSpec" and "fieldList" operate in a similar manner to that described for "SGetCollection". The 'klnternalizeResults' option may apply to this script.

■) $InstantiatePersisteαtRef(ET_PersistentRef *persistentRef) This script is called in order to instantiate into persistent storage (if necessary) a record for the persistent reference passed which contains a name but no ID. The script should check for the existence of the named Datura and create it if not found. In either case the ID field of the persistent reference should be updated to contain the reference ID. The actions necessary to instantiate values^{' •} into persistent storage vary from one data type to^" another and hence different scripts may be registered for each data type. The 'stringH' field of the persistent reference may also contain additional information specific to the fields of the storage to be created. The SSetPersReflnfoO function can be used during mining to append to this field. Any string assignement to a persistent reference field during mining results in setting the name sub-field. In the preferred embodiment, this script would clear the 'stringH' field after successful instantiation.. ' - ' .^' . ^'.

^• " SInstaαtiateCollectioα T CollectionRef *collectionRef) This script is called in order to instantiate into persistent storage (if necessary) all records implied by the collection field passed. The process is similar to that for "SlnstantiatePersistentRef ' but the script ^•would preferably be aware of the existence of the 'aStringH¹ field of the collection reference with may contain a text based list of the implied record names. Any string assignment to a collection field during mining results in appending to the 'stringH' field. This field coul also ; be explicitly set using the SSetPersRefln&O function. In the preferred embodiment, this ^• script would clear the 'stringH' field after successful instantiation. $DefauItValue(charPtr defaultValue) This script/function allows the default value of a type field to be set. If the field has a "SDefaultValue" annotation this is passed as a ^" parameter to the function, otherwise this parameter is null. In the absence of a "SDefaultValue" script, any "SDefault alue" annotation found will be passed to TM_StringToBinary(delimiter="\n") which can be used to initialize fields, including structures to any particular value required. The assignment of default values preferably^" occurs within calls to TM_NewPtrO, TMJNewHdlO, or TM_InitMemO so type memory would also be allocated using one of these functions if default values are being used. If no default value is specified, the memory is initialized to zero. A field may also be explicitly set to its default value by calling TM^SetFieldToDefaultO- .. J

* ^" ' -

SAddO This script/function is invoked to add a typed record to persistent storage (i.e, database(s)). In most cases -the record being added will be within a collection that has been extracted during mining OP which has been created manually via operator input.

SUniquelDO This script is called to assign (or obtain) the unique ID for a given record prior to adding/updating that record (by invoking SAdd) to the database. The purpose of this script it to examine the name field (and any other available fields) of the record to see if a record of the same type and name exists in storage and if it does fill out the ID field of the record, otherwise obtain and fill out a new unique ED. Since^" the ID field preferably serves as the link between all storage containers in the local system, it is essential that this field is set up prior to any container specific adds and prior to making any SMakeLink script (described below) calls. : - • -

SMakeLink(ETjCollectionHdl refColIection.ETjOffset ref£lement,charPtr refField) This script is called after SUniq^'uelD and before SAdd when processing ^"data in a collection for addition/update to persistent storage. The purpose of this script is to set up whatever cross-referencing fields or hidden linkage table entries are necessary to make the link ' • ' specified. If the referring field is a persistent reference, it will already have been set up to contain the ED and relative reference to the refeσed structure^'. If additional links are required (e.g., as implied by 'echo' fields), however, this script would be used to set them up prior the SAdd being invoked for all Datums in the collection.

. SSetFieldValue(anonPtr *newValue,long *context,int32 entry) This script could called whenever the value of a field is altered. Normally setting a field value requires no script in order to implement, however, if a script is specified, it will be called immediately prior to actually copying the new value over with the value of 'entry¹ set to' true. This means that the script could change the 'newValue' contents (or even replace it with a alternate ' 'new Value' pointer) prior to the copy. After the copy is complete and if 'context' is non-zero, the scπpt may be called again with 'entry' set to false which allows any context stored via 'context¹ to be cleaned up (including restoring the original 'newValue' if appropriate). Because of this copying mechanism, SSetFieldValue scripts would preferably not alter the field value in the collection, but rather the value that is found in 'newValue' This script is also a logical place to associate any user interface with the data underlying it so that updates to the UI occur automatically when the data is changed.

SDrag(ControlHandle aControlH,EventRecord *eventP,ET_DragRef *dragRef) This scπpt is called to start a drag.

SDrop(ControlHandle aControlH,ET_DragRef dragRef) This script is called to perform a drop. The options parameter will have bit-0 set true if the call is for a prospective drop, false if the user has actually performed a drop by releasing the mouse button. A prospective drop occurs if the user hovers over a potential drop location, in this case a popup menu may be automatically displayed in order to allow the user to select one of a set of possible drop actions (for example, "copy link", "insertsicon" etc). This same menu may also be produced on an actual drop if it is not possible to determine automatically what action is required. The DragAndDrop implementation provides a set of API calls for constructing and handling the drop action menu,

$EIementMatch(ET_Offset elemem\BooIean *match) This script is called to compare two elements to see if they refer to the same item. See TCJFindMatchingElementsO for details. Preferably, the Boolean result is returned in the 'match' field, true to indicate a match and false otherwise.

Annotations are arbitrarily formatted chunks of text (delimited as for scripts and element tags) that can be associated with fields or types in order to store information for later retreival from code or scripts. The present invention utilized certain predefined annotations (listed below) although additional (or fewer) annotations may also be defined as desired: SfilterSpec - This annotation (whose format is not necessarily currently defined by the environment itself) is passed to the SGetCollection and SGetPersisteiftCollectioπ scripts in order to specify the parameters to be used when building the collection.

StableSpec - This annotation (whose format is not necessarily currently defined by environment itself) is used when creating persistent type storage.

SDefaultValue - See the description under the SDefaultValue script.

SBitMask - This annotation may be used to define and then utilize bit masks associated with numeric types and numeric fields of structures. The format of the annotation determines the appearence in auto-generated UI. For full details, see the description or the function TM_GetTypeBιtMaskAnnotation().

S istSpec - In the preferred embodiment, this field annotation consists of a series of lines, each containing a field path within the target type for a collection reference. These field paths can be used to define the type and number of columns of a list control provided by the TypesUI API which will be used to display the collection in the UI. The elements of the SListSpec list would preferably correspond to valid field paths in the target type.

A function, hereinafter called TS_SetTypeAnαotation()j could be provided which adds, removes, or replaces the existing "on" condition annotation for a type. This routine may also be used to add additional annotations to or modify existing annotations of a type.

A function, hereinafter called TS_SetFieldAnnotation()_> could be provided which adds, removes, or replaces the existing annotation associated with a field. This routine may also be used to add additional annotations to or modify existing annotations of a type field. Preferably, annotations always apply globally. In such an embodiment, annotations could be divided into annotation types so that multiple independant annotations can be attached and retrieved from a given field.

A function, hereinafter called TSjGetTypeAnnotatioα , could be provided which obtains the annotation specified for the given type (if any). In the preferred embodiment, the following options are supported:

kNoInheπtance — dont inherit from ancestral types etc. kNoNodelnherit - dont inherit from ancestral nodes in the collection ^v A function, hereinafter called TS_GetFieldAnnotatioα(), could be provided which obtains the annotation text associated with a giyen field and annotation type. If the annotation and annotation" type cannot be matched, NULL is returned. In the preferred embodiment, options include:^' . ^'.

kNoInheritance — dont inherit from ancestral types etc. -

^•• kNoNodelnherit — dont inherit from ancestral nodes in the collection • kNoReflnherit ~ dont inheriffor reference fields

A function, hereinafter called TSjGetFieldScriptO, could be provided which obtains the script associated with a given field and action. If the script and action cannot be matched, NULL is returned. Preferably, the returned result would be suitable for input to the function TS_DoFieldActionScriptO- Note that field scripts may be overriden locally to the process using TS_SetFieIdScript(). If this is the case, the 'isLocal' parameter (if specified) will be set true. Local override scripts that wish to execute the global script and modify the behavior may also obtain the global script using this function with 'globalDef Only' set TRUE,- and execute it using TS JDoFieldActioπScriptO. If the script return actually corresponds to an action procedure not a script then the script contents will simply contain an -' character ' ' followed by a single hex number which is the address of the procedure to be called. This is - also valid input to TS_DoFieldActionScriptO which will invoke the procedure. If the 'inherif parameter is TRUE, upon failing to find a script specific to the specified field, this function will attempt to find a script of the same name associated with the enclosing type (see TMjGetTypeActionScript) or any of its ancestors. This means that it is possible to specifiy default behaviors for all fields derived from a given type iii one place only and then only override the default in the case of specific field where this is necessary. If the field is a reference field, a script is only invoked if it is directly applied to the field itself, all other script inheritance is suppressed. In the preferred embodiment, the following options wδuld \ be supported:

kNoInheritance — dont inherit from ancestral types etc. kNoNodelnherit — dont inherit from ancestral nodes in the collection kNoReflnherit - dont inherit for reference fields kGlobalDefnOnly — only obtain global definition, ignore local overrides The search order when looking for field scripts is as follows: 1) Look for a field script associated with the field itself. 2) If 'inherit* is TRUE:

A) If 'aFieldName' is a path (e. , fieldl ,field2.field3), for each and every

_ ancestral field in turn (from the leaf node upwards - 2,1 in the example above): a) If there is an explicit matching field script (no-inheπtance) associated with that field, use it

B) If the field is a 'reference' field (i.e., *,**,@,@@, or #), search the referred to type for a matching type script

C) Search the enclosing type ('aTypelD') for a matching type scπpt.

A function, hereinafter called TS_SetTypeScriptO, could be procided which adds, removes, to or replaces the existing "on" condition action code within an existing type script. For example, this routine could be used to add additional behaviors to or modify existing behaviors of a type. In the preferred embodiment, if the 'kLocalDefnOnly' option is set, the new action script definition applies within the scope of the current process but does not in any way modify the global definition of the type script. The ability to locally overπde a type action script is very useful in modifying the behavior of certain portions of the UI associated with a type while leaving all other behaviors unchanged. If the TcProcNotScripf option is set, 'aScπpt' is taken to be the address of a procedure to invoke when the script is tπggered, rather than a type manager script. This approach allows arbitrary code functionality to be tied to types and type fields. While the use of scripts is more visible and flexible, for certain specialized behaviors, the use of procedures is more appropπate.

A function, hereinafter called TSjSetFieldScriptfJ, could be provided which adds, removes, or replaces the existing "on" condition action code within an existing field script. For example, this routine may be used to add additional behaviors to or modify existing behaviors of a type field. If the 'kLocalDefnOnly' option is set, the new action script definition applies within the scope of the current process, it does not in any way modify the global definition of the field's script. As explained above, this ability to locally override a field action script is very useful in modifying the behavior of certain portions of the UI associated with a field while leaving all other behaviors unchanged. If the TcProcNotScripf option is set, 'aScripf is taken to be the name of a script function to invoke when the script is triggered, rather than an actual type manager script. This allows arbitrary code functionality to be tied to types and type fields. Script functions can be registered using TS_RegisterScriptFnO. A function, hereinafter called TS jGetTypeScript0₇ could be provided which obtains the script associated with a given type and action. If the type and action cannot be matched, NULL is returned. Preferably, the returned result would be suitable for input to the function TS_DoTypeActionScript(). Note that in the preferred ambodiment type scripts may be overriden locally to the process using TS_SetTypeScriptO. If this is the case, the 'isLocal' parameter (if specified) will be set true. Local override scripts that wish to execute the global script and modify the behavior somehow can obtain the global scπpt using this function with 'kGlobaDefnOnly' option set, and execute it using TSJDoTypeActionScripcQ. If the script return actually corresponds to an action procedure not a script then the script contents will simply contain an '=' character followed by a single hex number which is the address of the procedure to be called. This is also valid input to TS_DoTypeActionScript() which will invoke the procedure. If the 'kNoInheritance' option is not set, upon failing to find a script specific to the type, this function will attempt to-find a script of the same name associated with the enclosing type or any of its ancestors. Using this function, it is possible to specifiy default behaviors for all types (and fields - see TMjGetFieldActionScript) derived from a given type in one place only and then only override the default in the case of specific type/field where this is necessary. Options for this function are identical as described with respect to the function TSjGetFieldScriptO-

A function, hereinafter called TSJfavokeScriptfJ, could be provided which invokes the specified field action scπpt or script function. Note that because the 'fieldScript' parameter is explicitly passed to this function, it is possible to execute arbitrary scripts on a field even if those scnpts are not the script actually associated with the field (as returned by

TS jGetFieldScπpt). This capability makes the full power of the type scripting language available, to program code whilst allowing arbitrary script or scπpt function extensions as v, desired. Unlike most field related functions in this API, this function does not necessarily support sprintfQ type field expansion because the variable arguments are used to pass parameters to the scripts. When invoking a type action script without knowledge of the field involved, the 'aFieldName' parameter should be set to NULL.

A function, hereinafter called function TS_RegisterScriptFn(), could also be provided which could be used to to register a script function symbolically so that it can be invoked if encountered within a field or type script. In the preferred embodiment, when TS_InvokeFieldActionScriptO encounters a script beginning with an '- character and of the form "=scriptFnName" where "scriptFnName" has been registered previously using this procedure, it resolves "scriptFnName" to obtain the actual function address and then invokes the function. ' . ^• • ^' -

The foregoing description of the preferred embodiments of the invention has been presented for the purposes of illustration and description. For example, although described with respect to the C^'programming language, any programming language could be used to implement this invention. Additionally, the claimed system and method should not be limited to the particular API disclosed. The descriptions of the header structures should also not be limited to the embodiments described. While the sample pseudo code provides examples of the code that may be used, the plurality of implementations that could in fact be developed is nearly limitless. For these reasons, this description is not intended to be exhaustive or to limit ' ;' the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

SYSTEM AND METHOD FOR AUTOMATIC GENERATION OF SOFTWARE PROGRAMS Inventor: John Fairweataer

BACKGROUND OF THE INVENTION

In any complex information system that accepts unstructured or semi-structured input (such as an intelligence system) for the external work, it is obvious that change is the norm, not the exception. Media and data streams are often modified and otherwise constantly change making it difficult to monitor them. Moreover, in any system involving multiple users with divergent requirements, even the data models and requirements of the system itself will be subject to continuous and pervasive change. By some estimates, more than 90% of the cost and time spent on software is devoted to maintenance and upgrade of the installed system to handle the inevitability of change. Even our most advanced techniques for software design and implementation fail miserably as the system is scaled or is otherwise changed. The reasons for this failure arise, at least in part, from the very nature of accepted software development practice/process.

Referring now to Figure 1, the root of the problem with the current software development process, which we shall call the "Software Bermuda Triangle" effect, is shown. Conventional programming wisdom holds that during the design phase of an information processing application, programming teams should be split into three basic groups. The first group is labeled DBA (for Database Administrator) 105. These individuals 105 are experts in database design, optimization, and administration. This group 105 is tasked with defining the database tables, indexes, structures, and querying interfaces based initially on requirements, and later, on requests primarily from the applications group. These individuals 105 are - highly trained in database techniques and tend naturally to pull the design in this direction, as illustrated by the small outward pointing arrow. The second group is the Graphical User Interface (GUT) group 110. The GUI group 110 is tasked with implementing a user interface to the system that operates according the customer's expectations and wishes and yet complies exactly with the structure of the underlying data (provided by the DBA group 105) and the application(s) behavior (as provided by the Apps group 115). The GUI group 110 will have a natural tendency to pull the design in the direction of richer and more elaborate user interfaces. Finally the applications group 115 is tasked with implementing the actual functionality required of the system by interfacing with both the DBA and the GUI and related Applications Programming Interfaces (APIs). This group 115, like the others 105,110

173

APPENDIX 3 tends to pull things in the direction or more elaborate system specific logic. Each of these groups tends to have no more than a passing understanding of the issues and needs of the other groups. Thus during the initial design phase, assuming a strong project and software management process rigidly enforces design procedures, a relatively stable triangle is formed where the strong connections 120, 125, 130 enforced between each group by management are able to overcome the outward pull of each member of the tnangle. Assuming a stable and unchanging set of requirements, such a process stands a good chance of delivering a system to the customer on time.

The problem, however, is that while correct operation has been achieved by each of the three groups 110, 105, 115 m the onginal development team, significant amounts of undocumented application, GUI, and Database specific knowledge has likely been embedded into all three of the major software components. In other words, this process often produces a volatile system comprised of these subtle and largely undocumented relationships just waiting to be tπggered. After delivery (the bulk of the software life cycle), in the face of the inevitable changes forced on the system by the passage of time, the modified system begins to break down to yield a new "triangle" 150. Unfortunately, in many cases, the original team that built the system has disbanded and knowledge of the hidden dependencies is gone. Furthermore, system management is now in a monitoring mode only meaning that instead of having a rigid framework, each component of the system is now more likely to "drift". This drift is graphically represented by the dotted lines 155, 160, 165 During maintenance and upgrade phases, each change hits primarily one or two of the three groups. Time pressures, and the new development environment, mean that the individual tasked with the change (probably not an original team member) tends to be unaware of the constraints and naturally pulls outward in his particular direction. The binding forces have now become much weaker and more elastic while the forces pulling outwards have become much stronger. A steady supply of such changes impacting this system could well eventually break it apart. In such a scenario, the system will grind to a halt or become unworkable or un-modifiable. The customer must either continue to pay progressively more and more outrageous maintenance costs (swamping the original development costs), or must start again from scratch with a new system and repeat the cycle. The latter approach is often much cheaper than the former. This effect is central to why software systems are so expensive. Since change of all kinds is particularly pervasive in an intelligence system, any architecture for such systems would preferably address a way to eliminate this "Bermuda Triangle" effect. Since application specific logic and it's implementation cannot be eliminated, what is needed is a system and environment in which the 'data' wfthin the system can be defined and manipulated in terms of a world model or Ontology, and for which the DBA and GUI portions of the programming tasks can be specified and automatically generated from this Ontology thereby eliminating the triangle effect (and the need for the associated . programming disciplines). Such an approach would make the resultant system robust and - adaptive to change. , ' " . • ' ^" '

SUMMARY OF INVENTION

' The present invention provides a system capable of overcoming this effect and ' provides a system that is both robust and adaptive to change. The preferred base language upon which this system is built is the C programming language although other languages may be used. In the standard embodiment using the C programing language, the present invention is composed of the following components:

^' a) Extensions to the language that describe and abstract the logic associated with interacting with external. 'persistent' storage (i.e., non-memory based). Standard programming languages do not provide syntax or operators for manipulating , i persistent storage and a formalization of this capability is desirable. This invention provides these extensions and the "extended" language is henceforth^' \ referred to as C*. C*, in addition to being a standard programming language, is also an ontology definition language (ODL). .

b) Extensions to the C* language to handle type inheritance. Ln an ontology based _ system, the world with-which the system interacts is broken down based on the kinds of things that make up that world, and by knowledge of the kind of thing ^" ' involved, it becomes possible to perform meaningful calculations on that object without knowledge of the particulars of the descendant type. Type inheritance in this context therefore more accurately means ancestral field inheritance (as will be described later). ' -'

c) Extensions to the C* language to allow specification of the GUI content and layout d) Extensions to the C* language to allow specification and inheritance of scriptable actions on a per-field and per-type basis. Similar extensions to^'allow arbitrary annotations associated with types and fields are also provided.

' e) A means whereby the data described in the C* language can be translated automatically into generating the corresponding tables and fields in external databases and the queries and actions necessary to access those databases and read/write to them.^" This aspect of the invention enables dynamic creation of databases as data is encountered

f) A high level ontology designed to facilitate operation of the particular application being developed. In the examples below and in the preferred^' embodiment, the

^* application being developed will address the problem of 'intelligence' i.e., the understanding of 'events' happening in the world in terms of the entities involved,

- their motives, and the disparate information sources from which reports are . obtained. "

g) A means to tie types and their access into a suite of federated type or

- container/engine specific servers responsible for the actual persistence of the data.

- A necessary prerequisite foτ tackling the triangle problem is the existence of a runtime accessible (and modifiable) types system capable of describing arbitrarily complex binary structures and the references between them. In the preferred embodiment, the invention uses the system has been previously described in Appendix 1 (hereinafter, the ' "Types Patent"). Another prerequisite is a system for instantiating, accessing and sharing aggregates of such typed data within a standardized flat memory model and for associating , inheritable executable and/or interpreted script actions with any and all types and fields- within such data. In the preferred embodiment, the present invention uses the system and method that is described in Appendix 2 (hereinafter, the "Memory Patent"). The material presented in these two patents are expressly incorporated herein. Additional improvements and extensions to this system will also be described below and many more will be obvious to , . those skilled in the art. . BRIEF DESCRIPTION OF THE FIGURES

Figure 1 shows the root of the problem with the current software development process, which we shall call the "Software Bermuda Triangle" effect.

Figure 2 shows a sample query-building user interface (UI).

^■ Figure 3 shows a sample user interface providing access to the fields within the type .^' "country." ^' . . ; ^' • - _, ,

Figure 4 shows a sample user interface providing access to a free format text field within the type. "country." ^' . •^{■ '}

- - . 'Figure 5 shows a sample user interface providing access to a fixed sized text field within the type ''country."- '^'''

' Figure 6 A shows an example of how a short text field or numeric field (such as those handled by the RDBMS container described above) might be displayed in a control group.

Figure 6B shows one method for displaying a date in a control group.

Figure 6C shows !an example of an Islamic Hijjrah calendar being displayed.

Figure 7A shows the illustrated control group of how one might display and interact with a persistent reference field ('#'). - ' '

Figure 7B)Shows an example of one way that a collection reference field ('@@') might be displayed in an auto-generated„user interface. .. ^' •.. - -

Figure 8 shows one possible method for displaying variable sized text fields (referenced via the char @ construct). . ^' -

Figure 9 shows the manner in which an image reference (Picture @picture) field could be displayed in an auto-generated user interface.

Figure 10 shows a sample screen shot of one possible display of the Country record in the same UI layout theme described above (most data omitted)

Figure 11 shows a sample embodiment of the geography page within Country. ' Figure 12 shows a sample embodiment of the second sub-page of the geography page within country. . .. -:

Figure 13 shows an example of one part of a high-level ontology targeted at intelligence is shown. • •^" -

DETAILED DESCRIPTION OF THE INVENTION

As descπbed a^"bove, a necessary prerequisite for tackling the tπangle problem is the existence of a run-time accessible (and modifiable) types system capable of descπbing arbitraπly complex binary structures and the references between them. In the preferred embodiment, the invention uses the system descnbed in the Types Patent. Another prerequisite is a system for mstantiatmg, accessing and sharmg aggregates of such typed data withm a standardized flat memory model and for associating inheπtable executable and/or interpreted script actions with any and all types and fields within such data. In the prefened embodiment, the present invention uses the system and method that is descπbed in the Memory Patent. The material presented in these two patents are expressly incorporated herein and the functions and features of these two systems will be assumed for the purposes, of this invention. ,

As an initial matter, it is important to understand some of the langauge extentions that are needed in order to create an Ontology Description Language (ODL). In the preferred embodiment, the following operators/symbols are added to the basic C language (although other symbols and syntax are obviously posstble without changing the basic nature of the approach) in order to provide basic support for the items described herein:

script — used to associate a script with a type or field annotation — used to associate an annotation with a type or field

@ - relative reference designator (like '*' for a pointer)

@@ - collection reference designator

# — persistent reference designator

<on> — script and annotation block start delimiter

<no> — script and annotation block end delimiter x - echo field specification operator

: — type inheritance

Additionally, the syntax for a C type definition has been extended to include specification of the "key data-type" associated with a given ontological type as follows:

typedef struct X 'XXXX' { .. } ; .Where the character constant 'XXXX' specifies the associated key data-type. The persistent reference designator '#' implies a singular reference to an item of a. named type held in external storage. Such an item can be referenced either by name or by unique system- wide ID and given this information, the underlying substrate is responsible for obtaining the actual data referenced, adding it to the collection, and making the connection between the referencing field and the newly inserted data by means of a relative reference embedded within the persistent reference structure. Preferably, the binary representation of a persistent reference field is acomplished' using a structure of type 'ET_PersistentRef as defined below:

typedef struct ETMJniquelD

1 ,

OSType ' . . . •, systeπt; / .system id is 32 bits unsln 54^' id; // local id is 64 bits

} ETJϋtaiquelD; - ^'.^'

typedef struct ETJPersistentRef

STJCollectionHdl members ; // member collection charHdl .._ stringH; // String containing mined text _ • ^', ^{' "} - ' '

ETJTypelD aTypelD,- ', // type ID ' ^'

ET_Offset elementRef; // rel. ref. to data (NULL- if ! etched)

ETJOffset memberRef; • ^': // rel. ref. to member coll. (or NULL) anonPtr memoryRe ; // 'pointer to type data

(NULL if N/A) . - ^'_ _

-ET Uni-quelD - - ^'' id; ^•-' • - // unique ID char name [kPersRefNameSize] ; // name of reference

} ^'ET_PersistentRe , *ET_PersistentRe£Ptr;

The type ET JniquelD consists of a two part 96-bit reference where the 64-bit 'id' field refers to the unique ID within the local 'system' which would normally be a single logical installation such as for a particular corporation or organization. Multiple systems can exchange data and reference between each other by use of the 32-bit 'system' field of the unique ID. The 'members' field^'of an ETJPersistentRef is used by the system to instantiate a collection of the possible items to which the reference is being made and this is utilized in the user interface to allow the user to pick from a list of possibilities. Thus for example if the persistent reference were "Country nationality" then the member collection if retrieved would be filled with the names of all possible countries from which the user could pick one which would then result in filling in the additional fields required to finalize the persistent reference.

In normal operation, either the name or ID and type is known initially and this is sufficient to determine the actual item in persistent storage that is being referenced which can then be fetched, instantiated in the collection and then referenced using the 'elementRef field. The contents of the 'stringH' field are used during data mining to contain additional rnformating relating to resolving the reference. The 'aTypelD' field initially takes on the same value as the field type ID from which the reference is being made, however, once the matching item has been found, a more specific type ID may be assigned to this field. For example if the referencing field were of the form "Entity #owner" (a reference to an owning entity which might be a person, organization, country etc.) then after resolution, the 'aTypelD' field would be altered to reflect the actual sub-type of entity, in this case the actual owning entity. The 'memoryRef field might contain a heap data reference to the actual value of the referenced object in cases where the referenced value is not to become part of the containing collection for some reason. Normally however, this field is not needed.

As an example of how the process of generating and then resolving a persistent reference operates, imagine the system has just received a news story referring to an individual who's name is "X", additionally from context saved during the mining process, the system may know such things as where "X" lives and this information could be stored in the 'stringH' field. At the time the reference to "X" is instantiated into persistent storage, a search is made for a person named "X" and, should multiple people called "X" be found in the database, the information in 'stringH' would be used in a type dependant manner to prune the list down to the actual "X" that is being referenced. At this point the system-wide ID for the specific individual "X" is known (as is .whatever else the system knows about X) and thus the 'id' field of the reference can be filled out and the current data for "X" returned and referenced via "elementRef. If no existing match for "X" is found, a new "Person" record for "X" is created and the unique ID assigned to that record is returned. Thus it can be seen that, unlike a memory reference in a conventional programming language, a persistent reference may go through type specific resolution processes before it can be fully resolved. This need for a 'resolution' phase is characteristic of all references to persistent storage.

Like a persistent reference, the collection reference '@@' involves a number of steps during instantiation and retrieval. In the preferred embodiment, a collection reference is physically (and to the C* user transparently) mediated via the ΕTjCollectioπRef type as set forth below

typedef struct ET jCollectionRef {

ET_jCollectionHdl collection; // member collection charHdl stringH; // String containmg mined text

ETJTypelD aTypelD; // collection type ID (if any)

ETjOffset elementRef; // relative reference to collection root

ET_StringList cList; // collection member list (used for UI)

} ETjCollectioπRef, *ET_CollectιonRefPtr;

The first four fields of this structure have identical types and purposes to those of the ET_PersistentRef structure, the only difference being that the 'collection' field in this - structure references the complete set of actual items that form part of the collection. The 'cList' field is used internally for user interface purposes. The means whereby the collections associated with a particular reference can be distinguished from those relating to other similar references is related to the meaning and use of the 'echo field' operator 'x\ The following extracts from an actual ontology based on this system serve to reveal the relationship between the 'x' operator and persistent storage references:

typedef struct Datum ' DTUM ' It Ancestral type of all pers . storage

I »

{

NumericID - host D; It unique Host system ID

(0=»local) uns!ntS4 id; It unique ID char name [25S] ; II full name of this

Da am char datumType [32] , II the type of the datum

Numer cID securi yLevel ; II security level char updatedBy [30] ; // person updating/creating this Datum - ^* .^{' •} '

- Date ' . ' dateEntered; - J // date first entered

. Date dataσpdated; // date of last update

Feed #source; ., // information source for this Datum ^•• - *

Language ^' . - ilanguage;- _, ^• // language for this

Datum record struct '

^' { ^{" '}

Datum , -.- •

^{' '} ' ^'

( changes to Datum .

Category ©©membershi ; // Groupings Datum is a member of ' . . ' char ©sourceNotes; , // notes information source (sj - ^'•

• uns!ntS4' sourcelDref; // ID reference in original source

} notes; • *

'Symbology #symbology; // symbology used

Place #place; ' . // 'where' for the datum

(if taiown) _' ' ,

} Datum , *DatumPtr; -

typedef struct^' NoteRelating:Observation 'CXRF' // Relationship between two daturns _* ^'

{ ^' , ' _, ^' Datum #regarding >< notes. relatedFrom; // 'source' item char itemType [S4] ; // Datum type for regarding item ,

Datum #related >< notes. relatedTo; // ' arge ' item

183

SUBSTITUTE SHEET R L char . relatedType [64] ; // Datum type for related item , .

RelationType tJrelationType ,- // The type of the ^' relationship ' -^" ' •

Percent relevance ,- • . // strength of •^' relationship (1. .100). : - char author [12S] ; //. Author of the Sticklt

Relating note • ^• char - ^' . : - ' - title [2SS] ; ' // Full Title of Sticklt

} NoteRelating;

In the preferred embodiment, 'Datum' is the root type of all persistent types. That is, every other type in the ontology is directly or indirectly derived from Datum and thus inherits all of the fields of Datum. The type 'NoteRelating' (a child type of Observation) is the ancestral type of all notes (imagine them as stick-it notes) that pertain to any other datum. Thus an author using the system may at any time create a note with his observations and , opinions regarding any other item datum held in the system. The act of creating such a note causes the relationships between the note and the datum to which it pertains to be written to and persisted in external storage. As can be seen, every datum in the system contains within its 'notes' field a sub-field called 'relatedFrom' declared as "NoteRelating @@relatedFrom x regarding". This is interpreted by the system as stating that for any datum, there is a collection of items of type 'NoteRelating' (or a derived type) for which the 'regarding' field of each 'NoteRelating' item is a persistent reference to the particular Datum involved. Within each such 'NoteRelating' item there is a field 'relating' which contains a reference to some other datum that is the original item that is related to the Datum in question. Thus the 'NoteRelating' type is serving in this context as a bi-directional link relating any two items in the system as well as associating with that relationship a 'direction', a relevance or strength, and additjonal information (held in the @text field which can be used to give an arbitrary textual description_, of the exact details of the relationship). Put another way, in order to discover all elements in the 'relatedFrom' collection for a given datum, all that is necessary is to query storage/database for all 'NoteRelating' items having a 'regarding' field which contains a reference to the Datum involved. All of this information is directly contained within the type definition of the item itself and thus no external knowledge is required to make connections between disparate data items. The syntax of the C* declaration for the field, therefore, provides details about exactly how to construct and execute a query to the storage container(s)/database that will retrieve the items required. Understanding the expressive power of this syntax is key to understanding how it is possible via this methodology to eliminate the need for a conventional database administrator and/or database group to be involved in the construction and maintenance of any system built on this methodology

As can be seen above, the 'regarding' field of the 'NoteRelating' type has the reverse 'echo' field, i.e., "Datum ^regarding x notes.relatedFrom;". This indicates that the reference is to any Datum or derived type (i.e., anything in 1iιe ontology) and that the "notes.relatedFrom" collection for the referenced datum should be expected to contain a reference to the NoteRelating record itself. Again, it is clear how, without any need for conventional database considerations, it is possible for the system itself to perform all necessary actions to add, reference, and query any given 'NoteRelating' record and the items it references. For example, the 'notes.relatedTo' field of any datum can reference a collection of items that the current datum has been determined to be related to. This is the other end of the 'regarding' link discussed above. As the type definitions above illustrate, each datum in the present invention can be richly cross referenced from a number of different types (or derivatives). More of these relationship types are discussed further herein.

For the purposes of illustrating how this syntax might translate into a concrete system for handling references and queries, it will assumed in the discussion below that the actual physical storage of the data occurs in a conventional relational database. It is important to understand, however, that nothing in this approach is predicated on or implies, the need for a relational database. Indeed, relational databases are poorly suited to the needs of the kinds of system to which the technology discussed is targeted and are not utilized in the preferred embodiment All translation of the syntax discussed herein occurs via registered script functions (as discussed further in the Collections Patent) and thus there is no need to hard code this'system to any particular data storage model so that the system can be customized to any data container or federation of such containers. For clarity of description, however, the concepts of relational database management systems (RDBMS) and how they work will be used herein for illustration purposes. ^{• •} Before going into the details of the behavior of RDBMS plug-in functions, it is worth examining how the initial connection is made between these RDBMS algorithms and , functions and this invention. As mentioned previously, this connection is preferably established by registering a number of logical functions at the data-model level and also at the level of each specific member of the federated data container set. The following provides a sample set of function prototypes that could apply for the various registration processes:

Boolean DB_SpecιfyCallBack ( // Specify a persistent storage callback .. ^' short ; , ' aFuncSelector, ' . II I: Selector for the logical function - ^' . . ^' -. ^'•

ProcPtr aCallBackFα . // ItAddress of the callback , . function

•' ) // R:TRt/E for success, FALSE' otherwise

ftdefine kFαFillCollaction 1 // ET_FillCollectionFn - ^* ^' // Fn. to fill collection with data for a given a hit list ' ■ , ^'-- . , - , - ftdefine kFnFetchRecords 2 // ET_FetchRecordsFn -

// Fn. to query storage and fetch matching records .to colln. ftdefine kFnGetNextUπiquelD 3 // ETJjettfaigtieldFn -

^' . // Fn. to get next unique ID from local persistent storage _ ftdefine kFnStoreParsedDatums .4 // ET_StoreParsedDatumsFn -

. - . // Fn. to store all extracted data in a collection ftdefine kFn riteCollβction Ξ // ET_WriteCollectioπFn -

II Fn. to store^' all extracted data in a collection ftdefine kFnDoesIdExist S // ET_DoesIdExistFn -

II Fn. to determine if a given ID exists in persistent storage -

#define kFnRegisterlD 7 // ET_RegisterIDFn -

- // Fn. to register an ID to persistent storage ftdefine kFnRe ovelD 3 // ET_RamoveIDFn -

, // Fn. to remove a given ID from the ID/Type registery . - , ftdefine kFnFetchRecordToColl 9 // ET_FetchRecordToCollFn -

•_*• // Fn. Fetch a given persistent storage item into a colln . ^• ^ , ftdefine kFnFetchField ^'.. 10 // ETJFetchFieldFπ - • ^•

//' Fn. Fetch a single f eld from a single persistent record ftdefine kFApplyChanges _ 11 // EX_ApplyChangesFn -

^•..-•. ^•_ ^{■ '} _^' //^'Fn. to apply changes . - ftdefine kFnCancalChanges '• ■ 12 - ' . // ET_ _CancelChangesFn -

.-^'. ^{' '} • // Fn. to cance] . changes ftdefine kFnCαuntTypeltems 13 // ET. _CountIteκιsFn -

^'•-. .. ^{' '}. ft Fn. tα count items for a type (and descendant types) . .- _ ' - ftdefine kFnFetchToElements _v_ 14 // ET_ _FetchToElementsFn -

- ^' . . ^• . ^' ".// Fa. to fetch values into a speci ied set of elements/nodes _j • ^' ftdefine kFRcrsvHitListQuery IS // ET_ _RcrsvHitListQueryFn -

. , . - ^' - // Fn. create a hit list from a type and it's ' descendants . ftdefine JFnGatNextValidlD ^{' '} IS • _, ' // ET_ JSetNextValidlDFn -

// Fn. to find next valid ID_, of a type ^'after a given

ID

Boolean DB_DefineContainer ( // Defines a federated container , .

. charPtr name // I: Name o .container

// R: Error code (0 = no error) , ^" ;

Boolean DBJDefinePluginFunctionl- • ^' • Defines container plugin

_ // fn.

' _,. charPtr name. // I: Name of container int 2 unctiαπType, // I: Which function type

ProcPtr functionAddress // I: The address of the function

^{" '} ' . ^{} ;} // R: Void _ ^*.

ftdefine kCreateTypeStorageFunc 29 ' // Create storage for a container ftdefine klnsertElementsFunc • 30^' . // insert container data ftdefine kU dateRecordsFroraEleraentsFunc 31 // update container from data ftdefine kDeleteElementsFuπc , 32 // delete elements from container ^" - ftdefine kFetchRecordsToElementsFunc 33 // fetch container data ftdefine JInsertCollectionRecordFunc 34 // insert container data to elements ^• , ftdefine kUpdateCαllactionRecordFunc 35 // update collection from container - ftdefine kDeleteCollectionRecordFunc 3S // delate collection record

_, ftde ine kFetchRecordsToCollectionFunc 37 _,// fetch container record to colln. ^" .. • ^' - ^•

187

ftdefine kCheckFieldType 33 // determine f field is container' s

In this embodiment, whenever the environment wishes to perform any of the logical actions indicated by the comments above, it invokes the function(s) that have been registered using the function DB_SpecifyCallBackO to handle the logic required. This is the first and most basic step ra disassociating the details of a particular implementation from the necessary logic. At the level of specific members of a federated collection of storage and querying containers, another similar API allows container specific logical functions to be registered for each container type that is itself registered as part of the federation. So for example, if one of the registered containers were a relational database system, it would not only register a 'kCreateTypeStorageFunc' function (which would be responsible for creating all storage tables etc. in that container that are necessary to handle the types defined in the ontology given) but also a vaπety of other functions. The constants for some of the more relevant plug-ins at the container level are given above. For example, the 'kCheckFieldType' plug-in could be called by the environment in order to determine which container in the federation will be responsible for the storage and retrieval of any given field in the type hierarchy. If we assume a very simple federation consisting of just two containers, a relational database, and an inverted text search engine, then we could imagine that the implementation of the 'kCheckFieldType' function for these two would be something like that given below:

// Inverted file text engine:

Boolean DTXJCheckFieldType ( // Field belongs to 'TEXT" 1

ETJTypelD aTypelD, // I: Type ID charPtr fieldname // I: Field name

) // R: Error code

(0 a no error) ^N {

ETJTypelD fType, baseType, ιn 32 rType;

Boolean ret;

fType = TM_GetFieldTypeID(NU I., aTypelD, field ame) ; ret = NO; ' if { TM_TyρeIsReference(NULIj, fType, &rType, &baseTyp 2) && baseType ==

IcIntaType && (rType == kPointerRef.il rType « kHandleRef |^"| rType kRelativeRef ) . ) ret = YES return re ,-

} ^'

// Relational database:

Boolean DSQ_CheckFieldType _,- ( //• Field belongs to ^ΛRDBM' ? ,_ _, . ^{' •} ^• ETJTypelD _, _ aTypelD, //' I: Type ID . ■ ' charPtr ^" ' fieldname // I: Field name

^{' '}. " • ^] // R: Error code

(0 = no error) '. > ^{• '} - _,

ETJTypelD fTypei baseT; . int32 ref ; Boolean ret;

fType = TM_GetFieldTypeID (NULL, aTypelD, fieldname);. ref =« TMJTypelsReference (NULL, fType, &refT,&baseT) ,- ^■ ret => NO; ^'■ if ( ref &&. refT == kPersistentRef ) // We'll handle pers. Refs. , ^' ' . ret =■ YES ,- '. ' ^■ . ' else if ( ϊ'ref && ( , , . ^' . ' . - ^• . // We do:

TM_IsTypeDescendant (NULL, fType, klntaType) || // char arrays, fType^' == TMjGetTypelD (NULL, "Date") || // Dates,

TM_IsTypeDβscendant (NULL, Type, klntegerNumbersType) II ^' //

Integers and ^{" •'}

T _IsTypeDescendant (NULL, fType, RealNumbersType) ) ) . ^' //

Floating point^' #'s

.^' ret = YES; ' return ret; .^' ' ^"" - , , •

} ^'. . .

As the pseudo-code above illustrates, in this particular federation, the inverted text engine lays claim to all fields that are references (normally '@') to character strings (but not fixed sized arrays of char) while the relational container lays claim to pretty much everything else including fixed (i.e., small sized) character arrays. This is just one possible division of responsibility is such a federation, and many others are possible. Other containers that may be members of such federations include video servers, image servers, map engines, etc. and thus a much more complex division of labor between the various fields of any given type will occur in practice. This ability to abstract away the various containers that form part of the persistent storage federation, while unifying and automating access to them, is a key benefit of the system of this invention.

Returning to the specifics of an RDBMS federation member, the logic associated with the 'kCreateTypeStorageFunc' plug-m for such a container (assuming an SQL database engine such as Oracle) might look similar to that given below:

static EngErr DSQ_CraateTypeStorage ( // Build SQL tables

ETJTypelD theType // I: The type

) // R: Error Code

(α = no error) { char sqlStatemen [25S] , filter [256] ;

err = DSQ_CruιseTypeHιerarchy( heType, DSQ_CreateTypeTable) ; sprintf (filter, // does linkage table exist?

"owner= (select username from all_users where user_ιd=uιd) and "

"table name= ' LINKAGE TA3LES$'"),-

ifi (#records found("all_tables", filter)) // If not, then create x !

{ sprintf (sqlStatement , "create table LINKAGE JTABLES$

(DYN_NAME varchar2 (S0) ,ACT_NAME varchar2 (50) ) tablespace data" ) ,- err =« SQL_ExecuteStatetnent (Q , sqlStatement , NULL, α , . NULL) ;

} err - DSQ_CruιseTypeHιerarchy(theType, DSQ_CreataLmkageTables) ; — any other logic required return (err) ;

In this example, the function DSQ_CruiseTypeHierarchyO simply recursively walks the type hierarchy beginning with the type given down and calls the function specified. The function DSQjCreateTypeTableO simply translates the name of the type (obtained from TMjGetTypeName) into the corresponding Oracle table name (possibly after adjusting the name to comply with constraints on Oracle table names) and then loops through all of the fields in the type determining if they belong to the RDBMS container and if so generates the conesponding table for the field (again after possible name adjustment). The function DSQ_CreateLinkageTablesO creates anonymous linkage tables (based on field names involved) to handle the case where a field of the type is a collection reference, and the reference is to a field in another type that is also a collection reference echoing back to the original field. After this function has been run for all types in the ontology, it is clear that the external relational database now contains all tables and linkage tables necessary to implement any storage, retrieval and querying that may be implied by the ontology. Other registered plug-in functions for the RDBMS container such as query functions can utilize knowledge of the types hierarchy in combination with knowledge of the algorithm used by DSQjCreateTypeStorageO, ^such ^as knowledge of the name adjustment strategy, to reference and query any information automatically based on type.

Note that some of the reference fields in the example above do not contain a 'x' operator which implies that the ontology definer does not wish to have the necessary linking tables appear in the ontology. An example of such a field (as set forth above) is "Category ©©membership". This field can be used to create an anonymous linkage table based on the type being referenced and the field name doing the referencing (after name adjustment). The linkage table would contain two references giving the type and ID of the objects being linked. When querying such an anonymous table, the plug-ins can deduce its existence entirely from ^'the type information (and knowledge of the table creation algorithm) and thus the same querying power can be obtained even without the explicit definition of the linking table (as in the example above). Queries from the C* level are not possible directly on the.fields of such a linkage table because it does not appear in the ontology, however, this technique is preferably used when such queries would not necessarily make sense.

By using this simple expedient, a system is provided in which external RDBMS storage is created automatically from the ontology itself, and for which subsequent access and querying can be handled automatically based on knowledge of the type hierarchy. This has effectively eliminated the need for a SQL database administrator or database programming staff. Since the same approach can be adopted for every container that is a member of the federation, these same capabilities can be accomplished simultaneously for all containers in the federation. As a result, the creator of a system based on this technology can effectively ignore the whole database issue once the necessary container plug-ins have been defined and registered. This is an incredibly powerful capability, and allows the system to adapt in an automated manner to changes in ontology without the need to consider database impact, thus greatly increasing system flexibility and robustness to change. Indeed, whole new systems based on this technology can be created from scratch in a matter of hours, a capability has been up until now unheard of. Various other plug-in functions may also be implemented, which can be readily deduced from this description.

The process of assigning (or determining) the unique ID associated with instantiating a persistent reference resulting from mining a datum from an external source (invoked via the SUniquelD script as further described in the Collections Patent) deserves further examination since it is highly dependant on the type of the data involved and because it further illustrates the systems ability to deal with such real-world quirks. In the simple federation described above, the implementation of the SUniquelD script for Datum (from which all other types will by default inherit) might be similar to that given below:

static EngErr PTS_AasιgnUnιqueID ( // SUniquelD script registered with Datum

ETJTypeDBHdl aTypeDBHdl , // I.-Type DB handle (NULL to default) -

ETJTypelD typelD, // I:Type ID charPtr fleldName, // I: ield name/path (else

NULL) charPtr action, // I:The script action being invoked • charPtr script, // I:The script text anonPtr d taP r, // I: ype data pointer

ET_CollectιonHdl aCollection ,// I:The collection handle

ET_Offset o offset, // I: Collection element reference

* mt32 options, // I : arious logical options , ETJTypelD fromWho, // I:Type ID, 0 for field or unknown . ' ' va_list . ' ap // I:va list for additional parameters * '■' . . '

) . // R:0 for success, else error #

ET_UniqueID uniquelD;

TCJGetUniquelD (aCollection,.0, offset, &uniqueID) ; . TCJetCStringFieldValue (aCollection, 0,0, o fset, name, sizeof (name) , "name")

elemTypelD => TC_GetTypeID (aCollection, 0, offsett) ;

TM_BreakUniqueID (uniquelD, &localID, isys) ; if ( locallD ) return 0; ^' - // we've already got an ID, e ' re done ! . . ^' ■ scrubbedStrPtr =■ mangle name according to SQL name mangling algorithm force scrubbedStrPtr to upper case sprintf (filterText, kStartQueryBlock kRelationalDB " :upper (name) = '%s'" kEndQueryBlock, scrubbedStrPtr) ; // Create the filter criteria '. hitList = construct hit list of matches count = # hits in hitList; ^' // how many hits did we get

' // Should issue a warning or dialog if more than one hit here if (hitList £& hitList [0] ,_id) ,

uniquelD = TM_MakeUniqueID (hitList [0] ._id,hitList [0] ._system) ,- existingElemTypelD =« hitList [0] ._type; exists = TRUE;

} ^{' "• '} - . ^* _ if (!uniquelD. id) uniquelD = TM_MakeUniqueID (DB_GetNextLocalUniqueID ( ) , 0) ,- ' ^■ • if ( !TC_HasDirtyFlags (aCollection, 0 , 0 , offset) ) call TC_EstabHshEmptyDirtyState (aCollection, 0 , 0, offset , O) ) TC_SetUniqueID (aCollection, 0 , offset, uniquelD) ; // set the id return err; }

This is a simple algorithm and merely queries the external RDBMS to determine if an item of the same name already exists and if so uses it, otherwise it creates a new ID and uses ^' that. Suppose that the item involved is of type "Place". In this case, it would be helpful to be more careful when determining the unique ID because place names (such as cities) can be repeated all over the world (indeed there may be multiple cities or towns with the same . within any given country). In this case, a more specific SUniquelD script could be registered with the type Place (the ancestral type of all places such as cities, towns, villages etc.) that might appear more like the algorithm given below:

s tatic EngErr PTS_AssignPlacsϋniqueID ( // $UniqueID script registered with Place

ETJTypeDBHdl aTypeDBHdl, // I:Type DB handle (NULL to default) ' ]

ETJTypelD typelD-, ^• / .I:Type ID charPtr fieldName, // IsField name/path (else NULL) ' . _, •

> _ charPtr action, ' // I:The script action being invoked charPtr . script, // I:The script text anonPtr dataPtr, // i:Type data pointer . .^' ET_CollectionHdl aCollection,// I:The collection handle

ET Offset offset, // I:Collection element reference int32 options, ' // I: arious logical options

ETJTypelD fromWho, // I:Type ID,-0 for field_, or unknown- va list ap // I:va_list for additional parameters // R:0 for success, else error #

. ΞT_UniqueID uniquelD;

TCjGetUπiquelD (aCollection, 0, offset, δuniquelD) ;

TCjGetCStringFieldValue (aCollection, 0, 0, offset, ame, sizeof (name) , "name")

TC SetCStringFieldValue (aCollection, 0,0, offset, thisPlace, 123, "placeType"

)';

TC SetFieldValue (aCollection, 0,0, offset, ithisLon, "location, longitude") ; TCjSetFieldValue (aCollection, 0, 0, offset, &thisLat, "location. latitude") ,- elemTypelD = TCJetTypelD (aCollection, 0, of set) ; pT a TM_IsTypeProxy (elemTypelD) ; _ ^' .

. if ( pT ) elemTypelD -» pT; . '•'.

TM_BreakUniqueID(uniqueID,&localID,NULL) ; if ( locallD ) return 0; _ // we've already got an

ID, we' re done! • ^•, " ' - .._,

scrubbedStrPtr =» mangle name according, to SQL name mangling algorithm force scrubbedStrPtr to upper case sprintf (filterTaxt, kSttarttQueryBlock kRelationalDB " :upper(name) = '%s'" kEndQueryBlock, .scrubbedStrPtr) ; sprintf (fieldList, "placeType, location, country") ; tmpCollection =» fetch all matching items to a collection

TC_Coun (tmpCollection, ValuedNodesOnly, rootElem,.&count) ; -

// if we got one or more we need further study to see if it is in fact this place

// a place is unique if the place type, latitude and longitude are the same^{' ■} •^" " ' " ,^" • placeTypeld = TM_KeyTypeToTypeID( 'PLCE' ,NULL) ; pplaceTypeld » TM_KeyTypeToTypeID (' POPP ' ,NULL) ; - , .^' . , if (count) ^•

{ ; ^' ' - ^{' ■} anElem =0; ' , while (tmpCollection &£ TC_Visi (tmpCollection, kRecursiveOperation + kValuedNodesOnly, 0, SanEle , false) )

{ ^{•' '} . -^' . if ( TMJTypesAreCompatible (NULL, TCJSetTypelD ( tmpCollection, 0, anElem) ; ' • , .

..pplaceTypeld) £& .. ^' ,

TMJTypesAreCompatible (NULL, elemTypelD, pplaceTypeld) )

( // both populated places, check country • • '

^{^'} TCjGetFieldValue (tmpCollection, 0,0, nElem, iprfl, "country") ; TC_GetFieldValue (aCollection, 0, 0, offset, &pr , "country") ; if (strcmp(prfl.name,ρrf2.name) ) // different country! ' ' continue;

TC GetCStringFieldValue (tmpCollection, 0,0, anElem^", &placeType, 128, "placeTy pe")_; • ^' if ( !strcmp( hisPlace, laceType) ) // same type ( if ( ' ^' - ^' ' ^' •

TC lsFieldEmpty(tmpCollection,0, 0, anElem, "location. longitude") )

{ _, ^{• ■} ■ II this is the same place!

TC_GetUniqueID ( tmpCollection, 0, anElem, SuniquelD) ; •

TM_BreakϋniqueID (uniquelD , SlocallD , NULL) ; ^" existingElemTypelD = , •

TC_GetTypeID (tmpCollection, 0 , anElem) ; exists =- (existingElemTypelD != 0) ;

- ■ break;

} else • •

' { . . ^{• • • •} . . ^{■ ' "'} ^'TC_GetFieldValue(tmpCollection, 0, 0, anElem, ilongitude,

I . "location. longitude" ) ; if (ABS(thisLon - longitude) < 0.01)

{ ^' // at similar longitude

TC_GetFieldValue (tmpCollection, 0,0, anElem, Platitude, ^{• '}.

» "location. latitude") ; ^' if (ABS(thisLat - latitude) < 0.01)

{ ~ . 11 and similar latitude!

TC_GetUniqueID (tmpCollection, 0, anElem, iuniquelD) ,- TM_BreakUniqueID (uniquelD, S ocallD, ULL) ; existingElemTypelD ■ , •

TC_GetTypeID (tmpCollection, 0, anElem) ; . ^' exists β (existingElemTypelD != 0) ; break; - ^•

} ^• . ^{" ■} - ^■' - ^'

: } . ^{' ' •'} - •

- ^{" ■}. } ^■ } ^"

J ^"> ^{' '} \ .

} ^'

- if ( ilocallD ) ^' _. . . . ^' uniquelD = TM_MakeϋniqueID (DB GetNextLocalϋniquelD ( ) , 0) ; "~ else _, uniquelD = TM_MakeUniqueID (local ID, 0) ; ^" if ( !TC_HasDirtyFlaga (aCollection, 0, 0, of set)) call TC_EstablishEmptyDirtyState (aCollection, 0,0, off set, O) )

196

STITUTE SHEET E 26) TC_SetUnιqueID (aCollection, 0 , off set , uniquelD) ; // set the id return err

This more sophisticated algoπfhm for determining place unique IDs attempts to compare the country fields of the Place with known places of the same name. If this does not distinguish the places, the algorithm then compares the place type, latitude and longitude, to further discriminate. Obviously many other strategies are possible and completely customizable withm this framework and this example is provided for illustration purposes only. The algoπthm for a person name, for example, would be completely different, perhaps based on age, address, employer and many other factors.

It is clear from the discussion above that a query-building interface can be constructed that through knowledge of the types hierarchy (ontology) alone, together with registration of the necessary plug-ins by the various containers, can generate the UI portions necessary to express the queries that are supported by that plug-m. A generic query-building interface, therefore, need only list the fields of the type selected for query and, once a given field is chosen as part of a query, it can display the UI necessary to specify the query. Thereafter, using plug-in functions, the query-building interface can generate the necessary query in the native language of the container involved for that field.

Referring now to Figure 2, a sample query-building user interface (UI) is shown. In this sample, the user is in the process of choosing the ontological type that he wishes to query. Note that the top few levels of one possible ontological hierarchy 210, 215, 220 are visible in the menus as the user makes his selection. A sample ontology is discussed in more detail below The UI shown is one of many possibly querying interfaces and indeed is not that used in the preferred embodiment but has been chosen because it clearly illustrates the connections between containers and queries.

Referring now to Figure 3, a sample user interface providing access to the fields within the type "country" is shown. Having selected Country from the query-building UI illustrated in Figure 2, the user may then chose any of the fields of the type country 310 on which he wishes to query. In this example, the user has picked the field 'dateEntered' 320 which is a field that was inherited by Country from the base persistent type Datum. Once the field 320 has been selected, the querying interface can determine which member of the

197

SUBSTITUTE SHEET (RULE έ6) container federation is responsible for handling that field (not shown). Through registered plug-in functions, the querying language can determine the querying operations supported for that type. In this case, since the field is a date (which, in this example, is handled by the RDBMS container), the querying environment can determine that the available query operations 330 are those appropriate to a date.

Referring now to Figure 4, a sample user interface providing access to a free format text field within the type "country" is shown. In this figure, the user has chosen a field supported by the inverted text file container. Specifically, the field "notes.sourceNotes" has been chosen (which again is inherited from Datum) and thus the available querying operators 410 (as registered by the text container) are those that are more appropriate to querying a free format text field.

Referring now to Figure 5, a sample user- interface providing access to a fixed sized text field within the type "country" is shown. In this figure, the user has chosen the field "geography .landAreaUnits" 510, which js a fixed sized text field of Country. Again, in the above illustration, this field is supported by the RDBMS container so the UI displays the querying operations 520 normally associated with text queries in a relational database.

The above discussion illustrated how container specific storage could be created from the ontology, how to query and retrieve data from individual containers in the federation, and how the user interface and the queries themselves can be generated directly from the ontology specification without requiring custom code (other than an application independent set of container plug-ins). The other aspects necessary to create a completely abstracted federated container environment relate to three issues: 1) how to distribute queries between the containers, 2) how to determine what queries are possible, and 3) how to reassemble query results returned from individual containers back into a complete record within a collection as defined by the ontology. The portion of the system of this invention that relates to defining individual containers, the querying languages that are native to them, and how to construct (both in UI terms and in functional terms) correct and meaningful queries to be sent to these containers, is hereinafter known as MitoQuest™. The portion of the system that relates to distributing (federating) queries to various containers and combining the results from those containers into a single unified whole, is hereinafter known as MitoPlex™. The federated querying^' system of this invention thus adopts a two-layer approach: the lower layer (MitoQuest™) relates to container specific querying, the upper layer (MitoPlex™) relates to

198

SUBSTITUTE SHEET (RULE έ6) 03/06518

distributing queries between containers and re-combining the results returned by them. Each will be described further below (in addition to the patent application referenced herein). ^• ^{• ■} , I - .

Each container, as a result of a container specify query, constructs and returns a hit- list of results that indicate- exactly which items match the container specific query given. Hit lists are zero terminated lists"that, in this example, are constructed from the type ET_Hit, which is defined as follows:

typedef struct ETJHit , - // list of query hits returned by a server _/ " -

OSType _system; - ' // system tag

• unslnt64 d; // local unique item ID

ETJTypelD _, ype; // type lD ^{• •} _λ - ^' iπt32 . _relevance; // relevance value 0..100

} ETJHit; .

^• • - - ^{' •}

• As can be seen, an individual hit specifies not only the globally unique ID of the item that matched, but also the specific type involved and the relevance of the hit to the query. The specific type involved may- be a descendant of the type queried since any query applied to a type is automatically applied to all its descendants since the descendants "inherit" every field of the type specified and thus can support the query given. In this embodiment, relevance is encoded as an integer number between 0 and 100 (i.e., a percentage) and its computation is a container specific matter. For example, this could be calculated by plug-in functions within the servers) associated, with the container. It should be noted that the type ET_Hit is also the parent type of all proxy types (as further discussed in the Types Patent) meaning that all proxy types contain sufficient information to obtain the full set of item data if required. ^{" '}

When constructing a multi-container query in MitoPlex™, the individual results (hit lists) are combined and re-assembled via the standard logical operators as follows:

AND - For a hit to be valid, it must occur in the hit list for the container specific - query occurring before the AND operator and also in the hit list for the container specific query that follows the AND.

OR - For a hit to be valid, it must occur in either the hit list before the operator, or the one after the operator (or both).

AND THEN - This operator has the same net effect as the AND operator but the hit- list from before the operator is passed to the container executing the query that follows the operator along with the query itself. This allows the second container to locally perform any pruning implied by the hit list passed before returning its results. This operator therefore allows control over the order of execution of queries and allows explicit optimization of performance based on anticipated results. For example if one specified a mixed container query of the form "[RDBMS.date is today] AND THEN [TEXT.text contains "military"]" it is clear that the final query can be performed far quicker than the effect of performing the two queries separately and then recombining the results since the first query pre-prunes the results to only those occurring on a single day and since the system may contain millions of distinct items where the text contains "military". For obvious reasons, this approach is considerably more efficient

AND {THEN} NOT - This operator implies that to remain valid, a hit must occur in the hit-list for the query specified before the operator but not in the hit-list for the query after the operator.

Additional logical operators allow one to specify the maximum number of hits to be returned, the required relevance for a hit to be considered, and many other parameters could also be formulated. As can be seen, the basic operations involved in the query combination process involve logical pruning operations between hit lists resulting from MitoQuest™ queries. Some of the functions provided to support these processes may be exported via a public API as follows:

Boolean DB_NextMatchInHιtLιst ( // Obtain the next match in

ETJ.it* aMatchValue , // I:Hit value to match

ETJEitList *aHιtLιst, // 10: Pointer mto hit l st ιπt32 options // I : options as for

DB_PruneK tLιst() ¹ ); // R:TRUE if match found, else FALSE

. ^' . -

Boolean DB_ -elongsIriHitList ( // Should hit be added to a hit list?

ET_Hit+ ' • aHit, // I: Candidate hit

^•'ETJϊitList aPruneList, // I: Pruning hit list, zero

ID term. , , ^' int32 options .// I:pruning options word

^'); / R:TRUE to add hit,^" FALSE otherwise

. ^{' '}

•

ETJϊitList DBJ?runeHitList (^' // prunes two hit lists

' . ET_HitList aHitList,^' // I: Input hit.lis , zero

ID terminated

. ET_HitList -.^' aPruneList, ^• // I:Pruning hit list, zero

ID term. , ^• - \

, '. ' int32 options, // I:pruning options word

^'int 2 , maxHits // I: aximum # hits tα return (or 0)

^' >; // R:Resultant hit list, 0

ID term.

-

'

In the code above, the function DB_NextMatchInHitList 0 would return the next match according to specified sorting criteria within the hit list given. The matching options are identical to those_for DB_PruneHitList(). The function DB_BelongsIπHitList() can be used to determine if a given candidate hit should be added to a hit list being built up according to the specified pruning options. This function may be used in cases where the search engine returns partial hit sets in order to avoid creating unnecessarily large hit lists only to have them later pruned. The function DB_PruneHitList() can be used to . prune/combine two hit lists according to the specified pruning options. Note that by exchanging the list that is passed as the first parameter and the list that is passed as the second parameter, it is possible to obtain all possible behaviors implied by legal combinations of the MitoPlex™ AND, OR, and NOT operators. Either r both input hit lists may be NULL which- means that this routine can be used to simply limit the maximum number of hits in a

201

hit hst or alternatively to simply sort it. In the preferred embodiment, the following pruning options are provided:

kLimitToPruneList - limit returned hits to those in prune list (same as

MitoPlex™ AND) kExclusiveOfPruneList - remove prune list from "hits' found (same as

MitoPlex™ AND NOT) kCombineWithPruneList - add the two hit lists together (default - same as MitoPlex™ OR)

The following options can be used to control sorting of the resultant hit list:

kSortByTypelD — sort resultant hit list by type ID kSortByUniquelD — sort resultant hit list by unique ID kSortByRelevance — sort resultant hit list by relevance kSortlπlncreasingOrder — Sort in increasing order

In addition to performing these logical operations on hit lists, MitoPlex™ supports the specification of registered named MitoQuest™ functions in place of explicit MitoQuest™ queries. For example, if the container on one side of an operator indicates that it can execute the named function on the other side, then the MitoPlex™ layer, instead of separately launching the named function and then combining results, can pass it to the container involved in the other query so that it may be evaluated locally. The use of these 'server- based' multi-container queries is extremely useful in tuning system performance. In the preferred embodiment of the system based on this invention, virtually all containers can locally support interpretation of any query designed for every other container (since they are all implemented on the same substrate) and thus all queries can be executed in parallel with maximum efficiency and with pruning occumng in-line within the container query process. This approach completely eliminates any overhead from the federation process. Further details of this technique are discussed in related patent applications that have been incorporated herein.

It is clear from the discussion above that the distribution of compound multi-container queries to the members of the container federation is a relatively simple process of identifying the containers involved and launching each of the queries in parallel to the server(s) that will execute it. Another optimization approach taken by the MitoPlex™ layer is to identify whether two distinct MitoQuest™ queries involved in a full MitoPlex™ query relate to the same container. In such a case, the system identifies the logic connecting the results from each of these queries (via the AND, OR, NOT etc. operators that connect them) and then attempts to re-formulate the query into another form that allows the logical combinations to instead be performed at each container. In the preferred' embodiment, the system performs this step by combining the separate queries for that container into a single larger query combined by a container supplied logical operator. The hit-list combination logic in the MitoPlex™ layer is then altered to reflect the logical re-arrangements that have occurred. Once again, all this behavior is possible by abstract logic in the MitoPlex™ layer that has no" specific dependency on any given registered container but is simply able to perform these manipulations by virtue of the plug-in functions registered for each container. These registered plug-in functions inform the MitoPlex™ and MitoQuest™ layers what functionality the container can support and how to invoke it. This approach is therefore completely open-ended and customizable to any set of containers and the functionality they support. Examples of other container functionality might be an image server that supports such querying behaviors as 'looks like', a sound speech server with querying operations such - as 'sounds like', a map server with standard GIS operations, etc. All of these can be integrated and queried in a coordinated manner through the system described herein.

The next issue to address is the manner in which the present invention auto-generates . and handles the user interface necessary to display and interact with the information defined jn the ontology. At the lowest level, all compound structures eventually resolve into a set of simple building-block types that are supported by the underlying machine architecture. The same is true of any type defined as part of an ontology and so the first requirement for auto- generating user interface based on ontological specifications is a GUI framework with a set of - 'controls' that can be used to represent the various low level building blocks,. This is not difficult to achieve with any modem GUI framework. The following images and descriptive text illustrate just one possible set of such basic building blocks and how they map to the low level type utiføed within the ontology: ^" ' .

Referring now to Figure 6A, an example of how a short text field or numeric field (such as those handled by the RDBMS container described above) might be displayed in a control^' group. . ^l ' Referring now to Figure 6B, one method for displaying a date in a control group is shown. In this Figure, the date is actually being shown in a control that is capable of displaying dates in multiple calendar systems. For example, the circle shown_^on the control could be displayed in yellow to mdicate the cuπent calendar is Gregorian. Referring now to Figure 6C, an example of an Islamic Hijjrah calendar bemg displayed is provided. The UI layout can be chosen to include the calendar display option, for example

Referring now to Figure 7A, the illustrated control group is an example of how one might display and interact with a persistent reference field ('#'). The text portion 705 of the grouping displays the name field of the reference, in this case 'InsuregencyAndTerrorism', while the list icon 710 allows the user to pop up a menu of the available values (see the 'members' field discussion under ETJ^PersistentRef above), and the jagged arrow icon 715 allows the user to immediately navigate to (hyperlink to) the item being referenced.

^</

Referring now to Figure 7B, 7B provides an example of one way that a collection reference field ('@@') might be displayed in an auto-generated user interface. In this case the field involved is the 'related' field within the notes field of Datum. Note also that the collection this case is hierarchical and that the data has been organized and can be navigated according to the ontology.

Referring now to Figure 8, one possible method for displaying variable sized text fields (referenced via the char @ construct) is shown. Note that in this example, automatic UI hyperlink generation has been turned on and thus any known item within the text (in this case the names of the countries) is automatically hyperlmked and can be used for navigation simply by clicking on it (illustrated as an underline). This hyperlinking capability will be discussed further in later patents but the display for that capability may be implemented in any number of ways, including the manner in which hyperlinks are displayed by web browsers.

Referring now to Figure 9, this figure illustrates the manner in which an image reference (Picture @picture) field could be displayed in an auto-generated user interface.

Many other basic building blocks are possible and each can of course be registered with the system via plug-ins in a manner very similar to that described above. In all cases, the human-readable label associated with the control group is generated automatically from the field name with which the control group is associated by use of the function ' TM_jCleanFieldNameO described in the Types Patent Because the system code that is generating and handling the user interface in this'manner has full knowledge of the type being displayed and can access the data associated with all fields within using the APIs descπbed previously, it is clear how it is also possible to automatically generate a user interface that is capable of displaying and allowing data entry of all types and fields defined in the ontology. The only drawback is the fact that user interfaces laid out in this manner may not always took 'professional' because more information is required in order to group and arrange the layout of the various elements in a way that makes sense to the user and is organized logically. The system of this invention overcomes this limitation by extracting the

- necessary additional information from the ontological type definition itself. To illustrate this behavior, a listing is provided in Appendix A that gives the pseudo-code ontological type definition for the type Country (which inherits from Entity and thereby from Datum described above) in the example ontology.

As can be seem from the listing above, the ontology creator has chosen to break down the many fields of information available for a country into a set of introductory fields followed by number of top-level sub-structures as follows:

geography - Information relating to the country's geography people - Information relating to the country's people government. - Information relating to the country's government economy - Information about the country's economy communications- Information on communications capabilities transport - Transport related information military - Information about the country's military forces medical - Medical information i education - Education related information issues - Current and past issues for the country involved

Because the code that generates the UI has access to this information, it can match the logical grouping made in the ontology.

Referring now to Figure 10, a sample screen shot of one' possible display of the Country record in the same UI layout theme descπbed above (most data omitted) is provided. L the illustrated layout the first page of the country display shows the initial fields given for country in addition to the basic fields inherited from the outermost level of the Datum definition. The user is in the process of pulling down the 'page' navigation menu 1020 which has been dynamically built to match the ontology definition for Country given above. In addition, this menu contains entries 1010 for the notes sub-field within Datum (the ancestral type) as well as entries for the fields 1030 that country inherits from its other ancestral types. In the first page, the UI layout algorithm in this example has organized the fields as two columns in order to make best use of the space available given the fields to be displayed. Since UI layout is registered with the environment, it is possible to have many different layout strategies and appearances (known as themes) and these things are configurable for each user according to user preferences.

Referring now to Figure 11 , a sample embodiment of the geography page within Country is shown. Presumably, the user has reached this page using the page navigation menu 1020 described above. In this case, the UI does not have sufficient space to display all fields of geography on a single page, so for this theme_^it has chosen to provide numbered page navigation buttons 1110, 11120, 1130 to allow the user to select the remaining portions of the geography structure content. Once again, different themes can use different strategies to handle this issue. The theme actually being shown in this example is a Macintosh OS-9 appearance and the layout algorithms associated with this theme are relatively primitive compared to others.

Referring now to Figure 12, a sample embodiment of the second sub-page of the geography page within country is shown. As shown, the natural resources collection field 1210 is displayed as a navigable list within which the user may immediately navigate to the item displayed simply by double-clicking on the relevant list row. More advanced themes in the system of this invention take additional measures to make better use of the available space and to improve the appearance of the user interface. For example, the size of the fields used to display variable sized text may be adjusted so that the fields are just large enough to hold the amount of text present for any given record. This avoids the large areas of white space that can be seen in Figure 12 and gives the appearance of a custom UI for each and every record displayed. As the window itself is resized, the UI layout is re-computed dynamically and a new appearance is established on-the-fly to make best use of the new window dimensions. Other tactics include varying the number of columns on each page ' depending on the information to be displayed, packing small numeric fields two to a column, use of disclosure tabs compact content and have it pop-up as the mouse moves over the tab concerned, etc. The possibilities are limited only by the imagination of the person registering the plug-ins. To achieve this flexibility, the UI layout essentially treats each field to be displayed as a variable sized rectangle that through a standard interface can negotiate to change size, move position or re-group itself within the UI. The code of the UI layout module allows all the UI components to compete for available UI space with the result being - the final layout for a given ontological item. Clearly the matter of handling user entry into fields and its updating to persistent storage is relatively straightforward given the complete knowledge of the field context and the environment that is available in this system.

Referring now to Figure 13, an example of one part of a high-level ontology targeted at intelligence is shown. This ontology has been chosen to facilitate the extraction of meaning from world events; it does not necessarily correspond to any functional, physical or logical breakdown chosen for other purposes. This is only an example and in no way is such ontology mandated by the system of this invention. Indeed, the very ability of the system to dynamically adapt to any user-defined ontology is one of the key benefits of the present invention. The example is given only to put some of the concepts discussed previously in context, and to illustrate the power of the ontological approach in achieving data organization for the purposes of extracting meaning and knowledge. For simplicity, much detail has been omitted. The key to developing an efficient ontology is to categorize things according to the semantics associated with a given type. Computability must be independent of any concept of a 'database' and thus it is essential that these types automatically drive (and conceal) the structure of any relational or other databases used to contain the fields described. In this way, the types can be used by any and all code without direct reliance on or knowledge of a particular implementation.

Datum 1301 — the ancestral type of all persistent storage.

Actor 1302 — actors 1302 participate in events 1303, perform actions 1305 on stages 1304 and can be observed 1306.

<

Entity 1308- Any 'unique' actorl302 that has motives and/or behaviors, i.e., that is not passive Country 131 — a country 1315 is a unique kind of me a.-organιzatιon with semantics of its own, in particular it defines the top level stage 1304 within which events 1303 occur (stages 1304 may of course be nested)

Organization 1316— an organization 1316 (probably hierarchical)

Person 1317— a.person!317

SystemUser 1325— aperson 1317 who is a user of the system

Widget 1318— an executable item (someone put it there for a purpose/motive')

Object 1309— A passive non-unique actor 1302, i.e., a thing with no inherent drives or motives

Equipment 1319- An object 1309 that performs some useful function that can be described and which by so doing increases the range of actions 1305 available to an Entιtyl3Q8.

Artifact 1320— An object 1309 that has no significant utility, but is nonetheless of value for some purpose.

Stage 1304— This is the platform or environment where events 1303 occur, often a physical location. Stages 1304 are more that just a place. The nature and history of a stage 1304 determines to a large extent the behavior and actions 1305 of the Actors 1302 within it What makes sense in one stage 1304 may not make sense in another.

Action — actions 1305 are the forces that Actors 1302 exert on each other during an event 1303. All actions 1305 act to move the actor(s) 1302 involved within a multidimensional space whose axes are the various motivations that an Entity 1308 can have (greed, power, etc.). By identifying the effect of a given type of action 1304 along these axes, and, by assigning entities 1308 'drives' along each motivational axis and strategies to achieve those drives, we can model behavior.

Observation - an observation 1306 is a measurement of something about a Datum 1301, a set of data or an event 1303. Observations 1306 come from sources 1307.

General 1310- A general observation 1301 not specifically tied to a given datum 1301. Report 1321— a report 1321 is a'(partial) description from some perspective generally relating to an .£Ve7irl303.. .

Story 1326— a news story describing an eve«t 1303. . .

Image 1327- a still image of an event 1303. - ^' • '

Sound 1329— a sound recording of an event 1303. ' • -

Video 1328- a video of an event 1303.

Map 1330— a map of an event 1303, stage 1304, or entity 1308.

Regarding 1311— an observation regarding a particular datum 1301.

Note 1322- a descriptive text note relating to the datum 1301.

CrossRef 1323— an explicit one-way cross-reference indicating some kind of named 'relationship' exists between one datum 1301 and another, preferably also specifying 'weight' of the relationship. ^"

Delta 1324— an incremental change to all or part of a datum 1301, this is how the effect of the time axis is handled (a delta 1324 of time or change in time).

Relating 1312- A bi-directional link connecting two or more data together with additional information relating to the link.

Source 1307— A source is a logical source of observations 1306 or other Data.

Feed 1313— Most'jrσurcay 1307 in the system consist of Client/Server servers that are receiving one or rπore streams oϊ observations 1306 of a given type, that is; a newswire server is a source that outputs observations 1306 of type Story. In the preferred embodiment, feed sources 1313 are set up and allowed to ran on a continuous basis.

Query 1314— sub-type of source 1307 that can be issued at any time, returning a collection of observations 1306 (or indeed any Datum 1301 derived type). The Query source type corresponds to one's normal interpretation of querying a database. Event 1303— An event is the interactions of a set of actors 1302 on a stage 1304. Events 1303 must be reconstructed or predicted from the observations 1306 that describe them. It is the ability to predict events 1303 and then to adjust actions 1305 based on motives (not shown) and strategies that characterizes an entity 1308. It is the purpose of an intelligence system to discover, analyze and predict the occuπence of events 1303 and to present those results to a decision maker in order that he can take Actions 1305. The Actions 1305 of the decision maker then become a 'feed' to the system allowing the model for his strategies to be refined and thus used to better find opportunities for the beneficial application of those strategies occurring in the data stream impinging on the system.

Once the system designer has identified the ontology that is appropriate to allow the system to understand and manipulate the information it is designed to access (in the example above - understanding world events), the next step is to identify what sources of information, published or already acquired, are available to populate the various types defined in the system ontology. From these sources and given the nature of the problem to be solved, the system designed can then define the various fields to be contained in the ontology and the logical relationships between them. This process is expressed through the C* ontology defimtion and the examples above illustrate how this is done. At the same time, awareness of the desired user interface should be considered when building an ontology via the C* specifications. The final step is to implement any ontology-specific scripts and annotations as described in the Collections Patent. Once all this is done, all that is necessary is to auto- generate all storage tables necessary for the system as described and then begin the process of mining the selected sources into the system.

Having mined the information (a very rapid process), the system designer is free to evolve this ontology as dictated by actual use and by the needs of the system users. Because such changes are automatically and instantaneously reflected throughout the system, the system is now free to rapidly evolve without any of the constraints implied by the Bermuda Triangle problem experienced in the prior art. This software environment can be rapidly changed and extended, predominantly without any need for code modification, according to requirements, and without the fear of introducing new coding errors and bugs in the process. Indeed system modification and extension in this manner is possible by relatively un-skilled (in software terms) customer staff themselves meaning that it no longer requires any involvement from the original system developer. Moreover, this sytem can, through the ontology, unify data from a wide variety of different and incompatible sources and databases into a single whole wherein the data is unified and searchable without consideration of source. These two capabilities have fer^' years been the holy grail of all software development processes, but neither has been achieve — until now.

The foregoing description of the preferred embodiments of the- invention has been presented for the purposes of illustration and description. For example, although described with respect to the C programming language, any programming language could be used to implement this invention. Additionally, the claimed system and method should not be limited to the particular API disclosed. The descriptions of the header structures should also not be limited to the embodiments described. .While the sample pseudo code provides examples of ■ the code that may be used, the plurality of implementations that could in fact be developed is nearly limitless. For these reasons, this description is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the mvention be limited not by this detailed description, but rather by the claims_, appended hereto. ^• - -

2.11

A SYSTEM AND METHOD FOR MINING DATA Inventor John Fairweather

BACKGROUND OF THE INVENTION

The data ingestion and conversion process is generally known as data mining, and the creation of robust systems to handle this problem is the subject of much research, and has spawned the creation of many specialized languages (e g., Perl) intended to make this process easier. Unfortunately, while there have been some advances, the truth of the matter is that none of these 'mining' languages really provides anything more than a string manipulation library embedded into the language syntax itself. In other words, such languages are nothing more than shorthand for the equivalent operations written as a series of calls to a powerful subroutine library. A prerequisite for any complex data processing application, specifically a system capable of processing and analyzing disparate data sources, is a system that can convert the structured, semi-structured, and un-structured information sources into their equivalent representation in the target ontology, thereby unifying all sources and allowing cross-source analysis.

For example, in a cuπent generation data-extraction script, the code involved in the extraction basically works its way through the text from beginning to end trying to recognize delimiting tokens and once having done so to extract any text within the delimiters and then assign it to the output data structure. When there is a one-to-one match between source data and target representation, this is a simple and effective strategy. As we widen the gap between the two, however, such as by introducing multipFe inconsistent sources, increasing the complexity of the source, nesting information in the source to multiple levels, cross referencing arbitrarily to other items within the source, and distributing and interspersing the information necessary to determine an output item within a source, the situation rapidly becomes completely unmanageable by this technique, and highly vulnerable-to the slightest change^'in source format or target data model. This mismatch is at the heart of all problems involving the need for multiple different systems to intercommunicate mearύngful information, and makes conventional attempts to mine such information prohibitively expensive to create and maintain. Unfortunately for conventional mining techniques, much of the most valuable information that might be used to create truly intelligent systems comes from publishers of various types. Publishing houses make their money from the infonnation that they aggregate, and thus are not in the least bit interested in making such information

212

APPENDIX 4 03/065180

available in a form that is susceptible to standard data mining techniques. Furthermore, most publishers deliberately introduce inconsistencies and eπors into their data in order both to detect intellectual property rights violations by others, and to make automated extraction as difficult as possible. Each publisher, and indeed each title from any given publisher, uses different formats; and has an aπangement that is custom tailored to the needs of whatever the publication is. The result is that we are faced with a variety of source formats on CD-ROMs, databases, web sites, and other legacy systems that completely stymie standard techniques for acquisition and integration. Very few truly useful sources are available in a nice neat tagged form such as XML and thus to rely on markup languages such as XML to aid in data extraction is a woefully inadequate approach in real-world situations.

One of me basic problems that makes the extraction process difficult is that the control-flow based program that is doing the extraction has no connection to the data itself (which is simply input) and must therefore invest huge amounts of effort extracting, and keeping track of its 'state' in order to know what it should do with information at any given time. What is needed, then, is a system in which the content of the data itself actually determines the order of execution of statements in the mining language and automatically keeps track of the cuπent state. In such a system, whenever an action was required of the . extraction code, the data would 'tell' it to take that action, and all of the complexity would melt away. Assuming such a system is further tied to a target system ontology, the mining - problem would become quite simple. Ideally, such a solution would tie the mining process to compiler theory, since that is most powerful formalized framework available for mapping source textual content into defined actions and state in a rigorous and extensible manner. It would also be desirable to have an interpreted language that is tied to the target. ontology (totally different from the source format), and for which the order of statement execution could be driven by source data content

SUMMARY OF INVENTION

' The system of this invention takes the data raining process to a whole new level of power and versatility by recognizing that, at the core of our past failings in this area, lies the . : fact that conventional control-flow based programming languages are simply not suited to the desired system, and must be replaced at the fundamental level a more flexible approach to- software system generation. There are two important characteristics of the present invention that help create this paradigm shift. The first is that, in the prefeπed embodiment, the system of the present invention includes a system ontology such that the types and fields of the ontology can be directly manipulated and assigned within the language without the need for explicit declarations. For example, to assign a value to a field called "notes.sourceNotes" of a type, the present invention would only require the statement "notes.sourceNotes = ". An ontology is an explicit formal specification of how to represent the objects, concepts and other entities that are assumed to exist in some area of interest and the relationships that hold among them. The second, and one of the most fundamental characteristics, is that the present invention gives up on the idea of a control-flow based programming language (i.e , one where the order of execution of statements is determined by the order of those statements within the program) in order to dramatically simplify the extraction of data from a source. In other words, the present invention represents a radical departure from all existing "control" notions in programming.

The present invention, hereinafter refeπed to as MitoMine™, is a generic data extraction capability that produces a strongly-typed ontology defined collection referencing (and cross referencing) all extracted records. The input to the mining process tends to be some form of text file delimited into a set of possibly dissimilar records. Mitomine contains parser routines and post processing functions, known as 'munchers' The parser routines can be accessed either via a batch mining process or as part of a running server process connected to a live source. Munchers can be registered on a per data-source basis in order to process the records produced, possibly writing them to an external database and/or a set of servers. The present invention embeds an inteφreted ontology based language within a compiler/interpreter (for the source format) such that the statements of the embedded language are executed as a result of the source compiler 'recognizing' a given construct within the source and extracting the coπesponding source content In this way, the execution of the statements in the embedded program will occur in a sequence that is dictated wholly by the source content. This system and method therefore make it possible to bulk extract free- form data from such sources as CD-ROMs, the web etc. and have the resultant structured data loaded into an ontology based system.

In the prefeπed embodiment, a MitoMine™ parser is defined using three basic types of information:

1) A named source-specific lexical analyzer specification ^{• '} 2) A named BNF specification for parsing the source , ', • ^•

3) A set of predefined plug-in functions capable of interpreting the source information via C** statements. ^•

Other improvements and extentions to this system will be defined herein.

215

SUBSTITUTE "ST (RUL 26) BRIEF DESCRIPTION OF THE FIGURES

[NONE]

216

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention is built upon this and, m the preferred embodiment, uses a number of other key technologies and concepts. For example, these following patent applications (which are expressly incorporated herein) disclose all the components necessary to build up a system capable of auto-generating all user interface, storage tables, and querying behaviors required in order to create a system directly from the specifications given in an ontology description language (ODL). These various building-block technologies have been previously descπbed in the following patent applications:

1 ) Appendix 1 - Memory Patent

2) Appendix 2 - Lexical Patent

3) Appendix 3 - Parser Patent

4) Appendix 4 - Types Patent

5) Appendix 5 - Collections Patent

6) Appendix 6 - Ontology Patent

In the Parser Patent, a system was described that permits execution of the statements in the embedded program in a sequence that is dictated wholly by the source content, in that the 'reverse polish' operators within that system are executed as the source parse reaches an appropriate state and, as further described in that patent, these operators are passed a plug-in hint string when invoked. In the preferred embodiment, the plug-in hint string will be the source for the interpreted ontology-based language and the plug-ins themselves will invoke an inner level parser in order to execute these statements. The Ontology Patent introduced an ontology based language that is an extension of the C language known as C*. This is the preferred ontology based language for the present invention. We will refer to the embedded form of this language as C**, the extra '*' symbol being intended to imply the additional level of indirection created by embedding the language withm a source format interpreter. The Output of a mining process will be a set of ontology defined types (see Types Patent) within a flat data-model collection (see Memory Patent and Collection Patent) suitable for instantiation to persistent storage and subsequent query and access via the ontology (see patent reference

6). ^{• ■ ■ • • •} . ^' , - . ^• . . ^• ; ^{' • ■}

In the preferred embodiment, a MitoMine™ parser is defined using three basic types of information: - ^{• •} - .^{• "} ₍

1) A named source-specific lexical analyzer specification .

2) A named BNF specification for parsing the source

3) A set of predefined plug-in functions capable of interpreting the source information via C** statements. ^' '- - -

The BNF format may be based upon any number of different BNF specifications. MitoMine™ provides the following additional built-in parser plug-ins which greatly facilitate the process of extracting unstructured data into run-time type manager records:

<@1:1> _,. . ^{' * '} . ^' .

<@1:2> • ^{• '} , - . . _, ' . ^' .

These two plug-ins delimit the start and end of an arbitrary possibly multi-lined string to be assigned to the field designated by the following call to <@1 :5:fieldPath = $>. This is the method used to extract large arbitrary text fields. The token sequence for these plug-ins is always of the form <@1:1> <l:String> <@1:2>, that is any text occurring after the appearance of the <@1:1> plug-in on the top of the parsing stack will be converted into a single string^' token (token # 1) which will be assigned on the next <@1 :5> plug-in. The arbitrary text will be terminated by the occurrence of any terminal in the language (defined in the .LEX specification) whose value is above 128. Thus the following snippet of BNF will cause the field 'pubName' to be assigned whatever text occurs between the token <PUBLICATION> and <VOLUME/ISSUE> in the input file:

<PUBLICATION> <@l:l> <l:String> <@l:2><@l:5:pubName = $> ^{• '}

In the preferred embodiment, when extracting these arbitrary text fields, all trailing and leading white space is. removed from the string before assignment, and all occurrences of • ^* LINE_FEED are removed to yield a valid text string. The fact that tokens below 128 will not ^• terminate the arbitrary text sequence is important in certain situations where a particular string is a terminal in the language and yet might also occur within such a text sequence where it should not be considered to have any special significance. All such tokens can be assigned token numbers below 128 in the .LEX specification thus ensuπng that no confusion arises. The occurrence of another <@ 1 : 1 > or a <@ 1 :4> plug-in causes any previous <I:Stπng> text accumulated to be discarded. A <@1:5> causes execution of a C** statements that generally cause extracted information to be assigned to the specified field and then clears the record of the accumulation. If a plug-in hint consisting of a decimal number follows the <@1.1> as in <@1:1:4> that number specifies the maximum number of lines of input that wύl be consumed by the plug-in (four in this example) This is a useful means to handle input where the line number or count is significant.

<@1:3>

In the preferred embodiment, the occurrence of this plug-in indicates that the extraction of a particular record initiated by the <@1:4> plug-in is complete and should be added to the collection of records extracted.

< 1 :4:typeName>

In the preferred embodiment, the occurrence of the plug-in above indicates that the extraction of a new record of the type specified by the 'typeName' string is to begin. The , "typename" will preferably match a known type manager type either defined elsewhere or within the additionally type definitions supplied as part of the parser specification.

<@ 1 :5. C ** assigπment(s)>

In the preferred embodiment, the plug-in above is used to assign values to either a field or a register. Within the assigned expression, the previously extracted field value may be referred to as '$'. Fields may be expressed as a path to sub-fields of the structure to any depth using normal type manager path notation (same as for C). As an example, the field . specifier "description[Saa].u.equιp.specifications" refers to a field within the parent structure that is withm an array of unions. The symbol 'Saa' is a register designator. There are 26*26 registers 'ϋjaa' to 'Szz¹ which may be used to hold the results of calculations necessary to compute field values. A single character register designator may also be used instead thus ' '$a' is the same as 'Saa', 'Sb' is the same as 'Sba' etc. Register names may optionally be followed by a text string (no spaces) in order to improve readability (as in Saa-mylndex) but this text string is ignored by the C** interpreter The use of registers to store extracted information and context is key to handlmg the distributed nature of information in published sources. In the example above, '$a' is being used as an index into the array of 'description' fields. To increment this index a "<@1 :5.Sa = Sa + 1>" plug-in call would be inserted in the appropriate part of the BHF (presumably after extraction of an entire 'descnption' element). All registers are initially set to zero (integer) when the parse begins, thereafter their value is entirely determined by the <@1:5> plug-ins that occur during the extraction process. If a register is assigned a real or string value, it adopts that type automatically until a value of another type is assigned to it Expressions may include calls to functions (of the form SFuncName), which provide a convenient means of processing the inputs extracted into certain data types for assignment. These functions provide capabilities comparable to the string processing libraries commonly found with older generation data mining capabilities.

When assigning values to fields, the <@1.5> plug-in performs intelligent type conversions, for example: , .

1) If the token is a <1 :String> and the field is a 'charHdl', a handle is created and assigned to the field. Similarly for a 'charPtr'. If the field is a fixed length character array, the string is copied into it. If it won't fit, a bounds error is flagged. If the field is already non-empty (regardless of type) then the <@1:5> plugin appends any new text to the end of the field value (if possible). Note that registers do not append automatically unless you use the syntax $a = Sa + "string".

2) If the field is numeric, appropriate type conversions from the extracted value occur. Range checking could be automatic. Multiple assignments may be separated by semi-colons. The full syntax supported within the 'assignment' string is defined by the system BNF language "MitoMine" (described below).

Note that because the order of commutative operator (e.g., "+") evaluation is , guaranteed to be left-to-right, multiple non-parenthesized string concatenation operations can be safely expressed as a single statement as in:

fieldname = "Hello " + SFιrstCapOnly($a) + " do you like " + $b + " V The' <@1 :5> plug-in may also be used to support limited conditional statements which may be performed using the 'if and 'ifelse' keywords.^' The effect of the 'if is to • conditionally skip the next element of the production that immediately follows the <@1:5> containing the 'if (there can be only one statement within an 'if or 'ifelse' block). For example: • . . . . ' ' . ^•

<@l:5:if(l == 0)> <@l:4:rypeName> • ^' ' .

would cause the <@1:4> plug-in to be discarded without interpretation Similarly:

- <@l:5:ifelse(l = 0)> <@l:4:typeNamel> <@l:4:typeName2>

causes execution of the second of the two <@ 1 :4> plug-ins while:

<@l:5:ifelse(0 *-> 0)> <@l:5:$a = Sa + 1; $b = 1> <@I:5:$a = $a - 1; $b = 0>

causes execution of the first block to increment Sa and assign Sb to 1.

More significantly, since it is^'possible to discard any element from ^"the production in this manner, the prudent use of conditional <@1:5> evaluation can be used to modify the recognized syntax of the language. Consider the following production: - '

^_ ^• myProduction ::= <@ 1 :5:ifelse ( $a >= 0 )> positive_prod negative_prod

In this example, the contents of register '$a' is determining which of two possible .. productions will get evaluated next. This can be a very powerful tool for solving non- context-free language ambiguities (normally intractable to this kind of parser) by _^ remembering the context in one of the registers and then resolving the problem later when it' occurs. The results of misusing this capability can be very confusing and the reader is referred to the incorporated materials of the Parser Patent for additional details. That having been said, the following simplified guidelines should help to ensure correctness:

For any production of the form:

prod ::= <@l:5:ifelse (expression)> thenClause elseClause • '

Ensure: , ^' , - , - ^'

1) FIRST(thenClause) = FIRST(elseClause) ^•

221

SUBSTITUTE SHEET-(RULE 26 2) Either both thenClause and elseClause are NULLABLE, or neither is

' ^• 3) If elseClause is not NULLABLE, and if necessary (depending on other occurences of thenClause), . ,.

include a production elsewhere {that may never be executed} to ensure that • ^{'• '} FOLLOW(thenClause) includes FOLLOW(elseClause)

' For any production of the form: . , ,

• prod ::= prevElement <@l:5:if(expression)> thenClause nextElement

Ensure that if thenClause is not NULLABLE, and if necessary (depending on other i ^• occurences of nextElement), include a production elsewhere {that may never be - • ' executed} to ensure that FTRST(nextElement) is entirely contamed within ^"• FOLLOW(prevElement). ^'- . . I ^' , ^', '

Note that all plug-ins may contain multiple lines of text by use of the <cont> symbol ^' (see Parser patent). This may be required in the case where a <@I:5> statement exceeds the ' space available on a single line (e.g., many parameters to a function). The maximum size of any given plug-in text in the preferred embodiment is 8KB. <

The present invention also permits the specification of the language specific parser to include any user dialogs and warnings that might be required for the parser concerned, any additional type definitions that might be required as part of parser operation, and^' ny custom annotations and scripts (see Collections Patent) that might be necessary. '• -

Within the <@1:5> plug-in, in addition to supporting conditionals, additive, multiplicative and assignment operators, this package preferably provides a number of built-. in functions that may be useful in manipulating extracted values in order to convert them to a form suitable for assignment to typed fields. These functions are loosely equivalent to the string processing library of conventional mining languages. Function handlers may be J registered (via a registry API - see Parser Patent for further details) to provide additional ^' built in functions. In the built-in function descriptions below, for example, the type of a given parameter is indicated between square brackets. The meaning of these symbols in this example is as follows: ■ ^'■ . ■ ^{• ' ■ '} ' ^' [I] ^{• '} - Integer value (64 bit) ^{• '} ' ^' . - ^',^• - ' : -

fp] ^' • ] - Floating point value (double) -., • ^' ,

[S] ; ' - String value ^• _r ■

The following is a partial list of predefined built-in functions that have been found to be ' useful in different data mining situations. New functions may be added to this list and it is expected that use of the system will often include the step of adding new functions. In such a case, if a feature is not provided, it can be implemented and registered as part of any • particular parser definition. On the other hand, none of the features listed below are required meaning that a much smaller set of functions could also be used. In the preferred embodiment, however, the following functions (or ones having similar functionality) would be available. , . • , ^' ■ '-' '^" , ^•

1) (Fj SDateO . ■ _, . - . ^' , ~

get current date/time into a date-double ^" - ^{• '}

2) [F] SStringToDate([S] dateString,[S] calendar)

convert "dateString" to date/time double, current date if date string format invalid. The currently supported calendar values are "G" - Gregorian, "J" - . Julian etc. Note that in the Gregorian calendar you may specify the date string - - in a wide variety of formats, in any otheτ calendar it must be in the following format: "yyyy:mm:dd [ hh:mrn[:ss] [AM/PM]]" . ^' . .

3) [S] STeχtAfter([S] srcStrJS] delimStr)

Return the string portion after the specified delimiter sequence. Returns "" if. - ^' not found. ^"

4) [S] STextBefore([S] srcStr,(S] delimStr)

' Return the string portion before the specified delimiter sequence. Returns "" if . ^■ . not-found. ^' . - - ,

5) [S] STextBetweea([S] srcStr,(S] startScr,[S] endStr)

223

. . - Return the string portion between the specified delimiter sequences. Returns "" - -. . ^' . _t • if not found. .^' • ^' • ^" ' ^% ' • . •

.6) [I] Slnteger S] aString) ^{" " '}. ^' . ^■; ^{' "}'

Convert the specified string to an integer (decimal or hex)

7) [F] SReaI([S] aString) . ' ^• \ ' ; -

Convert the specified string to a real number , •-. " ^"

6) [I] SIntegerWithin([S] aString,rrj n)

Extract the n'th integer (decimal or hex, n = 1...) within the specified arbitrary . ' . , string ^■ , ' ,

7) [F] SRealWithin([S] aString,[I] n) ^{' '} . ^' ' . .

Extract the n'th real (n = 1...) within the specified arbitrary s.tring

8) [S] SStripMarkupflS] aString) ,

Strip any Markup language tags out of a string to yield plain text.

-9) [S] SSourceNameO

• Inserts the current value of languageName'

10) [S] $SetPersRefInfo([S] aString) >

This function allows you to append to the contents of the 'stringH' field of a . - persistent reference field rather than assigning to the name. The function

■ ^' result is equal to 'aString' but the next assignment made by the parser will be - to the 'stringH' sub-field, not the 'name' sub-field.

11) [S] SFtrstCapOπly([S] aString) '

Converts a series of words in upper/lower case such that each word starts with an upper case character and all subsequent characters are lower case.

12) [S] STextNotAfterdS] srcStr,[S] delimStr) Similar in operation to STextBeforeO except if 'delimStr' is not found, the original string is returned un-altered.

13) [S] $TextNotBefore([S] srcStr,[S] delimStr)

Similar in operation to STextAfterO except if 'delimStr' is not found, the original string is returned un-altered.

14) [S] $TextNotBetween([Sl srcStr,[S] startStr,[S] endStr)

Returns what remains after removing the string portion between the specified delimiter sequences (and the delimiter sequences themselves). If the sequence is not found, the original string is returned un-altered.

15) [S] $TruπcateText([S] srcStr,[I] numChars)

Truncated the source string to the specified number of characters

16) [S] $TextBeforeNumber([S] srcStr)

, This function is similar in operation to STextBeforeO but the 'delimStr' is taken to be the first numeric digit encountered.

17) [S] STextWithout([S] srcStr,[S] sequence)

This function removes all occurrences of the specified sequence from the - source string.

18) [S] SWordNumber([S] srcStr,[I] number)

This function gets the specified word (starting from 1) from the source string. If 'number¹ is negative, the function counts backward from the last word in the source string.

19) [S] SAsk([S] promptStr)

This function prompts the user using the specified string and allows him to enter a textual response which is returned as the function result.

20) [S] STextWithoutBIock([S] srcStr,[S] startDelim^' SI endDeum) ' This function removes all occurences of the delimited text block (including , ^"• delimiters) from the source string. ^{• "} « ^", ^' . .

21) [S] SRepIaceSequence([S] srcStr,[S] sequence, [S] nuSequence)

This function replaces all occurences of the target sequence by the sequence 'nuSequence' within the given string. - - ^*

22) [S] SAppeπdlfNotPresentQS] srcStr,[S] endDeUm)

-This function determines if 'srcStr¹ ends in 'endDelim' and if not appends . 'endDelim'- to 'srcStr¹ returning the result. .

23) [S] SProperNameFilter([SI srcStr,(I] wordMax,[S] delim)

This function performs the following processing (in order) designed to facilitate the removal of extraneous strings of text from 'delim' separated lists of proper - names (i.e., capitalized first letter words):

' a) if the first non-white character in ^"a 'delim' bounded block is not upper - case, remove the entire string up to and including the trailing occurence of • 'delim' (or end of string). ,^"

b) for any 'delim'. bounded block, strip off all trailing words that start with lower case letters.

,c) if more than 'wordMax' words beginning with a lower case letter occur ' consecutively between two occurrences of 'delim', terminate the block at the point where the consecutive words occur.

24) [S] $Sprintf([S] formatStr,...)

This function performs a C language sprintfO function, returning the generated string as its result.

25) [S] SShiftChars([Sl srcStr,[I] delta) ' ^'

This function shifts the character encoding of all elements of 'srcStr' by the amount designated in 'delta' returning the shifted string as a result This functionality^'can be useful for example when converting between upper and ^' lowercase. . ' . . , - • - - ^•

26) [S] SFlipChars([S] srcStr) ^{' ' *}

- . This function reverses the order of all characters in 'srcStr¹.

27) [S]^' SReplaceBlockDelimstfS] srcStr,[S] startDelim,^~[S] endDelim,[S] nuStartDelim,[S] nuEndDelimjfl] occurence, [I] reverse) ^'

This function replaces the start and end delimiters of one or more delimited ' blocks of text by the new delimiters specified. If 'occurence' is zero, all blocks found are processed, otherwise just me block specified (starting from 1). If 'reverse' is non-zero (i.e.,1), this function first locates the ending delimiter and ^{■ '} . then works^' backwards looking for^' the start delimiter. Often if the start delimiter is something common like a space character (e.g., looking for the last word of a sentence), the results of this may be quite different from those obtained using 'reverse' = 0. , , - .

28) [S] SRemoveIfFollows([S] srcStr,[S] endDelim)

^' ' This function determines if 'srcStr* ends in 'endDelim' and if so removes 'endDelim' from 'srcStr¹ returning the result.

29) [S] SRemoveIfStarts([S] srcStr,[S] startDelim) ,

This function determines if 'srcStr' starts with 'startDelim' and if so removes 'startDelim' from 'srcStr¹ returning the result.

30) [S] $PrependKNotPresent([S] srcStr, [S] startDelim)

This function determines if 'srcS r' starts with 'startDelim' and if not prepends ^' . 'startDelim' to 'srcStr¹ returning the result

31) [S] SNoLowerCaseWords([S] srcStr)

This function eliminates all words beginning with lower case letters from 'srcStr¹ returning the result. 32) [S] SRepIaceBIocks([S] srcStr,[S] startDelim, [S] eαdDeHm,(TJ occurence,[S] nuSequence) ,

" ^' This function replaces one or all blocks delimited by the specified delimiter sequences with the replacement sequence specified. If 'occurrence' is zero, all

' blocks are replaced, otherwise the occurrence is a one-based index to the block ', ^• - - .. -

- to replace. ' ^• , ' -

33) [S] SAppeπdIfNotFolIows([S] srcStr,(S] endDelim) ^'

This function determines if 'srcStr¹ ends in 'endDelim' and if not appends - ^' 'endDelim' to 'srcStr' remrning the result.

.34) [I] SWordCounttfS] srcStr) . ^' - ^■ "

. ^• This function counts the number of words in the source string, returning the ^' numeric result ^' .

35) [S] SPreserveParagraphs([S] srcStr) - - ^{' " '}. - ^' - .

. This function eliminates all line termination characters (replacing them by

- spaces) in the source string other than those that represent paragraph breaks. • Source text has often been formatted to fit into a fixed page width (e.g., 80 characters) arid since we wish the captured text to re-size to fit whatever display area is used, it is often necessary to^" eliminate the explicit line , ^' formatting from large chunks of text using this function. A paragraph is identified by a line termination immediately followed by a tab or space character (also works with spaces for right justified scripts), all other explicit - line foπnatting is eliminated. The resulting string is returned.

- 36) [I] SSrringSetJ_ndex([S] srcStr,(I] ignoreCase,(S] setStrl ...[S] setStrN)

. This function compares 'srcStr' to each of the elements in the set of possible match strings supplied, remrning the index (starting from 1) of the match ^' -. • ; string found, or zero if no match is found. If 'ignoreCase' is non-zero, the comparisons are case insensitive, otherwise they are exact.

37) [S] SlndexStringSetQIj iπdex, S] setStrl ...[S] setStrN). This function selects a specific string from a given set of strings by index (1 - . ^' based), returning as a result the selected string. If the index specified is out of range, an empty string is returned. " .

38) [S] SRεplaceChars([S] srcStr,[S] char,[S] nuChar) .

This function replaces all occurrences of 'char¹ in the string by 'nuChar¹ returning the modified string as a result.

39) [S] $Seπtence([S] srcStr,rj] index) ^' .' _, .

This function extracts the designated sentence (indexing starts from 0) from . . - the string, returning as a result the sentence. If the index specified is negative, the index counts backwards from the end (i.e., -1 is the last sentence etc.). A • sentence is identified by any sequence of text terminated by a period.

40) [S] SFindHyperlinkQS] srcStr,[S] domain, [I] index) -

This function will extract the iπdex'th hyperlink in the hyperlink domain • . specified by 'domain' that exists in 'srcStr¹ (if any) and return as a result the extracted hyperlink name. This technique can be used to recognize known things (e.g., city or people names) in an arbitrary block of text If no matching " hyperlink is found, the function result will be an empty string.

41) [S] SAssignRefType([S] aString) • , ' -

This function allows you to assign directly to the typelD sub-field of a -^' persistent reference field rather than assigning to the name. The function r result is equal to 'aString' but the next assignment made by the parser will be to the typelD sub-field 'aString' is assumed to be a valid type name), not the 'name' sub-field.

42) (I] SRecordCountO

, This function returns the number of records created so far ' during the current mining process. ^' ' '

43) [S] SExitQS] aReasoπ) ^' ' .. Calling this function causes the current parsing run to exit cleanly, possibly - ' '" displaying a reason for the exit (to the console) as specified in the 'aReason' string (NULL if no reason given). ^■, • ^' - ^"

44) p] SMaxRecordsO ^' . ^■ " ^{' '} ■ ^'

This function returns the maximum number of records to be extracted for this - run. This value can either be set by calling SSetMaxRecordsO or it may be set • by external code calling MN_SetMaxRecordsO-

45) [I] $SetMaxRecords([I] max)

This function sets the maximum number of records to be extracted for this.run. .- See SMaxRecordsO for details. ^'

46) p] SFieldSize([S] fieldName) . ..

This function returns the size in bytes of the field specified in the currently active type record as set by the preceeding <@1 :4:typeName> operator. - • ^■ Remember that variable sized string fields (i.e., char @fieldName) and similar will return a size of sizeof(Ptr), not the size of the string within, it.

47) [I] STextContains([S] srcText,[S] substring)

This function returns 0 if the 'srcText' does not contain 'substring', otherwise it . - ^" returns the character index within 'srcTexf where 'substring' starts + 1.

48) [I] $ZapRegisters([S] minReg,[S] maxReg)

This function empties the contents of all registers starting from 'minReg' and • - ending on 'maxReg¹. The parameters are simply the string equivalent of the register name (e.g., "Saa"). When processing multiple records, the use of SZapRegistersO is often more convenient than explicit register assignments to ensure that all the desired registers start out empty as record processing begins. The result is the count of the number of non-empty registers that were zapped.

49) P] SCRCString([S].srcText) This function performs a 32-bit CRC similar to ANSI X3.66 on the text string supplied, returning the integer CRC result. This is can.be useful when you _ want to -turn an arbitrary (i.e., npn-alphanumeric) string into a form that is (probably!) unique for name generating or discriminating purposes. _, '

Note that parameters to routines may be either constants (of integer, real or string type), field specifiers referring to fields within the cuπent record being extracted, registers, S (the currently extracted field value), or evaluated expressions which may include embedded calls to other functions (built-in or otherwise). This essentially creates a complete programming language for the extraction of data into typed stractures and collections. The C** programming ^"language provided by the <@ 1 :5> plug-ins differs from a conventional programming language in that the order of execution of the statements is determined by the BNF for the language and the contents of the data file being parsed. In the preferred embodiment, the MitoMine™ parser is capable of recognizing and evaluating the following token types:

- 3 - Declnt — syntax as for a C strtouIO call but ignores embedded commas.

' 4 - Real — real - as for C strtodO ^' -

5 - Real — real scientific format - as for C strtodO

The plug-in 5 MitoMine™ parser, in addition to recognizing registers, $, Sfunction names, and type field specifications, can also preferably recognize and assign the following token types: ^r ,

2 - character constant (as for C)

7 - Hex integer (C format)

3 - decimal integer (as for C strtoul)

10 - octal integer (as for strtoul) ' ^' ;

- ^" J 4 - real (as for strtod) .

5 - real with exponent (as for strtod) -

12 - string constant (as for C)

231 ^' . . ^{' '} . ^' Character constants can be a maximum of 8 characters long, during input, they are not sign extended. The followmg custom parser options would preferably be supported:

kTraceAssignments (0x00010000) - Produces a trace of all <@1'5> assignments on the console <

kpLineTrace (0x00020000) - Produces a line trace on the console

kTraceTokens (0x00040000) - Produces a trace of each token recognized

These options may be specified for a given parser language by adding the coπesponding hex value to the parser options line. For example, the specification below would set kTraceAssignments+kpLineTrace options in addition to those supported by the basic parse package"

^' = 0x30000+kPreserveBNFsymbols+kBeGreedyParser

The lexical analyzer options line can also be used to specify additional white-space and delimiter characters to the lexical analyzer as a comma separated list. For example the specification below would cause the characters 'a' and V to be treated as whitespace (see LX_AddWhiteSpace) and the characters Υ and 'Z' to be treated as delimiters (see LX_AddDelimiter).

= kNoCaseStates+whιtespace(a,b)+delimiter(Y_rZ)

Appendix A provides a sample of the BNF and LEX specifications that define the syntax of the <@ 1.5> plug-in (i.e., C* *) within MitoMine™ (see Parser Patent for further details). Note that most of the functionality of C** is already provided by the predefined plug-in functions (plug-in 0) supplied by the basic parser package. A sample implementation of the <@1:5> plug-in one and a sample implementation of a corresponding resolver function are also provided.

As described previously, the lexical and BNF specifications for the outermost parser vary depending on the source being processed (example given below), however the outer parser also has a single standard plug-in and resolver. A sample implementation of the standard plug-in one and a sample implementation of a coπesponding resolver function are also provided in Appendix A. The listing below gives the API interface to the MitoMine™ capability for the preferred embodiment although other forms are obviously possible. Appendix A provides the sample pseudo code for the API interface.

In the prefeπed embodiment, a function, hereinafter called MNJVlakeParserO, initializes an instance of the MitoMine™ and returns a handle to the parseT database which is required by all subsequent calls A 'parserType' parameter could be provided to select a particular parsing language to be loaded (see PS_LoadBNF) and used.^"

In the prefeπed embodiment, a function, hereinafter called MN_SetRecordAdderO determines how (or if) records once parsed are added to the collection. The default record adder creates a set of named lists where each list is named after the record type it contains.

In the prefeπed embodiment a function, hereinafter called MN_SetMineFunc , sets the custom mine function handler for a MitoMine™ parser. Additional functions could also ₍ be defined over and above those provided by MitoMine™ within the <@l:5:...> plugin context. A sample mine function handler follows:

static Boolean myFunc // custom function handler ETJ?arseHdl aParseDB, //IO:handle to parser DB int32 aContex ID //I: context

) // .TRUE for success p a (myContextPtr)aContextΙD; // get our context pointer αpCount =» PS_GetOpCouπt (aParseDB, TOP) ; // get # of operands tokp ≠ PS_GetTokeπ (aParseDB, opCount) ; // get fn name for ( i = 0 ; 1 < opCount , ι++ ) if ( IPS Evalldent (aParseDB, i) ) // eval all elements on stack { res = NO7 goto BadΞxit;

} if ( lUΞ_strcm ( ok , "$myFuncName" ) ) // function name {

-- check operand count and type -- implement function

-- set resulting value into stack OpCount¹ e.g.: PS_Set Value (aParseDB , opCoun , result) ,- } else f ( !US_strcmp(tokp, "$another function") )

In the prefeπed embodiment, a function, hereinafter called MJN_SetMaxRecordsθ₅ sets the maximum number of records to be mined for a MitoMine™ parser. This is the number returned by the built-in function SGetMaxRecordsO- If the maximum number of records is not set (i.e., is^'zero), all records are mined until the input file(s) is exhausted. In the prefeπed embodiment, a function, hereinafter called MN_SetMineLineFnOj sets the MitoMine™ line processing function for a given MitoMine™ parser. A typical line processing function might appear as follows:

static void myLmeFn ( // Built-in debugging mine-line fn

ET ParseHdl "^" aParseDB, // Parser DB int32 aContextLD, // 1 Context int32 lineNum, // LCuπent line number charPtr lineBuff, // IO- Cuπent line buffer charPtr aMineLineParam // String parameter to function

) // R:void

These functions can be used to perform all kinds of different useful functions such as altering the input stream before the parse sees it, adjusting parser debugging settings, etc. The 'aMineLineParam' parameter above is an arbitrary string and can be formatted any way you wish in order to transfer the necessary information to the line processing function. The current value of this parameter is set using MN_SetMineLineParamO.

In the prefeπed embodiment, a function, hereinafter called MNjSetMineLinePararnOj sets the string parameter to a MitoMine™ line processing function.

In the prefeπed embodiment, two functions, hereinafter calleds MN_SetParseTypeDB and MN_GetParseTypeDBO, can be used to associate a type DB (probably obtained using MN_GetMineLanguageTypeDB) with a MitoMine™ parser. This is preferable so that the plug-ins associated with the extraction process can determine type information for the structures unique to the language. In the prefeπed embodiment, the function MN_GetParseTypeDB would return the cuπent setting of the parser type DB.

In the prefeπed embodiment, a function, hereinafter called MNjSetFilePathO, sets the cuπent file path associated with a MitoMine™ parser.

In the prefeπed embodiment, a function, hereinafter called MN_GetFileP thO, gets the cuπent file path associated with a MitoMine™ parser.

Lα the prefeπed embodiment, a function, hereinafter called MNjSetCustomContextO, may be used to get the custom context value associated with a given MitoMine™ parser. Because MitoMine™ itself uses the parser context (see ' PS_SetContextID), it provides this alternative API to allow custom context to be associated ' with a parser. - ' ^• , . • - -_ ,

In the prefeπed embodiment, a function, hereinafter called ^' . .

MN_GetCustomContextO, may be used to get the custom context value associated with a given MitoMine™ parser. Because MitoMine™ itself uses the parser context (see PS_SetContextID), i provides this alternative API to allow custom context to be associated with a parser. '- -„ .^'

In the prefeπed embodiment, a function, hereinafter called • MN_GetParseCollectionO, returns the collection object associated with a parser, MN_SetParseCollectionO allows this value to be altered. By calling . MN_SetParseCoUection(...,NULL) it is possible to detach a collection from the parser in cases where you wish the collection to survive the parser teardown process. . ^■

In the prefeπed embodiment a function, hereinafter called N_SetParseCollectionOj returns the collection object associated with a parser. MN_SetParseCollectionO allows this value to be altered. By calling - - MN_SetParseCollection(...,NULL) it is possible to detach a collection from the parser. This would be useful in cases where it is preferable to permit the collection to survive the parser teardown process. ■, ^{■ '} •

In the prefeπed embodiment a function, hereinafter called MN_GetMmeLanguageTypeDBO, returns a typeDB handle to the type DB describing the structures utilized by the specified mine language. • If the specified typeDB already exists, it is simply returned, otherwise a new type DB is created by loading the type definitions from the designated MitoMine™ type specification file. - .

In the prefeπed embodiment, a function, hereinafter called MNJKilLParserO, . disposes of the Parser database created by MN_MakeParser - A matching call to

MN_KillParser must exist for every call to MN_MakeParserO. This call would also invoke MN_CleanupRecordsO for the associated collection. . '

In the prefeπed embodiment, a function, hereinafter called MNJParseO, invokes the MitoMine™ parser to process the designated file. The function is passed a parser database created by a call to MNJMakeParserO- When all calls to MN_ParseO are complete, the parser database must be disposed using MN CillParserO-

In the prefeπed embodiment, a function, hereinafter called MNJ uπMitoMineO, creates the selected MitoMine™ parser on the contents of a string handle. An parameter could also be passed to the MNJVfakeParserO call and can thus be used to specify vaπous debugging options.

In the prefeπed embodiment, a function, hereinafter called MN_CleanupRecordsO, cleans up all memory associated with the set of data records created by a call to MN_Rur-MitoMineO.

In the prefeπed embodiment, a function,, hereinafter called MN egisterMineMuncherO, can be used to register by name a function to be invoked to post process the set of records created after a successfuU MitoMine™ run. The name of the registered Muncher function would preferably match that of the miαing language (see MNJParse for details). A typical mine-muncher function might appear as follows:

static ET_CollectιonHdl myMuncher( // My Mine Muncher function

ET_MineScanRecPtr scanP, // IO:Scanning context record

ET_CollectionHdl theRecords, // ^Collection of parsed records char typeDBcode, // LThe typeDB code charPtr parserType, // LThe parser type language name

ET_Offset root, // Root element designator charPtr customString // LAvail pass cstm strig to muncher

) // R:The final collection

The 'scanP' parameter is the same 'scanP' passed to the file filter function and can thus be used to communicate between file filters and the muncher or alternatively to clean up any leftovers from the file filters within the 'muncher'. Custom 'muncher' functions can be used to perform a wide variety of complex tasks, bdeed the MitoMine™ approach has been used successfully to extract binary (non-textual) information from very complex sources, such as encoded database files, by using this technique.

In the prefeπed embodiment, a function, hereinafter called MN_DeRegisterMineMuαcher0₅ de-registers a previously registered mine muncher function.

236

SUBSTITUTk SHEET (RULE 26) - . In the prefeπed embodiment, a function, hereinafter called ^' • . ' ^:,

-MN_InvokeMineMuncher , invokes the registered 'muncher" function for the records output by a run of MitoMine (see MN_RuπMitoMine). If no function is. registered, the records and all associated memory are simply disposed using MN_CleanupRecordsO- -

L the prefeπed embodiment, a function, hereinafter called MN_RegisterFileFilterO, can be used to register by name a file filter function to be invoked to process files during a MitoMine™ run. If no file filter is registered, files are treated as straight text files, otherwise the file must be loaded and pre/post processed by the file filter. A typical file filter function might appear as follows: - - .

static EngEπ myFileFilter ( // Scan files and mine if appropr

HFilelnfo ^v *aCata!ogRec, . // IO:The catalog search record

- ' . . int32Ptr ' flags, // IO:available for flag use

ET MineScanRecPtr scanP // IO:Scanning context record

) . . // R:zero for success, else eπor #

In the prefeπed embodiment, a function, hereinafter called ^' MN_ istFileFiltersO-obtams a string list of all know MitoMine™ file filter functions.

In order to illustrate how MitoMine™ is used to extract information from a given source and map it into its ontological equivalent, we will use the example of the ontological definition of the Country record pulled from the CIA World Fact book. The extract provided in Appendix B is a portion of the first record of data for the country Afganistan taken from the 1998 edition of this CD-ROM. The. format of the information in this case appears to be a variant of SGML, but it is clear that this approach applies equally to almost any input format. The lexical analyzer and BNF specification for the parser to extract this source into a sample ontology are also provided in Appendix B. The BNF necessary to extract country information into a sample ontology is one of the most complex scripts thus far encountered in MitoMine™ applications due to the large amount of information that is being extracted from this source and preserved in the ontology. Because this script is so complex, it probably best illustrates a less than ideal data-mining scenario but also demonstrates use of a large number of different built-in mining functions. Some of the results of running the extraction script below can be seen in the Ontology patent relating to auto-generated UI. -

Note that in the BNF provided in Appendix B, a number of distinct ontological items are created, not just a country. The BNF starts out by creating a "Publication" record that identifies the source of the data injested, it also creates a "Government" record, which is descended from Organization. The Government record is associated with the country and forms the top level of the description of the government/organization of that country (of which the military branches created later are a part). In addition, other records could be created and associated with the country, for example the "opt_figure" production is assigning^' ^• a variety of information to the 'stringH' field of the "maplmage" field that describes a persistent reference to the file that contains the map image. When the data produced by this

• parse is written to persistent storage, this image file is also copie to the image server and through the link created, can be recalled and displayed whenever the country is displayed (as is further demonstrated in the UI examples of the Ontology Patent). In fact, as a result of extracting a single country record, perhaps 50-100 records of different types are created by this script and associated in some way with the country including government personel, international organizations, resources, poulation records, images, cities and ports, neighboring countries, treaties, notes, etc. Thus it is clear that what was flat, un-related information in the source has been converted to richly interconnected, highly computable and usable ontological information after the extraction completes. This same behavior is repeated for all the diverse sources that are mined into any given system the information from all such sources becomes cross-correlated and therefore infinitely more useful that it was in its separate, isolated form. The power of this approach over conventional data mining technologies is clear. . -

The foregoing description of the prefeπed embodiments of the invention has been presented for the purposes of illustration and description. For example, although described with respect to the C* programming language, any programming language that includes the appropriate extentions could be used to implement this invention. Additionally, the claimed ^■ ^• system and method should not be limited to the particular API disclosed. The descriptions of the header structures should also not be limited to the embodiments described. While the sample pseudo code provides examples of the code that may be used, the plurality of

' - implementations that could in fact be developed is nearly limitless. For these reasons, this description is not intended to be exhaustive or to limit the invention to the precise form '. disclosed. Many modifications and variations are possible in light of the above teaching. It . is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

Claims

. . ^{' ' '}- CLAIMS ^' . ; ^■ \ ^'. . ^' .. •^'" '

1) A large scale distributed multimedia system comprising the following: - - - . ' '

. a) One or more server(s), wherein each server includes i) a mam server thread; ii) one or more threads capable of monitoring ^'incoming data feeds and instantiating the resultant . data into the system (monitoring threads), and iii) one or more threads capable of handling incoming data requests (request threads); and iv) cache memory.

■ b) logically connected to such servers), a mass storage system (MSS), wherein such MSS . stores information concerning the arrangement of one or more server(s) and is capable of controlling one or more robotic autoloader systems (robots);

c) logically connected to such server(s), a types system for defining data types at a binary level and for associating those data types with one or more server(s) such that the mapping maps data types with the servers that must be addressed to obtain the coπesponding data; and. ^' ' .. . . . , ■

d) logically connected to such server(s), a query system for executing data queries on servers ' . ^' mapped to the data type being queried. ^' • _

2) The system of claim 1, further comprising an output folder logically connected to such server, wherein such folder stores the multimedia portion of any data feeds acquired by the server.

3) The system of claim 2, wherein such output folder stores batches of information to be migrated to the mass storage system. ^{' •} -

4) The system of claim 1, wherein each server includes the same core software.

5) The system of claim 5, wherein each server is configured at run-time to' support different functionality. ^' ,

6) The system of claim 6, wherein each server is customized using registered plug-ins.

7) The system of claim 1, wherein at least one server has a main server and a drone server that ^: transparently operate as part ,of a single logical server cluster.

8) The system of claim 8, wherein the drone server is tasked by the main server.

9) The system of claim 8, wherein at least one server has two drone servers, wherein one drone server is tasked by other drone server. ^{• •} - 10) The system of claim 8, wherein" toe'logicaT'servei* -cluster !^'pfiys-ιcatιy-^I,Qistrιi3tι^!te '^,-&ver a-wiβe-" --¹ area. • . . ' ^'; ' • • . ' ^• ;

11) The system of claim 8, wherein each server in the logical server cluster communicates with other servers (including other servers in the logical server cluster) using standard communication protocol, such as TCP/IP. ^• ■ . . . . .

12) The system of claim -1, wherein the MSS stores information concerning the logical aπangement of one or more server(s). .^' . .■ ^{' '}. . • ^' - • ■

13) The system of claim 1, wherein the MSS stores information concerning the physical location of one or more servers). • ' ^• ' .' _'■ ^•

14) The system of claim 1, wherein the MSS is capable of fulfilling one or more information requests from a server by directing a robot to mount the drive that stores the coπesponding information. ^■

15) The system of claim 1, wherein the MSS includes logical operations that enable migration of data from cache based stored to storage managed by one or more robots. . ^•' . ' .

16) The system of claim^' 16, wherein the MSS includes definitions for drivers of one or more devices,

• such definitions mapping the logical operations provided by the MSS to one or more commands [ necessary to implement such logical instructions for a given device.

17) The system of claim 17, wherein the MSS includes driver definitions that map the logical operations provided by the MSS to one or more logical operations provided by the device.

18) The system of claim 1, wherein the types system is the types system described in the Types Patent. . •^{' •} ' . . - - ^{" '}

19) The system of claim 1, wherein the query system includes the ontological framework described in - the Ontology Patent. . ^" . -

20) The system of claim 1, the system further comprising one or more client machines that are logically connected to one or more server(s) and that use this logical connection to submit one or . more query(s) to the query system. . ^{' '}

21) The system of claim 21, wherein the query system is capable of distributing portions of a query received. from a client machine. (each a distributed query) to one or more server(s) based on the functionality supported by such server(s). . • 22) The system of claim 22, wherein the quer ^''sys eirT're s'semBle^'s ϊhe^' esu^'lts^Eό^'f^'sucrf^Jdisti ^te^!a query(s) into a unified list. , --,' - ^• - . ^• . ^' . ^' , ^' •

23) The system of claim 23, wherein the query system is capable of saving and executing query(s) on - a continuous basis against any new data acquired from the feed(s) being monitored by the

' monitoring threads. ₍ , • '

24) The system of 23, wherein the query(s) saved and executed by the query system includes one or more "interest profiles" coπesponding to one or more client machines such that whenever new data arrives from a feed matches the interest profile, the conesponding client is notified by the query system. , ^■ - •

25)' The system of claim 21, wherein each client machine includes server functionality such that the client machine is capable of making data available to the system. ^{' "} ' ^{" • •}

26) The system of claim 26, wherein the server functionality stored on the client machine permits any other machine on the network to discover any data type(s) server by such client machine (if any).

27) The system of claim 1, wherein at least one server enables direct access to one or more databases.

28) The system of claim 28, wherein at least one server enables direct access to one or more databases using the collections model described in the Collections Patent. -

29) The system of claim 1, further comprising an input folder, wherein data coming from one or more feed(s) that are monitored by the monitoring thread(s) are stored in such input folder in a hierarchical manner.

30) The system of claim 30, wherein the server(s) includes one or more monitoring threads that . monitors the input folder for changes.

31) The system of claim 31, wherein such monitoring threads further supports defined 'aliases' (or ^' references) to data on other machines such that, responsive to a request from a client machine, such monitoring threads will acquire such data and transfer it to the input folder for processing.

32) The system of claim 30, further comprising a rejects folder, wherein such rejects folder stores any . mcoming data that cannot be processed by the input folder. .- -

33) The system of claim 1, wherein the server(s) further includes a maintenance interface, such interface permitting registration and execution of arbitrary maintenance functions. nxc&d g

" - ' ^' . .

35) The system of claim 35, wherein the custom command and APIs registered ^'on one or more " ' server(s) are based, at least in part, on the functionality associated with such server(s).

36) The system of claim 36, wherein the query system further includes one or more registered custom commands and APIs for a given data type. . • ' . ._ ^{' •} . . .

37) The system^' of claim 2, wherein the query system is capable of searching and retrieving data stored on a robot using customizable aliases stored in the output folder. ^'

38) The system of claim 8, wherein such_.server(s) further include one or more routing algorithms to . route commands and responses to/from one or more machines that make up a logical server

^■ cluster. ^' . ^•■ •. . , ,