WO2003065171A2

WO2003065171A2 - A system and method for managing dataflows

Info

Publication number: WO2003065171A2
Application number: PCT/US2003/003032
Authority: WO
Inventors: John Fairweather
Original assignee: John Fairweather
Priority date: 2002-02-01
Filing date: 2003-02-03
Publication date: 2003-08-07
Also published as: AU2003214975A1; US20040073913A1; US7555755B2; US7308449B2; US7328430B2; US7240330B2; WO2004002044A3; US20030171911A1; US20030182529A1; WO2004002044A2; US20040024720A1; US20070112714A1; US7103749B2; US7685083B2; WO2003065177A3; WO2003065173A9; AU2003269798A1; WO2003065213A1; US20040031024A1; EP1527414A2

Abstract

System and method for implementing a data-flow based system. The system includes three basic components: a data-flow based scheduling environment that balances the needs of data initiated program execution as a result of flows with other practical considerations such as user responsiveness, event driven invocation, user interface considerations, and the need to also support control-flow based paradigms where required; a visual programming language, based on the flow of strongly-typed run-time accessible data and data collections between small control-flow based locally and network distributed functional building blocks, known as widgets; and a formalized pin based interface to allow access to data-flow contents from the executing code within the widgets. The pins on the widgets include both pins used to control execution of a widget as well as pins used to receive data input from a data flow.

Description

A SYSTEM AND METHOD FOR MANAGING DATAFLOWS

BACKGROUND OF THE INVENTION

For complex systems, such as those designed for multimedia intelligence and knowledge management applications, the current 'control flow' based design methods are totally unsuitable. Once a system is broadened to include acquisition of unstructured, non-tagged, time-variant, multimedia information (much of which is designed specifically to prevent easy capture and normalization by non-recipient systems), a totally different approach is required. In this arena, many entrenched notions of information science and database methodology must be discarded to permit the problem to be addressed. We shall call systems that attempt to address this level of problem, 'Unconstrained Systems' (UCS). An unconstrained system is one in which the source(s) of data have no explicit or implicit knowledge of, or interest in, facilitating the capture and subsequent processing of that data by the system. The most significant challenges that must be resolved with the UCS is based on the following realities:

a) Change is the norm. The incoming data formats and content will change. The needs and requirements of the users of the data will also change. This will be reflected not only in their demands of the Ul to the system, but also in the data model and field set that is to be captured and stored by the system.

b) An unconstrained system usually only samples from the flow going through the information pipe. The UCS is neither the source nor the destination for that flow, but simply a monitoring station attached to the pipe capable of selectively extracting data from the pipe as it passes by.

c) In a truly unconstrained system, the information can only be monitored and the system may react to it — it cannot be controlled.

This loss of control over data is one of the most difficult challenges in the prior art. The prior art clearly suggests that software consists of a 'controlling' program that takes in inputs, performs certain predefined computations, and produces outputs. Nearly every installed system in the prior art complies with this approach. Yet it is obvious from the discussion above that this model can only hold true on a very localized level in a UCS. The flow of data through the system is really in control. It is illustrative to note that the only example of a truly massive software environment is the Internet itself. This success was achieved by defining a rigid set of protocols (IP, HTML etc.) and then allowing Darwinian-like and unplanned development of autonomous but compliant systems to develop on top of the substrate. A similar approach is required in the design of unconstrained systems.

In the traditional programming world, a programmer would begin by defining certain key algorithms and then identify all of the key inputs into the system. As such, the person or entity supplying the data is often asked to comply with very specific data input requirements impacting the format, length, field definitions, etc. The problem with this approach, however, is that predicting needed algorithms or approaches that are appropriate to solving the problem of 'understanding the world' is simply too complex. Once again, the conventional approach of defining processing and interface requirements, and then breaking down the problem into successively smaller and smaller sub-problems becomes unworkable. The most basic change that must be made, then, is to create an environment that operates according to data-flow rules, not those of a classic control-flow based system.

In spite of the prevalence of control based programming frameworks, various data-flow based software design and documentation techniques have been in usage for many years. In these techniques, the system design is broken into a number of distinct processes and the data that flows between them. This breakdown closely matches the perceptions of the actual system users/customers and thus is effective in communicating the architecture and requirements. Unfortunately, due to the lack of any suitable data-flow based substrate, even software designs created in this manner are invariably translated back into control-flow methods, or at best to message passing schemes, at implementation time. This translation begins a slippery slope that results in such software being of limited scope and largely inflexible to changes in the nature of the flow. This problem is at the root of why software systems are so expensive to create and maintain.

At the most fundamental operating system scheduling level, we need an environment where the presence of suitable data initiates program execution, not the other way round. More specifically, what is needed is a substrate through which data can flow and within which localized areas of control flow can be triggered by the presence of certain data. Additionally, such a system would ideally facilitate easy incoφoration of new plug-in control flow based functions or routines and their interface to data flowing through the data-flow based substrate so that it will be possible for the system to 'evolve'. In essence, the users, knowingly or otherwise, must teach the system how they do what they do as a side effect of expressing their needs to it. No two analysts will agree completely on the meaning of a set of data, nor will they concur on the correct approach to extracting meaning from data in the first place. Because all such perspectives and techniques may have merit, the system must allow all to co-exist side by side, and to contribute, through a formalized substrate and protocol, to the meta-analysis that is the eventual system output.

SUMMARY OF INVENTION

The present system and method provide such a system. To implement a data-flow based system, three basic components must be created and integrated:

a) A data-flow based scheduling environment that balances the needs of data initiated program execution as a result of flows with other practical considerations such as user responsiveness, event driven invocation, user interface considerations, and the need to also support control-flow based paradigms where required.

b) A visual programming language, based on the flow of strongly-typed run-time accessible data and data collections between small control-flow based locally and network distributed functional building-blocks, known henceforth as widgets.

c) A formalized pin-based interface to allow access to data-flow contents from the executing code within the widgets.

The pins on the widgets include both pins used to control execution of a widget as well as pins used to receive data input from a data flow. The system and method further include a debugging environment that enables visual debugging of one or more widgets (or collections s widgets). Data control techniques include the concepts of "OR" and "AND" consumption thereby permitting either consumption immediately or only after all widget inputs have received the token. Additional extensions to this framework will also be described that relate to the environment, the programming language and the interface.

BRIEF DESCRIPTION OF THE FIGURES

Figure 1 illustrates an example of a conditional statement.

Figure 2 illustrates an example case statement using an atomic widget.

Figure 3 illustrates a sample embodiment of a Widget Editing Mode (WEM) window 300.

Figure 4 illustrates a full-featured icon-editor that allows alteration of the widget icon.

Figure 5 illustrates a sample embodiment of a file menu 415 of the WEM.

Figure 6 illustrates a sample Add menu 417 of the WEM.

Figure 7 illustrates a sample Align menu 418 of the WEM.

Figure 8 illustrates a sample Display menu of the WEM.

Figure 9 illustrates a sample setup menu 421.

Figure 10 illustrates a sample embodiment of the debug menu 422 of the WEM.

Figure 11 illustrates a sample Pin information dialog 1100.

Figure 12 illustrates a user in the process of choosing the type 1210 of a constant symbol in the WEM diagram within the type pop-up menu 1210 of the constant information window 1200.

Figure 13 illustrates a dialog window 1300 generated in connection with selection of the view or widget information window in the setup menu 421.

Figure 14 illustrates a simple calculator widget/view 1400.

Figure 15 illustrates the structural components of the calculator widget/view 1400.

Figure 16 illustrates an internal diagram of the calculator as it would appear in the preferred Widget Editing Mode window. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The system described herein may be used in conjunction with a number of other key technologies and concepts that represent the preferred embodiments of the present invention. These various building-block technologies have been previously described in the following patents attached hereto as Appendix 1, hereinafter referred to as "Memory Patent" and Appendix 2, hereinafter referred to as "Types Patent":

As set forth above, the system is preferably comprised of the following components:

The requirements for and implementation of each of these required components will be addressed in the sections that follow. All of the structures defined and used below (widgets, pins, flows, constants, etc.) are preferably implemented within the flat memory model described in the Memory Patent and are contained within a loadable and executable memory allocation known as a 'view'. Only complete and correct views can be 'run' by the data-flow scheduler. There may be any number of different views in the system, and any number of instances of a given view.

As an initial matter, it is helpful to describe the "building blocks" involved in the present invention. A 'widget' is the fundamental building block of the system. A widget contains certain functionality encapsulated along with a definition of the inputs and outputs necessary to interface to that functionality. An atomic widget contains compiled code that generally cannot be either examined or altered within the framework of the environment. A compound widget contains an inner structure that defines any subordinate widgets that are required to implement the required functionality, together with the data flows between these contained widgets. In general compound widgets can be opened, examined, and altered by system users. Compound widgets may themselves be combined with other widgets (both atomic and compound) to yield higher-level compound widgets, to any arbitrary level of nesting. At the uppermost level, widgets are combined into 'views' that may be thought of as complete mini-applications that preferably include all necessary Ul functionality.

It is the views, and the widgets that they contain, that are loaded into the environment at execution time. Thereafter, the widgets are scheduled and executed according to the control and data flows defined in the widgets themselves. Atomic widgets can be grouped into functionally related sets known as widget packs.

In the preferred embodiment, widget packs appear and are manipulated in the Widget Editing Mode (WEM) diagram as a single unit, but each of the members of the unit can be executed independently of the other members. The principal functional member of a widget pack (i.e., the 'do it' function) is known as the formal widget, all other widgets in the pack are degenerate (known as degenerate widgets). The pack metaphor is necessary to support asynchronous access to elements or attributes of the internal state of a logical functionality implemented by the pack. Without packs, data-flow is essentially a synchronous metaphor where widgets do not run until all necessary inputs have arrived. The support of 'exclusive' pins (described later) is another exception to this rule.

Compound and atomic widget inputs and outputs as displayed during WEM, are collectively referred to as pins. A formal pin is one that must be connected in order for the widget to operate correctly, a degenerate pin need only be connected if required in the particular use of the widget and may otherwise be left unconnected. Degenerate pins come in two varieties; those that assume a default value within the widget when unconnected (defaulted degenerate pins), and those that do not (un-defaulted degenerate pins).

A View has associated with it a specialized compound widget known as a view widget which contains a collection of atomic or compound widgets, each of which may have at most one user interface region known as a pane. These regions range.from buttons, windows, controls, etc. to arbitrarily complex closed shapes. In addition to the view widget, a view contains the layout information that specifies the arrangement of display panes within the enclosing window. The entire view is enclosed in a view window. The system is capable of accessing and transferring between large numbers of different views both under menu control, and by use of suitable view change widgets. Like widgets, views may be shared between users or may be unique to a particular user. Like other widgets, view widgets may have data flow inputs and outputs, but in the case of views, these are physically mediated by network events/messages that are sent to or received from other views, either in the same workstation or another node on the network. A dataflow environment built on this metaphor is thus transparently distributed.

It is possible to execute a widget without making connections to any degenerate inputs or outputs that a widget contains; in this case the default values (if specified) will be used for the inputs, and the output(s) will be discarded. If no default is defined for a degenerate input, then within that widget, no tokens will appear from that input pin and hence any widgets connected to that input cannot become eligible for scheduling. Degenerate widgets I/O pins and the defaults associated with them can be explicitly overridden by connecting the inputs to an appropriate source/sink of the type required. Default values can be read and edited as part of the widget editing process. The interface provides a semi-automated and convenient method of resolving type conflicts and inserting the appropriate type conversion widgets. Type conversion widgets generally have many degenerate inputs and outputs, each of which will interface to a particular type. The interface is able to recognize type conversion widgets for what they are (via a dedicated flag), and when a type conflict occurs, searches all available type converters for those that meet the necessary input and output criteria. When all suitable type converters have been identified, the user is able to select the most appropriate from a list of all converters that meet the criteria.

Widget data-flow inputs and outputs can be connected to other data-flow inputs and outputs (of a compatible type) in widget editing mode (WEM) in order to define the required widget functionality. For example, a single widget data output can be connected to multiple data-flow inputs. When a multiple input connection is made to a single output, the interface allows the user to choose whether the output is consumed by the first widget that has all inputs available including the input in question (OR consumption logic), or whether it is only consumed when all connected widgets have run (AND consumption logic). Conversely, multiple widget outputs can be connected to a single widget input, in which case the input accepts and consume each widget output as it becomes available. This situation occurs commonly in user control panels where a number of buttons effect the state of a single widget/display. It is possible (though uncommon) for multiple widget outputs or sources to be connected to multiple widget inputs or sinks. This capability may be important for widget mediated load sharing across multiple server processes, for example. Every widget has the potential to accept a single control flow input and to generate a single control flow output; these pins are degenerate (i.e., are ignored unless actually connected). In the preferred embodiment, control pins cannot have defaults associated with them. Like data-flow inputs and outputs, control-flow pins can be wired up to other control flow pins, but not to data flow pins (unless of Boolean type). Control flow wiring carries an implicit Boolean value indicating that the control flow criteria concerned has or has not been met. If the control flow condition has not been met, then control flow wiring carries the value false, and does not trigger any connected control flow inputs. If the condition has been met, the wiring carries a true value and triggers any connected control flow inputs. Unless explicitly altered within a widget definition, a widget's control flow output goes true immediately upon completion of execution of that widget, it goes false immediately after execution of the widget begins. If a widget's control flow input is connected, then execution of that widget cannot begin until the control flow signal to which it is connected is asserted. Normally, it is likely that compound widgets can be constructed entirely based on data flow programming and without the explicit use of control flow pins. However, there are a number of situations, especially those involving the synchronization of multiple server processes, which may require use of the control flow pins. The system also permits a tie between multiple control flow outputs and a given control flow input in which case the associated widget cannot begin execution until all data flow inputs are satisfied and, either the AND or the OR of all control inputs is asserted (depending on the type of control input used). Selection of either a control flow OR/AND for a widget control input is generally performed when connecting control flow signals. The system also supports connections between a single widget control flow output to multiple different control flow inputs. In this case, all widgets whose inputs are so connected cannot execute until the control flow output is asserted. Unlike data-flow connections, control flow signals are not 'consumed' by the receiving input pin, but remain asserted until source widget activity drives them false. This means that, in general, control flow signals can be multiply sourced and synced without the potential for confusion as to what will happen. In the preferred embodiment, all logical operations on the control flow signals are the responsibility of the engine/interface; this knowledge does not propagate into the widget itself. It is possible to connect a control flow signal or pin to any data flow signal of the system defined type Boolean; connection to any other data flow type is generally forbidden.

The degenerate widgets of a widget pack are capable of accepting and producing both formal and degenerate data flow I/O pins as well as the standard control flow pins. Individual members of a widget pack can be invoked independently of other members of the pack but all members of the pack share the same storage area; this storage area is allocated at the time the widget pack is instantiated (generally via the initialize entry point of the formal widget), and is passed by reference to each member of the pack as it is invoked by the engine. As with all other widget types, a degenerate widget of a widget pack only executes when all of its inputs become available. Degenerate widgets need not provide any entry points other than the 'execute¹ entry point, the engine invokes the entry points associated with the formal widget of the pack when using entry points other than execute. All widgets of a widget pack are stored together in a single file and for the purposes of copying and other activities using the WEM menus are treated as a single unit. It is generally not possible to treat a degenerate widget of a pack as if it were a fully defined atomic widget within the normal WEM environment.

Within a pack, the various members can communicate with each other via the data area and can also directly invoke other members of the pack. As a result, it is valid for degenerate widgets of a pack to contain nothing but outputs which are presumably produced when the internal state of the pack meets certain criteria.

A View can be thought of as a mini application, whose functionality is defined by the widgets it contains and the data flows between them. A view's appearance is most easily defined by superimposing the display components of all widgets in the view on a background that is an image. Views provide the framework within which it becomes possible to instantiate and execute the various atomic and compound widgets. No widget can execute unless it is either explicitly or implicitly contained in a view. Views are preferably stored in view definition files which may be accessed and initiated via the environment's view menu. Each view definition file contains as a minimum the following components:

• A specialized compound widget (a view widget) associated with the view that defines the control and data flows between the widgets that make up the view

• Layout information describing the size of the view window, and the location, shape and size of all display and user interface components within that window

• an image that describes the background for the view.

Views may be in one of two states: active or inactive. An active view is one that is currently executing and has therefore loaded all contained widgets into the engine where they are currently executing. An inactive view contains no executing widgets and is not currently loaded into the widget engine. Every view has associated with it a window which may or may not be visible at any given time. For the purposes of this description, the view that is associated with the front window on the user's screen is known as the front view and only one view can be the front view at any given time. This would not necessarily be the case in alterative display environments that permit 2+ dimensional views, however. Unlike widgets, views are generally not nested within (in a visual sense) other views. A view is always the outermost component or container for a given user defined functionality and usually has no relation to the current screen position of any other view. Views also can be combined into groups called view packs and these views packs share a logical context much in the manner of widget packs.

As set forth above, a view has associated with it a compound widget defining the data and control flows between the display and functional widgets that go to make up the view. This compound widget is known as a view widget and is similar to any other compound widget. Because it is part of a view, however, its data flow I/O pins are connected to other views by means of network events. The majority of view widgets will have zero inputs and outputs. Certain specialized views, however, may be controlled from other views and such cases the controlling view will have data flow outputs while the controlled view will have corresponding data flow inputs. View outputs may also optionally include a target view and network node in order to route the event to the intended destination. If no such qualifiers are included, the event will be sent to any views in the local environment that contain inputs whose name and type exactly match that of the view output. If no such inputs exist, the data flow output is discarded. Unlike other widget types, a view widget is scheduled (or rescheduled) whenever any of its inputs becomes available. Internal scheduling of the view, however, may be suspended should other required inputs still be undefined.

Any component of a view widget may also have (one or more) panes associated with it. In such cases, the view causes a marquee or image of that pane to appear in the view layout window where it may be re-sized (within limits) and relocated as part of the view layout process. Panes are normally rectangular but it is possible to create and interact with panes that occupy any arbitrary closed region. A view may be comprised of many panes, each of which represents the display region of the widget responsible for interpreting or displaying the control/display that the pane relates to. For widgets whose appearance is fully determinable at layout time (e.g., named buttons), the final widget appearance is shown in the pane during view layout.

As described above, the WEM window provides the interface in which views and widgets may be edited, modified or displayed. For example, within the WEM window, the subordinate widgets and the data flow between them is displayed. In the preferred embodiment, colored lines that join the pins of widget icons/symbols to other pins in the diagram represent data flow. The color of the line can be used to convey type information. Wherever one or more lines in a WEM window is joined in terms of data or control flow, this is represented by a standard line junction symbol. By default, data-flow outputs are consumed when the widget associated with every connected input has been triggered. The user has the option to select that the output be consumed when the first widget that has a connected input is triggered. These two forms of signal consumption logic are referred to in this document as "OR consumption" (first triggered widget input consumes output) and "AND consumption" (output is only consumed when all widgets with triggered inputs have been run). Different forms of data flow output consumption logic may also be implemented by the scheduling engine. Control flow signals are never 'consumed'. They remain asserted until renewed widget activity causes them to go false. There is an implicit latch on every widget data input so that regardless of the consumption logic operating outside the widget, within the widget, the input may operate using either OR/AND Consumption, the default is AND Consumption. When multiple data-flow outputs are connected together to one or more data flow inputs, the same consumption logic applies to all connected widgets. During execution, a second output value will not be applied to the interconnect signal by the engine until any previous output has been consumed, thus forming an automatic queuing mechanism.

The representation of various standard programming constructs (such as the loop, switch, and conditional statement) are also supported within a WEM window. For example, the conditional statement (i.e., if then [else]) is provided by an atomic widget that accepts a string defining an arbitrary expression in terms of the widget inputs that resolves to a Boolean value either directly (because all inputs are Boolean), or as a result of use of a comparison operator within the expression. This atomic widget has two degenerate outputs which are automatically displayed when the widget is placed. The first corresponds to the YES condition, the second to the NO condition. The conditional widget only generates one or other degenerate outputs when it runs. As a result, any data flow connected to the un-generated output will not be executed. The conditional widget has a single formal input that accepts the Boolean expression, and a large number (up to 26) of degenerate inputs each of which will accept any scalar numeric value or a derived type. Each connected degenerate input can be referred to in the expression by its lower case letter or signal name. The figure below shows an example conditional statement:

Referring now to Figure 1, an example of a conditional statement is shown. In this example, a compound conditional statement of the form: "if (a = b) && (c * d < e) then ... else [if (a > 0.3) then ... else ...]" is provided. Because the outputs 110, 115, 120, 125 of the conditional widgets are both degenerate, either the 'then' clause or the 'else' clause may be omitted simply by not connecting the corresponding output 110, 120. This gives the user the freedom to create any conditional statement that he wishes simply by combining simple 'if blocks as desired. Note also that if the user simply wanted to connect a widget that did not expect a Boolean parameter as input to either the then or the else clause of a conditional widget, he can do so simply by connecting the Boolean output from the conditional widget to the control flow input of the widget required. This is because, in the preferred embodiment, control flow pins may be connected directly to data-flow signals of the system supplied type Boolean. Since control flow inputs are only triggered when a Boolean true value exists, any widget thus connected will only run when the appropriate clause is satisfied. In example illustrated in Figure 1, the else clause of the first conditional widget 130 will produce no output unless the first condition is met when it will output a true value. This means that the expression "a && (b > 0.3)" 140 is not required in the negative case since the widget 135 will not even run unless condition 130 is true.

The case or switch construct is provided by an atomic widget that takes as input two values. The first value is a comma-separated list of constant integer expressions (includes characters) or ranges, the second is an integer value to be evaluated against the list. The output of the case widget is a variable number of degenerate Boolean outputs (each of which preferably represents one of the case conditions being satisfied) and the first of which is always the default case (i.e., no other condition satisfied). Only one case widget output will be generated on any given execution of the widget, and as for the conditional widget, it outputs the Boolean value true. If no specified condition is met then an output will be produced on the default pin.

Referring now to Figure 2, an example case statement using an atomic widget is shown. In this example, the case statement within the widget 210 evaluates the integer input 220 against 5 different values or ranges. Only one of the outputs 231, 232, 233, 234, 235 will be generated as a result of widget 210 execution. This is true even if more than one condition is met due to overlapping ranges. When ranges for a case widget overlap, only the first condition in the list to be satisfied will cause an output. First, in this case, would be represented by going from top to bottom among the outputs 231, 232, 234, 234, 235. Because case widget outputs behave identically to conditional widget outputs, they can be used to trigger other widgets via their control input pins in the same manner described above for the conditional widget. The interface preferably prevents the user from connecting different inputs of a downstream widget to signals that are directly or indirectly connected to different output pins of either a conditional or a case widget within the WEM window. This is because, since only one such pin can be triggered by definition, any downstream widget that relies on more than one such widget output, will by definition never execute. This condition is normally enforced by the block structure of standard programming languages but since, in a visual environment, this structure may not be so apparent, block structuring is preferably enforced by the WEM environment. In the visual environment, the outputs of a case widget may be automatically labeled with the case value. This feature is depicted in the diagram above with respect to outputs 231, 232, 233, 234, 235. The WEM interface is also capable of highlighting all widgets and connections that are 'downstream' of a given selection in the WEM diagram by special Ul actions.

The WEM interface would also preferably prevent the user from connecting a data flow output of any widget (A) to any data flow input of a second widget (B) that either directly or indirectly is required to run in order for the widget (A) itself to become eligible to run. This requirement is made in order to prevent the user from accidentally setting up data flow diagrams that have hidden loops in them that mean they will not execute (or will always execute). Note that this requirement by itself makes it impossible to construct the 'loop' programming construct using pure data flow connections and widgets not expressly designed to implement loops. Loop constructs generally require the use of a variable to pass values from one loop iteration to the next. Once initialized, the variable value is always available and hence removes the possibility of creating a data flow input condition that cannot be satisfied. When a user attempts to create a data flow dependency loop in WEM mode, he is preferably warned of this fact and given the opportunity to create a variable that removes the disallowed dependency. In the event that a data or control flow input pin is multiply sourced, and one source is not dependent on execution of the widget concerned, then it is permissible to connect a downstream widget output to this widget input pin. There are a number of problems associated with loops that make them difficult, if not impossible to implement in a data flow design without enforcing block structuring (i.e., putting the loop body within a compound widget), these are:

• No data flow output can be tied back to a widget that appears earlier in the loop (including the same widget that produces the output) without creating a situation where neither widget can run because each depends on the other to trigger them (known as a deadly embrace). • A data flow signal that comes from code lying outside the loop cannot be directly connected to a data flow input within the loop because the loop may execute many times and will therefore consume the input on the first pass and then hang up.

• A data flow output that comes from code lying inside a loop cannot be directly connected to a data flow input lying outside the loop because the loop may execute many times and will therefore produce a series of outputs which will either unintentionally trigger the widget outside the loop on each pass, or more likely, if the widget outside the loop requires other non-loop inputs, will have to be queued up waiting for access to the data flow signal. It is probable that such a behavior would overload any queuing mechanism provided by the engine to handle multiply connected outputs since such a queue only needs to be as big as the number of connected outputs.

• It is visually very difficult to display the loop concept at a single level so that it is immediately obvious what is, and what is not part of the loop.

For the reasons described above, loops are implemented by one or more system supplied compound widgets that provide a number of degenerate universal input and output pins for passing data of any type into and out of the loop body and which allow specification of the loop behavior.

A suggested set of widget scheduling rules to be enforced by the engine is given in the table below. Many of these rules appear throughout the text above but they are summarized and augmented here in order to make the full rule set more apparent. The term widget is used below to imply both atomic widgets and compound (or view) widgets. When a distinction is required, the particular widget type is explicitly stated. The term signal is used below to refer to either a control flow or a data flow. Where a distinction is required, the type of signal (control or data) is explicitly stated. The term token is used below to refer to a data flow signal to which a value has been written, but has not yet been consumed. A linear compound widget is defined as a one that does not include any explicit connection to the control flow output. A cyclic compound widget is defined as a one that includes an explicit connection to the confrol flow output and which therefore may include explicit loops. A descendant widget Z of an ancestral widget Y within an enclosing compound widget X is defined as any widget within the WEM diagram for X that either directly or indirectly depends on data or control flow outputs from Y in order to become eligible for scheduling. Z is a formal descendant widget of Y if the dependency between Y and Z is mediated by signals connected only to formal widget outputs. Z is a degenerate descendant widget of Y if the dependency between Y and Z is mediated by at least one signal connected to any intervening degenerate widget output. The various descendancy terms described above may also be applied to the data or control flow signals attached to the widget Y. A pure data flow signal is defined as one that is not attached to a variable or constant symbol. A pure descendant data flow is one that can be traced back to a given ancestor through pure data flow or confrol flow signals alone.

Scheduling Rule

A widget is eligible for scheduling only when all of its connected data flow inputs have a token associated with them. A view may be scheduled whenever any of its data flow inputs have a token associated with them but may subsequently 'hang up' should other inputs be required and still be undefined.

If a widget has an explicit connection to its control flow input pin, then that widget (and any widgets it contains) is only eligible for scheduling when the value of the control flow pin is set to TRUE. This applies even if the widget is compound and has already started execution, (i.e., you can single-step a compound widget to any nesting level simply by explicit control of the highest level control pin).

Execution of a linear compound widget (and all widgets called by it) completes when all of its connected output pins have data tokens associated with them (i.e., have been assigned a value).

Execution of a cyclic compound widget (and all widgets called by it) completes when the control output pin is driven TRUE by the control or data flows within the widget. The implementation may encapsulate this functionality so that for standard loops, the user is unaware of the control pin connection.

Once a widget has been scheduled for execution, it cannot again become eligible for scheduling until execution completes.

If a widget input is specified as Only on update' then that widget only becomes eligible for scheduling each time a value is written to the data flow signal connected to that input. This applies even if that signal has an un-consumable token associated with it (see below). If multiple values are written to the input signal before the connected widget is scheduled, it is scheduled only once in response to the series of updates.

The following signals have tokens associated with them that cannot be consumed:

1) Any signal that is connected to a variable symbol once that variable becomes valued. If that signal is also attached (within the compound widget) to a variable symbol, then the input value may be overwritten by subsequent widget action. Variable values once written, persist across multiple executions of the same widget in the same context

2) Any signal that is connected to a constant symbol.

3) Any signal that is directly connected to a widget control flow output.

4) Any signal that is directly attached to a connected or defaulted data flow input of a compound widget.

A pure data flow that is specified as AND consumption logic (the default during construction), causes any tokens associated with that flow to be consumed only after all widgets that have input pins connected to that data flow have been scheduled and executed. If multiple widgets have input pins connected to the pure data flow signal in question, then each may be scheduled only once as a result of any given token appearing on the signal even if that token remains unconsumed.

A data flow that is specified as OR consumption logic (during construction), causes any tokens associated with that flow to be consumed by the first widget that has an input pin connected to that data flow, and which is scheduled and completes.

For a linear compound widget, the engine drives the control output pin FALSE at the time the widget is first scheduled, and TRUE when widget execution completes.

A trash can symbol will immediately consume any tokens associated with the data flow to which the trash can is attached (even if other widget input pins are also attached). Trash cans may not be attached to any signal which by definition cannot be consumed.

Scheduling Rule

Within any compound widget X, once a widget Y has been scheduled and executed, it cannot become eligible for scheduling again while any unconsumed token remains on a pure descendant data flow signal.

Any widget whose output is attached to a multiply sourced pure data flow signal that currently has a token associated with it, can be scheduled and executed (but not completed); the engine will not write any new token(s) produced by that widget until the pre-existing token on the backed-up output data flow signal has been consumed, nor will the engine permit the control flow output to go TRUE until this is the case. Furthermore, the widget (and any widgets it contains) is no longer eligible for scheduling and execution regardless of tokens appearing on its inputs, until the engine has transferred its previous output tokens onto the relevant data flow signals.

If a widget Y within a compound widget X is identified as an 'as needed' widget (for example a dialog widget) then in addition to any other rules that might apply, that widget will not be scheduled or executed until its output is required in order to cause another widget within X to become eligible for scheduling.

Any widget Y within a compound widget X whose outputs are exclusively connected directly or indirectly to unconnected degenerate outputs of X (or to a trash can), will not be scheduled or executed regardless of tokens appearing on its inputs. Furthermore, the widget Y is not considered in any consumption logic applicable to signals connected to its input pins. To all intents and purposes Y is completely ignored in all scheduling activities related to X. This rule can be enforced at load time, it is not dynamic.

Any widget Y within a compound or view widget X, and which has zero connected inputs, never becomes eligible for scheduling or execution. To cause such a widget to execute, its control input pin must be driven true by connecting it either directly to the control input of X or to another Boolean signal within X.

Once any widget Y has been scheduled and executed within an enclosing widget X, it is automatically moved to the last position in the scheduling check sequence so that in addition to all scheduling rules outlined above, the widget Y is not even checked again for scheduling eligibility until every other widget with in X has been checked on a subsequent pass.

All rules described above apply equally and simultaneously to every level of a view and the various subordinate compound widgets that it calls to any arbitrary level of nesting. Scheduling of the various components of each nested WEM diagram occurs independently of other levels except that each nested compound or atomic widget must complete before its output signals become available at the next higher (calling) level and thereby potentially cause one of the rules described above to be triggered at the higher level.

While the illustrated widgets give the appearance of passing widget inputs by value, for efficiency reasons the implementation (wherever possible) passes widget inputs and outputs by reference, not by value. Pass by value is only required in cases where the widget input is overwritten within the widget itself (by use of a variable symbol), and the signal is not simultaneously attached to a formal widget output within the WEM diagram. Because of the scheduling rules described in the table above, the internal values of any inputs to a widget that are supplied by pure data flow connections cannot change during the period of execution of that widget. Preferably, inputs that are externally attached to a variable symbol may potentially change during widget execution.

While the table given above describes the rules that relate to scheduling and execution within a particular context, it is important to understand that simultaneously there may be many active views, all of which are being scheduled and executed. This gives rise to a number of considerations with regards to prioritization between the various views that are illustrated in the scheduling algorithm.

The scheduling algorithm forms the core of the system architecture in that all system functionality, at some level or another, is initiated as part of a view launched within the scheduling environment (referred to as the 'widget engine'). The WEM user interface, including widget building and view editing, can be thought of as a separate application from the scheduling process itself, and could be replaced with other user interfaces without changing the widget engine itself. The widget engine is a strongly-typed data flow interpreter which can read in and subsequently execute atomic and compound widgets together with the data structures that define their I/O needs and characteristics.

Referring now to Figure 3, a sample embodiment of a Widget Editing Mode (WEM) window 300 is shown. In the illustrated example, the window 300 is a standard, titled, resizable window with scroll bars 305. The window 300 is used to define and edit data-flow functionality by connecting and configuring widgets 310. The two blocks 315, 320 connected by arrows to the edge of the window are the input and output bar of the compound widget being viewed. In this example, a single degenerate input 330 and single degenerate output pin 335 have been defined for this compound and connected to an embedded compound 310. In the preferred embodiment, this example could be created simply by clicking on the pin 330, dragging to the destination pin 340 it is to be connected to, and setting the type. The specialized pins 345, 350 at the top of the input 320 and output bars 315 are the control input pin 345 and control output pin 350 respectively. In the preferred embodiment, their type is always equivalent to 'Boolean'. As this diagram illustrates, the confrol output pin is wired to the constant 355, in this case true, which will enable the contents of the compound widget to run. The background widget 3.60 is a pre-supplied atomic widget serving the specialized purpose of defining the background visual appearance of the associated window. In this embodiment, the tool bar 370 provides (starting from the top):

• The arrow tool 371 is used as a toggle switch to turn options on/off, or it simply selects items.

• The lasso tool 372 can select freeform areas by clicking down and selecting the section.

• The hand 373 moves the entire contents of the WEM window in different directions.

• Label 374 can place a label on any element in the WEM window.

• The clipboard 375 is activated when information is copied or cut to the clipboard. Select it to use as a paste tool when clicked at the paste position.

• Click the widget tool 376 to insert a new compound widget.

• Click the variable input tool 377 to insert a variable.

• Click the constant input tool 378 to insert a constant.

• Use the lightening bolt tool 379 to delete elements in the window.

• Select the speaker 380 to set the volume.

• Window size boxes 381 allow enlarging of the window. The default setting is one white square. This setting will not allow the window to be enlarged further than a few inches. The available settings permit expansion of the window vertically, horizontally, or both ways.

• Grid tool 382 can be used to select to set the grid to which all data flow lines and objects snap in the window. It is possible to move a selected item from 1 to 8 pixels at a time.

Referring now to Figure 4, a full-featured icon-editor that allows alteration of the widget icon is shown. In this example, the result of double-clicking on the icon 390 in the Figure 3 is shown.

In the illustrated embodiment, a menu bar 410 is provided which provides an interface to a set of unique tools. Working across the menu bar at the top of the window (from left to right):

• The view-pack menu 411 allows navigation between various members of a view pack in order to allow single session editing of the entire pack/report. • Once widgets are created within widgets, it is possible to move up or down the widget hierarchy with a up/down menu item 412.

• The view menu 413 allows navigation and selection of any view stored on disk.

• The widget menu 414 allows selection of any widget stored on disk

• The file menu 415 allows standard save/load type actions to/from disk A sample embodiment of a file menu 415 of the WEM is provided in Figure 5.

• The Edit menu 416 is a standard Edit menu much like that found in any other application.

• The Add menu 417 allows new widgets to be chosen and added as well as allowing the addition of other objects in the WEM diagram. A sample Add menu 417 of the WEM is illustrated in Figure 6. In this example, a compound widget is selected.

• The Align menu 418 allows various aspects of the WEM diagram internal alignment to be configured. A sample Align menu 418 of the WEM is illustrated in Figure 7. In this example, the flow selected is "Flow left->right".

• The Display menu 419 allows various visual components of the WEM diagram and its contents to be configured as far as their appearance in the diagram is concerned. A sample Display menu of the WEM is illustrated in Figure 8. In this example, the sub menu "VEM Options" is selected.

• The Text menu 420 provides standard control over text appearance such as color, style, font, size, etc.

• The Setup menu 421 allows access to a number of additional 'daughter' windows that can be used to examine and edit the details of the various types of objects in the WEM diagram. A sample setup menu 421 is illustrated in Figure 9.

• The Debug menu 422 is used primarily during run-time debugging of widget execution and allows examination of the state of all flows, widgets and pins and the tokens and data on them. A sample embodiment of the debug menu 422 of the WEM is provided in Figure 10.

Referring now to Figure 11, a sample Pin information dialog 1100 is shown. In the preferred embodiment, this dialog 1100 is generated in response to either double-clicking on a pin in the diagram or using the menu 421. Various aspects of the pin, including it's type (as preferably determined by the run-time type system) and any other logic associated with the pin and it's dataflow behavior can be set from this dialog.

Referring now to Figure 12, a user in the process of choosing the type 1210 of a constant symbol in the WEM diagram within the type pop-up menu 1210 of the constant information window 1200 is shown. This further illustrates the connection between the types system provided by the substrate and the types of data on flows and pins.

Referring now to Figure 13, a dialog window 1300 generated in connection with selection of the view or widget information window in the setup menu 421 is shown. As can be seen, the dialog window 1300 allows adjustment of a wide variety of different aspects that apply to the view or widget including file path, security settings, operation behavior, visual dimensions, behaviors, and limits, and a variety of configuration and documentation descriptors.

It is clear, then, that the illustrated WEM and visual language described above allow users of the system to express and specify analytical processes in terms of data flowing between a set of computational blocks. The visual language of the present invention preferably provides the following basic features:

a) The ability to pass strongly, typed data through flows between a set of interconnected computation blocks (hereafter called widgets). Types are preferably run-time definable and examinable by the widgets themselves.

b) Widgets with typed input and output pins. Input pins provide the ability to specify default values if unconnected.

c) The ability to add arbitrary compiled code snippets to the collection of available widgets, shareable between users of the system. Such compiled code widgets are referred to as 'atomic'.

d) After creating an algorithm by wiring together widgets into an enclosing or 'compound' widget using WEM, permitting the compound widget itself to be used as a building block for higher-level widgets. That is the language would preferably allow arbitrary nesting depth of compound widgets. e) Because many widgets have associated Ul, the graphical environment provides the ability to lay out the Ul of various widgets appearing within the same window. In the preferred embodiment, within any atomic widget, an atomic GUI building environment is provided to allow layout of the atomic widget Ul components. All such information is saved as part of the widget definition for sharing and later re-loading purposes.

f) Normal looping and conditional constructs are supported as are junctions and associated logic joining flows connected to multiple endpoints.

g) When scheduling, the existence of a data token on a flow must precipitate code execution on the connected flow-consumer widgets.

h) The language preferably supports the overt graphical specification of variables and constants that participate in the wiring.

i) Finally, in the preferred embodiment, a debugging means is provided, at run-time, to examine the contents and state of flows and the state of execution of all widgets involved.

At the same time the data-flow wiring for a new view or widget is defined, the visual appearance is created. The interface that allows this is called View Editing Mode (VEM). Referring now to Figure 14, a simple calculator widget/view 1400 is shown. The figure displays the structure of the view 1400 of the calculator. Each section of this figure (such as the buttons, title bar, calculator display window) can be created separately, and the individual sections of the view result in the final view, which also may be created and edited in the View Editing Mode window.

Referring now to Figure 15, an illustrated of the structural components of the calculator widget/view 1400 is shown. In this case, broken-down structure comprises:

• Background 1510 - The plain background of the calculator where the number keypad will be added;

• View Layout 1520 - The view layout (the keypad itself);

• View 1530 - The final view , including the background 1510 and view layout 1520, complete with 0.0 shown as the initial output on the display is shown. Logically, something would need to be attached to the keys on the keypad to allow for performing calculations. Clicking the visually displayed numbers of the photograph of a calculator does not perform a function. These numbers would need to be attached to something that could actually read and manipulate them. In other words, a mere picture does nothing. Hence, once the drawing is done, the keys must be connected to an electronic device representing each individual key and its underlying meaning. This electronic equivalent, in our case, is known as a widget. Widgets are connected to the keys and act as valves that regulate the flow of electronic information. On the calculator, for instance, the widget for the 9 key would act as a valve for the constant valve 9; the plus sign (+) widget would act as the valve for the add operation. In the preferred embodiment, the VEM process allows the appearance of the calculator to be created. Once the physical appearance is complete, the internal connections must be made to enable arithmetic functions. Widgets, electronically connected to the numeric keypad of our calculator, display a different diagram, but are identical in performance to the calculator.

Referring now to Figure 16, an internal diagram of the calculator as it would appear in the preferred Widget Editing Mode window is shown. This figure illustrates a view known as a view widget, which contains a collection of atomic or compound widgets, each of which may have at most one user interface region, or pane. It is in this window that users can make modifications using the WEM capabilities described above.

For explanation purposes, each individual column in the diagram is described and numbered as a single item. It is described in this fashion because each element within the column performs the same general function.

• Column 1610 displays the input bar which has no connections in this case.

• Column 1620 consists of numbered rectangles that represent the constants 0 to 9 displayed on the calculator.

• Column 1630 consists of widgets that perform like valves, regulating the data flow or the flow of the constants 1620.

• Column 1640 displays rectangles with visual operands such as the plus sign (+), the equal sign (=).

• Column 1650 contains the widgets that perform like valves, regulating the flow of the constants.

• The widget 1660 performs the sum.

• Column 1670 includes the widgets that carry the sum to the total. • and that total is then sent to the output bar 1680.

The data-flow scheduling algorithm is the next important part of the system. The data-flow scheduling algorithm is described below by listing the algorithms for a series of recursively invoked functions. The global value "EG" is a complex structure containing various context and state information used by the environment and including the values utilized by the scheduling process.

The top level of the data-flow based scheduling algorithm is the routine SC_ScheduIer(3 which is called in a continuous loop from the main thread of the environment. This routine arbitrates the scheduling of the various active views currently loaded into the environment. The principal task performed by this routine is to enforce the levels of scheduling priorities associated with the active views. To do this SC_Scheduler() makes use of a global active views scheduling structure which contains a set of list headers each of which points to the first element in a singularly linked list of active views having the same priority level. Because this routine locks the view being scheduled, any code operating in the main application thread that is called via this routine can be sure that pointers to structures within the current view will not become invalid during procedure execution. Code running in other threads must either lock the view, or save/restore pointers in a relative form to ensure correct pointer values across scheduling boundaries or heap movements. SC_Scheduler() (and everything it calls) assumes that the application thread is in a critical section (i.e., can't be pre-emptively disturbed). While there are outstanding events to be processed in any view, this routine alternates between scheduling the view at the top of the pending events list (and then rotating the list) and scheduling a view according to the normal view priority scheme. In this manner the system ensures that events are processed as rapidly as possible without allowing event intensive views to suppress all other scheduling. The same mechanism is repeated in SC_ScheduleView() to the normal free walking algorithm.

void SC_Scheduler ( // data-flow scheduler void

) // R:void

{ bump the scheduler cycle count handle any pent-up interrupt level stuff after any slot j = rand ( ) ; // breaks certain deadly cycles! j = j | (1 << (kLowestEPriority+1) ) ; // set backstop bit

EG->Priority = find the first bit set in cycle count word j if ( EG->Priority > kLo estEPriority )

{ // run any background tasks do whatever regular monitoring etc. environment wants to do return; // at lowest priority possible. tmp = NULL; pnd = NULL; if ( ! (EG->MitopiaFlags & kVeventBasedSlot) && EG->PendEvtHdr )

{ // do views with events first pnd = tmp = EG->PendEvtHdr; EG->Priority = (*tmp) ->dPriority; // set priority to match view

EG->MitopiaFlags ]= kVeventBasedSlot; // remember we did an event slot

} else // alternate between event and

{ // normal slots on each pass

EG- >MitopiaFlags &= -kVeventBasedSlot ; // just a plain kind of slot tmp = EG- >ActiveViews [EG->Priority] [03 ; // front active view at priority pnd = 0 ;

}

SC_SetCurrent idget (tmp, iew widget (*tmp) ) // set the chosen view widget lock down the view (tmp) while we cruise around it and schedule below tmp = SC_ScheduleView(tmp, ...) ; // schedule a single view unlock the view (tmp) till the next time... rotate EG->Aσ iveViews [EG->Priority] [0] to tail of list at priority if ( (EG->MitopiaFlags & kVeventBasedSlot) && EG->PendEvtHdr == pnd && pnd)

{ // rotate pending list also move pnd to tail of EG->PendEvtHdr list // avoids greedy event problems }

}

As can be seen from the algorithm above, scheduling within any given view is handled entirely by the routine SC_ScheduleView() once the view itself has been selected based on pending event lists and priority. This routine is called by SC_Scheduler() to give a scheduling slot to a particular view. In most cases this amounts simply to a call to SC_ScheduleNode() for the view widget, but in addition this routine must deal with the special rules associated with starting and stopping views and the propagation of their tokens. Note that an exception to the fair handling of scheduling slots is made for any widgets that are waiting for non-timer related events which have occurred, these are scheduled immediately in order to ensure that events get eaten ASAP. An example of the logic that could be used to perform this routine is provided in Appendix A.

The two main routines that are called by SC_ScheduleView() above are SCJScheduleNodeO and SC_ScheduleANodeQ. SC_ScheduleNode() is the primary function that is responsible for enforcing the rules of data-flow in the system. This routine is recursive and is responsible for implementing the depth first tree walking scheduler algorithm. In order to facilitate navigation around the various levels of the compound widgets that make up the hierarchy associated with a given view, SC_ScheduleNode() makes use of three basic structures: the widget descriptor record (type ET_Widget), the flow descriptor record (type ET_Flow), and the pin descriptor record (type ETJPin). The following are the scheduler uses of the relevant fields in an ET_Widget record:

tokenHdr - This field contains the header into a dynamic list of active tokens associated with data or confrol flows inside a compound widget. tokenHdr = 0 if the list is empty. tokenTail - This field contains the tail (last element) of the tokenHdr list. flowHdr - This field is the header into a static list of control and data flows inside a compound widget. FlowHdr = 0 if the list is empty. sWidgHdr - This field is the header into a list of widgets within the current compound widget. cWidgHdr = 0 if the list is empty. sWidgNext - This field contains the link elements for the list header by sWidgHdr for the surrounding compound widget (i.e., it points to the next widget in the enclosing widgets sWidgHdr list). sWidgNext = 0 if no more elements in the list. The field sWidgPrev is the same but points in the reverse direction to sWidgNext. flags - This field contains various logical flags indicating to the scheduler the state of the current widget. Of particular relevance is 'hasChildTokens' which indicates that at least one widget in the sWidgHdr list has active tokens associated with it. The flag

'hasTokens' indicates that the current widget itself has a non-empty tokenHdr list.

The following are the scheduler uses of the relevant fields in an ET_Flow record:

tokenLink- This field is the link to the next element in the tokenHdr list of the enclosing compound widget. tokenLink = 0 if there are no more elements in the list. flowLink - This field is the link to the next element in the flowHdr list of the enclosing compound widget. flowLink = 0 if there are no more elements in the list. value - This field contains a handle to the heap storage allocation for the actual data value associated with the flow (if allocated). In the case of a control flow, value=0 and the flow state is stored in one of the Flags field bits. cIpinHdr - This field contains the header into a static list of contained widget input pins that are connected to the current flow. cIpinHdr = 0 if the list is empty. Note that an input pin to compound widget may be in the connected input pin list for a flow within the widget that contains it's parent while also being in the connected output pin list of a flow internal to its parent. cOpinHdr - This field contains the header into a static list of contained widget output pins that are connected to the current flow. cOpinHdr = 0 if the list is empty. Note that an output pin of compound widget may be in the connected output pin list for a flow internal to its parent, while also being in the connected input pin list of a flow within the widget that contains it's parent. flags - This field contains various logical flags indicating to the scheduler the state of the flow. 'isConfrolflow' indicates that the flow is a control rather than a data flow. 'hasToken' indicates that the flow has a token associated with it. 'useORlogic' specifies OR rather than AND consumption logic. 'unConsumable' indicates that tokens associated with the flow cannot be consumed. 'hasBreakPoint' and 'hasWatchPoint' are used to initiate debugging activities.

The following are the scheduler uses of the relevant fields in an ETJPin record:

cIpinLink - This field is the link to the next element in the cIpinHdr list of the connected flow. cIpinLink = 0 if no more elements in the list. cOpinLink - This field is the link to the next element in the cOpinHdr list of the connected flow. cOpinLink = 0 if no more elements in the list. parent - This field contains a reference to the widget record for the widget whose input/output pin this ETJPin record describes.

During normal processing, the scheduler algorithm attempts to walk any given free (view) in the environment's active views list and when it finds an active token that can potentially be consumed by running or resuming an atomic widget, it does so. The algorithm is predicated on the following facts:

• The mission of the scheduling algorithm is to attempt to consume any active data flow tokens in the view, thereby completing the view.

• Only atomic widgets actually do anything, so the algorithm must continue to walk the tree until it finds an atomic widget whose input is connected to an active data flow, and then execute it (if possible) in the hope that the widget will complete thereby consuming all outstanding tokens connected to it's inputs. • Many widgets produce more tokens on their outputs after consumption of the input tokens.

• Because a widget (atomic or compound) only consumes tokens on its inputs when it completes, this means that all ancestral (i.e., enclosing) compound widgets of any atomic widget that has not yet completed, must also have unconsumed tokens on their inputs. This in turn means that the scheduler is guaranteed to find all outstanding tokens in a view, no matter how deeply they are nested, simply by looking at only those enclosing widgets. For each level in the tree that has outstanding tokens, the enclosing widgets will have outstanding tokens on their inputs as well. The result is that rather than having to examine every compound widget in a WEM diagram tree, or even every data flow in the WEM tree, the scheduler can be sure that by examining only what is connected to data flows with active tokens, it has examined every widget in the view that has the potential to execute. Within any given compound widget, this list is referred to as the token list. These lists are preferably arranged in a tree structure that can be traced all the way back to the view widget itself. At any given moment, there are vastly fewer active tokens within a view than there are either widgets or data flows. Hence, the efficiency of this algorithm greatly exceeds other free traversal strategies.

• Whenever an atomic widget consumes its input tokens, it does so by removing them from the token list of the enclosing compound widget. If this token list becomes empty then the compound widget itself has completed and the scheduler should therefore consume any tokens on it's inputs and generate the necessary output tokens in the parent WEM of the compound widget. This process continues all the way up the free until the scheduler detects an empty tree at which time the view is complete.

The scheduler tree traversal algorithm is recursive, i.e., it calls itself repeatedly as it walks down the tree, starting at the view widget until it finds a leaf node (atomic widget) that can be scheduled. It then either starts or resumes that widget and when the widget completes, it returns back up the calling chain. As the algorithm climbs back up the path it rotates the token list for every level in the tree by moving the token that was at the front of the list (i.e., the one that determined which attached widget it chose in the downward path), to the back. The effect of this repeated descent and ascent algorithm is to allocate sequential time slices, at any given level of the tree, to widgets that are as far apart as possible in the tree. This is designed to prevent undesirable bunching of time allocations to a given compound widget. Atomic widgets that are higher up in the tree will get more time slices than those that are further down. This is as it should be since higher up atomic widgets generally correspond to Ul related displays and controls which must be as responsive as possible, and which will not actually be eligible to run unless the Ul event on which they are waiting has occurred. One possible side effect of the algorithm is that at any given level, the smaller branches of an unbalanced free get more scheduling slots than larger ones. A tree-balancing algorithm, if desired, could correct this behavior. This algorithm returns to the caller (SC Scheduler) after a single descent and ascent. SC_Scheduler() itself then selects another view and priority group and repeats the process. Thus available CPU time slots are distributed over all views in the system according to priority. Widgets in many other views may be scheduled before SC_Scheduler() again returns to this view and performs another descent and ascent of this tree. Sample code for one embodiment of SC ScheduleNode is provided in Appendix A.

The logic for SCjScheduleANode (schedule atomic node) is broken out separately from SC_ScheduleNode() so that nodes that are atomic can be forcefully scheduled based on non-dataflow related events. Sample code for SC_ScheduleANode() is provided in Appendix A.

The routine SCJStartWidgetO is responsible for checking that all the necessary conditions have been met for starting a particular widget. This routine is therefore responsible for enforcing the rules of data flow as well as the modifications to these rules described above. Once SC_StarfWidgetQ has determined that a widget is eligible to run, it actually launches it using either SC_StartAWidgetO or SCjStartCwidgetQ (depending on whether the widget is atomic or compound). If the widget concerned is ineligible to run for any reason, this routine returns FALSE otherwise it returns TRUE. This routine can also be called with the parameter 'JustCheckin' set to TRUE in which case it makes all necessary checks for eligibility (other than those for input availability) but does not actually cause the widget to be started. That is, it calls itself under certain circumstances in order to find out if widgets that are descendant from the current widgets (in terms of data flow, not hierarchy) have started or are ready to start. Sample code for this Algorithm is provided in Appendix A. This routine is recursive.

The function SCjStartAwidgetO is called by SCJStartWidgetO once it determines that all the necessary conditions have been met to actually initiate execution of an atomic widget. Initiating execution of an atomic widget involves creating a seperate execution thread for that widget to run in. In order to maintain this seperate thread, the thread manager software requires a seperate stack area which the atomic widget will use once launched. Every atomic widget contains a stackSize field which gives the maximum size of stack that the widget anticipates will be required in order to execute. Because the atomic widget retains control of the CPU once the thread has been launched, the scheduler has no way of preventing erroneous widgets from stepping outside their stated stack allocation and thus corrupting the heap. It is therefore very important that widgets ensure that this does not occur. This routine can however detect that such an error has occured after the fact and when widget execution completes, SC_StartAwidget() will report an appropriate error if stack debugging is enabled. The mechanism used is to place test patterns at various points within the allocated stack area, especially at the end point. When widget execution completes, these patterns will have been erased up to the deepest point that the widgets stack reached. If the test pattern at the end of the allocation has been overwritten then the widget is erroneous, otherwise the other test patterns may be used to determine actual stack requirements. Filling an area of heap with these test patterns consumes time and stack debugging should preferably be enabled only when developing new atomic widgets. Because a widgets initialization code may contain suspends, this routine may be re-entered a number of times for the same widget before initialization is complete. By returning a false for inclomplete initialization and not setting the "klsRunning" flag, we can be sure that SC_ScheduleNode() will keep calling us until done. Sample code for one embodiment of SC_StartAWidget is provided below:

Boolean SC_ _StartAwidget ( // Start atomic widge' execution

ET_ViewHdl aView, // I : iew handle

ET_WidgetPtr a idP, // I:widget record pointer

Boolean InitializeOnly // I: if TRUE, just initialization

) // R:TRUE if started

{ if ( a : LdP- >wThreadID ) // thread was already running

{ if ( ! (FLAGS (aWidP) & kHasBeenlnitialized) ) // resume/start initialize

SC_Resume idget (aView,aWidP,NO) ; // Resume the widget if ( !( (FLAGS (aWidP) & kHasBeenlnitialized) aWidP->wThreadID) )

// make a widget thread aWidP->wThreadID = create new thread err = yield to thread (aView, SaWidP,kInitializeEntryPt ...) ; if ( ! InitializeOnly && ! (FLAGS (*aView) & kKillThisView) ) if ( lerr && (FLAGS (aWidP) & kHasBeenlnitialized) ) err = yield to thread (aView, &aWidP, kExecuteEntryPt ...) ; return (err 0 && (FLAGS (aWidP) & kHasBeenlnitialized) ); The routine SC_StartCwidgetO starts execution of a compound widget once all the necessary preconditions have been satisfied. Since there is no code associated with a compound widget, the process of starting one essentially consists of copying the tokens on the external input pins into the internal flows of the compound widget so that these may in turn stimulate contained widgets into execution. Sample code for one embodiment of SC__StartCWidget is provided below:

Boolean SC_StartCwidget ( // Start compound widget execution

ET_ViewHdl aView, // I: View handle

ET_WidgetPtr aWidP // I: widget record pointer

) // R:TRUE if widget was started

{ oldFlags = FLAGS (aWidP) ;

FLAGS (aWidP) |= klsRunning + kHasBeenlnitialized; // set widget's init & run flags for (all formal and degenerate inputs)

{ mask = get the masks for available inputs i = get count of number of pins of that type while ( i ) // for all pins of this type

{ i--; pinName = i + 'A' or 'a'; pin = get the pin concerned iflow = get the flow connected to it within the compound widg. xflow = get the flow connected to it outside the compound widg. if ( iflow )

{

SC_AddAToken (iflow) ; // add a token to internal flow if ( iflow->value ) // dispose of old data

(if any)

TM DisposeHandle (0, (anonHdl) iflow->value, ... ) ; if ( Ixflow ) // if pin input ! available

{ // pin must have been defaulted iflow- >value = pin->pDefault;

} else iflow- >value = xflow- >value; // external to internal copy

} } } if ( ret ) SC_SpontaneousTokens (aWidP) ; // generate spontaneous tokens if ( aWidP == view widge (*EG->CurrentView) && ! (oldFlags & kHasBeenlnitialized) )

{ // routine to init. everything

SC InitializeCompoundAtomics (EG->CurrentView,aWidP) ;

} return ret;

} In the preferred embodiment, the routine SCJResumeWidgetO resumes execution of an atomic widget that has previously suspended by calling SCJSuspendO either directly or indirectly. This function assumes that all necessary conditions for execution have been checked at the time the widget was started and thus it does not need to repeat such checks. Resumption of an executing widget essentially consists of an explicit yield to the relevant thread. The only subtelty in SC_ResumeWidget() is its use to generate certain time based events (idle, tick, second) prior to resuming the thread if the specified time interval has elapsed and the widget is waiting for the time based event specified.

In the preferred embodiment, the routines SC_CheckCtrlStopO and SC_CheckCtrlStartO may be used to check a flow connected to a confrol pin to determine if it implies that a widget should be stopped/started. In the preferred embodiment, the routine SCJTimeToGobblelnputsO may be used by the scheduler to determine if it should schedule a given widget based on the state of it's input pins. The logic is complicated by the fact that 'exclusive' pins can cause widgets to fire even when only a subset of the inputs is available.

In the preferred embodiment, the routine SC_Trace2Inputs() can be used to check all the outputs of a given widget 'aWidP' to see if they directly or indirectly lead to completing the required inputs of a second widget 'cwidg'. The purpose of this is to implement the 'as needed' function whereby widgets that are marked as 'as needed' will only be scheduled by the environment when by running, their outputs might potentially cause another normal widget to become eligible to run. This is the behavior required of many Ul type widgets such as dialogs. See SC_StartWidgetO for usage.

In the preferred embodiment, the routine SC_StopCWidget() is called to complete execution of a compound widget. As for atomic widgets, completing compound widget execution involves propagating tokens onto the output flows. Since the output flows of the compound widget may be busy, there is a possibility that the token propagation routine (SC_PropagateTokens()) may hang up. To simplify this problem, a seperate temporary thread is created to perform the compound widget completion action thus allowing for the possibility of backup. Unlike conventional widget threads referenced from the wThreadlD field of the widget record, the threads associated with completing compound widgets are torn down and re-cycled as soon as the token propagation is complete, also these threads only execute internal environment code, not widget code. Note that because the thread may back up, it may be resumed in SC_ScheduleNode() many times before completing. In the preferred embodiment, the routine SC SpontaneousTokensO is called whenever a compound widget is started by the scheduler, and is responsible for generating any spontaneously produced tokens contained within the WEM diagram for that widget. Spontaneously produced tokens are generally associated with constant symbols. Note that although flows with variable symbols attached have unconsumable tokens on them, these tokens are not generated until the flow is first written to, i.e., they are not spontaneous. This routine does not check for whether the output flow is busy and hang up waiting for it to be clear. This is because this is not possible given the fact that this routine is called during widget starting at which time all internal flows are by definition free. Anyway to do so would be fatal to the main thread. In the preferred embodiment, the routine SC_AddAToken(3 adds a token to an existing flow.

In the preferred embodiment, the routine SC_InitializeCompoundAtomicsO is recursive and initializes all atomic widgets within the specified compound widget and any compound widgets it contains either by calling SC_StartAWidget() or by recursively calling itself as required.

Widget Pin Access API

The API definition below gives the basic public calls available to widgets/threads when accessing data on input pins and writing data to output pins. The API is intended to be illustrative only and is by no means complete. The header files for a sample API implementation is provided in Appendix B.

In the preferred embodiment, the function PCJNumDatalnputsO examines the Widget Input List contained in the specified (or defaulted) widget and returns the counts of the number of formal and degenerate input pins. In the preferred embodiment, the function PC_NumDataOutputs() examines the Widget Output List contained in the specified (or defaulted) widget and returns the counts of the number formal and degenerate output pins.

In the preferred embodiment, the function PC_GetDataInput() takes an input specifier ( 'A' <= char <= 'Z' or 'a' <- char <= 'z') and returns a handle to the storage value for that input or NULL if not found, or if the value of the connected flow is invalid. The handle returned by PC_GetDataInputQ would preferably NEVER be de-allocated by widget code. The handle returned may be subject to relocation or resizing by the scheduler across any scheduling boundary.

In the preferred embodiment, the function PC_GetDataInputTypeO takes an input specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns the type ID for that type. In the preferred embodiment, the function returns zero if the input was not found. The widget may use the returned type ID to obtain further information about the type using the routines provided by the type manager package. In the preferred embodiment, the function PC_GetDataOutputType(3 performs a similar function for output pins. In the preferred embodiment, the function PC_SetDataInputTypeO takes an input specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and a type ID, and sets the type field of the corresponding pin to match the type ID. PCjSetDataOutputTypeO may be used to do the same for output pins.

In the preferred embodiment, the function PC_GetDataInputName() takes an input specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns a handle to the name string for that input or NULL if not found or unnamed. The caller should dispose of the handle returned by this routine when the string is no longer required. In the preferred embodiment, the function PC_GetDataOutputName() is used for output pins.

^■ In the preferred embodiment, the function PC_IsDataInputConnected() takes an input specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns a Boolean indicating whether the input is connected or not. If the input does not exist, a FALSE is returned. Note that if the input is degenerate and PC_IsDataInputConnected() returns FALSE, the input may still have a default value assigned to it which can be retrieved using PC_GetDataInput(). Thus the combination of a FALSE from PC sDatalnputConnectedQ with a non-NULL result from PC_GetDataInρut() uniquely defines a defaulted degenerate input. A TRUE from PCJsDatalnputConnectedQ together with a null result from PC_GetDataInput() indicates an invalid data flow connected to the input. This routine is provided in order to allow atomic widgets to implement the logic associated with degenerate input pins. Because formal inputs are by definition connected (enforced by WEM), this routine simply returns TRUE when called for formal inputs.

In the preferred embodiment, the function PC_IsDataOutputConnected() takes an output specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns a Boolean indicating whether the output is connected or not. If the output does not exist, a FALSE is returned. This routine is provided in order to allow atomic widgets to implement the logic associated with degenerate output pins. Because formal outputs are by definition connected (enforced by WEM), this routine simply returns TRUE when called for formal outputs. In the preferred embodiment, the function PCJDoesOutputHaveTokenO can be used to determine if a particular output exists for a widget and has an unconsumed token value already assigned, TRUE is returned in this case, otherwise FALSE. In the preferred embodiment, the function PCjSetDataOutputO takes an output specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and assigns a new value to it. The function preferably returns a Boolean indicating whether the assignment was completed successfully or not. The value is passed to PCjSetDataOutputO as a void pointer reference, together with an integer parameter specifying the size of the output object to be created. This routine copies the value into the output value handle in the heap, creating or resizing the handle as necessary. If the output flow already has a token associated with it, this function creates a temporary storage allocation to hold the value. The scheduler will copy any temporary storage values into the connected output data flow when widget execution is complete and the connected output flow becomes free to accept new tokens. Once the value has been copied, the original passed in via the 'data' parameter may be discarded. The only output of this function is the updated value in the heap and flag settings in the widget record.

In the preferred embodiment, the function PCJSetControlOutputO sets the value of the control output pin to either true or false. Most normal atomic widgets will not need to use this function since the environment will by default set the control output to false when the widget begins execution and true on completion. Only those widgets that are performing loops or synchronizing functions and whose control output is intended for modifying the normal scheduling sequence of the WEM diagram within which the widget resides will explicitly control this pin using PCJSetControlOutputO. The effect of these values on the external WEM are:

TRUE - This will cause any externally connected widget to be eligible to run FALSE - Any externally connected widget will be ineligible to run

Note that because control flow values change in the external WEM diagram immediately they are written (as opposed to data flows which change when the writing widget terminates), this routine must also perform the necessary logic to maintain the environment flow and token lists as a result of any value change. For data flows, the Scheduler performs this logic when an atomic widget completes.

In the preferred embodiment, the function PC_GetInputFIowNameO takes an input specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns a handle to the name of the flow in the surrounding WEM diagram that is connected to that input or NULL if not found. The caller should dispose of the handle returned by this routine when the string is no longer required. In the preferred embodiment, the function PC_GetInputFIowTypeO takes an input specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns the type ID for the flow connected to that pin. In the preferred embodiment, the function returns zero if the input was not found. The widget may use the returned type ID to obtain further information about the type using the routines provided by the type manager package. Note that the type of the flow and the type of the input pin are normally the same and hence a call to PC_GetDataInpufTypeO would suffice, however certain widgets that accept a given parent type (e.g., scalar) may wish to examine the type of the flow actually connected in order to determine which descendant type was actually connected (e.g., double, int32 etc.).

In the preferred embodiment, the function PC_GetOutputFIowName0 takes an output specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns a handle to the name of the flow in the surrounding WEM diagram that is connected to that output or NULL if not found. The caller should dispose of the handle returned by this routine when the string is no longer required. In the preferred embodiment, the function PC_GetOutputFlowType0 takes an output specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z' ) and returns the type ID for the flow connected to that pin. In the preferred embodiment, the function returns zero if the output was not found. The widget may use the returned type ID to obtain further information about the type using the routines provided by the type manager package. Note that the type of the flow and the type of the output pin are normally the same and hence a call to PC_GetOutputType() would suffice.

In the preferred embodiment, the function PC_DoesInputExistO can be used to determine if a particular input exists for a widget, TRUE is returned if the input exists, otherwise FALSE. In the preferred embodiment, the function PCJDoesOutputExistO is similar for outputs.

In the preferred embodiment, the function PC_GetStaticDataInputO takes an input specifier ( 'A' <= char <= 'Z' or 'a' <= char <= 'z') and returns a handle to the storage value for that input or NULL if not found. Unlike the routine PC GetDatalnputO, this routine will also search on flows that have no associated token for an attached constant object or defaulted output and return any value found. This means that this routine will operate at initialize time as well as execute time. This routine also has the ability to monitor changes on an input flow at run time that do not necessarily have a token associated with them. This often occurs when a widget has written a value onto a flow but has not yet completed and thus has posted no tokens. Use of this "tokenless" communication path is strongly discouraged except in exceptional circumstances. The handle returned by PC_GetDataInput() should NEVER be deallocated by widget code. The handle returned may be subject to relocation or resizing by the scheduler across any scheduling boundary, widgets should be careful not to de-reference the handle and use the de-reference value across such a boundary. Widgets should avoid "Locking" data handles where possible since this will reduce the schedulers ability to resize the handle in response to new data.

The foregoing description of the preferred embodiments of the invention has been presented for the purposes of illustration and description. For example, although described with respect to the C programming language, any language could be used to implement this system. The descriptions of the header structures should also not be limited to the embodiments described. While the sample pseudo code provides examples of the code that may be used, the plurality of implementations that could in fact be developed is nearly limitless. For these reasons, this description is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

SYSTEM AND METHOD FOR MANAGING MEMORY Inventor: John Fairweather

. BACKGROUND OF THE INVENTION

The Macintosh Operating system ("OS"), like all OS layers, provides an 4API where applications can allocate and de-allocate arbitrary sized blocks of memory from a heap. There are two basic types of allocation, viz: handles and pointers. A pointer is a non- relocatable block of memory in heap (referred to as *p in the C programming language, hereinafter "C"), while a handle is a non-relocatable reference to a relocatable block of memory in heap (referred to as **h in C). In general, handles are used in situations where the size of an allocation may grow, as it is possible that an attempt to grow a pointer allocation may fail due to the presence of other pointers above it. In many operating systems (including OS X on the Macintosh) the need for a handle is removed entirely as a programmer may use the memory management hardware to convert all logical addresses to and from physical addresses.

The most difficult aspect of using handle based memory, however, is that unless the handle is 'locked', the physical memory allocation for the handle can move around in memory by the memory manager at any time. Movement of the physical memory allocation is often necessary in order to create a large enough contiguous chunk for the new block size. The change in the physical memory location, however, means that one cannot 'de-reference' a handle to obtain a pointer to some structure within the handle and pass the pointer to other systems as the physical address will inevitably become invalid. Even if the handle is locked, any pointer value(s) are only valid in the current machine's memory. If the structure is passed to another machine, it will be instantiated at a different logical address in memory and all pointer references from elsewhere will be invalid. This makes it very difficult to efficiently pass references to data. What is needed, then,-is a method for managing memory references such that a reference can be passed to another machine and the machine would be able to retrieve or store the necessary data even if the physical address of the data has been changed when transferred to the new machine or otherwise altered as a result of changes to the data.

APPENDIX 1 SUMMARY OF THE INVENTION

The following invention provides a method for generating a memory reference that is capable of being transferred to different machine or memory location without jeopardizing access to relevant data. Specifically, the memory management system and method of the present invention creates a new memory tuple that creates both a handle as well as a reference to an item within the handle. In the latter case, the reference is created using an offset value that defines the physical offset of the data within the memory block. If references are passed in terms of their offset value, this value will be the same in any copy of the handle regardless of the machine. In the context of a distributed computing environment, all that then remains is to establish the equivalence between handles, which can accomplished in a single transaction between two communicating machines. Thereafter, the two machines can communicate about specific handle contents simply by using offsets.

The minimum reference is therefore a tuple comprised of the handle together with the offset into the memory block, we shall call such a tuple an 'ET_ViewRef and sample code used to create such a tuple 100 in C is provided in Figure 1. Once this tuple has been created, it becomes possible to use the ETJViewRef structure as the basic relocatable handle reference in order to reference structures internal to the handle even when the handle may move. The price for this flat memory model is the need for a wrapper layer that transparently handles the kinds of manipulations described above during all de-referencing operations, however, even with such a wrapper, operations in this flat memory model are considerably faster that corresponding OS supplied operations on the application heap.

BRIEF DESCRIPTION OF THE FIGURES

Figure 1 illustrates sample code used to create the minimum reference 'tuple' of the . present invention;

Figure 2 illustrates a drawing convention that is used to describe the interrelationship between sub-layers in one embodiment of the present invention;

Figure 3 illustrates a sample header block that may be used to practice the- present invention;

Figure 4 illustrates a simple initial state for a handle containing multiple structures;

Figure 5 illusfrates the type of logical relationships that may be created between structures in a handle following the addition of a new structure;

Figure 6 illustrates a sample of a handle after increasing the size of a given structure within the handle beyond its initial physical memory allocation;

Figure 7 illustrates the manner in which a handle could be adapted to enable unlimited growth to a given structure within the handle;

Figure 8 illustrates the handle after performing an undo operation;

Figure 9 illustrates a handle that has been adapted to include a time axis in the header field of the structures within the handle;

Figure 10 illustrates the manner in which the present invention can be used to store data as a hierarchical tree; and

Figure 11 illusfrates the process for using the memory model to sort structures within a handle. DETAILED DESCRIPTION

Descriptive Conventions

In order to graphically describe the architectural components and interrelations that comprise the software, this document adopts a number of formalized drawing conventions. In general, any given software aspect is built upon a number of sub-layers. Referring now to figure 2, a block diagram is provided that depicts these sub-layers as a 'stack' of blocks. The lowest block is the most fundamental (generally the underlying OS) and the higher block(s) are successive layers of abstraction built upon lower blocks. Each such block is referred to interchangeably as either a module or a package.

The first, an opaque module 200, is illustrated as a rectangular in Figure 2A. An opaque module 200 is one that cannot be customized or altered via registered plug-ins. Such a form generally provides a complete encapsulation of a given area of functionality for which customization is either inappropriate or undesirable.

The second module, illustrated as T-shaped form 210 in Figure 2B, represents a module that provides the ability to register plug-in functions that modify its behavior for particular purposes. In Figure 2A, these plug-ins 220 are shown as 'hanging' below the horizontal bar of the module 210. In such cases, the module 210 provides a complete 'logical' interface to a certain functional capability while the plug-ins 220 customize that functionality as desired. In general, the plug-ins 220 do not provide a callable API of their own. This methodology provides the benefits of customization and flexibility without the negative effects of allowing application specific knowledge to percolate any higher up the stack than necessary. Generally, most modules provide a predefined set of plug-in behaviors so that for normal operation they can be used directly without the need for plug-in registration.

In any given diagram, the visibility of lower layers as viewed from above, implies that direct calls to that layer from higher-level layers above is supported or required as part of normal operation. Modules that are hidden vertically by higher-level modules, are not intended to be called directly in the context depicted. Figure 2C illustrates this descriptive convention. Module 230 is built upon and makes use of modules 235, 240, and 245 (as well as what may be below module 245). Module 230, 235 and 240 make use of module 245 exclusively. The functionality within module 240 is completely hidden from higher level modules via module 230, however direct access to modules 250 and 235 (but not 245) is still possible.

In Figure 2D, the Viewstructs memory system and method 250 is illustrated. The ViewStructs 250 package (which implements the memory model described herein) is layered directly upon the heap memory encapsulation 280 provided by the TBFilters 260, TrapPatches 265, and WidgetQC 270 packages. These three packages 260, 265, 270 form the heap memory abstraction, and provide sophisticated debugging and memory tracking capabilities that are discussed elsewhere. When used elsewhere, the terms ViewStructs or memory model apply only to the contents of a single handle within the heap.

To reference and manipulate variable sized structures within a single memory allocation, we require that all structures start with a standard header block. A sample header block (called an ETJHdr) may be defined in C programming language as illustrated in Figure 3. For the purpose of discussing the memory model, we shall only consider the use of ET Offset fields 310, 320, 330, 340. The word 'flags' 305, among other things, indicates the type of record follows the ET_Hdr. The 'version' 350 and 'date' fields 360 are associated with the ability to map old or changed structures into the latest structure definition, but these fields 350, 360 are not necessary to practice the invention and are not discussed herein.

Referring now to Figure 4, Figure 4 illustrates a simple initial state for a handle containing multiple structures. The handle contains two distinct memory structures, structure 410 and structure 420. Each structure is preceded by a header record, as previously illustrated in Figure 3, which defines its type (not shown) and its relationship to other structures in the handle. As can be seen from the diagram, the 'Nextltem' field 310 is simply a daisy chain where each link simply gives the relative offset from the start of the referencing structure to the start of the next structure in the handle. Note that all references in this model are relative to the start of the referencing structure header and indicate the (possibly scaled) offset to the start of the referenced structure header. The final structure in the handle is indicated by a header record 430 with no associated additional data where 'Nextltem = 0'. By following the 'Nextltem' daisy chain it is possible to examine and locate every structure within the handle. As the figure illustrates, the 'parent' field 340 is used to indicate parental relationship^'s between different structures in the handle. Thus we can see that stracture B 420 is a child of structure A 410. The terminating header record 430 (also referred to as an ET_Null record) always has a parent field that references the immediately preceding structure in the handle. Use of the parent field in the terminating header record 430 does not represent a "parent" relationship, it is simply a convenience to allow easy addition of new records to the handle. Similarly, the otherwise meaningless 'moveFrpm' field 330 for the first record in the handle contains a relative reference to the final ET_Null. This provides an expedient way to locate the logical end of the handle without the need to daisy chain through the 'nextltem' fields for each structure. . .

Referring now to Figure 5, Figure 5 illustrates the logical relationship between the structures after adding a third structure C 510 to the handle. As shown in Figure 5, structure C 510 is a child of B 420 (grandchild of A 410). The insertion of the new structure involves the following steps:

1) If necessary, grow the handle to make room for C 510, C's header 520, and the trailing ET_Null record 430;

2) Overwrite the previous ETJMull 430 with the header and body of structure C 510.

3) Set up C's parent relationship. In the illustrated example, structure C 510 is a child of B 420, which is established by pointing the 'parent' field of C's header file 520 to the start of structure B 420.

4) Append a final ETJSTull 530, with parent referenced to C's header 520.

5) Adjust the 'moveFrom' field 330 to reflect the offset of the new terminating ET_Null 530.

In addition to adding structures, the present invention must handle growth within existing structures. If a structure, such as structure B 420, needs to grow, it is often problematic since there may be another structure immediately following the one being grown (structure C 510 in the present illustration). Moving all trailing structures down to make enough room for the larger B 420 is one way to resolve this issue but this solution, in addition to being extremely inefficient for large handles, destroy, the integrity of the handle contents, as the relative references within the original B structure 420 would be rendered invalid once such a shift had occurred. The handle would then have to be scanned looking for such references and altering them. The fact that structures A 410, B 420, and C 510 -will generally contain relative references over and above those in the header portion make this impractical without knowledge of all stractures that might be part of the handle. In a dynamic computing environment such knowledge would rarely, if ever, be available, making such a solution impractical and in many cases impossible.

For these reasons, the header for each structure further includes a moveFrom and moveTo fields. Figure 6 illustrates the handle after growing B 420 by adding the enlarged B' structure 610 to the end of the handle. As shown, the original B structure 420 remains where it is and all references to it (such as the parent reference from C 510) are unchanged. B 420 is now referred to as the "base record" whereas B' 610 is the "moved record". Whenever any reference is resolved now, the process of finding the referenced pointer address using C code is:

src = address of referencing structure header dst = src + ET_Offset value for the reference if ( dst->moveTo ) dst = dst + dst->moveTo — follow the move

Further whenever a new reference is created, the process of finding the referenced pointer using C code is:

src = address of referencing structure header dst= address of referenced structure header if ( dst->moveFrom ) dst = dst + dst->moveFrom; ref value = dst - src

Thus, the use of the^'moveto and movefrom fields ensures that no references become invalid, even when stractures must be moved as they grow.

Figure 7 illustrates the handle when B 420 must be further expanded into B" 710. In this case the 'moveTo' of the base record 420 directly references the most recent version of the structure, in this example B" 710. Correspondingly, the record B" 710 now has a 'moveFrom' 720 field that references the base record 420. B's moveFrom 720 still refers back to B 420 and indeed if there were more intermediate records between B 420 and B" (such as B' 610 in this example) the 'moveTo' and 'moveFrom' fields for all of the records 420, 610, 710 would form a doubly linked list. Once each of these records 420, 610, 710 have been linked, it is possible to re-trace through all previous versions of a structure using these links. For example, one could find all previous versions of the record starting with B" 710 by following the 'movefrom' field 720 to the base record 420 and then following the 'nextltem' link of each record until a record with a 'moveFrom' referencing the base record 420 is found. Alternatively, and perhaps more reliably, one could look for structures whose 'moveTo' field references record 420 and then work backward through the chain to find earlier versions.

This method, in which the last 'grown' structure moves to the end of the handle, has the beneficial effect that the same stracture is often grown many times in sequence and in these cases we can optionally avoid creating a series of intermediate 'orphan' records. References occurring from within the bodies of stractures may be treated in a similar manner to those described above and thus by extrapolation one can see that arbitrarily complex collections of cross-referencing structures can be created and maintained in this manner all within a single 'flat' memory allocation.

The price for this flat memory model is the need for a wrapper layer that transparently handles the kinds of manipulations described above during all de-referencing operations, however, even with such a wrapper, operations in this flat memory model are considerably faster that corresponding OS supplied operations on the application heap. Regardless of complexity, a collection of cross-referencing structures created using this approach is completely 'flat' and the entire 'serialization' issue is avoided when passing such collections between processors. This is a key requirement in a distributed data-flow based environment.

In addition to providing the ability to grow and move structures without impacting the references in other structures, another advantage of the 'moveTo'/'moveFrom' approach is inherent support for 'undo'. FIGURE 8 illustrates the handle after performing an 'undo' on the change from B' to B". The steps involved for 'undo' are provided below:

src = base record (i.e., B) dst = locate 'moved' record (i.e. B") by following 'moveTo' of base record prev = locate last record in handle whose 'moveTo' references dst src->moveTo = prev - src;

The corresponding process for 'redo' (which restores the state to that depicted after B" was first added) is depicted below: src = base record (i.e., B) dst = locate 'moved' record (i.e. B') by following 'moveTo' of base record if ( dst->moveTo ) nxt = dst + dst->moveTo src->moveTo = nxt - src ;

This process works because of the fact that 'moveTo' fields are only followed once when referencing via the base record. The ability to trivially perform undo/redo operations is very useful in situations where the structures involved represent information being edited by the user, it is also an invaluable technique for handling the effects of a time axis in the data.

One method for maintaining a time axis is by using a date field in the header of each structure. In this situation, the undo/redo mechanism can be combined with a 'date' field 910 in the header that holds the date when the item was actually changed. This process is illustrated in Figure 9 (some fields have been omitted for clarity).

This time axis can also be used to track the evolution of data over time. Rather than using the 'moveTo' fields to handle growing structures, the 'moveTo' fields could be used to reference future iterations of the data. For example, the base record could specify that it stores the high and low temperatures for a given day in Cairo. Each successive record within that chain of stractures could then represent the high and low temperatures for a given date 910, 920, 930, 940. By using the 'date' fields 910, 920, 930, 940 in this fashion, the memory system and method can be used to represent and reference time-variant data, a critical requirement of any system designed to monitor, query, and, visualize information overtime. Moreover, this ability to handle time variance exists within the 'flat' model and thus data can be distributed throughout a system while still retaining variance information. This ability lends itself well to such things as evolving simulations, database record storage and transaction rollback, and animations.

Additionally, if each instance of a given data record represents a distinct version of the data designed for a different 'user' or process, this model can be used to represent data having multiple values depending on context. To achieve this, whatever variable is driving the context is simply used to set the 'moveTo' field of the base record, much like time was used in the example above. This allows the model to handle differing security privileges, data whose value is a function of external variables or state, multiple distinct sources for the same datum, configuration choices, user interface display options, and other multi-value situations. A 'flags' field could also be used in the header record and can be used to provide additional flexibility and functionality within the memory model. For example, the header could include a 'flag' field that is split into two parts. The first portion could contain arbitrary logical flags that are defined on a per-record type basis. The second portion could be used to define the stracture type for the data that follows the header. While the full list of all possible stracture types is a matter of implementation, the following basic types are ■ examples of types that may be used and will be discussed herein:

kNullRecord - a terminating NULL record, described above. kStringRecord - a 'C format variable length string record.

• • kSimplexRecord - a variable format/size record whose contents is described by a type-id. kComplexRecord - a 'collection' element description record (discussed below) kOrphanRecord - a record that has been logically deleted/orphaned and no longer has any meaning.

By examining the structure type field of a given record, the memory wrapper layer is able to determine 'what' that record is and more importantly, what other fields exist within the record itself that also participate in the memory model, and must be handled by the wrapper layer. The following definition describes a structure named 'kComplexRecord' and will be used to illustrate this method:

typedef struct ET_Complex // Collection element record

1

ET_Hdr hdr; // Standard header

ET_Offset /.* ET_SimplexPtr */ valueR; .// value reference

ET TypelD typelD; // ID of this type

ET_Offset /* ET_ComplexPtr */ nextElem; // next elem. link

ET_Offset /* ET_ComplexPtr */ prevElem; . // prev. elem. link

ET_Offset /* ET_ComplexPtr */ childHdr; // First child link

ET_Offset /* ET_ComplexPtr */ childTail; // Last child link

} ET_Complex

The structure defined above may be used to create arbitrary collections of typed data and to navigate around these collections. It does so by utilizing the additional ET Offset fields listed above to create logical relationships between the various elements within the handle. Figure 10 illustrates the use of this structure 1010 to represent a hierarchical tree 1020. The ET Complex structure defined above is sufficiently general, however, that virtually any collection metaphor can be represented by it including (but not limited to) arrays (multi-dimensional), stacks, rings, queues, sets, n-trees, binary trees, linked lists etc. The 'moveTo', 'moveFrom' and 'nextltem' fields of the header have been omitted for clarity. The 'valueR' field would contain a relative reference to the actual value associated with the tree node (if present), which would be contained in a record of type ET_S implex. The type ID of this record would be specified in the 'typelD' field of the ET_Complex and, assuming the existence of an infrastructure for converting type IDs to a corresponding type and field arrangement, this could be used to examine the contents of the value (which could further contain ET_Offset fields as well).

As Figure 10 illustrates, 'A' 1025 has only one child (namely 'B' 1030), both the 'childHdr' 1035 and 'childTail' 1040 fields reference 'B' 1030, this is in contrast to the 'childHdr' 1045 and 'childTail' 1070 fields of 'B' 1030 itself which reflect the fact that 'B' 1030 has three children 1050, 1055, 1060. To navigate between children 1050, 1055, 1060, the doubly-linked 'nextltem' and 'prevltem' fields are used. Finally the 'parent' field from the standard header is used to represent the hierarchy. It is easy to see how simply by manipulating the various fields of the ET_Complex structure, arbitrary collection types can be created as can a large variety of common operations on those types. In the example of the tree above, operations might include pruning, grafting, sorting, insertion, rotations, shifts, randomization, promotion, demotion etc. Because the ET_Complex type is 'known' to the wrapper layer, it can transparently handle all the manipulations to the ET_Offset fields in order to ensure referential integrity is maintained during all such operations. This ability is critical to situations where large collections of disparate data must be accessed and distributed (while maintaining 'flatness') throughout a system.

Figure 11 illustrates the process for using the memory model to "sort" various structures. A sample structure, named ETjString 1100, could be defined in the following manner (defined below) to perform sorting on variable sized stractures: typedef struct ET String // String Structure

{

ET Hdr hdr; // Standard header

ET_Offset /* ET_StringPtr */ nextString; // ref. to next string char . theString [ 0 ] ; // C string (size varies)

} ET_String; Prior to the sort, the 'nextString' fields 1110^ 1115, 1120, 1125 essentially track the 'nextltem' field in the header, indeed 'un-sort' can be trivially implemented by taking account of this fact. By accessing the strings in such a list by index (i.e., by following the 'nextString' field), users of such a 'string list' abstraction can manipulate collections of variable sized strings. When combined with the ability to arbitrarily grow the string records as described previously (using 'moveTo' and 'moveFrom'), a complete and generalized string list manipulation package is relatively easy to implement. The initial 'Start' reference 1130 in such a list must obviously come from a distinct record, normally the first record in the handle. For example, one could define a special start record format for containers describing executable code hierarchies. The specific implementation of these 'start' records are not important. What is important, however, is that each record type contain a number of ET_Offset fields that can be used as references or 'anchors' into whatever logical collection(s) is represented by the other records within the handle.

The process of deleting a stracture in this memory model relates not so much to the fields of the header record itself, but rather to the fields of the full structure and the logical relationships between them. In other words, the record itself is not deleted from physical memory, rather it is logically deleted by removing from all logical chains that reference it. The specific manner in which references are altered to point "around" the deleted record will thus vary for each particular record type. Figure 12 illustrates the situation after deleting "Dog" 1125 from the string list 1100 and 'C 1050from the tree 1020.

When being deleted, the deleted record is generally 'orphaned'. In order to more easily identify the record as deleted, a record may be set to a defined record type, such as 'kOrphanRecord'. This record type could be used during compression operations to identify those records that have been deleted. A record could also be identified as deleted by confirming that it is no longer referenced from any other stracture within the handle. Given the complete knowledge that the wrapper layer has of the various fields of the stractures within the handle, this condition can be checked with relative ease and forms a valuable double-check when particularly sensitive data is being deleted.

The compression process involves movement of higher structures down to fill the gap and then the subsequent adjustment of all references that span the gap to reduce the reference offset value by the size of the gap being closed during compression. Once again, the fact that the wrapper layer has complete knowledge of all the ET_Offset fields within the stractures in the handle make compression a straightforward operation.

The foregoing description of the preferred embodiment of the invention has been presented for the purposes of illustration and description. For example, the term "handle" throughout this description is addressed as it is currently used in the Macintosh OS. This term should not be narrowly construed to only apply to the Macintosh OS, however, as the method and system could be used to enhance any sort of memory management system. The descriptions of the header stractures should also not be limited to the embodiments described. While the defined header stractures provide examples of the stractures that may be used, the plurality of header structures that could in fact be implemented is nearly limitless. Indeed, it is the very flexibility afforded by the memory management system that serves as its greatest strength. For these reasons, this description is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. In particular due to the simplicity of the model/hardware based implementations can be envisaged. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

A SYSTEM FOR EXCHANGING BINARY DATA Inventor: John Fairweather

BACKGROUND OF THE INVENTION

In most modern computer environments, such as programming languages, and applications, the programming language compiler itself performs the job of defining data structures and the types and the fields that make them up. That type information is compile- time determined. This approach has the advantage of allowing the compiler itself to detect many common programmer errors in accessing compound data structures rather than allowing such errors to occur at run-time where they are much harder to find. However, this approach is completely inadequate to the needs of a distributed and evolving system since it is impossible to ensure that the code for all nodes on the system has been compiled with a compatible set of type definitions and will therefore operate correctly. The problem is aggravated when systems from different vendors wish to exchange data and information since their type definitions are bound to be different and thus the compiler can give no help in the exchange. In recent years, technologies such as B2B suites and XML have emerged to try to facilitate the exchange of information between disparate knowledge representation systems by use of common tags, which may be used by the receiving end to identify the content of specific fields. If the receiving system does not understand the tag involved, the corresponding data may be discarded. These systems simply address the problem of converting from one 'normalized' representation to another, (i.e., how do I get it from my relational database into yours?) by use of a tagged, textual, intermediate form (e.g. XML). Such text-based markup-language approaches, while they work well for simple data objects, have major shortcomings when it comes to the interchange of complex multimedia and non- flat (i.e., having multiple cross-referenced allocations) binary data. Despite the 'buzz' associated with the latest data-interchange techniques, such systems and approaches are totally inadequate for addressing the kinds of problems faced by a system, such as an intelligence system, which attempt to monitor and capture ever-changing streams of unstructured or semi-structured inputs, from the outside world and derive knowledge, computability, and understanding from the data so gathered. The conversion of information, especially complex and multimedia information to/from a textual form such as XML becomes an unacceptable burden on complex information systems and is inadequate for describing many complex data interrelationships. This approach is the current state of the art. At a minimum, what is needed is an interchange language designed to describe and

APPENDIX 2 manipulate typed binary data at run-time. Ideally, this type information will be held in a 'flat' (i.e., easily transmitted) form and ideally is capable of being embedded in the data itself without impact on data integrity. The system would also ideally make use of the power of compiled strongly typed programming languages (such as C) to define arbitrarily interrelated and complex structures, while preserving the ability to use this descriptive power at run-time to interpret and create new types.

SUMMARY OF INVENTION

The present invention provides a strongly-typed, distributed, run-time system capable of describing and manipulating arbitrarily complex, non-flat, binary data derived from type descriptions in a standard (or slightly extended) programming language, including handling of type inheritance. The invention comprises four main components. First, a plurality of databases having binary type and field descriptions. The flat data-model technology (hereinafter "Claimed Database") described in Appendix 1 is the preferred model for storing such information because it is capable of providing a 'flat' (i.e., single memory allocation) representation of an inherently complex and hierarchical (i.e., including type inheritance) type and field set. Second, a run-time modifiable type compiler that is capable of generating type databases either via explicit API calls or by compilation of unmodified header files or individual type definitions in a standard programming language. This function is preferably provided by the parsing technology disclosed in Appendix 2 (hereinafter "Claimed Parser"). Third, a complete API suite for access to type information as well as full support for reading and writing types, type relationships and inheritance, and type fields, given knowledge of the unique numeric type ID and the field name/path. A sample API suite is provided below. Finally, a hashing process for converting type names to unique type IDs (which may also incorporate a number of logical flags relating to the nature of the type). A sample hashing scheme is further described below.

The system of the present invention is a pre-requisite for efficient, flexible, and adaptive distributed information systems. BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 provides a sample implementation of the data structure ETJField;

Figure 2 provides a sample code implementation of the data structure ETJType;

Figure 3 is a block diagram illustrating a sample type definition tree relating ETJType and ET_Field data structures; and

Figure 4 provides a sample embodiment of the logical flags that may be used to describe the typelD.

DETAILED DESCRIPTION OF THE INVENTION

The following description provides an overview of one embodiment of the invention. Please refer to the patent application incorporated herein for a more complete understanding of the Claimed Parser and Claimed Database.

All type information can be encoded by using just two structure variants, these are the 'ETJField' structure, which is used to describe the fields of a given type, and the 'ETJType' structure, which is used to described the type itself. Referring now to Figure 1, a sample implementation of the ETJField structure 100 is provided. The fields in the ETJField structure are defined and used as follows:

"hdr" 102 - This is a standard header record of type ETJHdr as defined in the Claimed Database patent application.

"typelD" 104 - This field, and the union that surrounds it, contain a unique 64-bit type ID that will be utilized to rapidly identify the type of any data item. The method for computing this type ID is discussed in detail below.

"fName"106 - This field contains a relative reference to an ET_String structure specifying the name of the field.

"fDesc" 108 - This field may contain a relative reference to an ETjString stracture containing any descriptive text associated with the field (for example the contents of the line comments in the type definitions above).

"fieldLink" 110 - This field contains a relative reference to the next field of the current type. Fields are thus organized into a link list that starts from the "fieldHDR" 220 field 220 of the type and passes through successive "fieldLink" 110 links 110 until there are no more fields.

"offset" 112 - This field contains the byte offset from the start of the parent type at which the field starts. This offset provides rapid access to field values at run-time. "unitID" 114 - This field contains the unique unit ID of the field. Many fields have units (e.g., miles-per-hour) and knowledge of the units for a given field is essential when using or comparing field values.

"bounds" 116 - For fields having array bounds (e.g., and array of char[80]), this field contains the first array dimension.

"bounds2" 118 - For two dimensional arrays, this field contains the second dimension. This invention is particularly well-adapted for structures of a higher dimensionality than two, or where the connections between elements of a structure is more complex that simple array indexing.

"fScript" 120 - Arbitrary and pre-defined actions, functions, and scripts may be associated with any field of a type. These 'scripts' are held in a formatted character string referenced via a relative reference from this field.

"f Annotation" 122 - In a manner similar to scripts, the text field referenced from this field can contain arbitrary annotations associated with the field. The use of these annotations will be discussed in later patents.

"flaglndex" 124 - It is often convenient to refer to a field via a single number rather than carrying around the field name. The field index is basically a count of the field occurrence index within the parent type and serves this purpose.

"fEchoField" 126- This field is associated with forms of reference that are not relevant to this patent and is not discussed herein.

"flaglndexTypelD" 128- In cases where a field is embedded within multiple enclosing parent types, the 'flaglndex' value stored in the field must be tagged in this manner to identify which ancestral enclosing type the index refers to.

Referring now to Figure 2, a sample embodiment of the ETJType stracture 200 is provided. The fields of the ETJType structure 200 are defined and used as follows:

"hdr" 202 - This is a standard header record of type ETJHdr as defined in the Claimed Database patent application. "typelD" 204 - This field, and the union that surrounds it, contain a unique 64-bit type ID that will be utilized to rapidly identify the type of any data item. The method for computing this type ID is discussed in detail below.

"name" 206 - This is a relative reference to a string giving the name of the type.

"edit", "display" 208 - These are relative references to strings identifying the "process" to be used to display/edit this type (if other than the default). For example the specialized process to display/edit a color might be a color-wheel dialog rather than a simple dialog allowing entry of the fields of a color (red,green,blue).

"description" 210- This is a relative reference to a string describing the type.

"ChildLink" 212 - For an ancestral type from which descendant types inherit, this field gives the relative reference to the next descendant type derived from the same ancestor. Type hierarchies are defined by creating trees of derived types. The header to the list of child types at any level is the "childHdr" field 218, the link between child types is the "ChildLink" field 212. Because types are organized into multiple type databases (as discussed later), there are two forms of such links: the local form and non-local form. The non-local form is mediated by type ID references, not relative references (as for the local form), and involves the fields "childlDLink" 236, "childlDHdr" 238, and "parentID" 240 (which hold the reference from the child type to its parent). The parent reference for the local form is held in the "parent" field of "hdr" 202.

"cTypedef ' 216 - This field may optionally contain a relative reference to a string giving the C language type definition from which the type was created.

"childHdr" 218 - This field contains the header to the list of child types at any level .

"fieldHDR" 220 - Fields are organized into a link list that starts from the this field.

"keywords" 222 - This field contains a relative reference to a string contain key words by which the type can be looked up.

"bounds" 224, "bounds2" 226 - array dimensions as for ETJField

"size" 228 - Total size of the type in bytes. "color" 230- To facilitate type identification in various situations, types may be assigned inheritable colors.

"filelndex" 232 - used to identify the source file from which the type was created.

"keyTypelD" 234 - This field is used to indicate whether this type is designated a "key" type. In a full data-flow based system, certain types are designated 'key' types and may have servers associated with them.

"nextKeyType" 246 - This field is used to link key types into a list.

"tScript" 242,"tAnnotation" 244 - These fields reference type scripts and annotations as for ET_Field l00.

"maxFieldlndex" 248 - This field contains the maximum field index value (see ETJField 100) contained within the current type.

"numFields" 250 - This gives the total number of fields within the current type.

To illustrate the application of these structures 100, 200 to the respresentation of types and the fields within them, consider the type definitions below whereby the types "Cat" and "Dog" are both descendant from the higher level type "Mammal" (denoted by the "::" symbol similar to C++ syntax).

typedef struct Mammal

{

RGBColor hairColor; int32 gestation; // in days

} Mammal;

typedef struct Dog: :Mammal

{ int32 barkVol; // in decibels

} Dog;

typedef struct Cat::Mammal

{ int32 purrVol; // in decibels

} Cat;

Because they are mammals, both Cat and Dog inherit the fields "hairColor" and "gestationPeriod" which means the additional fιeld(s) defined for each start immediately after the total of all inherited fields (from each successive ancestor). Referring now to Figure 3, this portion of the type definition tree when viewed as a tree of related ETJType 200 and ETJField 100 structures is shown. In this diagram, the vertical lines 305 linking the types 315, 320 are mediated via the "childHdr" 218 and "parent" 240 links. The horizontal line 310 linking Dog 320 and Cat 325 is mediated via "ChildLink" 242. Similarly for the field links 330, 335, 340, 345 within any given type, the fields involved are "parentID" 240, "fieldHDR" 220, and "fieldLink" 110. It is thus very obvious how one would navigate through the hierarchy in order to discover say all the fields of a given type. For example, the following sample pseudo code illustrates use of recursion to first process all inherited fields before processing those unique to the type itself.

void LoopOverFields (ETJType *aType)

{ if ( aType->hdr.parent )

LoopOverFields(aType->hdr.parent) for ( fieldPtr = aType->fieldHdr ; fieldPtr ; fieldPtr = fιeldPtr->fιeldLink )

— do something with the field

}

Given this simple tree structure in which type information is stored and accessed, it should be clear to any capable software engineer how to implement the algorithms setr forth in the Applications Programming Interface (API) given below. This API illustrates the nature and scope of one set of routines that provide full control over the run-time type system of this invention. This API is intended to be illustrative of the types of capabilities provided by the system of this invention and is not intended to be exhaustive. Sample code implementing the following defined API is provided in the attached Appendix A.

The routine TM_CruiseTypeHierarchyO recursively iterates through all the subtypes contained in a root type, call out to the provided callback for each type in the hierarchy. In the preferred embodiment, if the function 'callbackFunc' returns -1, this routine omits calling for any of that types sub-types.

The routine TM_Code2TypeDB0 takes a type DB code (or TypelD value) and converts it to a handle to the types database to which it corresponds (if any). The type system of this invention allows for multiple related type databases (as described below) and this routine determines which database a given type is defined in.

TMJEnitATypeDBO and TMJTermATypeDBO initialize and terminate a types database respectively. Each type DB is simply a single memory allocation utilizing a 'flat' memory model (such as the system disclosed in the Claimed Database patent application) containing primarily records of ETJType 100 and ETJField 200 defining a set of types and their inter-relationships.

TM_SaveATypeDB0 saves a types database to a file from which it can be re-loaded for later use.

TM_AlignedCopyO copies data from a packed structure in which no alignment rules are applied to a normal output structure of the same type for which the alignment rules do apply. These non-aligned stractures may occur when reading from files using the type manager. Different machine architectures and compilers pack data into structures with different rules regarding the 'padding' inserted between fields. As a result, these data structures may not align on convenient boundaries for the underlying processor. For this reason, this function is used to handle these differences when passing data between dissimilar machine architecture.

TM_FixByteOrdering0 corrects the byte ordering of a given type from the byte ordering of a 'source' machine to that of a 'target' machine (normally 0 for the current machine architecture). This capability is often necessary when reading or writing data from/to files originating from another computer system. Common byte orderings supported are as follows:

• kBigEndian — e.g., the Macintosh PowerPC

• kLittleEndian — e.g., the Intel x86 architecture

• kCurrentByteOrdering — current machine architecture TMJFindTypeDBO can be used to find the TypeDB handle that contains the definition of the type name specified (if any). There are multiple type DBs in the system which are accessed such that user typeDBs are consulted first, followed by system type DBs. The type DBs are accessed in the reverse order to that in which they were defined. This means that it is possible to override the definition of an existing type by defining a new one in a later types DB. Normally the containing typeDB can be deduced from the type ID alone (which contains an embedded DB index), however, in cases where only the name is known, this function deduces the corresponding DB. This routine returns the handle to containing type DB or NULL if not found. This invention allows for a number of distinct type DBs to co-exist so that types coming from different sources or relating to different functional areas may be self contained. In the preferred embodiment, these type DBs are identified by the letters of the alphabet ('A' to 'Z') yielding a maximum of 26 fixed type databases. In addition, temporary type databases (any number) can be defined and accessed from within a given process context and used to hold local or temporary types that are unique to that context. All type DBs are connected together via a linked list and types from any later database may reference or derive from types in an earlier database (the converse is not true). Certain of these type DBs may be pre-defined to have specialized meanings. A preferred list of type DBs that have specialized meanings as follows:

'A' - built-in types and platform Toolbox header files

'B' - GUI framework and environment header files

'C - Project specific header files

'D' - Flat data-model structure old-versions DB (allows automatic adaption to type changes)

'E' - Reserved for 'proxy' types

'F' - Reserved for internal dynamic use by the environment

T - Project specific ontology types

TM_GetTypeID0 retrieves a type's ID Number when given its name. If aTypeName is valid, the type ID is returned, otherwise 0 is returned and an error is reported. TM_IsKnownTypeNameO is almost identical but does not report an error if the specified type name cannot be found.

TM_ComputeTypeBaseIDO computes the 32-bit unique type base ID for a given type name, returning it in the most significant 32-bit word of a 64-bit ETJTypelD 104. The base ID is calculated by hashing the type name and should thus be unique to all practical purposes. The full typelD is a 64-bit quantity where the base ID as calculated by this routine forms the most significant 32 bits while a variety of logical flags describing the type occupy the least significant 32-bits. In order to ensure that there is a minimal probability of two different names mapping onto the same type ID, the hash function chosen in the preferred embodiment is the 32-bit CRC used as the frame check sequence in ADCCP (ANSI X3.66, also known as FIPS PUB 71 and FED-STD-1003, the U.S. versions of CCITT's X.25 link- level protocol) but with the bit order reversed. The FIPS PUB 78 states that the 32-bit FCS reduces hash collisions by a factor of 10^Λ-5 over the 16-bit FCS. Any other suitable hashing scheme, however, could be used. The approach allows type names to be rapidly and uniquely converted to the corresponding type ID by the system. This is an important feature if type information is to be reliably shared across a network by different machines. The key point is that by knowledge of the type name alone, a unique numeric type ID can be formed which can then be efficiently used to access information about the type, its fields, and its ancestry. The other 32 bits of a complete 64-bit type ID are utilized to contain logical flags concerning the exact nature of the type and are provided in Appendix A.

Given these type flag definitions and knowledge of the hashing algorithm involved, it is possible to define constants for the various built-in types (i.e., those directly supported by the underlying platform from which all other compound types can be defined by accumulation). A sample list of constants for the various built in types is provided in Appendix A.

Assuming that the constant definitions set forth in Appendix A are used, it is clear that the very top of the type hierarchy, the built-in types (from which all other types eventually derive), are similar to that exposed by the C language.

Referring now to Figure 4, a diagrammatic representation of a built-in type is shown (where indentation implies a descendant type). Within the kUniversalType 405, the set of direct descendants includes kVoidType 410, kScalarType 415, kStractType 420, kUnionType 425, and kFunctionType 430. kScalarType also includes descendants for handling integers 435, descendants for handling real numbers 440 and descendants for handling special case scalar values 445. Again, this illustrates only one embodiment of built-in types that may be utilized by the present system.

The following description provides a detailed summary of some of the functions that may be used in conjunction with the present invention. This list is not meant to be exhaustive nor or many of these functions required (depending upon the functionality required for a given implementation). The pseudo code associated with these functions is further illustrated in attached Appendix A. It will be obvious to those skilled in the art how these functions could be implemented in code.

Returning now to Appendix A, a function TM_CIeanFieldNameO is defined which provides a standardized way of converting field names within a type into human readable labels that can be displayed in a Ul. By choosing suitable field names for types, the system can create "human readable" labels in the corresponding Ul. The conversion algorithm can be implemented as follows:

1) Convert underscores to spaces, capitalizing any letter that immediately follows the underscore

2) Capitalize the first letter

3) Insert a space in front of every capitalized letter that immediately follows a lower case letter

4) Capitalize any letter that immediately follows a '.' character (field path delimiter)

5) De-capitalize the first letter of any of the following filler words (unless they start the sentence):

"an","and","of',"the","or","to","is","as","a"

So for example:

"aFieldName" would become "A Field Name" as would "a_field_name"

"timeOfDay" would become "Time of Day" as would "time_of_day" A function, such as TM_AbbreveFieldNameO, could be used to provide a standardized way of converting field names within a type into abbreviated forms that are still (mostly) recognizable. Again, choosing suitable field names for types ensures both human readable labels in the corresponding Ul as well as readable abbreviations for other purposes (such as generating database table names in an external relational database system). The conversion algorithm is as follows:

1) The first letter is copied over and capitalized.

2) For all subsequent letters:

a) If the letter is a capital, copy it over and any 'numLowerCase' lower case letters that immediately follow it.

b) If the letter follows a space or an underscore, copy it over and capitalize it

c) If the letter is '.', '[', or ']', convert it (and any immediately subsequent letters in this set) to a single 'J character, capitalize the next letter (if any). This behavior allows this function to handle field paths.

d) otherwise disgard it

So for example:

"aFieldName" would become "j FiNa" as would "a_field_name" if 'numLowerCase' was 1, it would be 'AFieNam' if it were 2

"timeOfDay" would become "TiOfDa" as would "time of day" if 'numLowerCase' was 1, it would be 'TimOfDay' if it were 2

For a field path example:

"geog.city[3j\ population" would become "Ge_Ci_3JPo" if 'numLowerCase' was 1

Wrapper functions, such as TMJSetTypeEditO, TMjSetTypeDisplayO, TM_SetTypeConverterO, TM_SetTypeCtypedefO, TM_SetTypeKeyWordsO,TM_SetTypeDescriptionO , and TM_SetTypeColor(), may be used set the corresponding field of the ETJType structure 200. The corresponding 'get' functions are simply wrapper functions to get the same field. A function, TMjSetTypelconO, may be provided that sets the color icon ID associated with the type (if specified). It is often useful for Ul purposes to associate an identifiable icon with particular types (e.g., a type of occupation), this icon can be specified using TM SetTypelconO or as part of the normal acquisition process. Auto-generated Ul (and many other Ul context) may use such icons to aid in Ul clarity. Icons can also be inherited from ancestral types so that it is only necessary to specify an icon if the derived type has a sufficiently different meaning semantically in a Ul context. The function TM_GetTypeIconO returns the icons associated with a type (if any).

A function, such as TM_SetTypeKeyTypeO, may be used to associate a key data type (see TM_GetTypeKeyType) with a type manager type. By making this association, it is possible to utilize the full suite of behaviors supported for external APIs such as Database and Client-Server APIs, including creation and communication with server(s) of that type, symbolic invocation, etc. For integration with external APIs, another routine, such as TMJKeyTypeToTypelDOj may be used to obtain the type manager type ID corresponding to a given key data type. If there is no corresponding type ID, this routine returns zero.

Another function, TM_GetTypeNameO, may be used to get a type's name given the type ID number. In the preferred embodiment, this function returns using the 'aTypeName' parameter, the name of the type.

A function, such as TM FindTypesByKeywordO, may be used to search for all type DBs (available from the context in which it is called) to find types that contain the keywords specified in the 'aKeywordList' parameter. If matches are found, the function can allocate and return a handle to an array of type IDs in the 'thelDList' parameter and a count of the number of elements in this array as it's result. If the function result is zero, 'thelDList' is not allocated.

The function TM_GetTypeFileNameO gets the name of the header file in which a type was defined (if any).

Given a type ID, a function, such as TM_GetParentTypeIDO_? can be used to get the ID of the parent type. If the given ID has no parent, an ID of 0 will be returned. If an error occurrs, a value of -1 will be returned. Another function, such as TM_IsTypeDescendantO, may be used to determine if one type is the same as or a descendant of another. The TM_IsTypeDescendant call could be used to check only direct lineage whereas TM_jAreTypesCompatible checks lineage and other factors in determining compatibility. If the source is a descendant of, or the same as, the target, TRUE is returned, otherwise F.ALSE is returned.

Another set of functions, hereinafter referred to as TMJTypelsPointerO, TMJTypelsHandleO, TMJTypelsRelRefO, TMJTypelsCoIIectionRefO, TMJTypelsPersistentRefO_s maybe used to determine if a typelD represents a pointer/handle/relative etc. reference to memory or the memory contents itself (see typelD flag definitions). The routines optionally return the typelD of the base type that is referenced if the type ID does represent a pointer/handle/ref. In the preferred embodiment, when calling TM_TypeIsPtr(), a type ID that is a handle will return FALSE so the determination of whether the type is a handle, using a function such as TMJTypelsHandleO, could be checked first where both possibilities may occur. The function TMJTypelsReferenceO will return true if the type is any kind of reference. This function could also return the particular reference type via a paramter, such as the 'refType' parameter.

Another function, such as TMJTypesAreCompatibleO, may be used to check if the source type is the same as, or a descendant of, the target type. In the preferred embodiment, this routine returns:

+1 If the source type is a descendant of the target type (a legal connection)

-1 If the source type is a group type (no size) and the target is descended from it (also a legal connection)

0 Otherwise (an illegal connection)

If the source type is a 'grouping' type (e.g., Scalar), i.e., it has no size then this routine will return compatible if either the source is ancestral to the target or vice-versa. This allows for data flow connections that are typed using a group to be connected to flows that are more restricted.

Additional functions, such as TM_GetTypeSizeO and TM SizeOfO, could be applied in order to return the size of the specified data type. For example, TM_GetTypeSize() could be provided with an optional data handle which may be used to determine the size of variable sized types (e.g., strings). Either the size of the type could be returned or, alternatively, a 0 could be returned for an error. TM_SizeOf0 could be provided with a similar optional data pointer. It also could return the size of the type or 0 for an error.

A function, such as TM_GetTypeBoundsO_> could be programmed to return the array bounds of an array type. If the type is not an array type, this function could return a FALSE indicator instead.

The function TM_GetArrayTypeElementOffset0 can be used to access the individual elements of an array type. Note that this is distinct from accessing the elements an array field. If a type is an array type, the parent type is the type of the element of that array. This knowledge can be used to allow assignment or access to the array elements through the type manager API.

The function TM_InitMem initializes an existing block of memory for a type. The memory will be set to zero except for any fields which have values which will be initialized to the appropriate default (either via annotation or script calls - not discussed herein). The function TM_NewPtrO allocates and initializes a heap data pointer. If you wish to allocate a larger amount of memory than the type would imply, you may specify a non-zero value for the 'size' parameter. The value passed should be TM_GetTypeSize(...) + the extra memory required. If a type ends in a variable sized array parameter, this will be necessary in order to ensure the correct allocation. The function TMJNewHdIO performs a similar function for a heap data handle. The functions TMJDisposePtrO and TMJDisposeHdlO may be used to de-allocate memory allocated in this manner.

The function TMJLocalFieldPathO can be used to truncate a field path to that portion that lies within the specified enclosing type. Normally field paths would inherently satisfy this condition, however, there are situations where a field path implicitly follows a reference. This path truncation behavior is performed internally for most field related calls. This function should be used prior to such calls if the possibility of a non-local field path exists in order to avoid confusion. For example:

typedef struct tl

{ char x[16]; } tl; typedef struct t2

{ tl y; } t2;

then TM_LocalFieldPath(,t2,"y.x[3]",) would yield the string "y".

Given a type ID, and a field within that type, TM_GetFieldTypeIDO will return the type ID of the aforementioned field or 0 in the case of an error.

The function TM_GetBuiltInAncestor returns the first built-in direct (i.e., not via a reference) ancestor of the type ID given.

Two functions, hereinafter called TM_GetIntegerVaIue and TM_GetReaIValueO, could be used to obtain integer and real values in a standardized form. In the preferred embodiment, if the specified type is, or can be converted to, an integer value, the TM GetlntegerValueO would return that value as the largest integer type (i.e., int64). If the specified type is, or can be converted to, a real value, TM_GetRealValue() would return that value the largest real type (i.e., long double). This is useful when code does not want to be concerned with the actual integer or real variant used by the type or field. Additional functions, such as TMjSetlntegerValueO and TMJSetRealValueO, could perform the same function in the opposite direction.

Given a type ID, and a field within that type, a function, hereinafter called TM_GetFieldContainerTypeH)0, could be used to return the container type ID of the aforementioned field or 0 in the case of an error. Normally the container type ID of a field is identical to 'aTypelD', however, in the case where a type inherits fields from other ancestral types, the field specified may actually be contributed by one of those ancestors and in this case, the type ID returned will be some ancestor of 'aTypelD'. In the preferred embodiment, if a field path is specified via 'aFieldName* (e.g., field l.field2) then the container type ID returned would correspond to the immediate ancestor of 'field2', that is 'fieldl'. Often these inner structures are anonymous types that the type manager creates during the types acquisition process. A function, hereinafter called TM_GetFieldSize , returns the size, in bytes, of a field, given the field name and the field's enclosing type; 0 is returned if unsuccessful.

A function, hereinafter called TM_IsLegalFieldPath , determines if a string could be a legal field path, i.e., does not contain any characters that could not be part of a field path. This check does not mean that the path actually is valid for a given type, simply that it could be. This function operates by rejecting any string that contains characters that are not either alphanumeric or in the set '[',']7J, or '.'. Spaces are allowed only between '[' and ']'.

Given an enclosing type ID, a field name, and a handle to the data, a function, hereinafter known as TM_GetFieIdValueH , could be used to copy the field data referenced by the handle into a new handle. In the preferred embodiment, it will return the handle storing the copy of the field data. If the field is an array of 'char', this call would append a terminating null byte. That is if a field is "char[4]" then at least a 5 byte buffer must be allocated in order to hold the result. This approach greatly simplifies C string handling since returned strings are guaranteed to be properly terminated. A function, such as TM_GetFieldVaIueP _? could serve as the pointer based equivalent. Additionally, a function such as TM_SetFieldValueO could be used to set a field value given a type ID, a field name and a binary object. It would also return an error code in an error.

A function, such as TMjSetCStringFieldValueO, could be used to set the C string field of a field within the specified type. This function could transparently handle logic for the various allowable C-string fields as follows:

1) if the field is a charHdl then:

a) if the field already contains a value, update/grow the existing handle to hold the new value

b) otherwise allocate a handle and assign it to the field

2) if the field is a charPtr then:

a) if the field already contains a value:

i) if the previous string is equal to or longer than the new one, copy new string into existing pointer ii) otherwise dispose of previous pointer, allocate a new one and assign it

b) otherwise allocate a pointer and assign it to the field

3) if the field is a relative reference then:

a) this should be considered an error. A pointer value could be assigned to such a field prior to moving the data into a collection in which case you should use a function similar to the TM_SetFieldValueO function described above.

4) if the field is an array of char then:

a) if the new value does not fit, report array bounds error

b) otherwise copy the value into the array

A function, such as TM_AssignToFieldO, could be used to assign a simple field to a value expressed as a C string. For example, the target field could be:

a) Any form of string field or string reference;

b) A persistent or collection reference to another type; or

c) Any other direct simple or structure field type. In this case the format of the C string given should be compatible with a call to TM_StringToBinary() (described above) for the field type involved. The delimiter for TM_StringToBinary is taken to be "," and the 'kCharArrayAsString' option (see TM_BinaryToString) is assumed.

In the preferred embodiment, the assignment logic used by this routine (when the 'kAppendStringValue' is present) would result in existing string fields having new values appended to the end of them rather than being overwritten. This is in contrast to the behavior of TM_SetCStringFieldValue() described above. For non-string fields, any values specified overwrite the previous field content with the exception of assignment to the 'aStringH' field of a collection or persistent reference with is appended if the 'kAppendStringValue' option is present. If the field being assigned is a collection reference and the 'kAppendStringValue' option is set, the contents of 'aStringPtr' could be appended to the contents of a string field. If the field being assigned is a persistent reference, the 'kAssignToRefType','kAssignToUniqueID' or 'kAssignToStringH' would be used to determine if the typelD, unique ID, or 'aStringH' field of the reference is assigned. Otherwise the assignment is to the name field. In the case of 'kAssignToRefType', the string could be assumed to be a valid type name which is first converted to a type ID. If the field is a relative reference (assumed to be to a string), the contents of 'aStringPfr' could be assigned to it as a (internally allocated) heap pointer.

Given an enclosing type ID, a field name, and a pointer to the data, a function such as TM_SetArrFieldValue could be used to copy the data referenced by the pointer into an element of an array field element into the buffer supplied. Array fields may have one, or two dimensions.

Functions, hereinafter named TM_GetCStringFieIdValueB » TM_GetCStringFieldValuePO and TM_GetCStringFieIdValueHO, could be used to get a C string field from a type into a buffer/pointer/handle. In the case of a buffer, the buffer supplied must be large enough to contain the field contents returned. In other cases the function or program making the call must dispose of the memory returned when no longer required. In the preferred embodiment, this function will return any string field contents regardless of how is actually stored in the type structure, that is the field value may be in an array, via a pointer, or via a handle, it will be returned in the memory supplied. If the field type is not appropriate for a C string, this function could optionally return FALSE and provide an empty output buffer.

Given an enclosing type ID, a field name, and a pointer to the data, the system should also include a function, hereinafter name TM_GetArrFieldVaIueP , that will copy an element of an array field element's data referenced by the pointer into the buffer supplied. Array fields may have one, or two dimensions.

Simple wrapper functions, hereinafter named TM GetFieldBoundsO, TM_GetFieIdOffset0, TM_GetFieldUnitsO, and TM_GetFieIdDescription(), could be provided in order to access the corresponding field in ET Field 100. Corresponding 'set' functions (which are similar) could also be provided.

The function TM ForAHFieldsLoopO is also provided that will iterate through all fields (and sub-fields) of a type invoking the specified procedure. This behavior is commonplace in a number of situations involving scanning the fields of a type. In the preferred embodiment, the scanning process should adhere to a common approach and as a result a function, such as this one, should be used for that purpose. A field action function takes the following form:

Boolean myActionFn ( // my field action function

ET TypeDBHdl aTypeDBHdl, // 1: Type DB (NULL to default)

ETJTypelD 104 aTypelD, // 1: The type ID

ETJTypelD 104 aContainingTypelD, // 1: containing Type ID of field anonPtr aDataPtr, // 1: The type data pointer anonPtr context, // IO:Use to pass custom context charPtr fieldPath, // LField path for field

ETJTypelD 104 aFieldTypelD, // I:Type ID for field int32 dimensionl, // LField array bounds 1 (0 if

N/A) int32 dimension2, // LField array bounds 2 (0 if

N/A) int32 fieldOffset, // LOffset of start of field int32 options, // LOptions flags anonPtr internalUseOnly // LFor internal use only

) // R:TRUE for success

In this example, fields are processed in the order they occur, sub-field calls (if appropriate) occur after the containing field call. If this function encounters an array field (1 or 2 dimensional), it behaves as follows:

a) The action function is first called once for the entire field with no field indexing specified in the path.

b) If the element type of the array is a structure (not a union), the action function will be invoked recursively for each element with the appropriate element index(es) reflected in the 'fieldPath' parameter, the appropriate element specific value in 'fieldOffset', and 0 for both dimensionl and dimension2.

This choice of behavior for array fields offers the simplest functional interface to the action function. Options are:

kRecursiveLoop -- If set, recurses through sub-fields, otherwise one-level only kDataPtrlsViewRef — The 'aDataPtr' is the address of an ET_ViewRef designating a collection element

A function, hereinafter referred to as TM_FieldNameExistsO, could be used to determine if a field with the given name is in the given type, or any of the type's ancestral types. If the field is found return it returns TRUE, otherwise it returns F.ALSE.

A function, hereinafter referred to as TM_GetNumberOfFields0₅ may be used to return the number of fields in a given structured type or a -1 in the case of an error. In the preferred embodiment, this number is the number of direct fields within the type, if the type contains sub-structures, the fields of these sub-structures are not counted towards the total returned by this function. One could use another function, such as

TM_ForAllFieldsLoop(),to count fields regardless of level with 'kRecursiveLoop' set true and a counting function passed for 'aFieldFn' (see TM_GetTypeMaxFlagIndex).

Another function, referred to as TM_GetFieIdFIagIndexO, can provide the 'flag index' for a given field within a type. The flag index of a field is defined to be that field's index in the series of calls that are made by the function TM_ForAllFieldsLoop() (described above) before it encounters the exact path specified. This index can be utilized as an index into some means of storing information or flags specific to that field within the type. In the preferred embodiment, these indeces include any field or type arrays that may be within the type. This function may also be used internally by a number of collection flag based APIs but may also be used by external code for similar purposes. In the event that TMJForAllFieldsLoopO calls back for the enclosing structure field before it calls back for the fields within this enclosing structure, the index may be somewhat larger than the count of the 'elementary' fields within the type. Additionally, because field flag indeces can be easily converted to/from the corresponding field path (see TMJFlaglndexToFieldPath), they may be a useful way of referring to a specific field in a variety of circumstances that would make maintaining the field path more cumbersome. Supporting functions include the following: TM_FieldOffsetToFlagIndex0 is a function that converts a field offset to the corresponding flag index within a type; TMJFlaglndexToFieldPathO is a function that converts a flag index to the corresponding field path within a type; and the function

TM_GetTypeMaxFlagIndexO returns the maximum possible value that will be returned by TM_GetFieldFlagIndex() for a given type. This can be used for example to allocate memory for flag storage. Another function, referred to as TM FieldNamesToIndeeesO, converts a comma seperated list of field names/paths to the corresponding zero terminated list of field indeces. It is often the case that the 'fieldNames' list references fields within the structure that is actually referenced from a field within the structure identified by 'aTypelD'. In this case, the index recorded in the index list will be of the referencing field, the remainer of the path is ignored. For this reason, it is possible that duplicate field indeces might be implied by the list of 'fieldNames' and as a result, this routine can also be programmed to automatically eliminate duplicates.

A function, hereinafter name TM_GetTypeProxy , could be used to obtain a proxy type that can be used within collections in place of the full persistent type record and which contains a limited subset of the fields of the original type. While TM_GetTypeProxy() could take a list of field indeces, the function TMJMakeTypeProxyFromFieldsO could be used to take a comma separated field list. Otherwise, both functions would be identical. Proxy types are all descendant of the type ETJHit and thus the first few fields are identical to those of ET Hit. By using these fields, it is possible to determine the original persistent value to which the proxy refers. The use of proxys enables large collections and lists to be built up and fetched from servers without the need to fetch all the corresponding data, and without the memory requirements implied by use of the referenced type(s). In the preferred embodiment, proxy types are formed and used dynamically. This approach provides a key advantage of the type system of this invention and is crucial to efficient operation of complex distributed systems. Proxy types are temporary, that is, although they become known tliroughout the application as soon as they are defined using this function, they exist only for the duration of a given run of the application. Preferably, proxy types are actually created into type database 'E' which is reserved for that purpose (see above). Multiple proxys may also be defined for the same type having different index lists. In such a case, if a matching proxy already exists in 'E', it is used. A proxy type can also be used in place of the actual type in almost all situations, and can be rapidly resolved to obtain any additional fields of the original type. In one embodiment, proxy type names are of the form:

typeNameJProxy_n

Where the (hex) value of 'n' is a computed function of the field index list. Another function that may be provided as part of the API, hereinafter called TMJMakeTypeProxyFromFilterO) can be used to make a proxy type that can be used within collections in place of the full persistent type record and which contains a limited subset of the fields of the original type. Preferably, the fields contained in the proxy are those allowed by the filter function, which examines ALL fields of the full type and returns TRUE to include the field in the proxy or FALSE to exclude the field. For more information concerning proxy types, see the discussion for the function

TMJVΪakeTypeProxyFrornFieldsO.The only difference between this function and the function TM_MakeTypeProxyFromFields() is that TM_MakeTypeProxyFromFields() expects a commma separated field list as a parameter instead of a filter function. Another function, TM_IsTypeProxyO, could be used to determine if a given type is a proxy type and if so, what original persistent type it is a proxy for. Note that proxy type values start with the fields of ET Hit and so both the unique ID and the type ID being referenced may be obtained more accurately from the value. The type ID returned by this function may be ancestral to the actual type ID contained within the proxy value itself. The type ETJHit may be used to return data item lists from servers in a form that allows them to be uniquely identified (via the _system and _id fields) so that the full (or proxy) value can be obtained from the server later. ET Hit is defined as follows:

typedef struct ETJHit // list of query hits returned by a server

{

OSType _system; // system tag unslnt64 _id; // local unique item ID

ETJTypelD 104 _type; // type ID int32 relevance; // relevance value 0..100

} ETJΪit;

The function TM_GetNthFieIdTypeO gets the type of the Nth field in a structure. TM_GetNthFieldNameO obtains the corresponding field name, and TM_GetNthFieIdOffset0 the corresponding field offset.

Another function that may be included within the API toolset is a function called TM_GetTypeChildren . Th ^s function produces a list of type IDs of the children of the given type. This function allocates a zero terminated array of ETJTypelD 104's and returns the address of the array in 'aChildlDList'; the type ID's are written into this array. If 'aChildlDList' is specified as NULL then this array is not allocated and the function merely counts the number of children; otherwise 'aChildlDList' must be the address of a pointer that will point at the typelD array on exit. A negative number is returned in the case of an error. In the preferred embodiment, various specialized options for omitting certain classes of child types are supported.

A function, hereinafter referred to as TM_GetTypeAncestors _? may also be provided that produces a list of type IDs of ancestors of the given type. This function allocates a zero terminated array of ETJTypelD 104 and returns the address of the array in 'ancesfrallDs¹; the type ID's are written into this array. If 'ancesfrallDs' is specified as NULL then this array is not allocated and the function merely counts the number of ancestors; otherwise 'ancesfrallDs' must be the address of a pointer that will point at the typelD array on exit. The last item in the list is a 0, the penultimate item is the primal ancestor of the given type, and the first item in the list is the immediate predecessor, or parent, of the given type. The function TM_GetTypeAncestorPath produces a ':' seperated type path from a given ancestor to a descendant type. The path returned is exclusive of the type name but inclusive of the descendant, empty if the two are the same or 'ancestorlD' is not an ancestor or 'aTypelD'. The function TM_GetInheritanceChainO is very similiar to TM_GetTypeAncestors() with the following exceptions:

(1) the array of ancestor type ids returned is in reverse order with the primal ancestor being in element 0

(2) the base type from which the list of ancestors is determined is included in the array and is the next to last element (array is 0 terminated)

(3) the count of the number of ancestors includes the base type

In the preferred embodiment, this function allocates a zero terminated array of ETJTypelD 104's and returns the address of the array in 'inheritanceChainlDs'; the type ID's are written into this array. If 'inheritanceChainlDs' is specified as NULL then this array is not allocated and the function merely counts the number of types in the inheritance chain; otherwise 'inheritanceChainlDs' must be the address of a pointer that will point at the typelD array on exit. The last item in the list is 0, element 0 is the primal ancestor of the base type, and the next to last item in the list is the base type. The API could also include a function, hereinafter called TM_GetTypeDescendantsO, that is able to create a free collection whose root node is the type specified and whose branch and leaf nodes are the descendant types of the root. Each node in the tree is named by the type name and none of the nodes contain any data. Collections of derived types can serve as useful frameworks onto which various instances of that type can be 'hung' or alternatively as a navigation and/or browsing framework. The resultant collection can be walked using the collections API (discussed in a later patent). The function TM_GetTypeSiblingsO produces a list of type IDs of sibling types of the given type. This function allocates a zero terminated array of ETJTypelD 104's and returns the address of the array in 'aListOSibs', the type ID's are written into this array. If 'aListOSibs' is specified as NULL then this array is not allocated and the function merely counts the number of siblings; otherwise 'ancesfrallDs' must be the address of a pointer that will point at the typelD array on exit. The type whose siblings we wish to find is NOT included in the returned list. The function TM_GetNthChiIdTypeIDO gets the n'th child Type ID for the passed in parent. The function returns 0 if successful, otherwise it returns an error code.

The function TMJBinaryToStringO converts the contents of a typed binary value into a C string containing one field per delimited section. During conversion, each field in turn is converted to the equivalent ASCII string and appended to the entire string with the specified delimiter sequence. If no delimiter is specified, a new-line character is used. The handle, 'aStringHdl', need not be empty on entry to this routine in which case the output of this routine is appended to whatever is already in the handle. If the type contains a variable sized array as its last field (i.e., stufffj), it is important that 'aDataPtr' be a true heap allocated pointer since the pointer size itself will be used to determine the actual dimensions of the array. In the preferred embodiment, the following specialized options are also available:

kUnsignedAsHex — display unsigned numbers as hex

kCharArrayAsString — display char arrays as C strings

kShowFieldNames — prefix all values by fieldName:

kOneLevelDeepOnly — Do Not go down to evaluate sub-structures: An additional function, hereinafter referred to as TMjStringToBinaryO_? may also be provided in order to convert the contents of a C string of the format created by TM_BinaryToString into the equivalent binary value in memory.

The API may also support calls to a function, hereinafter referred to as TMJLowestCommonAncestorO, which obtains the lowest common ancestor type ID for the two type IDs specified. If either type ID is zero, the other type ID is returned. In the event that one type is ancestral to the other, it is most efficient to pass it as the 'typeID2' parameter.

Finally, a function, referred to as TM DefϊneNewTypeO, is disclosed that may be used to define a new type to be added to the specified types database by parsing the C type definition supplied in the string parameter. In the preferred embodiment, the C syntax typedef string is preserved in its entirety and attached to the type definition created so that it may be subsequently recalled. If no parent type ID is supplied, the newly created type is descended directly from the appropriate group type (e.g., structure, integer, real, union etc.) the typedef supplied must specify the entire stracture of the type (i.e., all fields). If a parent type ID is supplied, the new type is created as a descendant of that type and the typedef supplied specifies only those fields that are additional to the parental type, NOT the entire type. This function is the key to how new types can be defined and incoφorated into the type system at run time and for that reason is a critical algorithm to the present invention. The implementation is based on the parser technology described in Claimed Parser patent application and the lexical analyzer technology (the'Oaimed Lexical Analyzer") as provided in Appendix 3. As set forth above, those pending applications are fully incoφorated herein. The reader is referred to those patents (as well as the Claimed Database patent application) for additional details. The BNF specification to create the necessary types parser (which inteφrets an extended form of the C language declaration syntax) is provided in Appendix A. The corresponding lexical analyzer specification is also provided in Appendix A.

As can be seen from the specifications in Appendix A, the types acquisition parser is designed to be able to inteφret any construct expressible in the C programming language but has been extended to support additional features. The language symbols associated with these extensions to to C are as follows:

script — used to associate a script with a type or field

annotation — used to associate an annotation with a type or field @ — relative reference designator (like '*' for a pointer)

@@ — collection reference designator

# — persistent reference designator

<on> — script and annotation block start delimiter

<no> — script and annotation block end delimiter

>< — echo field specification operator

In order to complete the types acquisition process, a 'resolver' function and at least one plug-in are provided. A pseudo code embodiment of one possible resolver is set forth in Appendix A. Since most of the necessary C language operations are already provided by the built-in parser plug-in zero, the only extention of this solution necessary for this application is the plug-in functionality unique to the type parsing problem itself. This will be referred to as plug-in one and the pseudo code for such a plug in is also provided in Appendix A.

The foregoing description of the preferred embodiments of the invention has been presented for the puφoses of illustration and description. For example, although described with respect to the C programming language, any programming language could be used to implement this invention. Additionally, the claimed system and method should not be limited to the particular API disclosed. The descriptions of the header structures should also not be limited to the embodiments described. While the sample pseudo code provides examples of the code that may be used, the plurality of implementations that could in fact be developed is nearly limitless. For these reasons, this description is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

APPENDIX A Sample Code Logic for SC-Schedule View

ET_ViewHdl 3C_ScheduleView ( // schedule a view

ET_ViewHdl aView, // I: locked view handle ET_WidgetPtr force idget // I: idget to force, or

NULL

) // R:void

1 t vw = view widget (*aView) ; if ( ! (FLAGS (vw) & klsRunning) && ! (FLAGS (*aView) & kKillThisView) )

{ if ( view is the next in view starting up sequence )

1 res = SC_StartWidget (aView, w, YES) ; // check for potential to start if ( res ) // yes it will start SC_StartWidget (aView, w,N0) ; // start it!

}

} else // view is already running

{ wid = forceWidget if ( i//id ) // forced pick of a widget

SC _ScheduleANode (aView, wid) ; else

{ if ( ! (FLAGS (*aView) & kEventBasedSlot) ) // view event/normal flip-flop

{ // scan event pending list or ( wid = each item in view' s internal pending event list )

{ if ( ! (FLAGS (wid) & klsRunning) ) // non-running widget with events !

{ // can it be made to run if ( (wid->wFlags & kWidgetlsDeceased) )

{ // it just died! obliterate the offending event link to the widget report error break;

} tgt = wid; pwp,tgt = chain up to highest non-running ancestral widget stoppit = NO; // initialize to can run... cpin = get control output pin for widget cflow = connected internal control flow? if ( cflow ) stoppit = SC_CheckCtrlStop (cflow) ; if ( [stoppit )

{ // start containing compound rotate the view's pending list to avoid hang ups SCjStartWidget (aView, tgt,NO) ; // may start the ancestor FLAGS (*aView) |= kEventBasedSlot; return aView; } else if ( wid waiting for non-timer events? )

{ // time based events can wait ! cpin = get control intput pin for wid cflow = connected external control flow? if ( !SC_CheckCtrlStop (cflow) ) // ok, no control flow OR

TRUE

{ // viable- slot target SC_SetCurrentWidget (aView,wid) ; res = SC_ResumeWidget (aView, wid, YES) ; if ( res ) // resume previously running

{ rotate the pending list to avoid hang ups

FLAGS (*aView) |= kEventBasedSlot; return aView; } }

} } FLAGS (*aView) &= -kEventBasedSlot; // clear event based slot flag vw view widget (*aView) ; if ( vw- >wThreadID ) SC_StopCWidget (aView,vw) ;// propagate tokens thread el se { stoppit = NO; // initialize to can run... cpin = get control output pin for vw cflovϊ = connected internal control flow? if ( cflow ) stoppit = SC_CheckCtrlStop (cflow) ; else if (all output pin values available) // linear view widget stoppit = YES; // stop it and propagate tokens flags = FLAGS (vw) ; if ( stoppit ) SC_StopCWidget (aView,vw) ; // create/use dedicated thread el 3e // handle focus issues

{ // to make the Ul responsive wid = view focus widget (*aView) ; if ( ! (FLAGS (*aView) & kFocusBasedSlot) && ΞG->CurrentFocus == aView && wid &&

(wid->wFlags & kPriorityOnFocus) )

{

SC_ScheduleANode (aView, id) ; // give focus widget 1/2 the^' slot 3

FLAGS (*aView) |= kFocusBasedSlot;

} else // This is the normal behavior

{ // not interfere with things

SC_ScheduleNode (aView,vw) ; // could'nt stop this one FLAGS (*aView) &= -kFocusBasedSlot; // try to stop its children

}

} } if ( (*aView) ->hdr. flags & kKillThisView ) // summat nasty, kill off the view

{ terminate (aView) ; aView = NULL;

} return aView;

Code and Logic for SC-Schedule Node

void SC_ScheduleNode // schedule a compound node/widget

ET_ViewHdl aView, // I: iew handle ET_WidgetPtr aWidP // I:widget record pointer

) // R:void

{ stoppit = NO; SC_SetCurrentWidget (aView,aWidP) ; flow = get header to token list for this widget pin = follow connected inputs list to find a consumer widget while ( pin ) // till no more connected pins

{ wid = parent of the pin is the widget it is attached to, ignore WEM objects if ( RECORD_TYPE (wid) == kWidgetRecord && wid 1= aWidP ) { // pin attached to contained widget flags = FLAGS (wid) ; if ( flags & klsAtomic ) // contained widget atomic?

{

SC_SetCurrentWidget ( 0 , wid) ; if ( SC_ScheduleANode (aView,wid) ) break;

} else // attached to a compound widget { // descend and look for atomics

SC_SetCurrentWidget (aView, wid) ; if ( flags & klsRunning ) // and it's running ?

{ cpin = get control intput pin for widget cflow = connected external control flow? if ( !SC_CheckCtrlStop (cflow) ) // no flow OR set to TRUE

{ if ( wid->wThreadID ) // propagate tokens thread

{ if ( !SC_StopCWidget (aView,wid) ) break; } else

{ stoppit = NO; // initialize to can cpin = get control output pin for widget cflow = connected internal control flow? if ( cflow ) stoppit = SC_CheckCtrlStop (cflow) ; else if (all output pin values available) stoppit = YES; // stop and propagate tokens flags = FLAGS (wid) ; if ( stoppit ) SC_StopCWidget (aView,wid) ; else SC_ScheduleNode (aView,wid) ; // recurse to children break; // break outta loop

}

} else if ( SC_StartWidget (aView, wid,NO) ) break; pin = follow link to next pin on flow }

SC_SetCurrentWidget (aView,aWidP) ; rotate cIpinList of flow to ensure no pin always gets to go first rotate token list for aWidP if necessary return;

Sample Code and Logic for SC-Schedule A Node

Boolean SC_ScheduleANode ( // schedule an atomic node/widget

ET_ViewHdl aView, // I:View handle ET_WidgetPtr aWidP // I:widget record pointer

) // Rr oid

I ret = NO;

SC_SetCurrentWidget (0,aWidP) ; flags = FLAGS (aWidP) ; if ( flags & klsRunning ) // is it running ? if ( aWidP not waiting on events | | waiting for a NULL event | | waiting on any event that has arrived | | waiting on timer events )

{ cpin = get control intput pin for widget cflow = connected external control flow? if ( !SC CheckCtrlStop (cflow) ) // no flow OR set to TRUE

{ // valid slot target... if ( SC ResumeWidget (aView,aWidP, NO) ) ret = YES;

}

} } else if ( ! (FLAGS (*aView) & kKillThisView) &&

SC_StartWidget (aView, aWidP,NO) ) ret = YES; // not running so start it return ret ; }

Sample Code and Logic for SC-Start Widget

if ( SC TimeToGobblelnputs (aWidP ... ) ) // this widget might supply if { SC_StartWidget (aView, cwidg, YES ) runnit = YES;

} pin = = follow the link to the next connect! ≥d pin

}

} } else runnit = YES; // if not "as needed" then run if ( runnit && ! justCheckin ) // start this one for real so... '

1 runnit = ( flags & klsAtomic ) // if atomic allocate thread etc

^■? SC_StartAwidget (aView, aWidP,N0) // otherwise compound so copy input

SC_StartCwidget (aView,aWidP) ; // tokens to internal flows

} return 2runnit // indicate success of start }

Widget Pin Access API

void PC _NumDataInputs ( // count widget input pins

ET_WidgetPtr p, // I:widget pointer, or

NULL intPtr nFormal , // 0:# of formal input pins intPtr nDegenerate // 0:# of degenerate input pins

); // Rr oid void PC _NumDataOutputs ( // count widget output pins

ET_WidgetPtr wp, // I :widget pointer, or

NULL intPtr nFormal , // 0:# of formal output pins intPtr nDegenerate //^' 0:# of degenerate output pins

); // R:void

Handle PC_ _GetDataInput ( // gets a data input value

ET_WidgetPtr wp, // I: widget pointer, or

NULL char pin // I : input pin A-Z, a-z select

); // R:pin value or NULL

Handle PC_ _GetInputDefault ( // Get default input pin value

ET_WidgetPtr p, // I:widget pointer, or

NULL char pin // I : input pin A-Z, a-z select

⁾; // R: default value, NULL if none

ETJType ID PC_ _GetDataInputType ( // get input pin type

ET_WidgetPtr p, // I:widget pointer, or

NULL char pin // I: input pin A-Z, a-z select

); ^' // R:type ID, 0 if error

ETJTypelD PC_ _GetDataOutputType ( // get output pin type

. ET_WidgetPtr wp, // I:widget pointer, or

NULL char pin // I: output pin A-Z, a-z select

); // R:type ID, 0 if error

Boolean PC_SetDataInputType ( // set input pin type

ΞT_WidgetPtr p, // I :widget pointer, or

NULL char pin, // I : input pin A-Z, a-z select

ET_TypeID aTypelD // I:The type ID to set it to

); // RrTRUE for success Boolean PC__SetDataOutputType ( II set output pin type

ET_WidgetPtr wp, II I:widget pointer, or

NULL char pin, II I: output pin A-Z, a-z select

ETJTypelD aTypelD II I: The type ID to set it to

⁾; II R:TRUE for success charHdl PC_ _GetDataInputName ( II get input pin name (if any)

ET_WidgetPtr wp, II I:widget pointer, or

NULL char pin II I: input pin A-Z, a-z select

⁾; II R:pin name, NULL if error charHdl PC_ _GetDataOutputName ( II get output pin name

(if any)

ET_WidgetPtr wp, II I:widget pointer, or

NULL char pin II I:output pin A-Z, a-z select

⁾; II R:pin name, NULL if error

Boolean PC_ _IsDataInputConnected ( II is input pin ext . connected?

ET_WidgetPtr wp, II I: widget pointer, or

NULL char pin II I: input pin A-Z, a-z select

⁾; II R:TRUE if connected externally

Boolean PC_ _IsDataOutputConnected ( II is output pin ext. connected?

ET_WidgetPtr wp, II I:widget pointer, or

NULL char pin II I: output pin A-Z, a-z select

⁾; II RrTRUE if connected externally

Boolean PC_ _DoesOutputHaveToken ( II output pin token present?

ET_WidgetPtr wp, II I:widget pointer, or

NULL char pin II I:output pin A-Z, a-z select

); II RrTRUE if has token

Boolean PC_ _SetDataOutput ( II writes a value to an output pin

ΞT_WidgetPtr wp, II I rwidget pointer, or

NULL char pin, II I r output pin A-Z, a-z select void *data, II I pointer to data to write long size II I size in bytes of data

, >; 1/ RrTRUE on success void PC_SetControlOutput ( II Alter control output state

ET_WidgetPtr wp, II I rwidget pointer, or

NULL

Boolean pinstate II I state TRUE (go)

/FALSE (stop)

); II R oid charHdl PC_GetInputFlowName ( II Get input flow name

(if any)

ET_WidgetPtr wp, II I widget pointer, or

NULL char pin II I r input pin A-Z, a-z select

); II Rrflow name or NULL

ETJTypelD PC_GetInputFlowType ( II Get input flow type ID

ET_WidgetPtr wp, II I rwidget pointer, or

NULL char pin II I input pin A-Z, a-z select

); II RrType ID of flow, 0 if none charHdl PC_GetOutputFlowName ( II Get output flow name

(if any)

ETJWidgetPtr wp, II I rwidget pointer, or

NULL char pin II I output pin A-Z, a-z select

>; II Rrflow name or NULL

ETJTypelD PC_GetOutputFlowType ( II Get output flow type

ID

ETJWidgetPtr wp, II I r widget pointer, or

NULL char pin II I r output pin A-Z, a-z select

); II RrType ID of flow, 0 if none

Boolean PCJDoesInputExist ( II determine if input pin exists

ET_WidgetPtr wp, II I: idget pointer, or

NULL char pin II I r input pin A-Z, a-z select

>; II RrTRUE if input exists

Boolean PC_DoesOutputExist ( II determine if output pin exists

ETJWidgetPtr wp, II I rwidget pointer, or

NULL char pin II I output pin A-Z, a-z select

); II RrTRUE if output exists Handle PC_GetStaticDataInput ( // get input value

(tokenless)

ET_WidgetPtr wp, // I widget pointer, or

NULL char pin // I r input pin A-Z, a-z select

⁾; // R value handle or NULL

Claims

1) A system for implementing large-scale distributed data-flow based applications comprising the following:

a) A data-flow based scheduling environment, for managing the execution of one or more control-flow based functional building blocks based on one or more rules of data flow;

b) A visual programming environment, wherein such environment provides the ability to build and control the flow of data collections between small control- flow based functional building-blocks such as widgets within the scheduling environment;

c) A pin-based API that provides access to contents from the executing code within the widgets;

d) A strongly-typed run-time accessible type system; and

e) One or more building block widgets that perform one or more actions in response to data-flow arriving at widget input pins.

2) The system of claim 1, wherein the data-flow based scheduling environment includes one or more scheduler routines, wherein such routines help balance the needs of data initiated program execution with considerations such as user responsiveness.

3) The system of claim 1, wherein the data-flow based scheduling environment includes one or more scheduler routines, wherein such routines help balance the needs of data initiated program execution with considerations such as widget hierarchy.

4) The system of claim 1, wherein the building block widgets are network distributed building block widgets.

5) The system of claim 1, wherein the programming language supports the creation of the layout and specification of the visual appearance of the running widget(s) in a user interface 6) The system of claim 1, wherein each widget has one or more widget pins are interconnected via typed flows which transfer data from the output pin of the source to the input pin of the sink in a tokenized manner (i.e., data is consumed in the process).

7) The system of claim 1, wherein the visual programming environment can be used to develop one or more atomic widgets, such atomic widgets containing compiled executable code.

8) The system of claim 7, wherein the visual programming environment can be used to develop one or more compound widgets, such compound widgets including an inner structure that defines any subordinate widgets that are required to implement the compound widget's functionality.

9) The system of claim 8, wherein one or more atomic widgets is embedded into a compound widget.

10) The system of claim 9, wherein the visual programming environment can be used to develop one or more widgets that are instantiated within container objects (views) which associate the user interface (e.g., windows) to widgets.

11) The system of claim 10, wherein the visual programming environment can be used to assign one or more priorities to a view level and those priorities can be used by the scheduling environment.

12) The system of claim 1, wherein one or more of the building block widgets include a confrol input pin that determines whether the widget will run.

13) The system If claim 12, wherein one or more of the building block widgets include a confrol output pin.

14) The system of claim 12, wherein any confrol flow connections between building block widgets are level-based and cannot be consumed.

15) The system If claim 1, wherein one or more of the building block widgets include formal inputs and outputs.

16) The system If claim 1, wherein one or more of the building block widgets include degenerate inputs and outputs. 17) The system If claim 1, wherein one or more of the building block widgets include degenerate inputs and outputs having default values.

18) The system of claim 8, wherein a compound widget includes one or more constants and one or more variables within its data-flow definition.

19) The system of claim 8, wherein a compound widget includes a specification of the types of values associated with any flow, pin, constant, or variable based on the types defined within the run-time type system.

20) The system of claim 8, further comprising a graphical run-time debugging environment that permits examination and alteration of the state of all typed data internal to any compound widgets or widget pins.

21) The system of claim 1, wherein the types system is the types system claimed in the Types Patent.

22) The system of claim 1, wherein the data stractures created by the visual programming environment use the memory model claimed in the Memory Patent.

23) The system of claim 1, wherein the data-flow based scheduling environment supports specification of 'OR' consumption that results in the consumption of a data token by the first widget that has inputs available for the data token.

24) The system of claim 23, wherein the data-flow based scheduling environment supports specification of 'AND' consumption that results in the consumption of a data token only when all connected widgets have run.

25) The system of claim 24, wherein the data-flow based scheduling environment does not permit a token to be added to a flow until the existing data token is consumed.

26) The system of claim 10, further comprising widget packs, such that each widget in a packet shares a common context but can independently execute in accordance with the scheduling environment.

27) The system of claim 10, wherein the scheduling environment can ran any widget to enable another widget to run according to the rules of data flow.