WO2002052528A1

WO2002052528A1 - Method and system for assessment of user performance

Info

Publication number: WO2002052528A1
Application number: PCT/US2001/043056
Authority: WO
Inventors: Harry Kennedy Clark; James L. Boney; Kasempath Meesuk
Original assignee: Element K Online Llc.
Priority date: 2000-11-13
Filing date: 2001-11-13
Publication date: 2002-07-04
Also published as: EP1344202A1

Abstract

A method and system for assessing a user (115) regarding control of one or more devices that include comparing information regarding a configuration (105) of at least one of the devices against at least one evaluation criteria (103), comparing information regarding state information (106) for the device against at least one evaluation criteria, and assessing the user using the above comparisons.

Description

METHOD AND SYSTEM FOR ASSESSMENT OF USER PERFORMANCE

BACKGROUND OF THE INVENTION

The following describes systems and methods for assessing a user's proficiency regarding a device (or set of devices) by evaluating the user/student's control over the device(s). "Controlling the device(s)" includes their ability to correctly configure, troubleshoot, test, diagnose, initialize, set up, build, arrange, and analyze these devices.

Traditionally, students are assessed based on taking a test where they are asked multiple choice and/or true-false questions regarding the device or control of the device, thus testing their knowledge regarding control of the device. In an embodiment of the present invention, rather than simply asking a student questions regarding the device, the student is presented with a real world task regarding the control of the device (or set of devices). The student then exercises control over the one or more devices to perform the task. In an embodiment, the student may exercise control over one or more devices remotely over a network such as the Internet or a LAN. For example, the student may exercise control over the one or more devices using Mentor Technologies™ vLab™ system. For a more detailed description of a system for remote training on devices, see "Methods and Apparatus for Computer Based Training Relating to Devices," of T. C.

Slattery, et al., U.S. Patent Application No. 09/365,243, filed July 30, 1999, which is hereby incorporated by reference. After completing the task, the student is assessed on his/her performance or skills in controlling the device(s).

SUMMARY OF THE INVENTION

Methods and systems consistent with the present invention include systems and methods for assessing a user regarding control of one or more devices that include comparing information regarding a configuration of at least one of the devices against at least one evaluation criteria, comparing information regarding state information for the device against at least one evaluation criteria, and assessing the user using the above comparisons

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 illustrates an assessment system in accordance with methods and systems consistent with the invention;

Figure 2 illustrates a screen in accordance with methods and systems consistent with the invention;

Figure 3 illustrates an example output in accordance with methods and systems consistent with the invention;

Figure 4 illustrates an illustration of a results for report in accordance with methods and systems consistent with the invention;

Figure 5 illustrates an example analysis matrix report in accordance with methods and systems consistent with the invention;

Figure 6 illustrates an example configuration report in accordance with methods and systems consistent with the invention; Figure 7 illustrates an example non-configuration report in accordance with methods and systems consistent with the invention;

Figure 8 illustrates an example screen in accordance with methods and systems consistent with the invention; and

Figure 9 illustrates an example user administration screen in accordance with methods and systems consistent with the invention.

DESCRIPTION OF THE EMBODIMENTS

Figure 1 provides an illustration of an assessment system, in accordance with methods and systems consistent with the invention. In an embodiment, either during an assignment or once an assignment regarding control of a pod including one or more device(s) 101 is completed, the student can request that they be assessed based on their performance of the training exercise. For example, Figure 2 illustrates a screen that may be presented to a user performing an assignment using the vLab™ system. As shown, the user may be permitted to select an "Assess Me" button 210 to provide feedback in the middle of an assignment or exercise. By selecting this button 210, assessment may be initiated. In other embodiments, assessment may be initiated automatically upon completion of an exercise. This may occur, for example, in a classroom type setting where the assignment is performed as a test of the students' abilities. In addition, assessment may be performed in other types of systems where the student exercises actual control over devices on which they are being evaluated. Once assessment is initiated, various types of information may be gathered and transferred to a Grader Engine 107. This information may include device configurations 105, state information 106 regarding the device(s), SNMP results 104 from the devices in the pod 101 and other devices connected to the pod's devices, and/or other information. These various types of data will be collectively referred to as "Device Information". In addition, to device information, information is gathered regarding grading and/or Evaluation Criteria 103 ("Evaluation Criteria").

After collecting, analyzing and comparing the Device Information to the Evaluation Criteria 103, the Grader Engine 107 generates an output that may include a variety of reports with information regarding student performance. Figure 3 illustrates an example output 300 that may be presented to a vLab™ system user. As shown, this output may include a Results For report 310, an Analysis Matrix report 320, a Configurations report 330, and a Non- Configuration Information report 340. These various reports will be discussed in more detail below. As will be obvious to one of skill in the art, this is only one example of an output that may be presented to a user, and there are numerous other types of outputs containing various reports that may be presented to the user. This will depend in part on the type of system and types of devices using the present invention, in addition to other criteria.

In evaluating student performance the Grader Engine 107 may execute a series of diagnostic commands that captures the actual state of the network to which the device is connected, thus allowing the Grader Engine 107 to analyze real-time traffic such as ping, traceroute, adjacencies, routing tables, and other diagnostic commands regarding that device and/or network. The results and diagnostics commands can be issued either during the lab for "real time" evaluation, or at the end of the lab and stored in a database for future reporting.

Further, the Grader Engine 107 may use pattern matching and parsing technology to evaluate the Device Information (104, 105, and 106) against the Evaluation Criteria 103. The pattern matching and parsing technology may be presented in a hierarchy of "functions" for purposes of authoring the assessment. These provide a range of flexibility and power. For example, there can be "general-purpose" functions where the author of the assessment specifies the raw pattern match or parser, "wrapper" functions that are easier to use but less flexible, and "canned" functions that hide the parsing details but are specific in their use.

General-purpose functions involve the use of regular expressions, a pattern-matching language commonly used in UNIX and programming environments. Consequently, these functions are extremely flexible, but more difficult to use because they require the author to understand the regular expression language or other forms of pattern matching logic.

Wrapper functions take a regular expression and other forms of pattern matching logic supplied by the author and automatically "wrap" it inside of a larger regular expression, pattern matcher, or programming logic. Adding this "wrapper" makes the author's job considerably easier because it saves them from having to write complex expressions that only match in the desired context. For example, writing an expression that only matches an IP address on a given interface can be fairly tricky (it is easy for the IP address to inadvertently match on a different interface earlier or later in the config). The interface( ) wrapper function automatically limits the expression to the specified interface (or list of interfaces), allowing the author to concentrate on the much simpler process of matching on something inside that interface (for example, "ip address 1\.1.\1.\.1 255\.255\.0\.0" to ensure that the interface has an address of 1.1.1.1 and a /16 mask). Given that many interface (and related) matching tasks only require very basic (or no) wildcard characters, writing a criterion using a wrapper function is normally extremely simple. However, for more complex requirements, the author can always resort to the full power of regular expressions and other forms of pattern matching and parsing logic.

Canned functions are tailor-made to solve specific assessment requirements. Because they totally insulate the author from having to write complex expressions, they are extremely easy to use. However, their use is also considerably more limited than that of the "general-purpose" and "wrapper" functions. For example, the shCdpNeigh( ) function is only designed to process the output of the "show cdp neighbors" command. Although it is flexible enough to automatically determine if the command was issued with the "detail" option and automatically adjust its logic, it will never be useful for looking at other types of router information (for example, the routing tables). On the other hand, shCdpNeigh( ) is very easy to use: simply tell the function which devices you want to process CDP output from and a list consisting of: (i) a neighbor's name, (ii) the local interface used to reach the neighbor, (iii) the neighbor's interface used to reach the local router, and (iv) a list of Layer 3 protocols and addresses. This, and other, functions can allow "wildcards" to be specified by omitting various parameters.

The Evaluation Criteria 103 may be based on a set of desired learning objectives which are allocated differing amounts of grading points based on the relative importance of the specific learning objective. By comparing the Device Information (104, 105, and 106) to the Evaluation Criteria 103, the Grader Engine 107 may determine whether the student has met the relevant learning objects, award full or partial credit, deny credit altogether, and then generate an overall score.

In addition, the Grader Engine 107 may include a "land mine" feature that deducts points from as student's score when student enters certain commands into, or takes certain actions with respect to, the device, e.g., enters commands to try to circumvent the learning exercise. That is, the Grader Engine 107 may include the ability to look for certain types of actions that indicate that a student attempted to "cheat" the exercise.

Further, the Grader Engine 107 may include the capability to grant partial credit. The granting of partial credit may be made either based on pre- established criteria or new criteria established by the Grader Engine 107 based on specific Device Information (103, 104, and 105). This may be accomplished by the Grader Engine 107 using the above-described pattern matching and parsing technology, as well as by establishing a logical hierarchy between multiple criteria. This feature allows the Grader Engine 107 to assess a multitude of possible solutions a student may arrive at in trying to perform the designated tasks. Furthermore, use of pattern matching and parsing technology to permit an automated grading approach does not require that the author specifically address every possible solution to the learning exercise.

In addition, the system may include a Help Engine 108 that permits the student to link to other information related to a specific learning objective. These links may include technical notes, reference materials, and listings of classes or seminars that address that objective, among others. The Help Engine 108 is a software module that is triggered when the user selects a help link or function from one of the various types of feedback reporting produced by the Grader Engine 107 and its associated output modules. In generating the help information, the Help Engine 108 will access information in the Evaluation Criteria 103 and other possible sources such as a meta- database of remedial information and feedback.

The results generated by the Grader Engine 107 may be used to feed a variety of other outputs, such as an HTML Rendering Engine 109, XML Engine 111 , or other forms output 110, which in turn can, among other things, generate a variety of reports, including one that lists the learning objectives, number of maximum grading points allocated to each learning objective, and the actual number of points awarded to the student based on his or her performance. The HTML Engine 109 is a software process that generates information to be sent to a web browser via a network such as the Internet. The XML and other output engines are similar software processes, but they can output the results of the assessment information in a wide variety of report and data transfer formats.

In addition there may be sections of the report that a user may click on to link to information regarding specific learning objectives, the corresponding configurations, and/or state(s) resulting from the student's performance in the learning exercise. This may be useful in highlighting what the student did correctly or incorrectly. These sections that the user may click on may identified, for example, by shading certain words a particular color, underlining certain words, or by particular icons.

The system may also include a variety of security and administrative features 112 as well as other applications 113. Examples of these features include allowing the system administrator to prohibit a student from accessing the help function, viewing details of the lab before a testing situation, taking a test more than once, disabling various "mentoring" features in a testing situation, disabling certain levels of detail in the output report.

Figure 4 provides an illustration of the Results For Report 310 that was previously discussed in reference to Figure 3. This report may include overhead type information. For example, as illustrated this report may include the user's name 410, the title for the assignment 420, the time the assignment was purchased or selected by the user 430, the time it was started by the user 440, the time it was completed 450, the user's IP address 460, a title or identification for the pod used during the assignment 470, and the number of times the user attempted this particular assignment 480.

Figure 5 illustrates an example of the Analysis Matrix 320 Report that was previously discussed in regard to Figure 3. As illustrated, this report lists various learning objectives 510 that the user is assessed on. Each learning objective may include a key 520 that may include the words "Show Me" or a similar icon. For learning objectives where the key includes the words "Show Me," the user may click on these words to jump to relevant sections of the configuration code created during the assignment that enable the user to see what they did right and what they did wrong during the assignment. Further, these keys (e.g., Show Me) may be color coded or shaded a particular color. This color or shading may then be used as described below in reference to the below described configuration reports and non-configuration reports.

In addition, a description 530 may be presented for each learning objective. Further, a maximum score field 540 may be listed for each learning objective may be listed. This maximum score field shows the total point that may be awarded for this learning objective if it is completed successfully. In addition, a score field 550 may be listed for each learning objective. This score field 550 lists that score that the user was awarded for the learning objective. As shown, partial credit may be awarded to a user who is not completely successful in completing the learning objective. Also, a help link 560 may be presented for each learning objective. A user may click on this help link to view additional information regarding this learning objective, such as information concerning the technical notes, reference materials, classes, other distance learning components, etc. In addition, this report may include information regarding the maximum possible raw points 572, the user's raw points 574, the user's raw score 576, any administrative adjustment 578, and the user's final score 580.

Figure 6 illustrates an example of the Configuration Report 330 that was previously discussed in reference to Figure 3. As discussed above, with reference to Figure 5, a user may click on the text "Show Me" in the Analysis Matrix Report to jump to relevant sections of the configuration code. For example, by clicking on the "Show Me" text for a learning objective, the user may be presented with a Configuration Report 330 regarding the learning objective, such as illustrated in Figure 6. Further, various information in the Configuration Report may be identified by a color or shading corresponding to the learning objective for which the "Show Me" text was selected.

As shown, the Configuration Report 330 may include information regarding each of the devices in the pod 110. In the example illustrated, these devices include a router for Washington, DC 610, a router for Minot 620, and a router for Leesville 630. For each of these devices, the Configuration Report may include information regarding the configuration for the device. Figure 7 illustrates an example of the Non-Configuration Report 340 that was previously discussed in reference to Figure 3. As previously discussed, the Grader Engine 107 may execute a series of diagnostic commands that capture the actual state of the network. This therefore allows the engine to analyze real time traffic, such as ping, traceroute, adjacencies, routing tables, and other show commands. Further, as discussed above with regard to the Configuration Report 330, information in the Non-Configuration Report 340 may be identified by a particular color or shading. This shading or color preferably corresponds to the shading or color of the "Show Me" key for a particular learning objective. This helps a user to quickly identify the information in these reports that corresponds to the particular learning objective.

Figure 8 illustrates an example of a screen that may be presented to a user that clicks on one of the help links 560 illustrated in Figure 5.

In addition to the above, the user may access a User Administration screen. Figure 9 illustrates an example of a User Administration Screen 900 that may be presented to a user, teacher or system administrator. As shown, this screen may list the various users that performed particular assignments by last name 902, first name 904, login ID 906, and group 908. Further, this screen may list the descriptions 910 for the assignments performed, along with their score 912, and the attempt number 914 for the score. For users with more than one attempt, the score for each attempt may be listed by clicking on the attempt number and then selecting the attempt number for which the user desires to view the score. In addition, buttons may be presented that allow the user to view the report 916 and the user's options 918. Information obtained by selecting to view the user's options may include, for example, setting the administrative group the user belongs to as well as certain administrative flag that control behavior such as multiple attempts at a single exercise and removal of invalid test results.

Further, a data export button 920 may be presented to allow the data to be exported to a printer, floppy drive, some other storage device, or in a variety of formats that can be read by other systems, software packages, and databases. For example, this feature can be used to export the data to spreadsheet software. Further, scroll downs or filters, may be provided that allow a user to view the performances by individuals in a particular group 922, by the lab or assignment taken 924, the time or day during which the assignment was performed 1026. Also, a Hidden function 928 is illustrated that if selected hides or removes invalid test results from reports and export screens by default.

The above-defined methods may be performed by one or more computers or servers that is/are capable of obtaining the above described information. In addition, the above described method may be embodied in software that one or more processors in one or more computers are capable of executing

Also, although the above-described methods and systems were discussed with references to routers, they may also be used for any other type of devices, such as switches, computers, servers, PLCs, etc. Further, the above-described methods and systems also may be applied to assess a user with regard to software, such as NT, MSWord, UNIX, etc.

Appendix A presents various figures concerning an application of the above-described methods and systems as used in vLab™ systems with routers. Appendix B presents text corresponding to these figures.

While it has been illustrated and described what is at present considered to be the preferred embodiment and methods of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made, and equivalents may be substituted for elements thereof without departing from the true scope of the invention.

In addition, many modifications may be made to adapt a particular element, technique or, implementation to the teachings of the present invention without departing from the central scope of the invention.

If a user "purchases" and takes one of these assessment labs then they might see the "Assess Me!" button appear while they are in the console.

The "Assess Me!" button will only appear if the super-admin has selected the "Allow Assessment During Lab" flag. This feature can be enabled in situations where real-time feedback and guidance is appropriate - it's the ultimate form of "hints and tips" for the user! Once the user has completed the lab, a process automatically runs that evaluates the user's results and writes a grade to the database and grade log. When the user returns to their locker, they will now be presented with an "Assessment" option for the completed assessment labs.

0303. BSCN Assessment Completed | Assessment ^όj

If they select this and hit the "Gp!"-^utton, the Assessment Analysis Report will open.

Time Added To Locker Oct 1 1 12:49:54 2000.

Time Started: Oct 1 1 12:59:45 2000 A Time Completed: Oci 1 1 13:25:57 2000

User IP Address: 1.1.1.1

Pod Used: 1

Attempt Number: 3rd

K Description M: IX Score Help!

UI WJAM 'router rip' removed 30 10 ? -

Show Vic 'router igrp' present 45 45 ? nder IGRP 30 30 ? ~

-τ> ; 'network 10.0.0.0' present u β ft^' Bandwidth parameter present 30 10 ? ^'-^•

DC SerialO & Leesville Serial 1 on same subnet 15 15 ?

DC Serial 1 & Minot SerialO on same subnet 15 15 ? >

Minot Serial 1 & Leesville SerialO on same subnet 15 15 7

All 6 links use different subnets 20 20 ? >

All 9 addresses are unique 65 65 ? >

Show Me All 3 use same IGRP AS # 45 30 ? >

All addrs fall btw. 10.28.0.0 & 10.33.0.0 45 45 ? :-^■ onfiαurations:

WashingtonDC Minot Leesville

version 11.3 version 11.3 version 11.3 hostname WashingtonDC hostname Minot hostname Leesville enable password Cisco enable password cisco enable password Cisco

on-Confiαuratϊon Information

WashingtonDC: show cdp neighbors

Capability Codes: R Router, T - Trans Bridge, B - Source Route Bridge

4^* S Switch, H - Host, I - IGMP, r - Repeater

- /X du Device ID Local Iπtrfce Holdtrae Capability Platform Port ID PODSWl-4000196E77880Eth 0 148 T S 1900 1 lUJAUm-l Ser 0 165 R 2520 Ser 1

KinoC Ser 1 164 R 2520 Ser 0

WashingtonDC: ping (extended)

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to WHcimnw timeout is 2 seconds:

MP I Success rate is ___f_\ percent (5/5) , round-trip rain/avg/max = 32/34/36

WashingtonDC: ping (extended)

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to H<H-fWlf< timeout is 2 seconds:

Success rate is 1 percent (0/5i

Key Description lax i Score Help!

EflKffiy 'router rip' removed 30 10 ?> Show Me 'router igrp' present 45 45 ? > l≥ϊji ^'J f 'network 10.0.0.0' present under IGRP 30 30 ? > Show Me Bandwidth parameter present 30 10 ?> flHBH DC SerialO & Leesville Serial 1 on same subnet 15 15 ?> Show Me DC Serial 1 & Minot SerialO on same subnet 15 15 ?> HHHi Minot Serial 1 & Leesville SerialO on same subnet 15 15 ?>

All 6 links use different subnets 20 20 ? __>

All 9 addresses are unique 65 65 ? > Shovv Me_. All 3 use same IGRP AS # 45 30 ?>

All addrs fall btw. 10.28.0.0 & 10.33.0.0 45 45 ?>

AH interfaces use /16 subnet masks 90 90 ?> - IBRBMrJSl DC can ping EthernetO on Minot and Leesville 20 10 ?>

Minot can ping EthernetO on DC and Leesville 20 20 ?>

Leesville can ping -EthernetO on DC and Minot 20 20- ?> Minot and Leesvile see DC via CDP 20 20 ?> DC. and Leesville see Minot via CDP 20 20 ?>

DC and Minot see Leesville via CDP 20 20 ?>

Raw Points 565 500

Raw Score 88.5

Admin Adjustment 2.6

Final Score 91.1

ScSJBctO ( ) onfϊαuratϊons.

WashingtonDC Minot Leesville

I version 11.3 version 11.3 version 11.3

I ! ! hostname WashingtonDC hostname Minot hostname Leesville ! ! I enable password Cisco enable password Cisco enable password Cisco

nterface EthernetO interface EthernetO p address 10.28.0.1 255.255.0.0 ip address 10.30.0.1 255.255.0.0 I ! interface SerialO nterface EthernetO ip addressJQ.29Λ^'.22 ^~ 55255 i. p address 10.32.0.1 255.255.0.0 no shutdown I

! interface SerialO ! interface Seriall interface Seriall ip address;iQ.29.0;.r.255.255^".^'0:q clockrate 56000 no shutdown clockrate 56000 no shutdown ! - interface Seriall clockrate 56000 no shutdown

I network 10.0.0.0 !

router igr 2 banner motd *C --SA-jM-ls ICND Assessment vLab banner motd C ICND Assessment vLab ip classless

WashingtonDC

Minot end ! banner motd C end ICND Assessment vLab

Leesville end

on-Confiq vation Information:

WashingtonDC: show cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater

Device ID Local Intrfce Holdtme Capabili y Platform Port ID PODSWl-400Q19βE77880Eth 0 148 T S 1900 1

' 1S5 R 2520 Ser 1

MWino-tWW ^sS^ee^rr 1 164 R 2520 Ser 0

WashingtonDC: ping (extended)

Type escape sequence to abor .

Sending 5, 100-byte ICMP Echos to mmcnmi.il, timeout is 2 seconds:

M i l l

P« ^■'P Success rate is MPM percent (5/5 ) , round- trip mm/avg/max = 32/34/36 ms

WashingtonDC: ping (extended)

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to m«n ιιW timeout is 2 seconds:

Success rate is _] percent (0/5)

WashingtonDC: show ip route summary (bytes)

col

WashingtonDC: trace ip 10.32.0.1

Type escape sequence to abor . Tracing the route to 10.32.0.1

Minot: show cdp neighbors

<Η Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

Switch, H - Host, I - IGMP, r - Repeater

mentor} technologies-

X ^' A AςSςSFEςSSMMFE This test verifies that you correctly assigned IP addresses within your network.

IP addresses have two parts:

• A network portion

• A host portion

In most Layer 3 addressing schemes, the boundary between these parts is fixed. (For example, in IPX, the network portion is always 32 bits, while the host portion is always 48 hits.)

In IP, the boundary between the network and host portions of an address is variable. The subnet mask is used to determine where the boundary has been placed!

For details on the command to assign IP addresses, you may wish to refer to the following documentation link:

• IP address ip-address mask

For more information on IP, please refer to the following links:

Mentor Techolρgies Whiteboard Animation- IP Subnet Planning Cisco Technology Brief on IP

Click below for a description of the instructor led training course which covers this command:

ICND - Interconnecting Cisco Network Devicejs

menό^technoJogie-r VLAB ASSESSMENT ADMIN

-)l ASaSESbS iv

Assessment Summary:

Filters

Group:)^AI1 ~M Lab: All P_erio_drlWeek g _Hidden: π H tW

Last Name. FijstJVame Login ID Group Assessment Score Attempt Report User Options demo demo demo Unassigned 0800. A Demo Assessment — Mig |74.6 5th I Mew ocε jcAn tø«Λ _^ Unassigned 0800. A Demo Assessment — Mig 71.4 1st - Vfew- ά <z α

Unassigned 0800. A Demo Assessment — Mig 71.4 1st ^'View View λσe- o ^o U Unassigned 0800. A Demo Assessment ~ Mig 84.1 2nd r vifew

SC^QBΞΛ. ,;.

} > e. a \χ

Cell 1

INTRODUCTION^'

Special Effects Visual:

Mentor Technologies is pleased to

Keep theme of Gears and Knowedge from announce its "vLAB Assessment

Intro. Highlight. Technology - the world^'s first real-time,

^"VLAB ASSESSMENT TECHNOLOGY" skills-based assessment that gages a CISCO learner^'s true readiness to perform.

Words:

"It's not what you know. It's what you

It's not what you "know." It's what you '"do" '^•do" that counts." that counts.

Cell 2

Through our new "vLab Assessment Technology," we bring two ''new" Visual dynamics to Cisco learning - Words: "performance-based testing" and Performance-based testing "mentoring;. " Mentoring

Cell 6

This opens the Assessment Analysis Report Show report with 4 sections all together containing 4 primary sections. (Screen 2 )

The heart of the report is the '"Analysis

Matrix.^" Blow out of "Analysis Matrix" Portion of the report (2^nd area) (Screen 3)

Cell 7

Each row contains a description of the Burst out on an entire row. learning objective, and the learner's actual performance against that objective.

Cell 8

The "Max" field shows the total possible Highlight the "Max" column. points awarded if the objective is met successfully.

Cell 9

And the "Score" field reflects the learner's Highlight the "Score" Column. score in meeting that objective. Learners Burst out on a red- item where the score is quickly learn to gage their progress, as lower than the Max column. partial credit is reflected in the "Score" field if the objective is not totally met. •

Claims

WHAT IS CLAIMED IS:

1. A method for assessing a user regarding control of one or more devices, comprising: comparing information regarding a configuration of at least one of the devices against at least one evaluation configuration criteria; comparing information regarding state information for the device against at least one evaluation state criteria; and assessing the user using the above comparisons.

2. The method of claim 1 , further comprising the step of obtaining information using the Simple Network Management Protocol (SNMP); and comparing the information obtained using the Simple Network Management Protocol (SNMP) against at least one evaluation criteria, wherein the step of assessing the user includes using the comparison using the information obtained using the Simple Network Management Protocol (SNMP).

3. A method for assessing a user regarding control of one or more devices, comprising: comparing information regarding at least one of the devices against at least one evaluation criteria; assigning one or more weights to one or more of the evaluation criteria; generating at least one partial credit value in regard to the comparison; and assessing the user using the above comparisons and the one or more devices.

4. A method for assessing a user regarding control of one or more devices, comprising: comparing information regarding at least one of the devices against at least one evaluation criteria; generating at least one partial credit value based on the comparison; and assessing the user using the above comparisons and the at least one partial credit value.

5. The method of claim 1 , further comprising the step of: providing a report regarding the assessment.

6. The method of claim 5, wherein the report provides one or more of the following capabilities: linking to help information; linking to information regarding the configuration of the one or more devices; and linking to information regarding the states of the one or more devices.

7. The method of claim 1 , further comprising remotely accessing the one or more devices; and

exercising control over the device by the user to perform a training exercise, wherein the user is assessed based on their performance of the training exercise.