The quantitative analysis of the players' performance and collaborative work is
based on information from the simulation and on information from the players'
use of the information tools.
The goal of the quantitative analysis is to classify the information, enable frequency analysis,
time measurements and other quantitative measurements.
The information tools in the C3Fire system are the map (GIS),
the email and the diary. The nature of the information in the simulation and
GIS is simply to classify and can therefore easily be used
to obtain quantitative information such as statistics and frequency information.
The information classification of the email and diary use is a bit
harder but can be done by parsing the information in the messages.
The analysis of the simulation information has mainly been used for
performance measurements on how well the subjects have solved their task.
The analysis of the GIS information has mainly been used to observe the situation
awareness and some of the information distribution.
By using the session replay function, the training manager and the students can
perform a high-speed play-back of the whole session.
The replay shows simulated activities such as the fire and the
positions and activities of the fire-fighting units.
The main reason for having a session replay function is that research on simulation-based
and collaborative training shows that by using a reflection stages after a
session, increase the transfer effects and are an important step in a training process.
The session replay used in an after-action review provides a
good opportunity for the students to maximise group interaction
to investigate their view of the situations and what they learned.
The experience gained from using the session replay is that the use of
the replay is an important step in the learning. When the students observed the session replay,
they often starts to interact in presenting their view of what they thought had
happened during the session, and what to do in future engagements.
We believe that this function is one of the most important components
in the C3Fire environment and that the motivational appeal of the simulation
increases dramatically when the students can see a replay and discuss their collaboration.
It is important to note that the sessions is performed on several distributed workstations,
while the replay only can be performed on one single workstation.
Read more about the replay features at
Defining the monitoring goals and measurement methodes in a training session is a complex task.
Example on four monitoring goals that has been used in C3Fire during three studies (Shared representations in ROLF 2010
; Team Situation Awareness using Graphical or Textual Databases
; Collaboration in Serial and Parallel Organisations
) are: the effectiveness of the teams,
the information distribution in the organisation,
the students' situation awareness, and the students' work and collaboration methods.
Effectiveness of the Teams
One way to evaluate the sessions and see if the team has performed well is to use some kind of scoring system.
A scoring system should change depending on the training goal and the task given to the players.
In all three studies a score system that should tell us if the students
have succeeded in their task of extinguishing the forest fire has been created by
automatically analysing the extent of burned down area and destroyed objects.
The weakness of this type of scoring system is that it may not capture other requirements
on the team than fighting the fire.
In real life a command team has other responsibilities, such as creating an activity diary,
responding to incoming information from outside the organisation, etc.
Our experiences is that a simple scoring system that is only based on pure
effectiveness is not enough if the session is to be viewed as a good training
session when the environment and the task get as complex as it occurs in the C3Fire environment.
One way to measure the players is to observe how they collaborate.
This can be done by monitoring communication between the players.
In all three studies the communication and co-operation between students have been monitored
by message flow analysis and message classification of the email, GIS and diary based communication.
In all the studies quantitative message flow analysis has been carried out by classifying
the messages sent between students in the organisation during defined time periods.
When using message classes as commands, reports, questions, etc.
it is possible get an indication of the collaboration process used by the students.
Our experience is that it is possible compare different groups and see the different
types of communication patterns that they have used in the sessions.
To be able to perform a situation awareness study the C3Fire environment is equipped with a Map system
(GIS, Geographic Information System) where the subjects
could insert their view of the state during the session.
Based on this feature, the manager can compare the actual state in the simulation with the information that
the subjects had constructed in their GIS or communicated with the mail or diary tool.
The GIS tool makes it possibility to measure the accuracy in terms of the type
of information and the time difference between the time of
the simulated activity and the time of the insertion in the GIS.
Work and Collaboration Methods
We have observed that most of the examined monitoring strategies are affected by the work and
collaboration methods used by the students. We therefore believe that it is important to
create monitoring strategies that can capture these processes.
An example of a successful analysis of a group's work and collaboration methods arose in
a session in the third study. In this experiment we tried to make a deeper analysis of
the messages that were sent within the organisation to try to discover whether it was
possible to identify some work and collaboration methods. Our attempt was supported by
a military team-training teacher who had a group of students who had
received training in collaboration in the C3Fire environment.
During the students' trials the teacher studied their collaboration and
their comments during the briefing/debriefing between the trials.
The main and important phenomena that the teacher observed was that the group
used commando orders in the two first trials but used mission orders in the last trial.
During the debriefing between the second and the third trail the students also decided
that they should try to give mission orders. Based on the teacher's observation and
the students' collaboration plan, a special analysis was performed.
The goal was to identify the command level used by the staff.
The command level was divided into two classes, commando orders or mission orders.
The command level detection was based on mail classification where we identified
whether the commands contained specific resources or specific targets.
If so, then the order was classified as a commando order; otherwise the order was
classified as a mission order. An example of a commando order is 'fight fire at position x?12, y?24'.
An example mission order is 'fight fire at the south parts of the fire'.
The results are demonstrated in figure 2.
This shows that it is possible to observe the level of command using parsing
strategies and key?word detection. In trial 2 the staff send 11 commando orders
and 5 mission orders while in trial 3 the staff send only 1 command order and 11 mission orders.
Figure 2. Work procedure detection.