Monitoring

To be able to analyse the collaborative work in the C3Fire system, computer based monitoring are used. The monitoring is integrated in the simulation and all the information tools used by the players. During a session the C3Fire system creates a log with all events in the simulation and all computer mediated activities. Figure 1, shows a view of the log process. The log process receives information from the simulation about the current activities in the simulated world. It also receives information about individual work, in terms of marks in the personal Map (GIS), and on the collaborative work in terms of information about the e-mail communication and the use of the distributed GIS and diary. Read more about the user analysis of the log information at Analysis and about the log events at Log Event Types.



Figure 1. The log process in C3Fire.

When the log information is centralised to the session server, where it is used in two ways: quantitative analysis and session replay.

Quantitative Analysis
Session Replay
Monitoring Goals


Quantitative Analysis

The quantitative analysis of the players' performance and collaborative work is based on information from the simulation and on information from the players' use of the information tools. The goal of the quantitative analysis is to classify the information, enable frequency analysis, time measurements and other quantitative measurements. The information tools in the C3Fire system are the map (GIS), the email and the diary. The nature of the information in the simulation and GIS is simply to classify and can therefore easily be used to obtain quantitative information such as statistics and frequency information. The information classification of the email and diary use is a bit harder but can be done by parsing the information in the messages. The analysis of the simulation information has mainly been used for performance measurements on how well the subjects have solved their task. The analysis of the GIS information has mainly been used to observe the situation awareness and some of the information distribution.



Session Replay

By using the session replay function, the training manager and the students can perform a high-speed play-back of the whole session. The replay shows simulated activities such as the fire and the positions and activities of the fire-fighting units. The main reason for having a session replay function is that research on simulation-based and collaborative training shows that by using a reflection stages after a session, increase the transfer effects and are an important step in a training process. The session replay used in an after-action review provides a good opportunity for the students to maximise group interaction to investigate their view of the situations and what they learned. The experience gained from using the session replay is that the use of the replay is an important step in the learning. When the students observed the session replay, they often starts to interact in presenting their view of what they thought had happened during the session, and what to do in future engagements. We believe that this function is one of the most important components in the C3Fire environment and that the motivational appeal of the simulation increases dramatically when the students can see a replay and discuss their collaboration. It is important to note that the sessions is performed on several distributed workstations, while the replay only can be performed on one single workstation. Read more about the replay features at Replay.



Monitoring Goals

Defining the monitoring goals and measurement methodes in a training session is a complex task. Example on four monitoring goals that has been used in C3Fire during three studies (Shared representations in ROLF 2010 ; Team Situation Awareness using Graphical or Textual Databases ; Collaboration in Serial and Parallel Organisations ) are: the effectiveness of the teams, the information distribution in the organisation, the students' situation awareness, and the students' work and collaboration methods.

Effectiveness of the Teams
One way to evaluate the sessions and see if the team has performed well is to use some kind of scoring system. A scoring system should change depending on the training goal and the task given to the players. In all three studies a score system that should tell us if the students have succeeded in their task of extinguishing the forest fire has been created by automatically analysing the extent of burned down area and destroyed objects. The weakness of this type of scoring system is that it may not capture other requirements on the team than fighting the fire. In real life a command team has other responsibilities, such as creating an activity diary, responding to incoming information from outside the organisation, etc. Our experiences is that a simple scoring system that is only based on pure effectiveness is not enough if the session is to be viewed as a good training session when the environment and the task get as complex as it occurs in the C3Fire environment.

Information Distribution
One way to measure the players is to observe how they collaborate. This can be done by monitoring communication between the players. In all three studies the communication and co-operation between students have been monitored by message flow analysis and message classification of the email, GIS and diary based communication. In all the studies quantitative message flow analysis has been carried out by classifying the messages sent between students in the organisation during defined time periods. When using message classes as commands, reports, questions, etc. it is possible get an indication of the collaboration process used by the students. Our experience is that it is possible compare different groups and see the different types of communication patterns that they have used in the sessions.

Situation Awareness
To be able to perform a situation awareness study the C3Fire environment is equipped with a Map system (GIS, Geographic Information System) where the subjects could insert their view of the state during the session. Based on this feature, the manager can compare the actual state in the simulation with the information that the subjects had constructed in their GIS or communicated with the mail or diary tool. The GIS tool makes it possibility to measure the accuracy in terms of the type of information and the time difference between the time of the simulated activity and the time of the insertion in the GIS.

Work and Collaboration Methods
We have observed that most of the examined monitoring strategies are affected by the work and collaboration methods used by the students. We therefore believe that it is important to create monitoring strategies that can capture these processes. An example of a successful analysis of a group's work and collaboration methods arose in a session in the third study. In this experiment we tried to make a deeper analysis of the messages that were sent within the organisation to try to discover whether it was possible to identify some work and collaboration methods. Our attempt was supported by a military team-training teacher who had a group of students who had received training in collaboration in the C3Fire environment. During the students' trials the teacher studied their collaboration and their comments during the briefing/debriefing between the trials. The main and important phenomena that the teacher observed was that the group used commando orders in the two first trials but used mission orders in the last trial. During the debriefing between the second and the third trail the students also decided that they should try to give mission orders. Based on the teacher's observation and the students' collaboration plan, a special analysis was performed. The goal was to identify the command level used by the staff. The command level was divided into two classes, commando orders or mission orders. The command level detection was based on mail classification where we identified whether the commands contained specific resources or specific targets. If so, then the order was classified as a commando order; otherwise the order was classified as a mission order. An example of a commando order is 'fight fire at position x?12, y?24'. An example mission order is 'fight fire at the south parts of the fire'. The results are demonstrated in figure 2. This shows that it is possible to observe the level of command using parsing strategies and key?word detection. In trial 2 the staff send 11 commando orders and 5 mission orders while in trial 3 the staff send only 1 command order and 11 mission orders.



Figure 2. Work procedure detection.