The use of multimedia for training maintenance tasks

See the latest version: click here (dd. 13 febr. 2001)

Abstract:

Keywords: training, maintenance tasks, multimedia applications, Computer-Based Training, Virtual Reality

Biographical

notes: Mona van der Vloed - Claessens graduated in Cognitive Ergonomics at the University of Utrecht in the Netherlands. After that she worked at the TNO Human Factors Research Institute in Soesterberg, were she studied traffic behavior in order to adjust road design, traffic signs and ‘In vehicle devices’. Now she is working on her dissertation at the department of Educational Science and Technology at the University of Twente in the Netherlands. Here she runs a project in which she is investigating how information can ideally be presented to train procedural tasks. In the end suiting (combinations of) media and an effective and efficient situating of information, such as text, pictures, animations and interactive 3D models will be implemented in a Dutch company were maintainers are trained to perform preventive and corrective maintenance procedures on radar equipment.

1 Introduction

Until now, much research has been done about human-machine interaction. The nature of this interaction depends on both characteristics of the system and characteristics of its users. Systems or machines must meet certain requirements with respect to human cognitive abilities and weaknesses. In other words, knowledge about human information processing must be used to fully adjust a system to its users. For example, with the development of learning tools, it must be considered that humans can only store 7 plus/minus 2 information chunks of information in memory at once [1]. Besides research about restrictions on human memory, other important research has been done about the construction of memory [2], about human knowledge [3, 4] human performance [5, 6, 7, 8], and so on.

A successful interaction between system and user also depends on several other factors. The design of a system has to be adapted to the types of tasks users have to perform on this system. Different tasks demand different types of interaction between the user and the interface resulting in different demands and criteria as for their interface design. Criteria for displaying "In Vehicle Information" while driving a car differ from those applied to computer screens in the control room of a ship. An IVIS (In Vehicle Information System) is not allowed to force road users to pay visual attention to its display [9, 10], while with a vigilance task in a control room active interference of the system is needed to keep operators alert [11]. Except differences in tasks, also more context-based aspects such as the availability of (financial) resources can result in completely different design solutions.

In this study the interest was in the possible implementation of multimedia applications (‘machines’) to train maintainers (‘man’) within a company that produces naval combat systems and trains people to operate or maintain these systems. Therefore a good definition about the types of multimedia applications, in specific Computer–Based Training (CBT) and Virtual Reality (VR), the target group, the kind of tasks and the residual situation within the company had to be obtained. Here the goal was to explore the context within which these multimedia were to be used. Because this lead to questions that were not to be found in literature or ‘company-documentation’, such as: "what kind of training methods does each instructor use", "what are typical problems with students", and so on, interviews were taken from the instructors of the company. Hereby the concern was to learn more about the condition of the current situation, about eventual problems and about what was expected to be the ideal situation. On the basis of this information, necessary design requirements were obtained that could specifically be related to the particular circumstances the system had to be used in. In a following study experiments were planned to test these and other (gained from a literature study about the use of virtual objects and CBT for training procedural tasks [12]) suggested design requirements.

2 Interviews

Research design A qualitative method of investigation was maintained, because the goal was to explore the situation [13]. It was decided to use the expert knowledge of instructors by interviewing them about certain topics. First, ‘company-documentation’ [14, 15, 16, 17] was read, part of a course was followed and informal conversations with three instructors were held to further discover points of interest that seemed to be of influence on the implementation of multimedia in training. During these conversations the interviewer brought up subjects like the nature of the courses, the type of students and the learning material, without expressing her own opinion about these matters. On the basis of these conversations the checklist was extended to serve as a topic list for interviews with instructors. The interviews were semi-structured [18], that is that there were fixed formulated questions, but open answer possibilities. During the interview these questions were ‘discussed’. Table 1 shows the topics that were treated in the interviews.

Subjects The interviews were taken from instructors from the International Training Center of the company. All the available instructors were interviewed (some of them were sent on secondment or were abroad to give a course). Seven instructors participated. They all had a background of high technical education. Their ages ranged from 40 to 56 years of age, except for one of them who was 26 years old. Their training experience ranged from two years to 26 years.

Procedure The subjects were interviewed in a private room about the topics of the questionnaire. All respondents were given the same introduction to read in advance and during the interview the questions were formulated according to questions that were drawn up beforehand to increase the internal validity [14]. The interviewer adhered to the interview techniques as described in Emans [18]. As a consequence respondent’s answers were continually summarized to check mutual understanding. In addition, subjects were encouraged to explain their answers.

Data Analysis Answers to the questions were summarized, recorded and then classified on the basis of their contents. They were arranged in tables according to their contents and the frequency with which they were mentioned.

3 Results

A frequently recurrent subject during the informal conversations was that instructors and the teaching methods they used had to be flexible. This was said to be caused by the fact that student groups varied in cultural background (students from at least 8 different countries), size (small versus large groups), composition (low and high educated people versus low or high educated people) and entry level (low versus high). In addition, regardless of certain requirements from the involved company as to preliminary knowledge and skills, in practice customers decide for themselves what kind of people they will send for a course. Finally, the potential marked and wishes from the customer determined the kind of products or systems that this company produces.

Current situation

Systems There is a large scale of different systems and tasks available at this company. The systems that have to be supported and maintained are situated on naval vessels. There are a number of radar systems on the upper deck, there are several operator consoles (MOC’s) in the control room between-decks and on the lower deck there are cabinets that contain the hardware of the radar systems. Table 2 shows the systems mentioned that are placed on a frigate.

Equipment Equipment that has to be used differs per kind of job and apparatus. Nowadays, the use of measuring equipment becomes less because hardware is more and more operated by software. Table 3 shows the equipment that is used for the maintainer courses.

Tasks There are several levels (OLM, ILM, DLM) in which maintainers can be trained. These levels not always seemed to give information about the difficulty of a course. They are more bound to the location at which the maintenance takes place. The depth or difficulty level of a course will be determined in consultation with clients and depends mostly on the following two aspects:

• The kind of ship; the greater the ship the higher the training level
• The mission time; the longer the mission the higher the training level

Courses are divided into four difficulty levels: 1, 2, 3, and 3+. All levels include looking whether the system works or not. The first two levels include the transfer of very simple knowledge or skills. For example, cleaning or changing the system units due for replacement (preventive maintenance tasks). There is no technical education needed to learn this. Some teachers consider that the first two levels are insufficient to give students the feeling that they are capable of performing their maintenance tasks. Course 3 and 3+ both involve corrective maintenance (solving technical problems that can occur) with the help of measuring equipment. Students of level 3+ are also given enough background knowledge of the system to be able to solve problems that emerge unexpectedly and can not be solved with the available procedures. Reasoning is required and detailed schemes will be used. Table 4 shows the different difficulty levels of maintenance.

Tasks mentioned that maintainers have to learn were inventoried and shown in table 5.

Besides this inventory about the systems the company produces, the equipment they use and the scale of tasks they have to teach an inventory about teaching methods was made. They were ranked on the basis of the frequency with which they were mentioned.

Training goal(s) In general the goal of instructors is to train students the most important tasks (skills) (5) and knowledge (2) (5 and 2 are the numbers of teachers that expressed this opinion. Tasks that were mentioned are recognising whether the system works, faultfinding, interpreting fault reports, exchanging procedures, tuning procedures and learning to install. With respect to knowledge instructors demanded comprehension and overview, knowing technical details and knowing the meaning of buttons. Table 6 shows the goals that where mentioned by instructors and how many times they were referred to.

To test whether the aimed goal was reached at the end of a course instructors did not use formal verbal or written tests. This was because it would be an invasion on a student’s privacy. It could for example endanger a student’s career. As a consequence instructors used alternative manners to measure students capabilities. These informal tests are here referred to as observation judgements. Table 7 shows the nature of these observations.

Theory & practice: Officially a maintainer course should contain 50% of theory lessons and 50% of practice lessons. The time that instructors get to prepare a course is 1,5 times the length of the course. In advance there is contact with the client about what they want their personal to be trained in. The current situation is that these times vary now and than (see Table 8). Some instructors came up with ratio’s specific for certain tasks: Faultfinding: 20% theory - 80% practice, Fine-tuning: 100% practice and Operator course: 10% theory - 90% practice. Table 8 shows the ratio of theory and practice that instructors use with maintainer courses.

The amount of theory and practice instructors used in their lessons did vary from time to time according to the given circumstances. According to the teachers this was caused by the varying availability of the system (mostly not for 50 % of the time available, as it should be) (3) and the variation of students entry level that stressed the importance of theoretical education (3). Other reasons mentioned were the type and difficulty of tasks (for some tasks more theory lessons were needed) (2) and the fact that with large student groups less time could be spend with the system (maximally three students were allowed to rehearse with the system at once). Table 9 shows the reasons instructors had to maintain a certain ratio of theory and practice.

There were instructors that explicitly judged theoretical knowledge to be more important than practical skills (3). Reasons were that skills are easy to learn, so not many time had to be spend on this and the fact that build in tests (BITE, system checks itself on failures) from the systems are not perfect just as the procedures from procedure books. Other instructors considered knowledge as a basis (3). Students have at least to understand the system as a whole (what is in it, the location of all system parts/ units and how it functions). One instructor considered theory and practice as equally important. Table 10 shows reasons for the need for theoretical knowledge.

An equal amount of instructors stated that practice is more important than skills (3). "Practice is the final goal". The reasons given were that practically dealing with the system was the final goal and that in the end procedures had to be done without thinking (automatically). Table 11 shows reasons given for the need for practical skills.

Training materials: The materials the instructors used in their lessons were mostly the same for all instructors. For example, all the interviewed instructors used handouts (with block diagrams and service schemes or schemes to adjust), a blackboard and overhead projectors. On instructor said to use demonstration materials such as transmitters. Table 12 shows the training materials that were used for the maintainer courses.

Multimedia applications: Instructors were also questioned about more modern lesson materials, namely multimedia applications such as Virtual reality and Computer Based Training. Instructors agreed with the use of VR and CBT on the condition that they would not replace the instructor. Students need social contact and personal guidance. One instructor stated that possibly later students will get used to training without instructors, but now it would be too radical.

Table 13 shows the mentioned advantages and disadvantages of Virtual reality.

Almost all instructors put forward that Virtual Reality could not replace the real system (6). This is because VR is never the same as the ‘real thing’. One instructor suggested using VR as training tool before working on the real system, whereas another suggested using it as replacement for practice lessons.

Table 14 shows what subjects are appropriate to be learned with the help of Virtual Reality according to the instructors.

It did not become very clear how instructors would implement good educational Virtual Reality. They mentioned both passive VR (On a large screen, for example the Signaal VR room with moving 3D images) (5) and active VR (with head-mounted display and glasses, possibilities to walk around objects and enter them plus possibilities to perform actions) (5). However, many of them did not regard active VR as a realistic option in the near future. One instructor mentioned the importance of feedback by means of alarm noises that warn for illegal operations. And another advised to keep the sequence of actions intact.

Table 15 shows the mentioned advantages and disadvantages of Computer Based Training.

CBT is not seen as an application that will replace instructors. Instructors could refer to CBT so students could access the material at their own time. It can also be seen as a "tool" to practice maintenance in advance when the system is not available. Half of the students could use CBT while the other half practices with CBT. The amount of CBT can be adapted to the students’ affinity and / or experience with the subject matter. Another possibility is that CBT will serve / be sold as renewal course abroad. One instructor stressed the importance of instruction from a book.

According to the instructors there were several learning subjects that are appropriate to be learned with the help of Computer based training. Table 16 shows these subjects.

Instructors would provide good educational Computer based training with possibilities to interact ("pushing buttons resulting in actions") (2) and animations (2). In addition they would build in the possibility to record every act in enormous databases.

Problems

Course, Instructors, Students Table 17 shows mentioned problems with the course in general, the instructors and the students.

Student’s entry level (7) and their inability to comprehend or speak the official course language (English or Dutch) (5) were the biggest problems. Another complaint of instructors about the course was that the available time was not enough (4). Courses are too short to deal with all the subject matter in the difficulty level that was planned. A related problem that was mentioned is the lack of course (preparation) time (3). The more difficulty students had with the subject matter the more time was spend for extra training, further explanation etc. Another aspect that was said to interfere with a smoothly way of training is the hierarchy student groups have sometimes (3). Most groups exist of people with different ranks. This often complicates the way of training. Instructors have to be both cautious what to tell or ask students with a higher rank (to prevent them from loss of sight) and what to tell or ask students with a lower rank (to prevent them from sanctions). Problems with the availability of the system were stated to be caused because it was not ready or defect (3). The bad availability of information consisted of documentation or procedures that were not ready yet (2). Finally, a problem mentioned was that sometimes students became unmotivated during classes of previous instructors, and that this is very difficult to reverse (1). This was because of some student groups are trained in different subjects and because each instructor has its own specialty these students are brought into contact with different instructors.

There were also some questions about specific problems with students: Problems with the age of students, problems with their lack of computer experience/knowledge and with their cultural background. Three instructors stated that age is not of influence on the successful performance of students. In addition, it seemed that the advantages named of being young were stated advantages of being old. For example, while young students were said to have little practical experience, older students were labelled as experienced. Younger students were said to be less motivated (2), have less practical experience (2), have a shorter history of education, have more problems with hardware technique and have only knowledge about their own discipline. Older students were expected to have less experience with computers (3), need more time to learn (3) and to have more problems with newer systems and techniques.

Experience with computers was also found of small relevance to the performance of students. However it was mentioned as a convenient ‘characteristic’ with surplus value (4). "It is easier to understand when a student has experience with the computer". In addition, it was said to become more important nowadays. At the same time the number of students who do not know much about computers is decreasing.

Culture was already at the basis of some previously mentioned problems, such as problems with language and group hierarchy. The specific question about problems with cultural differences also elicited some other aspects. Students learning methods differed. From one culture students were used to learn everything by heart while students from another culture were used to reason and improvise (2). Students’ customs could also elicit problems. For example, normal behavior in one culture is considered as rude in another culture. Just as the entry level differed per culture the quality of their military and civil personal did too. Some country placed more importance upon their military personal and others on their civil personal.

Finally, affinity with technology and motivation were also seen as typical traits for students from some particular countries. It cannot be claimed that the problems that were mentioned are always necessarily caused by a cultural difference. Another fact is that problems cannot be totally generalized to all the students from one culture. Thus, problems mentioned were considered to be typical in general for one particular culture.

Ideal situation

Course, Instructors, Students Although it is reasonable to assume that an ideal course should contain all the characteristics opposite to the ones that were mentioned in the previous paragraph, there were also other aspects that were considered to be important for a good course, instructor or student. For example, enough time with the same student group. Sometimes more instructors give the same courses because all instructors have their own special expertise. Another aspect mentioned is that it is ideal to work with small student groups. This is because with rehearsing on the system only a maximum of three persons can practice at the same time. The question about what characteristics a good instructor should possess was answered more enthusiastically. The characteristics could be roughly divided into didactic skills, social skills and expert knowledge.

Table 18 shows characteristics that were considered to determine the quality of a course, instructor, student and the frequency with which they were mentioned

3 Conclusions and discussion

During the informal conversations it already became clear that in the first place things as the potential marked and wishes from the customer determine the kind of products or systems that this company produces. As a consequence it is already certain what will be the learning domain (subject matter) of the courses. Second, regardless of certain requirements from the involved company as to preliminary knowledge and skills, in practice customers decide themselves what kind of people they will send for a course. This was endorsed by the interview result that student’s entry levels in general appeared to be too low. Thus another unchangeable factor here is the target-group of students (which also proved to be a largely varying factor looking at the interview results). These aspects and the fact that despite this the available time for a course remains the same stresses the importance of instructors’ flexibility (which was also confirmed by the interview results).

From the interviews it appeared that the involved company possessed a large scale of systems, equipment and tasks. Although, further research was planned to investigate what types of design requirements are needed for each type of task (level) some general design criteria were derived from the interview results.

Multimedia in combination with instructors and real systems It was advised not to use multimedia applications independently from the real system or instructors. First, rehearsal with the real system during practice lessons was considered inevitable. Till now CBT or VR cannot simulate the real system completely (tactile information, technical details, etc) without a great amount of money and effort. Second, because of the varying target-group lessons should be adapted to each specific group of students. Potential CBT lessons should at least be selectively assigned to different students with respect to their contents and difficulty level. In addition social contact can be quite necessary to motivate students or make them feel at ease.

Multimedia as a practice tool The lack of rehearsal with the system (no system available, system broke, only small amount of students can rehearse with the system at the same time) could be decreased by means of simulations of the system with which students could interact. Especially tasks that occur often (recycling of multimedia) and that are difficult (extra practice with multimedia) or time consuming (multimedia unburdens costly time with system) are profitable too put into a multimedia application.

Multimedia with the use of visualizations The possibility of extra rehearsal (with simulations) is also useful because intended users of multimedia within this company usually have a low entry level and a bad control of the official course language (English). As a consequence multimedia applications should not contain large pieces of (complicated) text, but visualizations to avoid misunderstandings and to get rid of abstractions.

Multimedia for different difficulty levels Because of the varying entry levels and control of the course language, multimedia should be flexible with respect to speed and difficulty. Just as instructors have to adapt the speed of lessons to (the entry level of) students, multimedia design should allow the student to maintain his / her own learning speed. In addition, as previously mentioned the instructor should assign students to the right kind of lessons or difficulty level, unless the student is able to judge this for him/herself.

Multimedia to avoid group pressure The individual character of multimedia applications can partly take care of the ‘group hierarchy problem’. With ‘CBT-lectures’ students are spread across the classroom each behind their own computer. Students who were afraid to ask questions because of the presence of their superiors are now more easily able to ask something in private when the instructor makes his round.

Multimedia as an objective measurement tool In addition to the now used observation methods (Does the student asks intelligent questions? Does the student perform a task well during practice lessons?) multimedia can provide a more objective view on students competence. Registration of student’s performance can be used to get insight in their progress and to anticipate on the ‘educational needs’ of students.

In order to be able to test whether these design requirements hold when the target group of users gets into contact with this multimedia, it is aimed to build a piece of Computer Based Training with the use of virtual models. This prototype will contain the subject matter of a typical procedural task. Users will be provisioned with different designs (for example animation versus interactive models) and tested on their task performance afterwards. In addition, attention will be paid to students learning strategies.

Acknowledgements

I wish to thank the teachers of the involved company for sharing their expert knowledge.

References

[1] Miller, G. (1956). The magical number of seven, plus or minus two: Some limit on our capacity for

information processing. Psychological review, 63, 81-97.

[2] Wickens, C. (1984). Engineering psychology and human performance. Columbus, OH: Merrill.

[3] Collins, A. M. & Quillian, M. R. (1969). Retrieval time from semantic memory. Journal of Verbal Learning and Verbal behavior, 8, 240-247.

[4] Collins, A. M. & Loftus, E. F. (1975). A spreading activation theory of semantic processing. Psychological review, 82, 407-428.

[5] Rasmussen, J. (1983). Skills, rules and knowledge: Signals signs and symbols, and other distinctions in human performance models. IEEE Transactions on systems, Man, and Cybernetics, Vol. SMC-3, 3, 257-268.

[6] Rasmussen, J. (1982). Human errors. A taxonomy for describing human malfunction in industrial installations. Journal of occupational accidents, 4, 311-335.

[7] Rouse (1981). Models of human problem-solving: Detection, diagnosis and failures. Proceedings of IFAC conference on analysis, design and evaluation of man-machine systems. Baden-Baden, Germany.

[8] Reason, J. T. & Embrey, D. E. (1985). Human factors principles relevant to the modeling of human errors in abnormal conditions of nuclear and major hazardous installations. Dalton, England: Human reliability associates.

[9] Kaptein, N. A., Claessens, F. M. M., Janssen, W. (1998). Safety evaluation of in-vehicle devices that provide real-time traffic information. TNO-TM report: TM-98-C047. Human Factors Research Institute, Soesterberg: The Netherlands.

[10] Winsum, van W., Claessens, F. M. M. (1998). Effecten van mogelijke vormen van informatie presentatie van de OBU op rijgedrag en verkeersveiligheid. [The effects on possible kinds of information presentation of the On-Board Unit (OBU) on driving behavior and traffic safety. TNO-TM report: TM-98-C052. Human Factors Research Institute, Soesterberg: The Netherlands.

[11] Deatherage, B. (1972). Auditory and other forms of information presentation. In H. Van Cott and R. Kinkade (eds.), Human engineering guide to equipment design (rev. ed.), p124, Table 4-1, Washington, DC: Government printing office.

[12] Van der Vloed, F. M. M., Min, F. B. M. & Moonen, J.C.M.M. (2000). The use of Virtual objects in multimedia for training procedural tasks. Unpublished manuscript.

[13] Swanborn, P. G. (1987). Methoden van sociaal-wetenschappelijk onderzoek. Methods of social

scientific research. Boom, Amsterdam: The Netherlands.

[14] Signaal (1997). Van oorschelp tot radar; de geschiedenis van een bijzondere onderneming. [From auricle till radar; the history from a special company] Drukkerij Twente Hengelo, Hengelo: The Netherlands.

[15] Signaal (2000). Versatile Interactive Computer Based Training for Operation and Repair. VICTOR Program Management Plan. Hengelo: The Netherlands.

[16] Signaal (1999). Manual for corrective maintenance. Hengelo: The Netherlands.

[17] Signaal (1999). Manual for adjustment procedures. Hengelo: The Netherlands.

[18] Emans, B. (1990). Interviewen; Theorie, techniek en training. [Interviewing: Theory, technique and training] Wolters-Noordhoff, Groningen: The Netherlands.

Tables

Table 1: Topics that were used in the interviews

Table 2: Systems divided into systems on the upper deck, systems between-decks and systems on the lower deck.

Systems
Upper deck	Between-decks	Lower deck
Surveillance radar Track radar Navigation radar	TACTICOS Multifunction Operator Console (MOC)	Cabinets

Table 3: The equipment that is used for the maintainer courses

Equipment

Measuring equipment Maintenance Requirement Cards Procedure books Software installation manuals (SIM) PIOD (laptop) GSS

Table 4: The difficulty levels of maintenance

Difficulty levels of maintenance
Level 1 / 2	Level 3	Level 3+
Learning preventive maintenance No technical education needed Using simple procedures	Learning corrective maintenance Technical education needed Using procedures Solve envisioned malfunctions Using measuring equipment	Learning corrective maintenance Technical education needed Learning to maintain with and without procedures Solve envisioned malfunctions and malfunctions that did not occur yet Using measuring equipment

Table 5: Tasks that maintainers have to learn

Tasks

Start up the system Making the system ready for operation Noticing malfunctions Tracking defect units or malfunctions / Performing faultfinding procedures Exchanging defect units Performing adjustment procedures Functioning and tuning of the different cards Maintenance and repair of different software layers Communication between subsystems Performing preventive maintenance Installing software according to the procedures Make a distinction between hardware and software faults

Table 6: The goals that where mentioned by instructors and the frequency with which they appeared

Training goal	Number of goals mentioned
Training most important tasks Transferring understanding and knowledge about the functioning of the system Training students what is officially needed Training students what they expect See to it that students will have a nice time and learn at own ability. The instructor should have positive feelings about the degree of communication (non-verbal and verbal).	5 2 1 1 1 1

Table 7: The kind of (informal) ‘tests’ that were used by instructors to measure whether the training goal was reached and the frequency with which they were mentioned

Tests to measure whether the training goal was reached

Number of tests mentioned

Observing students during practice lessons / simulating defects Speed with which students perform tasks Independence with which students perform tasks Judging answers from students to questions during theoretical or practice lessons Quality of answers Speed of answers Judging questions from students during theoretical or practice lessons Quality of questions Amount of questions Observing students during theoretical lessons Quality of things they write down (own summary is good, literal transference is bad)

4     3     2     1

Table 8: The ratio of theory and practice that instructors use with maintainer courses and the frequency with which they were mentioned

Ratio theory-practice maintained by instructors	Number of ratio’s mentioned
Theory: 60 % - Practice: 40 % Theory: 50 % - Practice: 50% Theory: 40 % - Practice: 60 %	3 3 1

Table 9: The reason instructors had to maintain this ratio of theory and practice and the frequency with which they were mentioned

Reason for ratio theory-practice	Number of reasons mentioned
The availability of the system The entry level of the students Difficulty level of the tasks Size of the student group	3 3 2 1

Table 10: The need for theoretical knowledge and the frequency with which they were mentioned

The need for theoretical knowledge

Number of knowledge mentioned

Theoretical knowledge is more important than skills Knowledge as a basis Depends of insight, entry level and former experience of students Depends on whether theory has value in practical situations Knowledge gives more feeling for system Knowledge betters imaginative powers of what can go wrong (faultfinding) Knowledge elicits more pleasure and more motivation The ability to gain new information is more important than skills Affection with the subject is more important than skills Relevant technical knowledge and practical skills are equally important

3 3 1 1 1 1 1 1 1 1

Table 11: The need for practical skills and the frequency with which they were mentioned

The need for practical skills	Number of skills mentioned
Practice is most important Students do need some practical experience, otherwise it would take too much time to train subject matter The students have to be active, because reading is passive and can only be efficient for a short time. Interest is very important for practical skills	3 1 1 1

Table 12: The training materials that were used for the maintainer courses and the frequency with which they were mentioned

Training materials that are used	Number of training materials mentioned
Handouts Blackboard Overhead projectors ‘Signaal’ manuals Videos of maintenance procedures Commercial folders /documentation folders Demonstration material	7 7 7 4 4 2 1

Table 13: The advantages and disadvantages of Virtual reality that were mentioned by instructors and the frequency with which they were mentioned

Advantages of Virtual Reality	Number of advantages mentioned	Disadvantages of Virtual Reality	Number of disadvantages mentioned
Things can be easily edited It can save costs (the system will be needed less often) One can easily clear up and visualize things It can make the information more accessible and get rid abstractions It can be used when the availability of real system is critical It can be used when it is risky to break things in the real system It can be used for extra training, when students do not have the presumed entry level	1 1   1   1   1     1   1	Too expensive It may be too hard to simulate a very complex system	3 1

Table 14: Training subjects that are appropriate to be taught with the help of virtual reality and the frequency with which they were mentioned

Ideal subjects to train with the help of Virtual reality	Number of subjects mentioned
Corrective / Preventive maintenance procedures Faultfinding Tuning Exchanging procedures (replacing units) See through a gearbox (Mechanical jobs) Theoretical concepts Principles of radar energy / electrical beams Demonstrating physical properties of system (size, amount of parts or units) Operators tasks Tasks that occur often Tasks that are difficult to do in reality Time consuming tasks	5         2     1 1 1 1

Table 15: The (amount of) advantages and disadvantages of Computer Based Training that were mentioned by instructors and the frequency with which they were mentioned

Advantages of Computer Based Training	Number of advantages mentioned	Disadvantages of Computer Based Training	Number of disadvantages mentioned
You will not constantly need the real system Can be used when there is a risk to break things in the real system Can be used when parts of the system are difficult to reach Can be used for extra training When students do not have the presumed entry level when students with more affinity and experience want to learn extra	3           2	Too expensive One hour of training via CBT is 200 hours of programming. System differs per client so adaptations have to be done all the time	4     1

Table 16: Training subjects that are appropriate to be taught with the help of Computer based training and the frequency with which they were mentioned

Ideal subjects to train with the help of Computer Based Training	Number of subjects mentioned
Theoretical knowledge Principles of radar energy / Doppler effects Definitions Subjects that pop up in all courses Corrective maintenance procedures Faultfinding Standard procedures	5       2

Table 17: Problems with the course in general, instructors, students and the frequency with which they were mentioned

Problems with course, instructors and students

Number of problems mentioned

Course Course is too short Equipment is not available / difficult to find System is not available to rehearse with Preparation time is too short Bad availability of information Course is too difficult Documentation not correct Multiple instructors for one student group The amount of students that can rehearse on the system is limited Instructors Instructors who are stubborn Instructors who have difficulties dealing with another culture Instructors who go too quick through the learning material Instructors who are not able to motivate students Instructors who are not able to explain difficult things it in a simple manner Students Students who do not have the right entry level Students who cannot comprehend and / or express themselves in the official course language Students who need another approach due to cultural differences Students who are afraid to ask questions (group hierarchy) Students who do not want to admit they do not understand Students who have a lack of motivation

4 4 3 3 2 1 1 1 1   2 2 1 1 1   7 5 3 3 1 1

Table 18: Characteristics that were considered to determine the quality of a course, instructor, student and the frequency with which they were mentioned

Ideal situation

Number of characteristics mentioned

Course Enough time with the same student group Knowing in advance about students entry level Enough time with the same student group Enough time to prepare the course A small student group A group with students that al have the same entry level Training on the job afterwards Availability of tools / measuring equipment Availability of right course documentation in advance Instructors Instructors with expert knowledge Didactic skills Adapting course to entry level of students: 3 Adapting course to cultural background of students: 2 Adapting course to future responsibilities of students Skills to explain difficult things: 2 Ability to formulate in a logical way Transmitting information successfully Involve people Also discussing other topics Admitting lack of knowledge and looking up things you do not know Social skills Patience: 3 Good at contacts Knowing how to deal with people Intuitive in relations outside lessons Students have to trust you: 2 Students Students with good entry level Students with good control of official course language Motivated students Students who are interested Students who are disciplined Students who are responsive Students who are not afraid to make mistakes Students who are conscious about future responsibilities / tasks Students who are not too serious, not to precise (time absorbing) Students who have the ability to gain new information or knowledge

5 2 1 1 1 1 1 1 1   4 13                   8               7 5 4 1 1 1 1 1 1 1