Summative Evaluation Report
This document describes the summative evaluation of APOSDLE. Given the scope of APOSDLE it is very important that a comprehensive and well planned evaluation is carried out.
This summative evaluation is directed to the main goals of the project and the application partners. Based on previous work on the i*model, a set of goals is identified that are specific enough to make some kind of measurement possible. These goals are prioritized by the application partners, leading to four goals that are important for all application partners and seven goals that are important for a subset of the application partners. In the summative evaluation achievement of these goals is the central focus.
In addition, a more precise definition is given of what the “APOSDLE product”, or maybe better, the “APOSDLE solution” actually is. It would be a mistake to see only the software embedded in the final prototype to be deployed at a workplace as the “solution”. In the project the “APOSDLE solution” is seen as consisting of the following aspects
- Modelling the domain and the work processes
- Annotating documents and other sources of information available in the company repository
- Training the prospective users of the APOSDLE system
- Using the APOSDLE system at the workplace
This means that a comprehensive summative evaluation of the APOSDLE solution requires a summative evaluation of each of these aspects taking the goals into consideration. This is even more mandatory as the aspects depend on each other. If the domain modelling has not been done correctly, the annotation will fall short of what is needed; if the annotation is done badly retrieval of relevant information will be unsatisfactory; if the users are not well trained, their use of the APOSDLE system will be sub-optimal.
The summative evaluation was carried out at the Application Partner sites. The workplace evaluation took about three months and involved nineteen persons. A multi-method data collection approach was followed using a questionnaire, interviews, log data, user diaries kept while working with APOSDLE and site visits. This allowed for triangulation of results.
The conclusion of the summative evaluation is that the APOSDLE solution is very useful for learners, especially those in a highly-specialized, scientific and relatively new domain like EADS's Electromagnetism Simulation domain. In those circumstances, APOSDLE delivered an effective work-based learning solution that enabled relatively inexperienced knowledge workers to improve their knowledge in various ways. However, it proved less effective for relatively inexperienced knowledge workers in broad and well-established customer-driven domains where knowledge was shared to a large extent in person and was available in alternative company repositories, employees' computers and the Internet.
Experts in all three domains perceived APOSDLE as a tool aimed mainly at learners or novices in the domain. They valued its benefit as a training tool but compared to learners regarded it a tool they did not need to do their work. They acknowledged their role was to contribute high quality learning material (e.g., Snippets, documents and Learning Paths) but commented such responsibility may not be realistic given their busy schedule and, in some cases, time spent out of the office.
Overall APOSDLE supported the acquisition of new knowledge by the users by making them aware of learning material, learning opportunities and by providing relevant material. However, this conclusion is not based on more formal knowledge gaining tests, which are not opportune to administer in a working context, but is derived from self reports. Concerning specific support for learning it must be concluded that this is only of limited value in a working context. Apart from learning goals and learning paths, most other supporting facilities, like hints and notes, were rarely used. This shows that supportive measures derived from instructional theories which are focusing on formal learning contexts, are not very relevant for learning at work. This leaves open the question what kind of support, or maybe no more specific support at all, can contribute more to learning at work than providing good material and some help in planning learning and thinking about learning goals.
APOSDLE supported working mainly by making knowledge workers aware of material which helped them complete a task. Such material was discovered as a result of keyword search, browsing the list of topics or tasks or as a result of a topic being detected by the context detection component. In EADS especially, it was reported on numerous occasions in the user diary that explicit and implicit learning material enabled knowledge workers to gain useful insight, improve their knowledge and complete a task they were working on. This type of support was less evident in the two consultancies (ISN and CCI) were knowledge workers relied primarily on each other to support their work.
The workplace evolution of the collaboration component of APOSDLE was limited due to technical problems, which prevented it from being used throughout most of the evaluation period. It did function during the last two weeks of the evaluation and end users were able to test it and form a positive impression, though there no evidence it was used in a real-life scenario either to fill an explicit learning need or complete a task. Apart from any technical problems, it is worth noting that in ISN and CCI end users shared an office space or worked in close proximity. They were used to interacting with each other in person throughout most of the day as part of their work. Under such circumstances one must consider the possibility there was no real need for people working in a small team to structure a collaboration request with a colleague during the evaluation.
Finally, concerning challenges APOSDLE addressed it is concluded that the domain independent approach partly worked. Relying on existing material instead of tailor made learning material provided to be effective and is probably also cost efficient. Crucial for this is having good modelling tools, experienced modellers and high quality annotations of snippets. Providing domain independent learner support was less successful as most of the provided features were hardly used. Probably the gap between the generalized and rather abstract nature of these features and the very specific context of work is too large. It is proposed to put more effort in good annotations which probably brings more benefits than trying to refine or enhance specific learner support.