Background information for assessors

In 2003, I lead the validation team for a new pathway at Anglia Ruskin University, the BA (Hons), Learning, Technology, and Research (BALTR). It was my responsibility to synthesise input from the development team and to make the final decisions about the design and planning of the programme. This pathway was developed as a part of the Ultraversity Project.

The BALTR was designed to be delivered fully online with no face-to-face meetings. Internet technologies are deployed to offer an HE experience in new and creative ways for people in full-time employment. These individuals wish to study their current work discipline to further develop their career and to improve their performance in the workplace.

Over the past 4 years, I lead the project team of 24 including technologists, administrators, and learning facilitators to deliver the degree programme. In addition to this, I also worked as a learning facilitator to gain first hand experience of delivering the programme.

My CMALT application is primarily based around my work on the Ultraversity project and further evidenced by my professional blog - Thoughts Mostly About Learning.

Internet technologies

The Nesta Futurelab publication on Personalisation and Digital Technologies (2005), argues that there is a high degree of personalisation in the experience of lifewide learners, however in the formal context this is still largely unrecognised.

Downes (2006) observes that despite the rapid increase in educational institutions adoption of Internet technologies, most people who inhabit the online world are in fact elsewhere. He points out that there are a myriad of web2.0 spaces that enable them to generate and share their own content in ways that they chose to amongst their own ‘learning networks’.

Since 2003, Ultraversity has changed how we provide services for learners. This progression (see fig.1) took us from reliance upon purpose built locally hosted software, to the use of tailored OSS locally hosted, and recently the use of OSS ‘libre’ services that are externally hosted. Throughout this transition, choices made were intended to allow learners to take control and personalise their learning experience within the constraints of an accredited formal learning degree prgramme.

Fig.1 Evolution of Internet Technology Provision (Powell and Tindal 2006)

This latest development required the development of strategies to address Quality Assurance issues around submission of work for assessment. Common standards for interoperability between locally hosted and external services, should in the future allow for these issues to be addressed at technical level.

Two examples of the development of this approach are described below.

Plone:  locally hosted, tailored open source software
Technological, pedagogical and operational considerations placed the OSS content management platform Plone as the core learning technology for the project, our realisation of a VLE.

A key benefit of Plone is that it is a ‘multilayered’ technology providing a “low threshold and high ceiling” (Papart, 1980) user interface. This allows Individuals with relatively low levels of technological ability to easily master a rich set of creative online tools and to develop a ‘virtual spaces’.  As individuals’ skills increase, the powerful functions of Zope and programming in Python become accessible to them .

This ’symmetry’ was a key consideration in the selection of Plone; providing all users with a rich set of asynchronous tools so enabling teaching staff and students alike to build online learning environments. A range of individuals with widely differing levels of technical skills were able to develop the Plone environment (appendix 1 ).

A strong open source community supports the Plone development and ensures that it is a robust and likely to be long lived platform.

Gearbox: locally hosted, tailored open source software
The gearbox OSS tool was developed as means of enabling learners to critically reflect upon their abilities against specified competencies. It is easily adaptable and was integrated into the Plone environment. Gearbox involves an interactive graphic representation of competencies as spokes on a set of gear wheels (fig.2). The gearbox framework (a full set of cogs) enables learners to self-construct a competency based profile.

Fig 2. One gear on the BALTR Gearbox portal

Although OSS, it is unlikely that a community will develop to support the Gearbox tool. This illustrates well the importance of understanding that when selecting OSS that there are issues of viability that need careful consideration. Building teaching and learning approaches around particular tools can have significant implications in terms of the student learning experience, staff training, and programme development. If software cannot be relied upon this may not be justifiable approach outside of the research arena.

Development of Ultraversity online environment

The screen grab below illustrates the environment developed for Ultraversity using the Plone content management system introduced in section 2.1. The online learning environment was developed through collaboration between the Ultraversity technical team and myself with each individual bringing a particular skill set to the project.

The Plone software was designed for collaborative working between web designers and as such is capable of being set up with a high degree of symmetry between users with different roles. My responsibility was for the initial structuring and build of the online environment including decisions about access and permissions.

Figure 3 Online Environment (source: www.researchers.ultraversity.net)

The environment developed consists of several key areas with distinct learning purposes.

Cohort areas are designed for discrete student groups where module related discussions and activities are hosted. This is a ‘one-stop’ space where students can pick up vital module information such as start and submissions dates, when they can expect feedback and results, and links to module discussions and related resources.

Hotseat discussion and archive is an essential ingredient of the Ultraversity experience and is where external guests hold focused conversations with researchers. The space is set up to allow for universal access so that experts are available to all students and not restricted to certain groups (appendix 2).

e-Portfolio space is offered through the Plone ‘my folder’ where permissions have been set to allow students full access to the Plone tool set. The individual user can upload and store work and chose to allow access to particular views for collaboration with fellow students and for assessment purposes.

Development of new induction Blog

A second online environment I collaboratively developed for the Ultraversity project is the induction blog. Initially developed in the Drupal blog platform, with colleagues, we have just completed the development of a new blog using the Elgg software. Hosted by Elgg, this approach was taken to explore the possibilities of the new wave of web2.0 services.

Figure 4 Ultraversity Induction Blog

The purpose of the induction blog is to act as staging post for students who have registered with ARU, but for who their start date is some period of time away. In addition, Shirley Pickford has now developed a framework for an induction module that outlines the intended learning by students who participate in this voluntary activity.

The Elgg platform can be described as a personal learning environment (PLE). The key technology underpinning its functionality is RSS that enables the sharing of content that is then aggregating into different views or representations.

The design for the Ultraversity induction blog in Elgg (appendix 3) is a simple one with an identity Ultraversity content hosting the induction information and then individual learning facilitators pushing their personal posts to the shared blog. Students are able to either create their own Elgg account and fully utilise the platform functionality, or simply brows and access the static information but not access the induction activities for which an account is required.

The move into externally hosted software has both benefits and challenges. Clearly, we have less control over how it is developed and are not able to significantly ‘hack’ the interface to meet our needs. On the plus side, we no longer have to maintain servers and software which has significant advantages in terms of saving resources. The success of this approach will need evaluating after this induction period.

In 2006, the Ultraversity Project had an FTE of 19.1 learning facilitators, 3 technical researchers / support staff, and 2 administrative researchers.

The Ultraversity degree programme is delivered by a team of learning facilitators (lecturers) and is supported through online learning communities and a range of other online technologies such as e-portfolio and interactive and contributory learning resources. Although the facilitation team were experienced educators and, in some instances, had experience of working online, there was the need to develop new working practices to support the delivery of the Ultraversity degree programme.

To effect this change, I implemented a planned change management and CPD process where current practice is examined and the shifts required to meet the aims and objectives of the Ultraversity validation document identified. The attached file (appendix 4) indicates a typical f2f agenda for the 3/4 Ultraversity f2f meetings held per year since the projects inception in 2003. The aim of these f2f sessions were to integrate the development of the use of learning technologies integrated into the development of the BA. Learning, Technology and Research programme.

It was apparent from this process that the biggest challenge to embedding sustainable e-learning is not the use of the e-learning technology itself, but the challenge posed by the existing values and beliefs of lecturers that are at odds with learning which is centred upon personalisation and choice. The HEFCE - Strategy for E-learning (2005) is a useful document in setting out the direction over a 10 year time-frame for helping HEI to inform their action planning and change management. The document focuses on the student learning rather than developments in technology per se and also advocates an approach  “to promote learning research, innovation and development that begin with a focus on student learning rather than on developments in technology, enabling students to learn through and be supported by technology”.

The Ultraversity development of the BALTR is undertaken through an action research methodology which promotes a culture and practice of learning about teaching and implementing change.

To implement the above vision, typically, colleagues are asked to lead sessions at f2f with a heavy emphasis on practice, focusing on how will this work in the context of a remote team delivering a wholly online degree programme. This is a deliberate attempt to foster an approach that includes all in the development of the teams capacity and sharing responsibility for the development of the use of learning technology.

The BALTR pathway uses levelled assessment criteria in the form of a rubric that is based on ARU generic criteria. The rubric describes what would be expected of students achieving different levels against the assessment criteria (appendix 5 ultraversity validation). These generic criteria are then selected for appropriateness at module level and further developed to emphasise particular requirements of modules and guide students to develop particular aspects that they are required to learn.

The assessment regime we use for the BALTR supports the aim to widen access to HE. Part of this approach is to use alternative approaches to the essay and examination, as some of the students have had negative experiences of these in previous periods of education. Students are encouraged to express themselves through multimodality using an e-portfolio approach making use of rich media and technology such as video, audio, websites and weblogs.

The thinking behind this approach is in part informed by Winter’s “Patchwork Text” model, with its emphasis on a reflexive approach and the use of creative imagination, peer review and discussion, “It’s time we found an alternative to the student essay. For tutors across the country, it’s marking time again and, reading essays, we realise that many of our students have yet again taken refuge in ’surface learning” (Winter, 2003).

Students assemble pieces of work for their assessment e-portfolio with a ‘retrospective commentary’, which ’stitches’ them together linking ideas and forming conclusions. This concluding activity should provide and an honest view of the learning journey including learning from failures, celebration of success and identifying new questions for future inquiries.

One of my key roles is to model the giving of ‘critical feedback’ in the online communities so that learners become more adept at peer reviewing each other work. In pursuing this approach, it is also important to ensure that assessment criteria credit the approach I have designed in the summative assessment.

As the degree is wholly online with no face-to-face, the majority of individual feedback is given in writing. This requires it to be very thorough in nature, addressing each intended learning outcome (ILO) identified by the module definition form of a module using the assessment criteria identified for each ILO. This is given via a cover sheet that is split into three sections. Things students have done well, things students should consider, and targets for improvement. Each learner is encouraged to seek clarification on the cover sheet, but it is made clear that this is not negotiation about the assessment decision. It is an opportunity for them to better understand what and why they have done things well and things they need to focus on to improve in the future.

Secondly, at the end of modules that I lead, I write an ‘examiners report’ that seeks to highlight generic issues from across the module, including those identified by colleagues. This report is made available for the students and colleagues to discuss in the online communities to encourage them to explore with fellow students and facilitators the assessment process more widely.

Occasionally, individual students may request a phone conversation that I undertake only after we have tried to address issues, ideally through discussions in the online communities or if of a personal nature via the use of email. This is an important point that has implications for all online teachers; a model of 1:1 communications is not sustainable in terms of workload and those who work as lecturers/facilitators online must strive to use the power of many to many community discussions where possible.

Thirdly, phone conferences or synchronous online conversations enable us to look back over a previous module and forward to the module currently under study. This is driven by student requests and is and optional activity.

Evaluation of my learning and teaching activities occurs through the performance appraisal process in place at Anglia Ruskin and includes periodic meetings with my line manager and pathway leader. Discussions are informed through feedback on the design and delivery of modules from both students and fellow teaching staff via the University Field Pathway meetings. This is an opportunity for me to plan to address issues that have arisen as well as new ideas for developing innovative approaches and evaluate the implementation of my strategies at subsequent appraisal meetings.

As an action researcher, I seek to identify issues related to teaching and learning that I can have an impact upon. This can be at the level of reflection on my practices shared in my professional weblog and the Ultraversity/Ultralab online communities, or through formal action research inquiries that demonstrate a rigorous process of data collection, analysis, identification of findings and evaluation against research questions with conclusions that inform planned phases of action.

The BALTR approach encourages students and staff to engage in discussions about modules, their design, delivery, and approach to teaching. At one level, this is a very informal approach but it yields some of the most useful feedback on approaches I take including changes implemented to try and address issues identified. These interactions take place in online communities where focussed and planned discussions as well as informal observations and questions take place. Any changes or actions taken, often receive evaluative feedback especially when they have failed to achieve the desired outcome.

Ultraversity project meetings that take place three times a year over a three-day period support the above activities. Issues of pathway and module development from the starting point of the experience of the learner are addressed by the team as a whole and smaller working parties. As a project specifically set up to identify new and innovative ways for delivering a HE experience, this activity is at the heart of my working practice. At these meetings, discussions and activities to develop our practice are informed by ongoing evaluation and research.

Particular examples of how these findings have improved the student learning experience and learning activities include:

student feedback over the first year of the BALTR expressed concern about the running of modules concurrently. We identified this as a particular issue for online students studying on a programme like ours who do not have the discipline of a timetable to help structure their studies. As a result, we implemented changes to our timetable that meant only a maximum of two modules would run concurrently;
learning sets form an important strategy in the BALTR programme and student feedback has helped design how they operate. Initially, I selected students concerned that self-organisation would lead to complex management issues. Naturally, for students who had developed friendship groups this wasn’t ideal, although a case can be made in its favour in terms of developing their skills to work with different students, etc. However, feedback was strong that students wished to be in control of this process and in handing it over to them we have found that greater degree of variety of composition of sets decided by students works well;

dissertation at level 3 of the BALTR takes the format of a workplace exhibition to colleagues and stakeholders. This first cohort of students that undertook this module reported many issues and difficulties they faced. In response I developed an earlier module with a small-scale pilot exhibition that enabled students to ‘test the waters’ in their place of work. This approach has proved to be very successful as demonstrated by the better success of students in their assessment products for their dissertation and final exhibition.

I have broken this section into two parts. The first concentrating on accessibility considerations, and the second on the direction in which Jisc are pushing e-Learning in the UK.

The Disability Discrimination Act 1995 (DDA) sets the legal requirements surrounding accessibility and websites. From 2000, the Disability Rights Commission (DRC) have policed these requirements, publishing a Code of Practice in 2002 and again in 2006. Despite these measures put in place, a survey in 2004 by the DRC found that 81% of websites did not meet basic W3C accessibility guidelines. Arguably, this is a very complicated landscape to successfully navigate.

My own experience may be illustrative of why there has been a lack of progress in making websites accessible. In my role, it was important that I was aware of the legal requirements in terms of accessibility and ultimately it was my responsibility that project websites conformed. However, I do not have a specific and detailed understanding of the W3C Web Content Accessibility Guidelines and in my work role could realistically have no more than a general understanding of issues surrounding frames, images, and the use of colour to convey information, etc.

This problem was recognised by the DRC and in 2006 a Publicly Available Specification (PAS) was published in collaboration with the British Standards Institution (BSI). The PAS sets out a process for the commissioning of an accessible website and in so doing gives those commissioning work a set of principles and processes to apply when dealing with website developers. These include ensuring that those building the site have appropriate knowledge and skills, are the processes for accessibility testing are clearly defined, and how to ensure ongoing maintenance and development doesn’t result in accessibility degradation. Ensuring that appropriate quality assurance and reporting mechanisms are in place by those commissioning websites, would I believe have a significant impact on accessibility.

The Jisc Effective Practice With e-Learning - good practice in designing for learning guide, 2004 (see review) makes for an interesting read. It is an attempt to distill down the findings from research into e-learning since 2003, when the Jisc e-Learning and Pedagogy research strand began.

The document seeks to provide a framework for colleagues to “reflect upon the value of e-learning” which it does well providing we maintain a critical eye as in my view there are some omissions and wrong emphasis in parts.

Firstly, I applaud the approach of putting the focus firmly on learning and the designing of planned learning experiences, not the technology itself. The language of “designing for learning” with terms such as ‘intended learning outcomes’ brings a greater degree of focus to the planning of an appropriate student learning experience.

However, I believe that the flaw in the document is that it approaches e-learning as something separate from good teaching and learning practice. The starting point for the document is the question “What is e-learning?”, I would argue a better question might be, What supports effective learning? and then address this from both the student and teacher (used advisedly to encompass the role of the tutor, coach, mentor, etc) perspective.

More recently, the phrase learning ecology’ (George Siemens October 17, 2003) is being used to describe an holistic approach that sees the technology as an enabler, that is as a means of connecting and making links between information and people. We must also not forget the use of technology as a tool to support cognitive processes (modeling, etc) and as means for greater expressive and communicative power. These points are not new to the Ultralab way of thinking.

Figure 5 Jisc Report Summary

The diagram (fig. 5) is included in the summary of the Jisc report (2003). I believe that that there are two areas that are given insufficient attention.

First is the lack of an explicit recognition of the role of the facilitator/lecturer/teacher in the learning process itself. This is not just as the ‘instructional designer’, but as the mediator of learning. It is not sufficient to plan a programme of learning without an ongoing interaction with the learners and there is much evidence of the importance of the relationship building between the and the learner. At the heart of the HE learning experience should be the development of a critically reflective learner. Without the modeling of aspects that contribute to this such as critical reflection, challenging of assumptions, testing of ideas, discussion of analysis, etc.

An excellent discussion of some of these issues is given by Brockbank and McGill (1998) Facilitating Reflective Learning in Higher Education, OUP) and we could also look to Laurillards conversational model (1993) Rethinking University Teaching .

The Hefce model shows effective practice as an outcome of factors that contribute to the learning activity, but does not show the actions of the ‘teacher’ as part of the model. This is a potentially significant omission as the values, beliefs, and teaching and learning philosophy of those teachers will heavily influence the learning experience.

Secondly, I believe that there is a lack of emphasis on assessment in the model presented. Although it was mentioned often in the document I do not think it was addressed in sufficient depth. The assessment process must be well thought out if it is not to end up being the ‘tail that wags’ the teaching and learning dog! Unless this is addressed in terms of both the products for assessment and the purpose of the assessment, then e-learning or any other form of learning is not as effective. For example, is it appropriate to have the majority of assessment by the essay or examination, or should we be thinking about the opportunities offered by approaches such as e-portfolios and Patchwork Texts (Winter, R., 2003)?

So why is there not a strong emphasis on this in the Jisc report? Could it be that there is still a view that the technology really is the answer to everything and that if we get the infrastructure and processes right, then deep learning will follow as a matter of course?

It could be argued that the role of the facilitator and the need for formative and other forms of assessment is implicit throughout the report. Personally, I do not think that this is sufficient as the changes that are identified as being required by the Hefce Strategy for e-Learning represent a paradigm shift for HE. That is a change to an agenda of learner personalisation and choice, access through workplace learning, and making the rhetoric of life-long learning a reality.

Working collaboratively was integral to my 7 years employment at Ultralab. This included both working with co-located colleagues and, perhaps more significantly, in a team of distributed workers which brought together a practitioner community of around 40.

As a member of this community, I engaged in discussions and supported colleagues both on projects where I had direct responsibility but also on those which I had an interest or in response to requests.

This community was the focus of a paper and book chapter I co-authored, “€œBuilding a community of practice: technological and social implications for a distributed team”. http://69.89.27.211/~stephep6/blog/wp-content/uploads/2007/03/building-a-community-of-practice.pdf   The chapter is based upon the professional communities that work within Ultralab supported by a range of online technologies with the express purpose of developing the whole teams capacity in learning, technology, and research.

A specific example of the power of the Ultralab practitioner communities is demonstrated by the Ultraversity project. Although the team drawn together to work on the project were experienced educators, there was the need to develop new working practices of this remote team to support the delivery of the new BALTR degree programme.

This online community worked on a daily basis to address issues that naturally arise out of the development of a new pathway.

In addition to the ongoing online professional development, I convened three ‘mini’ conferences per year of 3 days duration that focus solely on the development of the BALTR programme. At these conferences, sessions address research, module development, issues of learning and teaching, and learning resource development.

I would describe myself as a practitioner researcher, and my research interests are in the disciplines of learning and teaching, education, and learning technologies. Papers and presentations that I have co-authored include:

* Bradshaw, P., Powell, S., Terrell, I. (2005). Developing Engagement in online community: lessons for higher education. Innovations in Education and Teaching International.
* Bradshaw, P., Powell, S., Terrell, I. (2004). Building a community of practice: technological and social implications for a distributed team. Chapter 16 of Knowledge Networks: Innovation through Communities of Practice. Edited by Paul Hildreth and Chris Kimble published by Idea Group Publishing.
* Powell, S. (2003). Leading a Geography Department. In, M. Brundrett & I. Terrell (Ed.), Learning to Lead in the Secondary School: Becoming an Effective Head of Department.
* Powell, S., Terrell, I. (2004). Across the continents - online community conferences as a mechanism for school improvement. Paper presented to ICSEI Conference.
* Bradshaw, P., Powell, S. (2003) - Online Communities for School Leaders. Paper presented to the VCI Conference, London. Pete Bradshaw and Stephen Powell.
* Bradshaw, P., Gee, A., Powell, S. (2002) - Virtual communities and professional learning across a distributed, remote membership. Paper presented to Virtual Communities Conference, University of Navarra, Barcelona.
* Bradshaw, P., Powell, S., Terrell, I. (2002). Online Communities - Vehicles For Professional Learning? Paper presented to BERA.
* Bradshaw, P., Powell, S., Terrell, I. (2002). Learning, Community and Technology: Ultralab’s recent experience. Paper presented to APU Learning and Teaching Conference.

I am a Reviewer for the British Journal of Educational Technology which I find to be an invaluable way of keeping abreast of developments in the field of learning technology and online learning or e-learning in general.

I maintain a professional weblog ‘Thoughts mostly about learning’ where I post reflections on my work on the BALTR programme, reviews of conference events I have attended, and research findings. This post evidences a discussion with a colleagues’ around a post.

In 2003, I lead the validation team for a new pathway at Anglia Ruskin University, the BA (Hons), Learning, Technology, and Research (BALTR - appendix 5 validation). In this role, I was responsible for synthesising input from the development team and to make the final decisions about the design and planning of the programme. This pathway was developed as a part of the Ultraversity project.

The BALTR was designed as a series of ‘open module frameworks’ of generic statements against which performance is assessed. Learners identify subject knowledge that is relevant to their own context and needs. Through a process of negotiation with teaching staff develop a set of learning activities that enable them to meet the requirements of the modules and assessment criteria. The processes of  learning and inquiry defines the content of the degree with a focus on developing a ‘practical understanding’ or ‘knowing why and how to’ in their chosen discipline.

A brief overview of the design of the model of learning developed and some of the approaches, activities, and techniques are given below.

Figure 6 Model Supporting Workbased Learning (Powell, S and Tindal, I 2006)

Personalised learning
This model of personalised learning has at its heart notions of learner control and choice. In this formal learning context, limits to the degree of personalisation are imposed by both reasonable requirements of the accreditation process as well as those that are in place for reasons of academic tradition, for example word count. This model uses Individual Learning Plans (ILP) and inquiry proposals where the learning activities are negotiated to be authentic and embedded in the daily work of the learner by design.

Inquiry based learning
This is based upon Action Research methodology that has an emphasis on critical reflection on an individuals work practices and inquiry into their work context. This leads to an action that is planned, implemented and evaluated with the intention of making a positive impact on their work, that is, ‘learning for performance’.

This approach is designed to enable students to effectively integrate study and workplace activities with the support of a ‘workplace advocate’ who is identified by the learner as someone who can help with work-place issues.

Online community
Researchers work and learn together in an online community of inquiry where social construction of knowledge is realised through collaboration and critical friendship between learners. Engaging in processes of inquiry together as a cohort makes it possible to share and support without plagiarising because learners are studying in their own work context.

The course designers valued unstructured or ‘chance dialogue’ (Powell, 2004) where learners initiate their own conversations, but also designed an experience that had opportunities for purposeful conversations initiated by teachers (Laurillard, 2002). This was achieved through the development of a facilitated online Ccommunity of Inquiry where a rich experience of challenge and debate, support, sharing findings, critical feedback, access to online library, and conversations with invited expert could take place.

Participation in this community is not punctuated by the delivery pattern of modules, or determined by the access restrictions applied by Virtual Learning Environments (VLE). The BALTR model allows for ongoing interaction between students 365 days of the year. In addition, the choice was made to allow learning resources to be available outside of the teaching time so that students could plan and take responsibility for their learning.

Experts join the communities to host focused conversations that engage learners in critical dialogue. This is not a lecture by an expert, but an opportunity for learners to direct conversation to meet their own needs.

Exhibition for dissertation
Towards the end of the programme, learners are required to construct an exhibition of their findings primarily based upon the final year of their studies but drawing on the whole three-year experience. The exhibition is given to an audience identified by the learner, wherever possible in their place of work.  This critical evaluation of the exhibition helps validate their findings.

Through this process learners demonstrate to themselves and stakeholders the progress they have made in terms of personal growth, and in their ability to perform in their work role. Initial findings indicate that students are engaging with the idea of being a lifetime learner.

Assessment for learning (see section 3.1)

Currently I am unemployed and am taking a couple of months out of formal work to reflect on where I want to be in a few years time. However, I am certain that I wish to take forward the ideas developed by Ultraversity as I believe that they have the potential to make a significant impact on HEI and in particular workbased learning.