Difference between revisions of "What are the main considerations for each of these roles?"

From Scarlet Wiki
Jump to: navigation, search
(→‎Project evaluator considerations)
 
(One intermediate revision by the same user not shown)
Line 76: Line 76:
  
  
−
One of the key things to consider when putting together a toolkit for any context, but particularly an educational one, is how you will evaluate the outcomes from the surveys and focus groups.  In a recent workshop, the SCARLET Team used a dialectical approach, initially to put down the key processes we used to evaluate those outputs, and then to imagine you could somehow talk to yourself in the past – the” you” who is just starting to put together an evaluative process – and then to tell your temporal doppelgänger what you need to know to make the process as easy as possible.
+
'''Evaluation Processes:'''
−
Reflecting on what is ostensibly a dialectic, what surfaced was the intuitive way we all worked and the difficulties involved in trying to reify those efforts into something usable to another.  We recalled that in working toward some sort of cohesive evaluative process for a project like SCARLET, it’s important to determine methods for making sense of how the project has developed, targeting key success factors and ideas for improvement.  There are a number of ways of doing just that, outside of more formal, academic assessment, which doesn’t necessarily point to the factors which might influence take-up of a particular project or pedagogical enterprise.  In addition, the evaluative process needs to be one which works toward accessing audience views before and after the experience of the project.  Initial surveys and focus groups were determined to be the best way for evaluation and were used in order to get an idea of what students knew about Augmented Reality and other technologies before, during, and after.
 
  
−
The focus group stage is often fraught with some of the biggest problems. Generally, it is difficult to get people to attend, and if you do, the idea of incentives can often create problems.  With regard to incentives, it is important that they are presented as a “thank you” for participating, and not perceived as some sort of payment.  More problematic is the make-up of the group; you don’t want to have a group made up of individuals who are too similar or too different.  Liz Spencer, lecturer for the Social Research Association at the University of Essex, talks about the fact that the best kind of results generally come from a heterogeneous collective of people who are largely unaware of the subject matter but who are interested in learning more – in our case, a group of 3rd year undergraduates. 
+
1. '''What were the objectives in evaluating the SCARLET Project?'''
  
−
The conceptual framework of a focus group, then, is one which helps the Team see and reflect on the design of the project’s platforms, technologies, content, and delivery, as opposed to getting an exit interview or simple feedback form we have all had to fill out after a workshop, e.g., “on a scale of 1-10, how likely are you to recommend this course to others?”.  That kind of information is valuable, to a degree, but it hardly tells you why somebody thought the workshop was valuable.
+
• To create a selection of data which showed what about the SCARLET application students found useful in their coursework
−
In addition, it is also key to understand that focus groups do not create quantitative data; they are not scientific studies, whose results we can point to and say, with confidence that, “95% of the users believe that brushing with Colgate improves their social standing at the office.”  Yes, they are participative in a branch of sociology, but focus groups again do not necessarily create that level of precise data.  Their inherent value lies in that they can potentially give us an idea of why 95% of the group felt or thought that way about a relatively ordinary toothpaste.
+
• To determine how “skilled up” students needed to be to use the app (i.e., did they have previous experience using Augmented Reality? Do they have a Smartphone? etc.)
 +
• To determine what worked well and what needed continued development for potential future stages of the Project
  
−
The next phase will be the exit interview of the academics, with whom we have worked.  In attempting to evaluate the overall success of a project, this bit of feedback is essential, particularly because there remains a strong emphasis on working with the HE sector.  As The Horizon Report of 2011 suggested, AR remains a technology to watch, both because of its relatively low cost, which makes it attractive to a range of projects, but also because it offers so many different kinds of applications to a range of possibilities, and education remains one which has only begun to be tapped in the UK markets.  Finding out how the academics perceived of the project, from its perception, through to its completion, is important to the continuing design of future projects, which involve AR, as well as the design and construction of the toolkit.  The interview’s design is straightforward enough and based upon comments that the academics have made, with regard to the project throughout, posing issues which underscore how it has affected their teaching, as well as the students’ learning.  As with the vast majority of the feedback gathered, these interviews will also be made publicly available through various means, most notably in the toolkit itself.
 
−
As a means of starting to get all these steps down for the toolkit, the project manager created a simple chart, with a “Process” column running down on the left-hand side.  For the group I was in, this column is where we plotted the various things that we have done to gather data, as I mentioned above: an initial survey of 3rd year undergraduate students participating in the pilot course and a focus group of those same students, approximately 12 weeks into their course, as well as writing up a brief explanation of that focus group, highlighting key “Lessons Learned”,  feeding those highlights back into a meeting for dissemination and internal evaluation, and writing up a case study on the project so far, and then a final online survey for the students who participated.  I wrote these down in the order that I did them, acknowledging now that the focus group may have been better placed earlier in the course and that the highlights could have been a simple blog post.
 
  
−
Working left to right on the same sheet, we worked toward that dialectic, to achieve some synthesis from the process: what would you tell yourself if you were starting the project fresh?
+
2. '''How was the SCARLET Project formally evaluated?'''
  
−
• '''Self-completion survey:'''
+
• The formative evaluative process involved an early stage focus group with 3rd year undergraduates in the pilot course, “The Book and Its Body”.  The session was approximately 1 hour, and offered the students a chance to be reminded how the app actually worked and what its use was to have been in special collections.
 +
• Following on from the focus group, a blog post was written up to highlight the specific lessons learned.  The students presented some interesting points regarding the use of the technology, along with some pertinent anxieties over how it might get in the way of the actual experience of working with the ancient editions of Dante’s The Divine Comedy.
 +
• An exit interview of the academics who were directly involved in the design and implementation of the Project
 +
• In addition, four more focus groups were conducted with groups of 1st, 2nd and 3rd year students in follow-on courses: in Dr. Armstrong’s ITAL 10300,“Contemporary Italian Culture,” and Dr. Roberta Mazza’s CLAH30320, “Advanced Greek 3” and RELT20232, “The Body and Society: Christianity and the West.”
 +
• An exit survey with the students was conducted as a part of the summative evaluative process.
 +
• Blog posts accompanied all of this work.
  
−
o '''Considerations:'''
 
−
• Agree the most appropriate format for the survey (paper-based or electronic) dependent on the audience and        situation
 
−
• Be careful to create good open-ended questions
 
−
• Include a mixture of open and closed questions. Closed questions are easier to answer and provide more structured data but open questions may provide more detailed responses
 
−
• Split questions which have multiple subjects into simpler questions
 
−
• Create questions which encourage students to participate
 
−
• Plan design carefully to ensure ease of use and maximum survey completion rate
 
−
• Aim for shorter surveys to increase response rate
 
−
• Create clear, unambiguous questions and clear instructions
 
  
−
o '''Risks to consider:'''
+
3. '''How was overall success of the SCARLET Project determined?'''
−
• Too few students participate
 
−
• Data is irrelevant or incomplete
 
−
• Questions did not encourage engagement, i.e., students simply answered “yes” or “no” without explanation, can’t prompt or probe
 
−
• Surveys generally provide lower response rates
 
  
−
• '''Final  Online Survey (using Bristol Online Surveys)'''
+
• Student feedback via the Focus Groups (formative assessment) illustrated overall satisfaction with the course offering and delivery
−
• Create open-ended questions
+
• Student feedback via the Online Survey (summative assessment) also showed overall satisfaction with the course and its delivery
−
• Ensure that each question focuses only on one item/issue
+
• Academic feedback from exit interviews (to be conduction on 28 May 2012)
−
• Encourage students to participate
 
−
• Get academic buy-in
 
−
 
 
−
o '''Risk to consider:'''
 
−
• Students no longer feel obligated to respond since the module is over
 
−
• Students feel resentful that they feel compelled to participate
 
−
• Too few respond regardless of encouragement
 
−
• Online survey malfunctions or is inaccessible
 
−
 
 
−
• '''Focus Group:'''
 
−
 
 
−
o '''Considerations:'''
 
−
• Select members which are heterogeneous
 
−
• Consider how many group sessions are required or feasible and the size of the group
 
−
• Develop a session guide or plan
 
−
• If students, ensure that you get them early enough in the module, otherwise they are likely to give skewed answers, influenced by the lecturer or library staff, rather than based on their own perceptions
 
−
• Consider recording and transcribing the session
 
−
 
 
−
o '''Risks to consider:'''
 
−
• The group was too homogeneous, so answers don’t tell you much
 
−
• The group lacked any cohesion – no bonding, no sharing
 
−
• The environment was not conducive to the session, i.e., too warm, too cold, room was too small
 
−
• The incentives offered didn’t work to produce anything meaningful
 
−
•
 
−
• '''Results and highlights:'''
 
−
o Write up the highlights from the survey and focus group as soon as possible
 
−
o Share with others who may have also been present, to ensure accuracy
 
−
o Work toward getting quotes, although do not attribute them (confidentiality)
 
−
o Make it relatively short so that others can re-use in blog posts, case studies, etc.
 
−
 
 
−
• '''Dissemination:'''
 
−
• Consider a variety of dissemination methods and channels including:
 
−
o Blog posts
 
−
o Professional publications,
 
−
o Case Studies
 
−
o Conferences, meetings, workshops and events
 
−
o News items, articles and features on websites and in journals
 
−
o Academic journals
 
−
o Risks to consider:
 
−
• Academic articles require a great deal of time and the direct input of the academic; therefore, this form of dissemination is often most problematic.
 
−
 
 
−
Not every factor has significant risks, but it is important to consider what risks there are prior to embarking on any kind of evaluative process.  It would seem obvious, but the toolkit is meant to emphasise any of the steps necessary to move toward completion.  It also maps nicely back to a general bid-writing process, which also has to consider potential risks and pitfalls.
 
−
In the end, the project is valuable and significant, responding to the Horizon Report’s call to watch AR as a pivotal technology in the coming years (2010).  We have produced something which uses technology to enhance the experience of working with a material object, AND which importantly does not replace or get in the way of that experience; as I mentioned in my previous blog post, most students felt that the use of AR with Special Collections was, indeed, valuable to their experiences with the texts in Special Collections. 
 
−
 
 
−
Moving forward, this toolkit will also benefit from a new group of 1st year students lined up for a similar course, along with a group focusing on a fragment of the Gospel of John, under the tutelage of Dr. Roberta Mazza, both of which have exhibited a great deal of interest and enthusiasm for the content and the means of delivery.  If as Confucius said that “success depends upon previous preparation”, this toolkit promises to be a useful and compelling aid in the creation of future projects involving Augmented Reality in a variety of places and contexts.
 
  
  
 +
4. '''What are some examples of links to additional helpful sources?'''
 +
• Determining whether there are related projects is helpful in presenting the project.  The examples below are indicative that our own project was both built on an existing ethos of course delivery, but remained unique among courses and modules using non-traditional means to delivery content.
 +
o University of Exeter LAYER: http://www.exeter.ac.uk/students/life/layar/
 +
o JUNAIO: http://www.junaio.com/
  
 
== Project management considerations ==
 
== Project management considerations ==

Latest revision as of 13:13, 11 July 2012

Library IT considerations

Access to devices

Whilst many users now have their own devices capable of using AR services, many do not. If inclusivity is an issue, therefore, the institution may need to consider providing devices for loan.

Issues to consider when providing devices include:

• The life span of the equipment purchased is likely to be short. Mobile device technology is changing quickly and you may need to put in place budget for the regular replacement of devices bought. • You will need to download apps from the appropriate app store. These may include paid for apps in which case you will need to provide details from a corporate credit card. You will need to identify an individual who can be associated with the account (probably someone within the local IT team). • Devices will need to be easily accessible but stored securely with proper reservation and loan procedures in place. Wireless access Whilst many users using augmented reality based services will use their own devices that come with network access many will not, or will be unwilling to use their own network do to the cost). It is therefore vital that good WIFI connectivity is available in the spaces where the AR service is to be used. Integration with University systems If most of the content you wish to provide access to is already delivered through a web interface, there are unlikely to be any significant integration issues. The main concern is likely to be display – it may be necessary to reformat output to make it more useable in a mobile environment. However it is worthwhile testing all integrations to ensure that data is pulled through and rendered correctly.

Where services require authentication this can cause serious usability issues and you may want to consider providing a non-authenticated view of the data pointed to. Accessing licensed content Staff identifying content for inclusion in a service may wish to include licensed content (such as a journal article provided on a publisher website). This can cause serious usability issues as authentication is likely to be required if IP address recognition is not in place or if users are using their own devices. You may want to discourage accessing such content and identify alternatives.


Library Special Collections considerations

Enriching the experience of using Special Collections

AR provides a wonderful opportunity for enriching the experience of users of Special Collections materials. It can transform it from being a rather passive learning experience – students being talked at by curators and tutors, or visitors reading exhibition captions, for example – into something that is much more engaging and interactive. It therefore fits in well with modern enquiry-based learning methodologies. We found that AR was most beneficial in introducing undergraduates and visitors to a subject or an aspect of our collections. It is not a research tool, and has limited relevance for experienced undergraduates and postgraduate students.

Although it wasn’t an element of the Scarlet project, AR also has the potential for linking difference parts of your collections together: for example maps and archival records, photographs and audio-visual content. Choose your materials carefully

AR works best with small quantities of distinctive materials, such as unique illuminated manuscripts and early printed books, which can serve as visual cues for the apps. It is less suited to large quantities of material, such as large archives, and run-of-the-mill printed items. Obviously the material needs to be intrinsically interesting. You also need to consider how to associate QR codes or other visual cues with the objects to which they relate. It obviously isn’t appropriate to attach a QR code directly onto a Special Collections item: you might need to attach it to an acid-free bookmark, or use a distinctive feature of the item itself, such as a title page or frontispiece. Consider the needs of your audiences carefully

There is a danger that the technology can be the driver of AR apps, not the needs of your audience. You need to know your audiences and their needs, and you will of course want to involve them in the development and evaluation of apps. Does an app fulfil a particular pedagogical need? What is it adding to the student’s or visitor’s experience? If the answer is ‘not much’, or ‘not sure’, it’s time for a rethink!

Working with academics and other experts

We found it a really rewarding experience to work with enthusiastic academics, and the Library staff learned a great deal from the process. In a way, this wasn’t particularly surprising, as we chose academics with whom we already had a good relationship, and who are interested in innovative pedagogy. Working with less IT-savvy academics or experts might be more challenging. Our experience does point out the importance of developing close relationships with academics and other key stakeholders for your organization.

Unexpected benefits

Special Collections sometimes have a reputation for being rather staid and stick-in-the-mud. The SCARLET project put the John Rylands Library’s Special Collections at the forefront of technological developments, and focussed attention on our remarkable collections. There are significant promotional and publicity opportunities involved in developing successful AR apps.


Library management considerations

Longevity

Developing successful AR apps takes time and effort, and you’ll want to maximize the benefits from that investment. It is therefore sensible to plan for your apps to have a reasonable shelf-life, of say two-three years. This means that you won’t want to depend upon ephemeral content. However, it isn’t reasonable to expect apps to have a longer shelf-life; the technology is advancing at such a rapid pace that what is leading-edge now may soon appear tired and old fashion. Think about refreshing existing apps from time to time with new content, or a visual makeover. If you are linking to external web resources, check the links regularly.

Sustainability

You will probably wish to develop the capacity to create and update apps within your organization. While outsourcing elements of the process may initially be quicker and easier to manage, and result in a more ‘polished’ product, there is a danger that you will never develop expertise in-house, and there is also a risk of being locked into expensive maintenance agreements. Of course there are also dangers of relying entirely on one or two key people within your organisation: ideally the expertise should be spread as widely as possible.

Cost-benefit analysis

The old adage that the more you put in, the more you’ll get out certainly applies to AR: the greater the effort and investment you put in, the more students and other users will get out of it. Simply throwing together an image and a few web links is not likely to satisfy most users of AR apps. We found that students really appreciated information that was tailored to the specific books and manuscripts being discussed, rather than being generic. They pointed out that they could obtain general information more easily from the web. They particularly appreciated the video presentations made by tutors on individual books and manuscripts.

On the other hand, in a period of severe financial constraint, no-one can lavish infinite time and money on AR apps. You may want to concentrate resources on a limited number of significant items, and to repurpose existing content as much as possible. The cost of apps can be surprisingly low. For example, rather than commissioning professional-made videos, we made our own. You could also make use of volunteers and interns: making an app would be a great project for media and heritage studies students.

AR apps can also serve to showcase and repurpose existing investments in digitisation and cataloguing activity, not only helping to justify the expenditure on such activities, but serving to promote them as well.

Partnerships

A success AR project requires a wide range of skills and expertise: collection knowledge, pedagogical expertise, technical skills, knowledge of your audiences and their needs, promotional skills, etc. It is therefore likely to require the active, enthusiastic participation of curators, external experts, teachers, IT specialists, project managers and marketing and communications staff. The experience of the Scarlet project was very rewarding, in that it helpful to consolidate relations between academics and students within The University of Manchester, John Rylands Library staff, and the project managers and IT experts in Mimas. Other AR projects may provide exciting opportunities for working with other institutions, community groups and individuals, enabling the sharing of expertise and development costs.

Evaluation

In order to justify continued investment in AR and to ensure that it meets customers’ needs, it’s important to evaluate your project, using a range of quantitative and qualitative measures. Have you achieved the outputs and outcomes that you forecast at the start of the project? What difference has it made? What could have been done better? Are there potential cost savings for future projects? Are there opportunities for partnerships with other organisations?


Project evaluator considerations

Evaluation Processes:

1. What were the objectives in evaluating the SCARLET Project?

• To create a selection of data which showed what about the SCARLET application students found useful in their coursework • To determine how “skilled up” students needed to be to use the app (i.e., did they have previous experience using Augmented Reality? Do they have a Smartphone? etc.) • To determine what worked well and what needed continued development for potential future stages of the Project


2. How was the SCARLET Project formally evaluated?

• The formative evaluative process involved an early stage focus group with 3rd year undergraduates in the pilot course, “The Book and Its Body”. The session was approximately 1 hour, and offered the students a chance to be reminded how the app actually worked and what its use was to have been in special collections. • Following on from the focus group, a blog post was written up to highlight the specific lessons learned. The students presented some interesting points regarding the use of the technology, along with some pertinent anxieties over how it might get in the way of the actual experience of working with the ancient editions of Dante’s The Divine Comedy. • An exit interview of the academics who were directly involved in the design and implementation of the Project • In addition, four more focus groups were conducted with groups of 1st, 2nd and 3rd year students in follow-on courses: in Dr. Armstrong’s ITAL 10300,“Contemporary Italian Culture,” and Dr. Roberta Mazza’s CLAH30320, “Advanced Greek 3” and RELT20232, “The Body and Society: Christianity and the West.” • An exit survey with the students was conducted as a part of the summative evaluative process. • Blog posts accompanied all of this work.


3. How was overall success of the SCARLET Project determined?

• Student feedback via the Focus Groups (formative assessment) illustrated overall satisfaction with the course offering and delivery • Student feedback via the Online Survey (summative assessment) also showed overall satisfaction with the course and its delivery • Academic feedback from exit interviews (to be conduction on 28 May 2012)


4. What are some examples of links to additional helpful sources? • Determining whether there are related projects is helpful in presenting the project. The examples below are indicative that our own project was both built on an existing ethos of course delivery, but remained unique among courses and modules using non-traditional means to delivery content. o University of Exeter LAYER: http://www.exeter.ac.uk/students/life/layar/ o JUNAIO: http://www.junaio.com/

Project management considerations

Funding:

The first step prior to developing a project is to source appropriate funding to support development. Identifying sources of funding, whether internal to your organisation or external is paramount. Proposals and funding applications should clearly outline the project aims and objectives. Telling a story can help to set the scene, provide a rationale for doing the project and emphasise the potential benefits.

Team:

It’s important to get the right balance of people within a team to ensure that the project functions effectively. Cross disciplinary projects and those involving a mixed team such as the SCARLET project need the right people with the right skills This can involve a significant amount of time at the outset of the project and this shouldn’t be underestimated. A successful AR project requires a variety of skills and expertise. The SCARLET project included library and special collections experts, academics, learning technologists and project managers. These diverse roles brought a much-needed mixture of skills and expertise to the project. An enthusiastic team with a shared focus and common aims is key to making progress on innovative projects where timescales are tight.

Project plan, budget and WP:

A project plan defining aims and objectives, means of achieving them, outputs, measures of success and approaches to project management is a useful working document for the whole team to help review and monitor the project. Detailed plans for work packages help all the team to understand the project and means of achieving the project aims. It’s important to involve all team members in development of the work packages and project plan to develop a shared vision and establish the viability of the project.

Disseminate:

When developing any AR project it’s important to consider how you will disseminate it. If dissemination activities are planned at the outset it’s easier to make sure that the right people are informed about progress, at the right time, using the most appropriate methods. Consider the purpose of the dissemination activity, think about what you want to achieve and how it will support or inform project development.

Create a schedule outlining possible dissemination events, articles, conferences and meetings where there will be opportunities to highlight the work of the project. It may be that additional funding will need to be secured in order to attend events but it’s useful to list all options should additional budget become available.

Retrieved from ‘http://wiki.digitalresources.jisc.ac.uk/scarlet/index.php?title=What_are_the_main_considerations_for_each_of_these_roles%3F&oldid=116’