Evaluation of ARTS 2012 Program

Evaluation of ARTS 2012
A. Evaluation forms
Each of the ARTS Seminars was evaluated on the final day of the seminar by means of questionnaires given to ARTS conveners and participants. The questionnaires included quantitative ratings and open-ended questions about participants’ and conveners’ expectations and reactions to the seminars, and also gathered information about ARTS publicity and financial support. Participant questionnaires were completed by 43 of the 45 program participants (by 14 participants in the ARTS-1: Coping with the Strains of Globalization; 14 participants in the ARTS-2: Analyses of Psychological Data with R; and 15 participants in the ARTS-3: Community Psychology. Convener questionnaires were completed by conveners or speakers in all three seminars.

Participant evaluation
Programme evaluation. ARTS participants came to the program with high expectations (average rating 4.1 of 5), and modest levels of prior knowledge of the topics (average rating 2.9 of 5). The overall rating for whether they would recommend ARTS to others was 4.6/5 (scale from not to definitely would): approximately 97% of the participants answered that they would recommend ARTS. Other ratings are given in Table 1 below (on a scale 1 [poor] to 5 [outstanding]). Items 2 to 5 in the Table indicate the expectation that what participants had learned would be useful in their research or teaching, and would be taken forward to their students and colleagues. ARTS participants gave the instructors overall very positive ratings, and, as in the earlier ARTS, the open climate and opportunities for establishing networks with other participants were highly acknowledged.

Open-ended comments. Participants were also asked to answer open-ended questions about what they saw as strengths of their ARTS seminar and what they would like to see done differently. Strengths included the instructors’ efforts for building rapport, interactive climate among participants, sharing their knowledge in both theoretical and applied levels, the broad range of knowledge and skills addressed, the opportunity to interact with other participants in social settings, the opportunity to establish a network for future research collaboration, and the overall collaborative nature of the whole process.

Several aspects of the organizational issues were evaluated in a positive way, but some areas received lower ratings and will need addressing in future ARTS programs. These issues received relatively low ratings: the housing conditions the lack of internet facilities; the duration of the seminars (at least three days were recommended).

Table 1: Participant evaluation ratings

Finding Out About ARTS. ARTS participants learned about the program from seeing it announced on the web (31% saw it on one of the Congress websites; 14% saw it on one of the sponsoring organization websites). In addition participants heard about ARTS from their colleagues (20%) and the organizers (10%). Less effective modes of dissemination were through national associations and email listservs.

While the number of applications for the 2012 ARTS Program was considerably greater than in previous years, it remains important to ensure that dissemination about the program is as varied and broad as possible. Routes for dissemination include engaging national psychology organizations and psychology departments, as well as professional organizations for those countries where psychology is not well developed. The program should receive consistent and broad publicity through the websites of the sponsoring organizations and their Congresses.

Convener evaluation
Conveners reported that they generally found information about ARTS and participant applications useful in planning their own seminar and providing applicant ratings. They appreciated Steering committee’s collaborative leadership, supportive attitudes, attentiveness and overall active involvement before and during seminars. More specifically,they acknowledged the selection criteria and process, recently elaborated for the 2012 ARTS Program by the Steering Committee. In a similar vein, conveners gave positive feedback by emphasizing the participants’ profiles in terms of their valuable contributions to the seminars. They gave generally good scores for all aspects apart from accommodation. The challenges mentioned included providing participants with an earlier engagement, materials and preparation prior to seminar.

Overall, they were positive about the 2012 ARTS and especially appreciated the ways that the groups cohered over the course of the program.

B. Round Table Meeting at ICP2012:
ARTS for early-career researchers from low-income countries

To build on the participants’ and conveners’ experiences and insights, a round table session was held at ICP2012, a few days after the conclusion of ARTS. This was convened by Nick Hammond (ARTS Coordinator) and facilitated by Tom Oakland (IAAP Liaison); David Sam Lackland (IACCP Liaison) and Gonca Soygüt (IUPsyS, Liaison), ARTS Steering Committee Members. Participants included conveners from all three Seminars and many of the 2012 ARTS Alumni.

This session explored the effectiveness of the ARTS Program in meeting its aims, drawing on evidence from both the 2012 program and earlier programs. The session addressed the benefits for individuals attending ARTS, the wider benefits of capacity building within the participants’ countries or regions, the international collaboration or networking resulting from ARTS, the benefits for sponsoring organizations, and how to sustain these benefits. As a parallel source of evidence to the responses from the evaluation forms reported above, reported benefits included particularly meeting and interacting with participants from diverse countries and cultures, sharing and knowing their views, comparing others’ views with one’s own, being challenged with new ideas, and the excellence of the presenters.

Several topics were proposed for the future ARTS Programs. These included:

  • Cognitive neuroscience (e.g., brain-behavior relationships
  • Meta-analysis
  • Multilevel data analysis,
  • Qualitative analyses (e.g. content, discourse)
  • How to access difficult to reach populations
  • Use of computer-based software that programs experiments
  • Organizational psychology (e.g. I/O employment issues)
  • Research methods in disaster psychology
  • Indigenous psychologies
  • Non-individual (e.g., group, systems) methods to address mental health needs in the community, psychological interventions with families
  • Adapting tests for local use
  • How to publish in English language journals

Finally the session considered alternative models for, or developments to, ARTS to help the program meet its aims more effectively and efficiently in the future. Many participants agreed that the success of this and other ARTS depends on the participant’s continuing involvement. 2012ARTS Alumni were encouraged to remain engaged with the topics discussed in their Seminar through continued networking with other participants and the conveners, and to convey their new knowledge and skills to others in their home countries, and by becoming active members of the ARTS alumni continue their academic and professional development. A further suggestion for consideration by future organizers and conveners was to consider how best ARTS participants can prepare for the Seminars, through, for example, engagements with the conveners on a regular basis beforehand or through preparatory reading or exercises.