Assessment

Over the years, I have read a lot of books, articles, and blogs and listened to many podcasts about assessment and research. I find it enlightening how terminology has evolved as the student affairs/higher education assessment field has matured. Language is important and nuanced.

I have heard/used the terms data-driven, data-informed, evidence-based and more to link the use of assessment to decision making and change. I have been reflecting on that as I think about how we not only develop narratives about what we do and who our students are, but also improvement and accountability. I also think about how student affairs assessment is inherently about relationships—about the relationship of multiple data points to each other, about data in relation to the people who create, analyze, and disseminate data, and the relationship between people in how data is used to improve stakeholders’ experiences and learning. It’s not always clear-cut.

These days, I’m less likely to use the term data-driven because it implies that data tells a story that we must follow in our decisions. On the other hand, data-informed and evidence-based indicate that data is one piece of the decision-making process. It allows for people to also include their own experiences, education, and biases, as well as other environmental factors that go into decisions.

In his podcast, ReThinking, Adam Grant had an episode called “Why ‘data don’t talk’ with data scientist and comedian Andrea Jones-Rooy.” It was based on Andrea’s essay, “I’m a data scientist who is skeptical about data.” They agreed that data actually don’t talk; data itself doesn’t say anything. Andrea made the analogy that data are like maps. We look at a map, but we decide which route to take. When I look at a map, I think about the shortest route (in time or distance), road construction, highway vs. roads, time of day, weather conditions, stops along the way, etc. I then decide about which route to take.

Getting back to assessment, people are the ones who say something—they collect and interpret data and then communicate and use the information. People make decisions about what data to collect when, what statistical or qualitative analysis to perform, and what changes are made. Staff also operate in an everchanging context: budgets, leadership, priorities, and policies inform actions. Assessment data is only one piece of the puzzle. I used to hear a lot of student affairs staff say something like, “I know that the students in my program are learning/satisfied/successful; I don’t need assessment to tell me that.” They are using their experience and anecdotal evidence to know that, and I don’t doubt they have a positive impact. And, I think that’s not the whole picture. I still hear that statement on occasion, but it is also usually accompanied by “I was able to use the results from my assessment to tweak a couple of things to make it even better.” I get really excited if someone tells me that they shared an assessment report with students and together they all interpreted the results and created collective change. Now, we’re really talking!

In summary, language gives meaning, and data doesn’t/shouldn’t take away your power to use your own wisdom and other pieces of information to make the best decision possible. Don’t be afraid to use your data-informed voice, since data don’t do the talking.

You have invited people to a focus group. You have created very insightful questions. You have food delivered (respecting any dietary concerns). You are ready for your focus group, right? While the preparation before the focus group is important, the actual focus group also takes planning and skill. Last month, I wrote about some of the preparations, so this blog addresses some of the intricacies during a focus group.

As people arrive to the great space you have reserved, you want to welcome them to the event and make them comfortable. Encourage them to get the refreshments you have provided. When everyone is settled, you’ll want to welcome the group, explain the purpose, and thank them for taking time to participate. You may also have them sign a consent form that also explains the purpose, how you will use the information, and where they can get further information.

Based on the time you have available and the number of questions you have, you will want to keep track of time. Hopefully, you have balanced the number of people with the time available and the number of questions you want to include. Each of those variables impacts the others.

As the facilitator, you have an important role in the success of the focus group, even if you are speaking the least amount of time. Your body language, eye contact, and verbal cues will encourage or discourage participation. Being comfortable with silence after asking a question will also encourage others to participate. You should also be familiar with phrases such as “tell me more about that” and “Is there anyone else who would like to contribute that hasn’t done so already?” to elicit more clear information or encourage quiet people to speak up. If you do not have experience in this area, you might want to talk to people who have more experience. In case the focus group veers to far away from your purpose, you need to be assertive to get the group back on track.

Ideally, you also have someone who is co-facilitating with you who can keep track of time, run the recorder (if you choose to do that), and take notes not only about the content but also any non-verbal signs from the participants. The co-facilitator can also give you signals about moving on when you are short on time or engaging other participants.

Asking questions is a skill in itself. In a structured or semi-structured focus group, you have a road map of what you want to know. When you first start with the questions (after doing introductions if appropriate), you probably want to ask something fairly easy, before moving on to deeper or more challenging questions. It’s helpful to be familiar with the questions and purpose, in case participants are confused by the questions. You can re-state the question in a different way that makes sense to them. At the same time, you do not want to be so verbose as to run on or bias participants to answer a particular way.

At the end of the time, you definitely want to thank people for their time and reiterate who to contact with questions. If you have recorded the session, you will want to explain how you will secure the data and ensure confidentiality or anonymity (if you are).

Focus groups are a great way to collect information about experiences, opinions, and ideas. As with any assessment technique, they have advantages and disadvantage in terms of ease, time, resources needed, and analysis. There are numerous books, articles, and videos about focus group best practices. While they may be challenging at first, they get easier with practice.

Focus groups are a great way to collect data (and it’s not using a survey!). A focus group brings together people who have a set of experiences and/or opinions about a topic of interest. The size of the focus group can vary, but usually six to eight people allows for balanced participation and a variety of views. The data collected will be narrow and deep, rather than wide and shallow from a survey because you will ask fewer questions, but have the ability to follow up to get more information. Participants can build on each other’s answers, but also take the conversation in unforeseen directions. For them to be successful, you need a good logistical plan in place. You need to plan ahead. Here are a few details that you want to address:

1. Clearly articulate what you want to know, who you want to hear from, and when you need to have the information in hand.

2. Reserve a space that is quiet, convenient, and accessible to your participants. The space should allow people to face each other to have a conversation. A conference table usually works well, as does having people sit in a circle.

3. Set a time that is convenient for your potential participants. You may want to schedule several options at a variety of times.

4. Invite more people than you think you need. Inevitably, the focus groups may not be at a time that is convenient for everyone, and sometimes other priorities come up. Follow up with them with several reminders.

5. Make the invitation meaningful and specific to why you think this person has information you are looking for and how you will use it. Give them plenty of time to get it on their calendar.

6. Consider using technology to audio record the session. Even if you don’t have the transcribed, it will allow you to listen again for anything you might have missed.

7. Think about having a co-facilitator, someone who can take notes if you are facilitating and jot down any body language contexts. They can also work the recording device.

8. Consider providing refreshments, especially if you are doing this near a mealtime. You might also consider providing a token of appreciation as you have the ability to do so. Because you are asking students to give up their time, it’s nice to reward them for that.

Those are a few of the logistical details that you need to consider. I haven’t yet touched on the content of focus groups, which will be a future blog post.

I recently attended the 2022 NASPA Student Success in Higher Education Conferences (composed of Assessment, Persistence, and Data Analytics; Dismantling Systemic Barriers to Student Success; and First-Generation Student Success conferences). It really is a quality professional development opportunity, so I highly recommend attending in the future if you have the opportunity. I presented a session on the politics and ethics of student affairs assessment. For this month’s blog, I will focus on the ethical issues.

What words or phrases come to mind when you think about ethics? You might think of values, morals, doing the right thing, integrity, rules of behavior, honesty, etc. When thinking about ethics specific for student affairs assessment, you might think about accuracy, protection, confidentiality, privacy, fairness, equity, etc. While ethics are an individual matter, they are also based in an organizational or group context. Kitchener (in Applied Ethics in Student Affairs, 1985) described five principles that should guide students affairs practice, and they can be applied to assessment specifically. Respecting autonomy allows people to choose whether to participate or not (i.e., no coercion). Doing no harm means that you would not risk hurting others. Thinking beyond physical harm, assessment harm could be about ensuring confidentiality and ensuring instruments are culturally appropriate. Benefiting others could be about the immediate benefit to participants but also providing benefits to others through decisions made from assessment results. Being just means being fair, equitable, and impartial. It could also mean providing programs and services to all students if assessment results show a benefit to students who were assessed in a program. Finally, being faithful is about being honest. Rarely would assessment involve deception, and potential participants should be notified about the purpose of the assessment (i.e., informed consent). The big takeaway is that people who are engaged in the assessment process do right by their participants, the organization, and the larger society.

National associations also have ethical standards and/or guiding principles they expect members to follow. The American Educational Research Association (AERA) has a code of ethics that address the work of educational researchers. ACPA—College Student Educators International also has a statement of ethical principles and standards for student affairs practitioners. I encourage you to look at websites of associations you belong to or that are similar to your work, in order to see the ethical and behavioral expectations.

Nearly all campuses have an Institutional Review Board that assists researchers in ethical behaviors and makes sure participants are not harmed. The U. S. Department of Health and Human Services regulates the protection of “human subjects” (i.e., participants) in the research process. Some institutions require all data collection projects to go through the review process, while others have a clear distinction between what is research and what is assessment. See my previous blog about IRBs, and always contact your institutional board if you have any questions. Even if your assessment project does not go through the IRB, it is important that you are trained on ethical principles.

I hope this helps you reflect not only on your personal ethical principles, but also how you see those principles applied in an organizational setting. If you have not already, you might reach out to colleagues to have discussions on this topic or even develop your own code of ethics. Student Affairs Planning, Assessment & Research created and continues to revisit our Standards of Ethical Practice.

Recently our staff was talking about celebrating assessment and ways to recognize the people who do it well. We are thoughtful people, though not always the rah-rah, touchy-feely people, especially when we see assessment as something that you just naturally do in your work. We put on our thinking caps and let the creativity flow. Here are a few ideas that you can incorporate into recognizing those folks who do assessment in your department.

Stress toys: For some, assessment can be anxiety-filled. Help them make it through their next project burning off that stress energy. If you have read and shared the article, “A Simple Model for Learning Improvement: Weigh Pig, Feed Pig, Weigh Pig,” from the great folks at James Madison University, you might purchase stress pigs to remind people that they need to do something more than the assessment to make change happen.

Video: You can record a short video on your phone and send it to them. Recognition doesn’t have to take long, and this would be from the heart, as well as something they could save to view again later.

Divisional recognition: If you have a division newsletter, you might be able to submit some sort of recognition for a particular project or person to thank them for the work they have done. There might be a formal recognition at the division or institutional level that you could apply for (or create your own award!).

Food: Who doesn’t appreciate something good to eat? Hopefully, you might have a list of staff’s favorites foods and know their dietary restrictions. It doesn’t have to be a full meal, but it could be cookies, snow cones, fresh fruit, etc.

Appreciation day: You can always designate a specific day to celebrate the people in your office who have done assessment in the past year. Decorate their office doors, send out an email to your staff, invite people to lunch, etc.

Thank you notes: Taking the time to write a hand-written note means a lot to some staff, especially in times of doing everything electronically.

I hope that gets you started thinking about creative ways to celebrate staff efforts toward assessment. If you need more ideas, feel free to reach out to Student Life Studies staff.

Do you see yourself as a data person? Do you think only the Student Life Studies staff and a few others are the data people in the Division of Student Affairs? Are you not quite sure of your data identity? If you checked the weather before getting dressed and going outside today, you are a data person (you know we can have four seasons in one day here!). If you step back and think about it, you consume pieces of data and information all day long, you probably have questions about students and your programs, and you think about what your program could look like in the future given your current environment.

[Read more…] about Are You a Data Person?