Katy Pearce

This is what it feels like to be teaching sometimes (made with Bing Image Creator, powered by Dall-E)

It seems that all I can think about lately is AI in the classroom. I wanted to share my summer 2023 AI/LLM policy statement for my class. Feel free to use with attribution.

Important to note:

• This is for a research methods class with very little writing, so if your class has writing in it, YMMV.
• I’m currently teaching remote and asynchronous classes, so I don’t have the same options for in-class assignments as others do.
• I have MANY other tools to discourage AI/LLM use in my courses. (Hypothesis social annotations, video reactions, etc.)
• I give FREQUENT reminders to students about this policy.
• This is a work-in-progress. I took a summer workshop on AI/LLM in the classroom that helped me refine it. I read through Aleksandra Urman‘s work on this. I am in dozens of AI/LLM in the classroom Facebook and subreddits. All of this contributed.
• It is ESSENTIAL that your policy aligns with your university policy and what your student conduct group’s policy is as well as how they react to such cases.
• AI detection software is quite flawed. A single tool for detection is insufficient as evidence of AI. And these tools are notorious for flagging non-native English writing as AI. More on this here.

Artificial Intelligence and Large Language Model Policy

We know that artificial intelligence text generators like ChatGPT and other tools like Grammarly and Quillbot are powerful tools that are increasingly used by many. And while they can be incredibly useful for some tasks (creating lists of things, for example), it is not a replacement for critical thinking and writing. Artificial intelligence text generators and editors are “large language models” – they are trained to reproduce sequences of words, not to understand or explain anything. It is algorithmic linguistics. To illustrate, if you ask ChatGPT “The first person to walk on the moon was…” it responds with Neil Armstrong. But what is really going on is that you’re asking ChatGPT “Given the statistical distribution of words in the publicly available data in English that you know, what words are most likely to follow the sequence “the first person to walk on the moon was” and ChatGPT determines that the words that are most likely to follow are “Neil Armstrong.” It is not actually thinking, just predicting.  Learning how to use artificial intelligence well is a skill that takes time to develop. Moreover, there are many drawbacks to using artificial intelligence text generators for assignments and quiz answers and proofreading and editing. 

Some of those limitations include: 

• Artificial intelligence text generators like ChatGPT are sometimes wrong. (For example, for our sampling assignment, I had ChatGPT generate lists of Pokemon that can evolve and not evolve and it was wrong for 15% of them.) If the tool gives you incorrect information and you use it on an assignment, you are held accountable for it. If the proofreading introduces terminology that is not as precise as the terminology in course materials or used differently than in course materials, you are held accountable for it.
• There is also a drawback in using artificial intelligence tools like Grammarly or Quillbot to “proofread” or “edit” your original writing – it may change your text so much that it no longer reflects your original thought or it may use terminology incorrectly. Further, in COM 382, you are not being evaluated on your writing, so there is no need to use extensive proofreading.
• The text that artificial intelligence text generators provide you is derived from another human’s original writing and likely multiple other humans’ original writing. As such, there are intellectual property and plagiarism considerations.
• Most, if not all, artificial intelligence text generators are not familiar with our textbook or my lectures and as such, will not draw from that material when generating answers. This will result in answers that will be obviously not created by someone enrolled in the course. It is likely that your assignment will not be graded as well if you’re not using course material to construct your writing. For example, AI does not understand the difference between measurement validity and study validity. AI does not understand the difference between ethics more broadly and research ethics. 
• Answers written by artificial intelligence text generators are detectable with software and we will use the software to review answers that seem unusual. We will have to be cautious in our use of such tools, but if multiple detectors find that something is likely to have been written with AI, that will be used as evidence of misconduct.

It is okay for you to use artificial intelligence text generators in this course, BUT:

• You must use them in a way that helps you learn, not hampers learning. Remember that these are tools to assist you in your coursework, not a replacement for your own learning of the material, critical thinking ability, and writing skills.
• The only acceptable use of AI on assignments (quizzes, tickets, etc.) in COM 382 is for proofreading (like Grammarly or Quillbot). This should only be for simple grammar checks, not extensive rewriting. And in COM 382 you are not being evaluated on your grammar, so we discourage this use, while acknowledging that some students want to use it.
• It is acceptable to use AI in COM 382 to provide you with other explanations of concepts or organize your notes and there is no need to disclose these. However, if the AI gives you incorrect information and you use that incorrect information on an assignment, you will be held accountable for it.
• Be transparent: If you used an AI tool for proofreading, you must include both your original writing and the AI-version so that I may see both and determine if the answer that you submitted reflects your original thought. And I expect that you will include a short paragraph at the end of the assignment or in the final 0 point question in the quiz/exam that explains what you used the artificial intelligence tool for and why. (For example: “I used Grammarly to give me feedback on my sentence structure on question 6. English is my 3rd language and I like using AI as a proofreading tool.” It is not required to disclose using AI for studying, but you can if you want to: “I read the book and listened to the lecture on measurement reliability and I didn’t fully understand it, so I asked ChatGPT to give me other examples which helped my understanding.” Or “I did not understand a term in the textbook and I asked ChatGPT to explain it to me.”)
• If you are using artificial intelligence tools to help you in this class and you’re not doing well on assignments, I expect that you will reflect upon the role that the tool may play in your class performance and consider changing your use.
• If artificial intelligence tools are used in ways that are nefarious or unacknowledged, you may be subject to the academic misconduct policies detailed earlier in the syllabus.

Although my research is primarily about technology and inequality in Armenia and Azerbaijan, I do dabble occasionally in studying games. Also all of my teaching, undergrad and grad, is on broader technology and society, so I keep up with the research.

I got into Wordle like many others did in January 2022 and I tweeted about it. A tech journalist saw me tweeting about it and contacted me for an email interview. I replied and gave some thoughts. This has turned into me being interviewed about Wordle quite a bit in the past few weeks. I’ll archive them here.

My main points:

• Wordle is really easy to pick up (no app, no login, etc.).
• Wordle is easy to get started with.
• Being forced to only play once a day on the official Wordle page is nice compared to other social media “breaks” where it is easy to get sucked in.
• It allows for a performance of being “smart” or “intellectual” by sharing results.
• During the pandemic in particular, people are really tired and don’t have a ton of bandwidth to interact with others, but sharing Wordle results allows people to be social with very little labor.
• One can feel part of a community of fellow Wordle players or part of the “in-crowd” or at least the “intellectual” crowd
• There are now clones in many languages and to me, this is getting very interesting – folks are playing in a second language, folks are playing in their heritage language.
• There are people trying to figure out the best starter word, which is fun.
• There is already backlash about sharing results, and I suspect that the sharing of results will die out soon.
• Now that the New York Times has bought Wordle, eventually they will put it behind their games paywall, which is currently $5/month. People online are annoyed about paying for it, but IMHO, NYT Games is probably the best home for it. Their existing games are really nice and Wordle fits in well. And it is nice that the inventor got paid.

What you should know: Wordle.The Hawk Newspaper. February 8, 2022.

Wordle and the future of the internet’s favorite word game. NPR’s On Point [radio interview]. February 4, 2022.

What Makes Wordle So Popular? Psychologists Explain Its Appeal. GameSpot. February 1, 2022.

Wordle. KNX In Depth [radio interview]. February 1, 2022.

Why Is Everyone Suddenly Playing Wordle? Psychologists Explain. Inc. January 30, 2022.

Wordle is a deceptively easy game for burnt-out pandemic shut-ins. Vox. January 20, 2022.

These are questions that are asked of me regarding self-funded professional masters programs in marketing and communication. I’m skipping the questions regarding how well I know the person, etc.

These questions are to be answered in addition to a full letter.

Program A’s questions

1. Please rate the applicant in the following areas:

• Motivation
• Analytical skills
• Intellectual capacity
• Communication skills
• Interpersonal skills

2. What do you consider the applicant’s strengths and/or weaknesses?

3. Describe a specific situation where you have observed the applicant using critical thinking skills or applied a new skill.

4. How would you describe the applicant’s leadership skills?

Program B’s questions

1. Please list three to five adjectives describing the applicant’s strengths.
2. Please compare the applicant’s performance to that of his or her peers.
3. What does the applicant do best?
4. If you were giving feedback to the applicant regarding his or her professional performance and personal effectiveness, in what areas would you suggest he or she work to improve?
5. How does the applicant accept constructive criticism or handle conflicts?
6. How effective are the applicant’s interpersonal skills in the workplace?
7. On the below scale, please rate the applicant’s individual vs. team orientation (1 = most effective as an individual contributor; 5 = focused exclusively on the team). Please elaborate on your rating:
8. Please give an example of how the applicant has demonstrated leadership.
9. Is there anything else you feel we should know?

Please evaluate the applicant by entering the following quality ratings for the traits below. Truly exceptional (Top 2%) Outstanding (Top 10%) Very good (Top 20%) Good (Top third) Average (Middle third) Poor (Bottom third).

• Intellectual Ability
• Maturity
• Quantitative Ability
• Analytical Skills
• Poise/Professionalism
• Initiative
• Personal integrity/ethics
• Interpersonal skills/ability to work well with others
• Sense of humor
• Verbal English Communication Skills
• Written English communication skills
• Self confidence
• Leadership ability
• Future managerial or business success
• Please provide us with your overall impression of the applicant

Program C’s questions

What are the applicant’s chief weaknesses or areas of growth

Rating 0-5

• Integrity
• Interpersonal Relations
• Oral Communications
• Self-Awareness
• Analytical Ability
• Research Ability
• Initiative
• Potential for Success in Chosen Field
• Maturity
• Self-Confidence
• Written Communication
• Overall Evaluation

Program D’s questions

Rate 0-5

• Communication and writing skills
• Intellectual curiosity, originality and independence in thinking
• Initiative
• Interpersonal skills, ability to work well with others
• Leadership ability and potential
• Academic and analytical ability
• Problem-solving ability
• Flexibility, adaptability, willingness to learn new skills
• Organizational ability
• Maturity and professionalism
• Research and reporting skills
• Integrity

Program E’s questions

Rating 0-5

• Academic performance
• Intellectual ability
• Written communication skills
• Oral communication skills, including willingness to contribute valuably to discussion/debate where applicable
• Analytical skills, including research and critical thinking skills where applicable

Word and page limits exist for a reason, but they present a challenge. These are my top tips for trimming manuscripts down.

1. Cut references

I know that this can be really difficult but these take up a lot of space. Hopefully you’re using a reference manager already, so cutting is a bit easier.

What I do is go to my references and search for the surname in the text.

I then will make a comment if it used only once. I usually add something like “But it is a meta-analysis” or “But it is in the journal that we’re submitting to” or sometimes I’ll delete it immediately. These one-offs can be really difficult. I will also ask myself if it is at all possible that some other reference that I used says the same thing or if I absolutely need to cite that paper only once.

I’ll also consider if I’m citing the same person/team multiple times in the same citation and ask myself if I have to do so. Perhaps one of the citations is the key theoretical piece, so I can’t avoid it. But if it is a smaller finding that they had in both a 2012 and a 2016 paper and perhaps the 2016 paper is in a better venue or is more widely cited, I’ll cut the 2012 reference.

Similarly with the same person, I’ll do a scan for them throughout the paper. For example, if in the entire paper I cited B, T, and R 2012 and T and B 2009 and R, T, and B 2016 each 4 times, but I only cited R, B, and T 2015 once. I’ll re-skim the 2015 paper and ask myself if I absolutely need to include it or if the finding was in one of the other papers as well.

Thinking about the venue is important too. For example, let’s say I’m working on a paper about walruses playing board games with an outcome of better walrus solidarity. And I’m submitting this paper to a journal that is really focused on board game playing and less on solidarity or walruses. While I cannot remove all of the citations that tie back to walrus or solidarity literature, I should prioritize the board game playing literature as that is the journal’s audience and reviewers will come from that field. But I do always have older versions of the paper that have all of the references in it just in case the reviewers ask why there isn’t more theorizing and literature from Walrus Studies.

Finally, the most heartbreaking reference cuts are studies with too many authors and/or really long titles. This is presuming that references count towards the word count.

2. Cut words

Do searches for common adverbs. Delete transitional words (this is painful for me).

3. Merge words

Sometimes hyphens are appropriate and can save space.

4. Wordiness

You’re probably being too wordy. Try to read the paragraph out loud to yourself or have your computer read it to you. It can be easier to hear the problems. I also sometimes do better when I’m editing a printed version of the paper versus on a screen. I think that taking a break from the manuscript also helps.

5. Text -> Table

Sometimes you can turn text into tables. This will reduce words and if the journal doesn’t count tables toward the word count, save you a ton of words. However, if page count is the issue, tables sometimes can be longer than text.

6. If qualitative/interview based, look for redundant participant quotes

It is easy to fall in love with a great direct quote or example from a participant. Sometimes they’re just so delicious and represent the theme so well. But I find that a lot of people will have 3 or even more examples for a particular theme. And that is okay, but one must also remember that some of them have to be cut eventually.

Ask yourself if the quote is absolutely necessary to illustrate the theme or is so perfect that it really sells the theme in a sincere way. Look at each quote in comparison with every other quote within the theme and ask yourself if they both need to be there.

Also keep a table of quote/example counts by participant. Sometimes some participants are chattier or articulate themselves better and we lean more heavily on their quotes. It is important to have a count at the end so you don’t accidentally have many more quotes from a few participants. This is not to say to purposefully manipulate your quote choices or include some participants artificially. Rather, seeing that you’re already quite heavy with quotes from “Alice” can help you make decisions between two quotes more easily.

7. Redundancy in the findings section

Sometimes we have a fairly complicated framework and we need to remind our reader what the conceptual definition of a particular theme was. However, this does take up a lot of space. This is another one that is painful for me.

8. Could this be two papers?

Sometimes there is so much going on that you can split the manuscript into two papers. Honestly, this happens to me about 75% of the time. This does require work to ensure that the theoretical scaffolding is different and that you’re not reusing findings. However, in some cases it might make sense to divide.

9. Have an editing partner

Certainly if there are multiple authors in a study, someone else can look at the manuscript. But you may also be able to have a friend with whom you trade editing/trimming tasks.

10. Check your conclusion/discussion

Sometimes we get a little bit freewheeling at the end of the manuscript. This can be a place where entire sections could be removed.

I received tenure in September 2018. If you see something out on the web that lists me as assistant professor, please let me know.