21 Aug

AI/LLM policy statements

This is what it feels like to be teaching sometimes (made with Bing Image Creator, powered by Dall-E)

Me, teaching to a room of robots

It seems that all I can think about lately is AI in the classroom. I wanted to share my summer 2023 AI/LLM policy statement for my class. Feel free to use with attribution.

Important to note:

  • This is for a research methods class with very little writing, so if your class has writing in it, YMMV.
  • I’m currently teaching remote and asynchronous classes, so I don’t have the same options for in-class assignments as others do.
  • I have MANY other tools to discourage AI/LLM use in my courses. (Hypothesis social annotations, video reactions, etc.)
  • I give FREQUENT reminders to students about this policy.
  • This is a work-in-progress. I took a summer workshop on AI/LLM in the classroom that helped me refine it. I read through Aleksandra Urman‘s work on this. I am in dozens of AI/LLM in the classroom Facebook and subreddits. All of this contributed.
  • It is ESSENTIAL that your policy aligns with your university policy and what your student conduct group’s policy is as well as how they react to such cases.
  • AI detection software is quite flawed. A single tool for detection is insufficient as evidence of AI. And these tools are notorious for flagging non-native English writing as AI. More on this here.

Artificial Intelligence and Large Language Model Policy

We know that artificial intelligence text generators like ChatGPT and other tools like Grammarly and Quillbot are powerful tools that are increasingly used by many. And while they can be incredibly useful for some tasks (creating lists of things, for example), it is not a replacement for critical thinking and writing. Artificial intelligence text generators and editors are “large language models” – they are trained to reproduce sequences of words, not to understand or explain anything. It is algorithmic linguistics. To illustrate, if you ask ChatGPT “The first person to walk on the moon was…” it responds with Neil Armstrong. But what is really going on is that you’re asking ChatGPT “Given the statistical distribution of words in the publicly available data in English that you know, what words are most likely to follow the sequence “the first person to walk on the moon was” and ChatGPT determines that the words that are most likely to follow are “Neil Armstrong.” It is not actually thinking, just predicting.  Learning how to use artificial intelligence well is a skill that takes time to develop. Moreover, there are many drawbacks to using artificial intelligence text generators for assignments and quiz answers and proofreading and editing. 

Some of those limitations include: 

  • Artificial intelligence text generators like ChatGPT are sometimes wrong. (For example, for our sampling assignment, I had ChatGPT generate lists of Pokemon that can evolve and not evolve and it was wrong for 15% of them.) If the tool gives you incorrect information and you use it on an assignment, you are held accountable for it. If the proofreading introduces terminology that is not as precise as the terminology in course materials or used differently than in course materials, you are held accountable for it.
  • There is also a drawback in using artificial intelligence tools like Grammarly or Quillbot to “proofread” or “edit” your original writing – it may change your text so much that it no longer reflects your original thought or it may use terminology incorrectly. Further, in COM 382, you are not being evaluated on your writing, so there is no need to use extensive proofreading.
  • The text that artificial intelligence text generators provide you is derived from another human’s original writing and likely multiple other humans’ original writing. As such, there are intellectual property and plagiarism considerations.
  • Most, if not all, artificial intelligence text generators are not familiar with our textbook or my lectures and as such, will not draw from that material when generating answers. This will result in answers that will be obviously not created by someone enrolled in the course. It is likely that your assignment will not be graded as well if you’re not using course material to construct your writing. For example, AI does not understand the difference between measurement validity and study validity. AI does not understand the difference between ethics more broadly and research ethics. 
  • Answers written by artificial intelligence text generators are detectable with software and we will use the software to review answers that seem unusual. We will have to be cautious in our use of such tools, but if multiple detectors find that something is likely to have been written with AI, that will be used as evidence of misconduct.

It is okay for you to use artificial intelligence text generators in this course, BUT:

  • You must use them in a way that helps you learn, not hampers learning. Remember that these are tools to assist you in your coursework, not a replacement for your own learning of the material, critical thinking ability, and writing skills.
  • The only acceptable use of AI on assignments (quizzes, tickets, etc.) in COM 382 is for proofreading (like Grammarly or Quillbot). This should only be for simple grammar checks, not extensive rewriting. And in COM 382 you are not being evaluated on your grammar, so we discourage this use, while acknowledging that some students want to use it.
  • It is acceptable to use AI in COM 382 to provide you with other explanations of concepts or organize your notes and there is no need to disclose these. However, if the AI gives you incorrect information and you use that incorrect information on an assignment, you will be held accountable for it.
  • Be transparent: If you used an AI tool for proofreading, you must include both your original writing and the AI-version so that I may see both and determine if the answer that you submitted reflects your original thought. And I expect that you will include a short paragraph at the end of the assignment or in the final 0 point question in the quiz/exam that explains what you used the artificial intelligence tool for and why. (For example: “I used Grammarly to give me feedback on my sentence structure on question 6. English is my 3rd language and I like using AI as a proofreading tool.” It is not required to disclose using AI for studying, but you can if you want to: “I read the book and listened to the lecture on measurement reliability and I didn’t fully understand it, so I asked ChatGPT to give me other examples which helped my understanding.” Or “I did not understand a term in the textbook and I asked ChatGPT to explain it to me.”)
  • If you are using artificial intelligence tools to help you in this class and you’re not doing well on assignments, I expect that you will reflect upon the role that the tool may play in your class performance and consider changing your use.
  • If artificial intelligence tools are used in ways that are nefarious or unacknowledged, you may be subject to the academic misconduct policies detailed earlier in the syllabus.

02 Feb

Wordle me this

Although my research is primarily about technology and inequality in Armenia and Azerbaijan, I do dabble occasionally in studying games. Also all of my teaching, undergrad and grad, is on broader technology and society, so I keep up with the research.

I got into Wordle like many others did in January 2022 and I tweeted about it. A tech journalist saw me tweeting about it and contacted me for an email interview. I replied and gave some thoughts. This has turned into me being interviewed about Wordle quite a bit in the past few weeks. I’ll archive them here.

My main points:

  • Wordle is really easy to pick up (no app, no login, etc.).
  • Wordle is easy to get started with.
  • Being forced to only play once a day on the official Wordle page is nice compared to other social media “breaks” where it is easy to get sucked in.
  • It allows for a performance of being “smart” or “intellectual” by sharing results.
  • During the pandemic in particular, people are really tired and don’t have a ton of bandwidth to interact with others, but sharing Wordle results allows people to be social with very little labor.
  • One can feel part of a community of fellow Wordle players or part of the “in-crowd” or at least the “intellectual” crowd
  • There are now clones in many languages and to me, this is getting very interesting – folks are playing in a second language, folks are playing in their heritage language.
  • There are people trying to figure out the best starter word, which is fun.
  • There is already backlash about sharing results, and I suspect that the sharing of results will die out soon.
  • Now that the New York Times has bought Wordle, eventually they will put it behind their games paywall, which is currently $5/month. People online are annoyed about paying for it, but IMHO, NYT Games is probably the best home for it. Their existing games are really nice and Wordle fits in well. And it is nice that the inventor got paid.

What you should know: Wordle.The Hawk Newspaper. February 8, 2022.

Wordle and the future of the internet’s favorite word game. NPR’s On Point [radio interview]. February 4, 2022.

What Makes Wordle So Popular? Psychologists Explain Its Appeal. GameSpot. February 1, 2022.

Wordle. KNX In Depth [radio interview]. February 1, 2022.

Why Is Everyone Suddenly Playing Wordle? Psychologists Explain. Inc. January 30, 2022.

Wordle is a deceptively easy game for burnt-out pandemic shut-ins. Vox. January 20, 2022.

15 Dec

Questions I’m asked as a recommendation writer

These are questions that are asked of me regarding self-funded professional masters programs in marketing and communication. I’m skipping the questions regarding how well I know the person, etc.

These questions are to be answered in addition to a full letter.

Program A’s questions

  1. Please rate the applicant in the following areas:
  • Motivation
  • Analytical skills
  • Intellectual capacity
  • Communication skills
  • Interpersonal skills

2. What do you consider the applicant’s strengths and/or weaknesses?

3. Describe a specific situation where you have observed the applicant using critical thinking skills or applied a new skill.

4. How would you describe the applicant’s leadership skills?

Program B’s questions

  1. Please list three to five adjectives describing the applicant’s strengths.
  2. Please compare the applicant’s performance to that of his or her peers.
  3. What does the applicant do best?
  4. If you were giving feedback to the applicant regarding his or her professional performance and personal effectiveness, in what areas would you suggest he or she work to improve?
  5. How does the applicant accept constructive criticism or handle conflicts?
  6. How effective are the applicant’s interpersonal skills in the workplace?
  7. On the below scale, please rate the applicant’s individual vs. team orientation (1 = most effective as an individual contributor; 5 = focused exclusively on the team). Please elaborate on your rating:
  8. Please give an example of how the applicant has demonstrated leadership.
  9. Is there anything else you feel we should know?

Please evaluate the applicant by entering the following quality ratings for the traits below. Truly exceptional (Top 2%) Outstanding (Top 10%) Very good (Top 20%) Good (Top third) Average (Middle third) Poor (Bottom third).

  • Intellectual Ability
  • Maturity
  • Quantitative Ability
  • Analytical Skills
  • Poise/Professionalism
  • Initiative
  • Personal integrity/ethics
  • Interpersonal skills/ability to work well with others
  • Sense of humor
  • Verbal English Communication Skills
  • Written English communication skills
  • Self confidence
  • Leadership ability
  • Future managerial or business success
  • Please provide us with your overall impression of the applicant

Program C’s questions

What are the applicant’s chief weaknesses or areas of growth

Rating 0-5

  • Integrity
  • Interpersonal Relations
  • Oral Communications
  • Self-Awareness
  • Analytical Ability
  • Research Ability
  • Initiative
  • Potential for Success in Chosen Field
  • Maturity
  • Self-Confidence
  • Written Communication
  • Overall Evaluation

Program D’s questions

Rate 0-5

  • Communication and writing skills
  • Intellectual curiosity, originality and independence in thinking
  • Initiative
  • Interpersonal skills, ability to work well with others
  • Leadership ability and potential
  • Academic and analytical ability
  • Problem-solving ability
  • Flexibility, adaptability, willingness to learn new skills
  • Organizational ability
  • Maturity and professionalism
  • Research and reporting skills
  • Integrity

Program E’s questions

Rating 0-5

  • Academic performance
  • Intellectual ability
  • Written communication skills
  • Oral communication skills, including willingness to contribute valuably to discussion/debate where applicable
  • Analytical skills, including research and critical thinking skills where applicable
20 Aug

Manuscript trimming tips

Word and page limits exist for a reason, but they present a challenge. These are my top tips for trimming manuscripts down.

  1. Cut references

I know that this can be really difficult but these take up a lot of space. Hopefully you’re using a reference manager already, so cutting is a bit easier.

What I do is go to my references and search for the surname in the text.

I then will make a comment if it used only once. I usually add something like “But it is a meta-analysis” or “But it is in the journal that we’re submitting to” or sometimes I’ll delete it immediately. These one-offs can be really difficult. I will also ask myself if it is at all possible that some other reference that I used says the same thing or if I absolutely need to cite that paper only once.

I’ll also consider if I’m citing the same person/team multiple times in the same citation and ask myself if I have to do so. Perhaps one of the citations is the key theoretical piece, so I can’t avoid it. But if it is a smaller finding that they had in both a 2012 and a 2016 paper and perhaps the 2016 paper is in a better venue or is more widely cited, I’ll cut the 2012 reference.

Similarly with the same person, I’ll do a scan for them throughout the paper. For example, if in the entire paper I cited B, T, and R 2012 and T and B 2009 and R, T, and B 2016 each 4 times, but I only cited R, B, and T 2015 once. I’ll re-skim the 2015 paper and ask myself if I absolutely need to include it or if the finding was in one of the other papers as well.

Thinking about the venue is important too. For example, let’s say I’m working on a paper about walruses playing board games with an outcome of better walrus solidarity. And I’m submitting this paper to a journal that is really focused on board game playing and less on solidarity or walruses. While I cannot remove all of the citations that tie back to walrus or solidarity literature, I should prioritize the board game playing literature as that is the journal’s audience and reviewers will come from that field. But I do always have older versions of the paper that have all of the references in it just in case the reviewers ask why there isn’t more theorizing and literature from Walrus Studies.

Finally, the most heartbreaking reference cuts are studies with too many authors and/or really long titles. This is presuming that references count towards the word count.

2. Cut words

Do searches for common adverbs. Delete transitional words (this is painful for me).

3. Merge words

Sometimes hyphens are appropriate and can save space.

4. Wordiness

You’re probably being too wordy. Try to read the paragraph out loud to yourself or have your computer read it to you. It can be easier to hear the problems. I also sometimes do better when I’m editing a printed version of the paper versus on a screen. I think that taking a break from the manuscript also helps.

5. Text -> Table

Sometimes you can turn text into tables. This will reduce words and if the journal doesn’t count tables toward the word count, save you a ton of words. However, if page count is the issue, tables sometimes can be longer than text.

6. If qualitative/interview based, look for redundant participant quotes

It is easy to fall in love with a great direct quote or example from a participant. Sometimes they’re just so delicious and represent the theme so well. But I find that a lot of people will have 3 or even more examples for a particular theme. And that is okay, but one must also remember that some of them have to be cut eventually.

Ask yourself if the quote is absolutely necessary to illustrate the theme or is so perfect that it really sells the theme in a sincere way. Look at each quote in comparison with every other quote within the theme and ask yourself if they both need to be there.

Also keep a table of quote/example counts by participant. Sometimes some participants are chattier or articulate themselves better and we lean more heavily on their quotes. It is important to have a count at the end so you don’t accidentally have many more quotes from a few participants. This is not to say to purposefully manipulate your quote choices or include some participants artificially. Rather, seeing that you’re already quite heavy with quotes from “Alice” can help you make decisions between two quotes more easily.

7. Redundancy in the findings section

Sometimes we have a fairly complicated framework and we need to remind our reader what the conceptual definition of a particular theme was. However, this does take up a lot of space. This is another one that is painful for me.

8. Could this be two papers?

Sometimes there is so much going on that you can split the manuscript into two papers. Honestly, this happens to me about 75% of the time. This does require work to ensure that the theoretical scaffolding is different and that you’re not reusing findings. However, in some cases it might make sense to divide.

9. Have an editing partner

Certainly if there are multiple authors in a study, someone else can look at the manuscript. But you may also be able to have a friend with whom you trade editing/trimming tasks.

10. Check your conclusion/discussion

Sometimes we get a little bit freewheeling at the end of the manuscript. This can be a place where entire sections could be removed.

25 Mar

Facebook in Armenia, March 2020

It has been awhile since I last blogged about Facebook use in the Caucasus. Again, here is a guide to how I get these data. Click on the tags for previous rates – here’s September 2017, here is May 2018. Here is December 2018.

As of March 2020, there are about 1,500,000 Facebook users in Armenia, according to Facebook. That is 50% of the total population, and 46% of the population over age 14 (Facebook technically isn’t available to those under 13.) There is a bit of growth since December when Facebook ads estimated 1,400,000 users (47% of the population).

As far as gender, 50% of the total male population, or 64% of males over age 14 are on Facebook. 53% of the total female population, or 49% of the female 14+ population. So there are some gender differences, but probably within the margin of error.

Just looking at the 15-24 year olds, 82% of them are on the site (this is a drop from 2 years ago — I suspect people have moved to Instagram), 82% of young men and 87% of young women.

Some trends to look for – as in the rest of the world, young people are moving more toward Instagram. Everyone is moving toward WhatsApp and other private messaging services.

25 Mar

2017 Internet access in Armenia

With the move toward online learning all over the world, someone asked me about Internet access in Armenia. The most recent publicly available data that we have is from the 2017 Caucasus Barometer. Here are a few relevant statistics. Please note that the survey respondents are a mix of household answers (like owning a computer) and individual answers (frequency of Internet use), so we are making some methodological leaps in claiming that this would reflect the access that young people would have.

It is difficult to ascertain the presence of children in the household using the Caucasus Barometer online analysis tool. When I have some time I will create a variable that subtracts the number of adults in the household from the total number of household residents to create a new variable called “Number of children” — I’d like to use that to look at all of the analyses described below in the future.

Another caveat – in homes across the world, adults working from home are having to share their technology with their children. There may also be greater demands on home Internet access and adult work use becomes prioritized over children’s.

Analysis

Overall, 29% of Armenia adults never access the Internet. This has remained fairly stable for the past few years. The 2017 CB did not ask about why people did not use the Internet, but in previous years, the answered varied and were not all tied to resource access issues.

Home Internet access is far more available in the capital city, although mobile Internet (and nearly two-thirds of Armenians have mobile Internet) certainly bridges that gap for many households.

Nonetheless, in 2017, over a third of rural respondents never accessed the Internet.

Mobile Internet does vary a bit by urbanness. Two-thirds or nearly two-thirds of Yerevan residents and regional urban center residents have mobile Internet, while a little over half of rural residents do.

While mobile phones have come a long way (and nearly all Armenians have owned a mobile phone for over a decade), no one can deny that some activities are conducted more easily using a personal computer. As of 2017 58% of households had a computer. This does not mean that people do not have access to computers at cafes, work, or school. However, in terms of considering distance learning or working from home, the lack of a computer may be a barrier for some.

Urbanness has always been an important part of the Armenian digital divide story. Personal computer ownership is far higher in the capital (67%) than in regional urban environments (56%) or rural areas (51%).

11 Dec

Facebook in Azerbaijan, December 2019

It has been quite awhile since I last blogged about Facebook use in the Caucasus. Again, here is a guide to how I get these data. Click on the tags for previous rates.

According to Facebook, as of December 2019, around 3,300,000 Azerbaijanis, about 35% of the total population, or more accurately, 32% of the population over age 14, are on Facebook.

Over half of all Azerbaijani men (over age 14) are on Facebook (well, 61%) and 29% of Azerbaijani women (over age 14) are on Facebook. This has been the trend for as long as I’ve been tracking this.

Looking at just youth, about 44% of Azerbaijanis ages 15-24 use Facebook (this is a drop from last year!). 60% of males that age and 27% of females that age.

As always, these numbers are to be taken with a grain of salt. This is information from Facebook ads.

12 Apr

Social media and bullying in Azerbaijan

This week a young woman in Azerbaijan took her own life as a result of possible bullying. Video of the act and subsequent events were widely shared and discussed on social media. As a result, many Azerbaijanis are discussing the problem of bullying in schools. Here’s a summary of the case in English.

Here’s a NodeXL hashtag analysis of #BullinqəSon (End Bullying)

Here’s a NodeXL hashtag analysis of #Elinaüçünsusma (Don’t be silent about Elina) — this is more popular

Some caveats here regarding any sort of hashtag analysis:

  •  Twitter data downloads like this are always incomplete, as it is impossible to get the full dataset.
  • The results are a little skewed because a lot of the users are tweeting at the President and First Lady to do something. Obviously they have a lot of followers, so a lot of these “importance” metrics are impacted by that.
  • Twitter isn’t a great venue to consider Azerbaijani public discussion of such topics, in particular this one that is of great interest to young people. I’ve seen far more on Facebook, Instagram, TikTok, etc. I can only imagine WhatsApp has a great deal of this as well.
  • Social media are always performative. The need to let one’s audience know that they care about this issue is not always the same as discussion about solutions.
  • Social media “influencers” sometimes feel compelled to have a hot take on the topic of the day and sometimes they’ll say provocative things because it leads to more engagement. There have been a few “celebrities” in Azerbaijan, especially on Instagram, doing this. I wouldn’t take this as the complete story.

And some general thoughts on social media campaigns, with a bullying angle. I’m not a bullying expert, but I’ve talked to people that are and these thoughts are influenced by that. (Thanks to them! Especially Lindsay Blackwell and her excellent work on cyberbullying.)

  • For campaigns to be taken seriously by young people, they need to feel sincere. I’m under the impression that some of these campaigns have been started by people that young people in Azerbaijan may not follow and that may not be entirely relatable. Decades of research shows that adult-created campaigns aimed at youth frequently fail. I worked on an environmental campaign aimed at middle school and high school students and I cannot even begin to tell you how much money was spent on materials that the young people laughed at. We ran focus groups to see what sort of messages resonated with the young people and where they were most likely to be influenced (both media channels and with peers) and it was entirely different from what the campaign organizers had done.
  • Campaigns need to be multifaceted, especially when there is a goal of behavioral change. There is currently criticism that schools are securing windows (the young woman jumped out of a window), but at the same time, such an act is probably a good idea and can be done immediately. It does not mean that the school is not working on other strategies and actions to reduce bullying and its effects.
  • Who is the target of the campaign? Is it bullies asking them to not bully other children? Is it those that are the victims of bullying telling them to be strong? Or is it potential bystanders, asking them to intervene? The messages will be different!
    • With bystanders in particular, which is the hypothetical largest audience, in any campaign, people need to be told what to do. For example, in the US, there was a campaign about forest fires that said “Only you can prevent forest fires” but the campaign did not actually tell people how to prevent forest fires! So, in the bullying case, potential bystanders need instructions about what to do if they see a classmate being bullied. (And at least at my own child’s school, there is an entire curriculum about this from the very early years.)
    • Victims need different messaging about what they can do as well as a sense that there are others out there experiencing bullying. According to my expert colleagues, bullying feels very isolating and it is hard to see that there are others in the same position. So campaigns whereby people disclose that they were bullied and what they did about it can be helpful.
    • The bullies themselves are also children and conventional wisdom says that children that are bullies are not infrequently subject to problems at home, have mental health issues, etc. – they also need help.
    • Schools and parents also need help! Law enforcement too!
  • For campaigns to be effective with young people, the message needs to also be relatable and from an authentic figure. For example, a gorgeous 30-year-old actress saying that she was bullied in school may not be believable in the eyes of a 13-year-old who sees that actress leading a glamorous life and looking beautiful. We see that a bit with the “It Gets Better” type campaigns with celebrity focuses versus “#metoo” whereby the majority of the messaging comes from “regular” people.
  • Figuring out who is influential among young people in Azerbaijan (and on what platform) but still relatable and if that person did in fact experience bullying (and it is believable that they did) or perhaps they were an intervening bystander would be a very good tactic. In fact, in the words of one of my bullying expert colleagues, that would be infinitely more helpful than a Ministry of Education campaign that takes months or years to design.
  • Decades of research shows that suicide has a potential copycat effect. It is incredibly irresponsible for media outlets (and individual users) to share the video where this young woman takes the action.
  • In Azerbaijan there are laws related to suicide and it is a criminal offense to “cause” someone to commit suicide. I have to admit that this seems strange to me as an American. Certainly it is not possible to demonstrate this beyond reasonable doubt and the mental health issues that those considering suicide are facing are numerous. There is a great deal of social media speculation about this young woman’s family, the role that the school administration played or didn’t play in her death, the young woman’s romantic and sexual life, etc. In my opinion, such discussions do little to help anyone – those grieving or those trying to reducing bullying and its effects.

As a takeaway, although tragic, as a result of this young woman’s death, the very real problem of bullying in schools is now being discussed more widely in Azerbaijan and that is a good thing. Those that want to try to help – both immediately and in the long term – would be well-advised to look at the existing work on anti-bullying campaigns before jumping in.

04 Dec

Facebook in Armenia, December 2018

fb

It has been awhile since I last blogged about Facebook use in the Caucasus. Again, here is a guide to how I get these data. Click on the tags for previous rates – here’s September 2017, here is May 2018.

As of December 2018, there are about 1,400,000 Facebook users in Armenia, according to Facebook. That is 47% of the total population, and 43% of the population over age 14 (Facebook technically isn’t available to those under 13.) There is a bit of growth since May 2018 when Facebook ads estimated 1,200,000 users (40% of the population), but I would be cautious to attribute this to the Velvet Revolution. About 100,000 new Armenians join Facebook each quarter or so anyway.

As far as gender, 44% of the total male population, or 57% of males over age 14 are on Facebook. 48% of the total female population, or 49% of the female 14+ population. So there are some gender differences, but probably within the margin of error.

Just looking at the 15-24 year olds, 87% of them are on the site, 86% of young men and 92% of young women. These numbers have been fairly steady for a few years, with many young women on the site for awhile. 2016 seemed to be the big leap here.

Some trends to look for – as in the rest of the world, young people are moving more toward Instagram. Everyone is moving toward WhatsApp and other private messaging services.

22 Oct

Associate!

I received tenure in September 2018. If you see something out on the web that lists me as assistant professor, please let me know.