Update: Will AI put me out of a job? Experiments with OpenAI’s GPT-3

2/13/2023: A friend who read this post responded: “I’m having a hard time jumping from ‘this seems adequate… noticed some problems’ to a grade of ‘F’.”

This comment made me realized that I had not emphasized a key word:

seems

Superficially, GPT-3’s responses appeared good enough. But they weren’t. That’s the worst of both worlds: if they were blatantly inadequate, writers and content developers wouldn’t have anything to worry about, at least for now.

Ever since I read about a college student using GPT-3 to create a blog that went viral, I have been wondering about the possible effects of AI on my ability to make a living as a writer and editor. OpenAI’s recent release of ChatGPT gave me the chance to go straight to the source with my question:

Screen capture of chat. AI gives error message when asked if GPT will make writing profession obsolete.

I decided I needed to do some experiments.

I used OpenAI’s Playground for most of my experiments. Like ChatGPT, the Playground uses the GPT-3 language model. However, ChatGPT is optimized for dialogue, while the Playground is set up to be used for a range of possible applications.

Currently, most of my writing is instructional (I’m an instructional designer at Harvard Online), so I focused on writing for teaching and learning. I’ll share my first experiments here, and more in upcoming posts.

Experiments in Quantum Mechanics

I’m working on a quantum mechanics unit for a first-year chemistry course, so I chose that as the topic for my first tests of GPT-3’s abilities.

Experiment 1: Write an explanation

Method and Results

Me: Explain Hertz's discovery of the photoelectric effect

OpenAI Playground: 

Hertz's discovery of the photoelectric effect was an important milestone in the development of quantum mechanics. He observed that when ultraviolet light was shone onto the surface of certain metals, electrons were emitted from the surface. This effect only occurred when the light was of a certain frequency, and Hertz observed that the number of electrons emitted was proportional to the intensity of the light. He also found that the kinetic energy of the electrons was dependent on the frequency of the light, but was independent of the intensity. This was the first observation of the photoelectric effect, and it showed that light behaves as both a particle and a wave.

Discussion

On a first pass, this seems adequate. But on a closer read, I noticed some problems.

Hertz's discovery of the photoelectric effect was an important milestone in the development of quantum mechanics. He observed that when ultraviolet light was shone onto the surface of certain metals, electrons were emitted from the surface. This effect only occurred when the light was of a certain frequency, and Hertz observed that the number of electrons emitted was proportional to the intensity of the light. He also found that the kinetic energy of the electrons was dependent on the frequency of the light, but was independent of the intensity. This was the first observation of the photoelectric effect, and it showed that light behaves as both a particle and a wave.
  • “a certain frequency…” This explanation glosses over the concept of threshold frequency. The word “certain” is ambiguous and, in this context, highly susceptible to misinterpretation. “a certain frequency” could be interpreted to mean a single specific frequency, which would be incorrect.
  • “This was the first observation…” “This” is probably referring to all of the observations, or possibly to “Hertz’s discovery”. (Colloquially, “discovery” and “observation” might be used synonymously, but in science education materials, the two are not equivalent.) When a writer uses “this” to refer to something other than the immediately preceding information, the reader can’t be sure what the writer meant unless they already know the content. In GPT-3’s explanation, there’s no way for a reader to be certain what was the first observation of the photoelectric effect.
  • “it showed that light behaves as both…” I’d want to check with a content expert, but I think Hertz’s experiment showed the particle properties of light, not both wave and particle properties.

But the biggest flaw is the complete lack of a connection between the content of the paragraph and the final sentence. How do those observations support the world-shaking conclusion that light can behave like a particle?

Grade: F

Experiment 2: Write a multiple choice item

Method and Results

Me: Write a multiple choice question about Hertz's discovery of the photoelectric effect

OpenAI Playground:

Which of the following was NOT a result of Hertz's discovery of the photoelectric effect?

A. Creation of a new type of electric current
B. A new understanding of the behavior of light
C. Increased electricity costs
D. Development of radio communication

Discussion

This is a trivial question that doesn’t test learner understanding of the important concepts.

The question stem also breaks an important rule of question-writing: Avoid using negatives, such as “NOT”, unless their use is essential to the concept you are assessing.

Grade: F

Job prospects

GPT-3 produces results that would appear good enough to a non-expert (in chemistry, education, or writing), but they aren’t good enough. I think that’s the worst possible news. But it’s possible that if I give more precise instructions, I’ll get better results. So tune in next time to see how that turns out.

%d bloggers like this: