Why Did AI First Come for Creative Jobs?

“Five or seven years ago, the kind of ironclad wisdom on AI was that it first comes for physical labor (e.g., working in the factory and truck driving), then this sort of less demanding cognitive labor; and then the really demanding cognitive labor like computer programming, and then very last of all, and maybe never – because maybe it is some deep special human sauce – was creativity. And of course we can look now and say it really looks like it is going to go exactly the opposite direction.” – Sam Altman in an interview in January 2023

A lot has been said about Generative AI since that interview and its impact on society, the future of work, and in particular the future of creative work. Many, including Sam Altman, have pointed out that these tools are here to assist the human professionals, rather than replacing them. They can amplify you, and unblock your creativity, and can have a transformative impact on the creative industry, and other jobs.

Obviously, this is not the first time that AI has achieved superhuman results. Earlier we saw AlphaGo mastering the game of Go, AlphaFold solving the protein folding problem, and simpler AI-powered systems before them assisting professionals in medicine, face recognition, and other domains. What is new is that transformer-based general-purpose generative models, which are closer to general AI, have now reached unexpected success in creative tasks.

This unforeseen success raises some questions. The one that I have been thinking about is the question of complexity. Are creative tasks more complex (e.g., writing) than routine mechanical tasks (e.g., truck driving, or working in assembly line)? Has AI now achieved a point where it can perform fundamentally more complex tasks?

Many mechanical tasks can be very complex. In particular, when precision and accuracy is required to correctly complete each step of the work. Consider driving for instance. There are many situational constraints and changing conditions that should be taken into account when driving a car.

When it comes to creative work, it is difficult to talk about its complexity, as creativity spans a wide range of tasks and activities. Nevertheless, creativity feels complex because we know very little about how it works. One aspect of creativity is novelty: to piece together new things, and in new ways. However, novelty is considered creative only if it results in something useful. The authors in 1 state that: “We distinguish creative ideas from bizarre ideas, which are also novel, because creative ideas take into account the parameters of a situation, the constraints. Novel productions that are in some way “useful” or “worthwhile” in a given context are, thus, creative.”

When it comes to creative content generated by AI, people often mean textual or visual content generated by the AI system in response to a prompt provided by the user. Luckily, for most of this content, there is no clear line between right or wrong, and valid or invalid. For textual content for instance, the main constraints that apply, and can rule out the usefulness of the generated content, are grammatical rules and a degree of relevance to the prompt. Consider instead generating a novel, perhaps more succinct, solution to a math problem. This one is a lot more complex, as satisfying the constraints of the problem is more difficult. You know exactly what is right and what is wrong. The space of possible solutions is much slimmer.

This lack of a clear line between right and wrong, is, in my opinion, the first factor that contributes to the success of AI in creative tasks. It significantly reduces the complexity of the task.

The second factor is that the usefulness and worth of creative and artistic content is a matter of taste and subjective opinion. Creative contents are greatly open to interpretation and their aesthetic appeal changes from one person to another. The human brain, as the consumer of the creative content, fills in the gaps and participates in assigning meaning to the creative content and making it worthwhile. This, which I call the smart recipient effect, is the second contributor to the early success of AI in creative tasks.

In the following, I’ll elaborate on these two factors! But the short answer to my two questions above is that (1) the creative tasks that we are talking about in the context of generative AI are not that complex, and (2) it does not seem like these large models can solve dramatically more complex problems for us.

No right or wrong

When it comes to artistic content, there is often no right or wrong. You don’t care about completeness, accuracy, or precision. The artist can violate certain constraints and rules, and still produce a worthwhile piece of art, maybe even introduce a new form of art!

The value of a piece of art is in the idea behind it and the feeling it provokes in you. In this case, AI is a great assistant. It has not replaced human producers, but is a great aid to amplify their abilities. It is a very good tool for expanding your imagination, and presenting you with new ideas.

Consider a paragraph generated by ChatGPT. What makes it worthwhile or useful? Certainly relevance to the prompt, good grammar, a coherent flow of thoughts, and factfullness. Apart from good grammar, the other criteria are not very well defined or easy to measure. For factfulness in particular, unless you are an expert on the topic; you cannot say how factual the generated text is. Often if the text is well-organized and is relevant to the topic at a high level, you are happy with it.

How about images generated by AI? What makes them worthwhile? You may say relevance to the prompt, good ratios and perspectives, lighting effects, clear shape boundaries, etc. In my experience, as soon as you have an unusual prompt, the generated image fails in all these aspects. Nevertheless, for all intents and purposes, a creative content producer might find the AI-generated image exactly what he or she has been looking for.

Here is a little experiment, from 22 April 2023, where I asked ChatGPT to generate a prompt for DALL·E 2:

Me: DALLE is an AI system that generates images from textual prompts. Produce a prompt that can be used as input to DALLE.

ChatGPT response:

Here's a prompt that can be used as input to DALLE:
"Imagine a peaceful countryside landscape with rolling hills, tall trees, and a
river flowing gently through the valley. The sun is setting in the distance,
casting a warm orange glow across the scene. A small cabin sits on the edge of
the river, smoke gently rising from the chimney. In front of the cabin, a wooden
dock extends out into the water, and a rowboat bobs gently in the current. A
family of ducks swims lazily by, enjoying the tranquility of the evening."

I had no idea you could provide such long inputs to DALL·E 2! Honestly, I could never come up with such a beautiful description of a scene. But there is nothing particularly novel about the scene described in the prompt. Anyway, I gave it to DALL·E 2, and here are the four images it produced:

The images look a lot better than any images I ever got from DALL·E 2 with my prompts! However, there are a lot of details in the prompt that are not present in the generated images. I don’t see smoke from a chimney, nor a bobbing boat, nor a family of ducks in any of the images. It is difficult to say which one of these photos ChatGPT had in mind, but I think you can agree with me that all these images somewhat fit the description.

I thought I’d ask ChatGPT for another prompt:

Me: That was very artistic. Do you have another one?

ChatGPT’s response:

Sure, here's another prompt for DALLE:
"Picture a futuristic cityscape with towering skyscrapers reaching towards the
sky, interlaced with networks of sleek flying vehicles weaving in between the
buildings. A bustling street below is filled with people of all ages and
backgrounds, moving quickly along the sidewalks. Neon lights and holographic
advertisements fill the air with color and light, and in the distance, a
massive artificial intelligence network can be seen, humming quietly in the
background. Amidst all this, a lone figure stands at the edge of a rooftop,
looking out over the vast expanse of the city and contemplating its infinite
possibilities."

While the previous prompt described a rather customary scene, this one has got more novelty in it. There are few things that are particularly interesting about this description. The humming massive artificial intelligence in the background, and the lone figure standing at the edge of a rooftop. To my disappointment, I don’t see any of them in the images generated by DALL·E 2. Nor are there any flying vehicles. I suppose this is because of the limited training samples. There has never been that much variety or volume for such futuristic cityscape images. Parts of the generated images are also blurred, which again I think is because of the smaller sample space.

The most interesting part however is that ChatGPT provides details of sound (the quietly humming AI) in its description. I don’t see why ChatGPT thinks this is relevant, but here is what ChatGPT has to say about that:

Me: How do you expect DALLE to draw the artificial intelligence network and its humming sound?

ChatGPT’s response:

As an AI language model, I cannot predict with certainty how DALLE will generate
images from textual prompts, as the specific details and style of the image will
depend on the model's training data and internal weights.

However, it's possible that DALLE could use its knowledge of common visual
representations for technology and sound to generate an image of an artificial
intelligence network, such as rows of computer servers, glowing data cables, or
a digital interface displaying lines of code.

As for the sound of the network humming, DALLE might generate an abstract
representation of sound, such as waves or lines emanating from the network, or
a visual metaphor such as the depiction of the network as a living organism that
emits a constant hum. Ultimately, the output of DALLE will be influenced by the 
way it has been trained on vast amounts of data and the creative interpretation
of the language prompt.

Regardless of its relevance and usefulness, the idea of including a description of a background noise in a prompt for DALL·E 2 is new to me, but it does not necessarily mean that it is an original idea that ChatGPT came up with. In fact, “The extent to which the work produced is novel can vary from being original only for the person who completed the work (this is the notion of reinventing ideas known already in the larger social context) to being original for a limited social group to being original for all of humanity” [1].

Finally, we have to try Bard too. This is what Bard said in response to the first prompt I gave to ChatGPT:

Bard’s response:

Sure, here are some examples of prompts that can be used as input to DALL-E:

* A photorealistic painting of a cat sitting on a couch in a living room with a
  coffee mug in its hand.
* A detailed infographic of the human respiratory system.
* A vintage, grainy, sepia, photograph of an adorned Native American man,
  Hassleblad, f4, 150mm, natural light.
* An iPhone made of terracotta from Harappa, Indus Valley Civilization, Pakistan;
  2600-1900 BC, studio light.
* A colorful surreal seamless pattern of a cat and a dog playing together.
* A film still from the movie "The Matrix" of Neo dodging bullets in slow motion.
* A detailed map of the city of New York with all the major landmarks labeled.
* A realistic 3D rendering of the planet Earth from space.
* A beautiful abstract painting in the style of Jackson Pollock.
* A funny cartoon of a dog chasing its tail.

These are just a few examples, and there are many other possibilities. The possibilities are endless!

These look more like what I’d expect people to provide as their prompts to DALL·E 2. I suppose this is because Bard has access to online content. There are some technical details in some of the prompts that I know nothing about, and cannot verify if DALL·E 2 could get that right! But I fed the second prompt to DALL·E 2, and the results are hilarious. It’s been known that DALL·E 2 cannot do text!

The smart recipient effect

“A stupid man’s report of what a clever man says can never be accurate, because he unconsciously translates what he hears into something he can understand.” – Bertrand Russell

This is true in the other direction too! In fact, similar to the quote above, we can say: A human brain’s report of what ChatGPT says can never be accurate, because it unconsciously fills in the gaps and assigns meaning to the words as it sees fit.

I believe, the ultimate measure of understanding something, is the ability to do something with it. In a conversation, or exchange of content, the smarter the recipient, the more likely for it to grasp the meaning in the content, and the easier for the producer to produce content that the recipient can act on.

A smart recipient fills in the gaps and automatically fixes inaccuracies based on its knowledge and model of the world. It actively participates in assigning more meaning to the content at hand, and asks for more details when needed. I have demonstrated this in an earlier post on playing with Lego with ChatGPT. I had to actively interpret ChatGPT’s instructions, making tweaks and adjustments along the way using my knowledge and observations.

If we were to ask ChatGPT to produce some code, for instance in Rust, while the resulting code would seem correct to the human eye at the first glance, and in simple cases the user would be able to run through the instructions and calculate the output of the code, it is very likely that the Rust compiler would not be able to do much with the code (unless it is a very small snippet without many dependencies). The Rust compiler is not as smart as the human user, and in particular cannot fill in the gaps.

For ChatGPT to produce input for the Rust compiler, it has to be familiar with all the rules of the language – the syntax and the semantics – or be able to interact¹ with the compiler the same way the Rust programmers do. Most Rust programmers do not master all details of the language, but can interact with the compiler, and learn from it, by adjusting and improving their mental model of the language. This need for precision and details, can be generalized to interaction with any other compiler, or tool that humans have built.

And that I think is the biggest difference between the creative industry and other industries: people working in the creative industry produce content for human interpretation, while people in other professions produce content that must obey strict rules, or be consumed by dumb tools.

One last thing to note is that, in a conversation, ChatGPT, like us, plays both the recipient and the producer roles. It is the recipient, when processing our requests, and has proven to be a very smart one. It has succeeded in understanding very complex prompts, and correctly following the instructions given to it. When responding, it plays the producer’s role, and has proven to produce content that the human brain can interpret and enjoy.

What does this mean for our creativity?

ChatGPT is now being branded as an assistant tool that can speed up or improve the quality of your work.That is true and great. Recently I asked ChatGPT to rewrite some of my every-day writings, and its rewrites were really better than my own. ChatGPT is certainly better at using phrases and expressions than I am. For my more creative writings, however, I have not been happy with ChatGPT’s rewrites. In those cases it felt like ChatGPT was trying to get the novelty out of the text and make it closer to what is deemed normal. This is the main danger I see in the use of these Generative AIs. There does not seem to be a great operational definition of creativity, let alone a measure of it. But I believe like all other natural phenomena there is a bell curve of creativity somewhere. Too much use of generative AIs will push the standard deviation of that curve to zero!

Last updated: 2023-05-04

1. 4 May 2023 update: I now hear that with GPT-4, people have developed systems that do exactly this. It learns from its interactions. I just got access to the GPT-4 API and have not yet had the chance to experiment with the interactive approaches that use the API. Others have reported developing a system, using GPT-4, that can read the API documentation for a tool, and based on that use the tool correctly. Based on what I’ve seen in my experiments with GPT-4 (via ChatGPT) I am still very skeptical of the level of complexity it can handle. ↩