Obsessed with finding optimal solutions, a colleague and I spent an afternoon pair programming, alongside our friend ChatGPT. Our self-imposed challenge driven by our sense of perfectionism was to refine an already working process. As geeks, it is always an absolute joy to engage with new technologies and explore and unravel their capabilities, pushing the boundaries of what we know and use.

However, that particular afternoon turned into a rather frustrating experience. Using ChatGPT, we were trying to extract some information and put it in a structured way from an input text but we kept falling short of what we needed. The prompt we ended up with was almost as long and complex as the input and yet it wasn’t giving us what we needed. Nevertheless, our results were way better than the ones traditional programming without AI gave us. 

We kept scratching our heads until we decided to be lazy and just give the input to ChatGPT and the output we wanted and let ChatGPT do the job and figure out the logic. Surprisingly it worked! With some tweaks we managed to get 100% accuracy and get exactly what we were after.

At first we were so amazed and fascinated by how much potential ChatGPT has in solving complex problems. Those feelings were very much mixed with the embarrassment of why we never thought about using it that way in the first place. We talked for an hour about how much time, effort and energy this approach could’ve saved us and also wondered how many people are on the dark side of ChatGPT, like us. 

We decided to plan an internal experiment and evaluate our assumptions. We invited 100 of our colleagues, ranging from developers to business analysts to sales and marketing. Considering our team's varied interactions with ChatGPT on a daily basis, I anticipated substantial engagement and feedback.

The main idea of these experiments was to create outputs using ChatGPT prompts.  Participants were allowed to write as many prompts as they wanted, without any limit on how long they could be as long as they could produce the desired outputs.

The first challenge we gave was to generate a certain output for a given number, in this case, the number “5”:

1 2 3 4 5

1 2 3 4

1 2 3

1 2

1

The first responses were really interesting. How people approached the challenge was influenced a lot by their roles. For example, developers tended to go for logical solutions, while others focused more on how the output looked visually. After some time playing around with it, many of the participants found the solution. There was a lot of happiness and satisfaction at this achievement. But they didn't know what was coming in the next rounds.

Some of the prompts participants came up with:

“Create a sequence of numbers from 1 to 5, output in rows, first row 1 to 5 , then 1 to 4 and continue until you get to 1”

“For x. display all number from 1 to x on first line and reduce one number in every line until it reached 1”

“Write me 5 lines of number containing ascending numbers starting from 1 equal to the number of the sentence”

Moving on, I introduced the second challenge. This task required them to generate the following output for a provided number “5”:

1

1 2

1 2 3

1 2 3 4

1 2 3 4 5

1 2 3 4

1 2 3

1 2

1

Watching their focused faces was a joy I couldn't resist. Some participants gave up fast, others came up with outputs that looked nothing like what I asked for. Slowly but surely, a few participants figured out the right answer. Some of the prompts were particularly interesting, for example, some participants (potentially from Canada?) started to use their charm in order to get a response from Chat GPT:

“Could you please count out the numbers one through five on one line, then count up to one number lower on each subsequent line until you finish with only the numeral one. Then please immediately begin the same process in reverse, starting directly with one and two rather than a second line with only one.”

It was time for the third and final challenge. Their task? To generate:

1

1

12

1

12

123

1

12

123

1234

1

12

123

1234

12345

The logic was a mystery to many, causing several to give up right away. Others showed signs of stress and confusion, moving in their chairs or thinking aloud. Finally, a few people figured it out, sparking celebrations of their success. But the prompts they came up with required a great deal of programming mindset and knowledge. For example, one of our bright developer’s answer was (feel free to share a selfie of when you see the output on ChatGPT :D):

“see this code for n=3 {1\n1\n12\n1\n12\n123} produce this for n = 10 - format the \n's nicely”

Before sharing the solution, I took a moment to feel the mood in the room. Everyone, knowing my style, could tell something unexpected was coming. With a long pause, I finally revealed the answer. Seeing it, their reactions were the same mix of surprise and wonder that my colleague and I had. They quickly accepted that they'd been going about it the wrong way and recognised ChatGPT's amazing abilities. 

The answer: 

Below is the output for "5" 

                           1 2 3 4 5

                           1 2 3 4

                           1 2 3

                           1 2

                           1

with the same sequence and create the output for "10"

Many participants initially tackled the problem by instructing ChatGPT on how to create this pattern. While they were successful, these process-based solutions were complex and time-consuming. As we increased the complexity of the challenge, fewer participants were able to find the solution, and the time taken to do so significantly increased.

Fewer participants were able to handle the tougher tasks, and the expected time to complete them shot up quite a bit. We didn't measure if there would be an increase in effort, but it wouldn't surprise me if there was a similar increase.

This experiment brought out an important part of working with AI - focusing on the end goal instead of detailing the process step by step can often result in better and more efficient solutions. This move from telling ChatGPT exactly what to do to just describing what we wanted not only saved us time, but also let ChatGPT uncover layers of logic we'd missed when we first tried to solve the problem.

The lesson from this experiment is clear: By concentrating on 'what' we want to achieve, rather than 'how' we do it, we can make full use of the advanced problem-solving capabilities of AI systems like ChatGPT.