Like any new technology, ChatGPT has generated a flurry of excitement. And while it will certainly change the future of AI, its launch hasn’t been problem-free.

ChatGPT users have complained about performance issues they’ve encountered accessing the tool. Many were only able to use it after multiple attempts due to high usage and load over the portal. The same thing happened to me when trying to access it.

To improve the user experience, Open AI will need to increase the processing capabilities and overall computing resources to satisfy the service high demand.

In this Insight Blog, I explore the limitations of the tool in its current state and its potential for general use and software development.

For general usage

When I used the tool to ask simple and straightforward questions, the model responded accurately and fast. When more complex and sequential questions were asked in the same message, it struggled to get it right or it got hung up on the request.

One important aspect to consider is the way the model was trained. Supervised learning in ML models relies completely on human interaction and feedback received from sample data. Without one unique source of truth, it’s difficult to always expect a precise answer. Moreover, AI trainers play a significant role in how the model will perform in answering questions precisely.

To concisely fine-tune the AI model, Open AI uses several methods and techniques such as Reinforcement Learning from Human Feedback (RLHF), Rewarding Models and Proximal Policy Optimization. The result leads to a nice chatting experience. Users hardly sense they are talking to a non-human receiver.

After playing around with the tool a couple of times, the way it followed up the conversation was surprising for me. For instance, I asked for general information about a historical character. After the initial response, I added some sequential questions without specifying to whom they were intended, and ChatGPT assumed it correctly. Also, there is no waiting time for question clarification, but a constant effort to understand the context and meaning of questions on the fly, which contributes to conversation fluency.

ChatGPT is sensitive to tweaks over the user input. The user needs to ask specific questions to get better answers. If the chatbot is not able to answer when an important level of ambiguity is being used, try removing it so the model can get a useful answer.

I noticed that the model is sometimes highly biased regarding some controversial topics, and it sometimes is unable to refuse inappropriate requests. For example, I asked ChatGPT to author a poem to a well-known politician. As I expected, it refused it, but it surprised me by providing a response for the same request with a different politician. Open AI claims that this is an expected behavior. Now, Open AI is making a significant effort to prevent it by using the “Moderation API” to warn or block certain types of unsafe or harmful content. Also, it is important to highlight that user feedback is always welcome to improve the system.

For software development

As a programmer, I was able to challenge ChatGPT on coding skills. I asked, for examples, best practices and code reviews and I received some good answers and feedback, but other responses were too general or they were not what I needed then. As reference, I asked for a complete docker-compose file and received just a fraction of it. Then I asked for a fully working example and it still was not able to provide a more useful code.

The system was able to find some code smells and potential runtime errors over chunks of code in a variety of programming languages, so I see potential if it’s used as a code advisor/reviewer.

While ChatGPT is incapable of producing a complete program, it will help users get started on the first code base. With its limitations regarding how it can be used to help programmers, there is still a long way to go to get the model at the desired state.

In summary

Surely, the upcoming investment in improving the model will eventually result in more specialized capabilities and a better user experience. With that said, I recommend using it as a personal assistant in the workplace. It can enhance performance and deliver value faster to employers. It might not solve all inquiries as expected, but the potential is certainly there.

Have a project you would like to discuss? Connect with us now.