Custom GPT, Smarter, Faster and Still Clueless
OpenAI flaunted with the Release of Custom GPT on 6th Nov 2023. This piqued most of the folks as it monetized your own GPT. If I see, this feature is just a ChatGPT Plugin with nice office suit to earn some bucks. Needless to say Plugin dint go well as expected. I’d some ideas on creating my first basic Custom GPT and trying out how things work. Its not difficult but its not straight forward. I will share my experience and understanding and also in between lets dive into the technical aspects of GPT which could provide a better view.
What, Why of Custom GPT#
Via official docs as of November 6, 2023. 
Some Basics of LLMs
Before understanding how the Custom GPT works, one need to understand the various capabilities and features of Open-AI Models or LLMs.
- Prompt Engineering
To get the best results out of LLM, its important to give as much as context and as much as data to the model. It is like if you are teaching a 5 year old kid to do some specific work. The official site has very nice and useful instructions. One of the important excerpts which I think pretty much sums up
- Write clear instructions
These models can't read your mind. If outputs are too long, ask for brief replies. If outputs are too simple, ask for expert-level writing. If you dislike the format, demonstrate the format you'd like to see. The less the model has to guess at what you want, the more likely you'll get it.
- Retrieval Augmented Generation powered by Embedding
RAG by itself , is a method introduced by Meta AI researchers that combines an information retrieval component with a text generator model to address knowledge-intensive tasks. Open-AI calls it Knowledge Retrieval. Essentially what it means is basically. From official docs
Retrieval augments the Assistant with knowledge from outside its model, such as proprietary product information or documents provided by your users. Once a file is uploaded and passed to the Assistant, OpenAI will automatically chunk your documents, index and store the embeddings, and implement vector search to retrieve relevant content to answer user queries.
Why we need RAG?
Generative AI technologies, though impressive, encounter limitations due to their confined knowledge base. Technologies such as Language Models (LLMs) like ChatGPT can execute various tasks but have inherent knowledge gaps stemming from the scope of their training data. When tasked with working on contextual data, these LLMs lack current information and context, leading to responses that can be at best inconsistent and at worst, inaccurate; calling it hallucinations. This is where Retrieval-augmented generation (RAG) becomes essential. RAG addresses these limitations by integrating real-time data retrieval into the response generation process. This enables the AI to access and incorporate the latest information, ensuring more accurate, relevant, and informed responses, particularly for queries about recent developments or current events.
RAG is backed by Vector Embedding. RAG and Embedding again are a huge topic and needs a separate discussion. Nevertheless there are tons of material available on these topics already. I will plan to write on Embedding and RAG sooner.
I will visit this part later again when we actually talk about the Custom GPT.
- Action
Action basically allows the GPT to talk the external APIs. From the official docs
Define custom actions by making one or more APIs available to the GPT. Like plugins, actions allow GPTs to integrate external data or interact with the real-world. Connect GPTs to databases, plug them into emails, or make them your shopping assistant. For example, you could integrate a travel listings database, connect a user’s email inbox, or facilitate e-commerce orders.
What’s the big deal?
Action also facilitates providing the OpenAPI schema which thereby enables the GPT to call the APIs. You don’t need to host your own OpenAPI spec but instead do if from the scratch on the OpenAI side.
Custom GPT Under the Hood#
I do not intend to showcase on how to create a custom GPT because there are already tons of blogs showing it. Here the focus is on how the LLM capabilities are co-related to the Builder and what are the limitations.
The Message to the GPT Builder(Instructions in Configure section) is multi facet process. If I take a short example of providing Restaurant information to the people of New-York
Firstly Instruction to the GPT Builder is providing the context with detailed system prompts. That’s the aspect of Prompt Engineering. Provide very detailed information/instruction. Below is just a sample example.Its 1/10th of the actual Prompt.
You are an agent specializing in providing restaurant information for New York. Identify a renowned restaurant in New York City, focusing on a specific cuisine (e.g., Italian, Japanese, etc.). Include the restaurant’s name, a brief description highlighting its unique features or popular dishes, and any notable awards or recognitions it has received. Specify its exact location, including street address and neighborhood. Additionally, provide detailed operating hours, mentioning if they vary on weekends or holidays. Note any special services like outdoor seating, private dining options, or delivery services. If possible, include average price range for a meal and any dress code requirements. Conclude with contact information for reservations
- Secondly, I may also attach the documents describing the details of the famous 7 star restaurants with their Menu cards. This is where the Knowledge Retrieval or RAG comes in picture. Once a file is uploaded and passed to the Assistant, OpenAI will automatically chunk your documents, index and store the embeddings, and implement vector search to retrieve relevant content to answer user queries. The "upload files" is the way of providing the knowledge base for the OpenAI.
- Thirdly, there are few restaurants which provide Rest API. However the instructions have to be provided to the builder on extracting the input parameters to the endpoint either via the document(knowledge base provided) or via the user input. I can either use the OpenAPI spec from the Restaurant or create my own from the scratch. For example a very basic menu API may look like.
openapi: 3.0.0
info:
title: Restaurant Menu API
version: 1.0.0
paths:
/getMenu:
post:
summary: Retrieves the menu of a specific restaurant
description: This endpoint provides the menu for a given restaurant based on its name, ID (if provided), or location.
operationId: getMenu
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
restaurantId:
type: string
description: Unique identifier for the restaurant (optional)
restaurantName:
type: string
description: Name of the restaurant
location:
type: string
description: Location of the restaurant (e.g., city, neighborhood)
required:
- restaurantName
responses:
'200':
description: Successfully retrieved the menu
content:
application/json:
schema:
type: object
properties:
menu:
type: array
items:
type: object
properties:
itemName:
type: string
description:
type: string
price:
type: number
Whats the Reality!!#
Now I think its fairly understood how the things works and are glued together. Building and configuring is not simple and sometimes feels wild Goose chase. Also since its in its initial nascent stage, lots of pieces are moving in parallel( Including the Sam's exit and entry at OpenAI as I write this blog ;) ) and things are in Beta.
1. Instructions/System Prompts
The GPT builder tends to forget/diverge from the instruction. As a solution it is always better to save your prompts in a private github repository. It saves times to fine tune your prompts when you want to play with the Builder. That said, I still see the missing correct answers. A week back I saw technical glitch while saving/updating the GPT. I even ask the GPT what it has learnt from the details provided and I see some missing instructions.
2. RAG/ Knowledge Retrieval
OpenAI supports these file types and also additionally markdown files. Note the difference in the file types for Retrieval vs Code Interpreter. I uploaded a tabular data in text format as well as markdown format which has 550k lines(records) with 4 columns, the GPT builder was unable to load the data. The file size is just 28MB.
Official file size supported is 512 MB.
3. Action Calling
This a pretty straight forward execution. The more clear the spec is, the better the results are. The complexity here is identifying the right parameters for the Endpoint via the output of RAG execution.
Considering the overall capabilities, I played around with the order of inputting the capabilities.
Provided the Knowledge Retrieval via the text files and markdown files together with the System Prompt/Instructions in one shot. This was too much of Information and I saw some wrong and don't know answers
Provided the Knowledge Retrieval via the text files and markdown files first and then interactively provided the instruction(system prompts) which was more comprehensible and iterative. This worked much better but still over the margin of errors.
Inferences#
The Custom GPT is very freshly baked out of oven and hence cant be expected way too much during its initial stages. That said, it may take a while for near to perfection results. Reason for perfection is expected as, we as developers/non developers are in full control of the context and the Knowledge base with finite set of data.
From the lens of creator(In context of creating the GPT), the effort and experience of developing the app is very much depending on scope and context of idea. Of course Custom GPT is just a flavor of plugin, which acts as a more of moderator/chatbot and not the preferred platform for developing a full fledged Enterprise app.
For a proper Enterprise application the APIs are still the right way going forward. As the developer can own the whole stack of his capabilities and extend/scale it as per the needs.
Confession#
I've been creating a custom GPT(hopefully coming soon) and trying out things for past 2 weeks. These are my own experience with the scope and context I am building it. Definitely the experience is enriching and not to mislead the future developers. Lots of things will change eventually in coming months and even my writing here may be obsolete. I would be very eager to receive some comments from your learnings.