New Lexica model and GPT-4 rumors

Generative Pre-trained Transformer (GPT) is a text generation deep learning model trained on the data available on the internet. It is used for question & answers, text summarization, machine translation, classification, code generation, and conversation AI.

There are endless applications for GPT models, and you can even fine-tune them on specific data to create even better results. By using transformers, you will be saving costs on computing, time, and other resources.

Before GPT-1, most Natural Language Processing (NLP) models were trained for particular tasks like classification, translation, etc. They all were using supervised learning. This type of learning comes with two issues: lack of annotated data and failure to generalize tasks.

GPT-1 (117M parameters) paper (Improving Language Understanding by Generative Pre-Training) was published in 2018. It has proposed a generative language model that was trained on unlabeled data and fine-tuned on specific downstream tasks such as classification and sentiment analysis.

GPT-2 (1.5B parameters) paper (Language Models are Unsupervised Multitask Learners) was published in 2019. It was trained on a larger dataset with more model parameters to build an even more powerful language model. GPT-2 uses task conditioning, Zero-Shot Learning, and Zero Short Task Transfer to improve model performance.

GPT-3 (175B parameters) paper (Language Models are Few-Shot Learners) was published in 2020. The model has 100 times more parameters than GPT-2. It was trained on an even larger dataset to achieve good results on downstream tasks. It has surprised the world with human-like story writing, SQL queries and Python scripts, language translation, and summarization. It has achieved a state-of-the-art result using In-context learning, few-shot, one-shot, and zero-shot settings.

According to Altman, GPT-4 won’t be much bigger than GPT-3. So, we can assume that it will have around 175B-280B parameters, similar to Deepmind’s language model Gopher.

The large model Megatron NLG is three times larger than GPT-3 with 530B parameters and did not exceed in performance. The smaller model that came after it reached higher performance levels. In simple words, a large size does not mean higher performance.

Altman said that they are focusing on making smaller models perform better. The large language models required a large dataset, massive computing resources, and complex implementation. Even deploying large models becomes cost ineffective for various companies.

Large models are mostly under-optimized. It is expensive to train the model, and companies have to make a trade between accuracy and cost. For example, GPT-3 was trained only once, despite errors. Due to unaffordable costs, researchers could not perform hyperparameter optimization.

Microsoft and OpenAI have proved that GPT-3 could be improved if they have trained it on optimal hyperparameters. In the findings, they have discovered that a 6.7B GPT-3 model with optimized hyperparameters has increased the performance as much as a 13B GPT-3 model.

They have discovered new parameterization (μP) that the best hyperparameters for the smaller models are the same as the best for the larger ones with the same architecture. It has allowed researchers to optimize large models at a fraction of the cost.

During the Q&A, Altman said that the GPT-4 won’t be multimodal like DALL-E. It will be a text-only model.

Why is that? Good multimodal is hard to build compared to language only or vision only. Combining textual and visual information is a challenging task. It also means that they have to provide better performance than GPT-3 and DALL-E 2.

So, we won’t be expecting anything fancy in GPT-4.

The GPT-4 will be more aligned than GPT-3. OpenAI is struggling with AI alignment. They want language models to follow our intention and adhere to our values.

They have taken the first step by training InstructGPT. It is a GPT-3 model trained on human feedback to follow instructions. The model was perceived to be better than GPT-3 by human judges. Regardless of language benchmarks.

The GPT-4 release date is still unconfirmed, and we can assume that the company is focusing more on other technology like text-to-image and speech recognition. So, you might see it next year or next month. We can’t be sure. What we can be sure of is that the next version will solve the problem of the older version and present better results.

GPT-4 will be a text-only large language model with better performance on a similar size as GPT-3. It will also be more aligned with human commands and values.

You might hear conflicting news on GPT-4 consisting of 100 trillion parameters and only focusing on code generation. But they all are speculation at this point. There is so much more that we don’t know about, and OpenAI has not revealed anything concrete about the launch date, model architecture, size, and dataset.

Just like GPT-3, GPT-4 will be used for various language applications such as code generation, text summarization, language translation, classification, chatbot, and grammar correction. The new version of the model will be more secure, less biased, more accurate, and more aligned. It will also be cost-efficient and robust.