So GPT-4's architecture has been leaked.

The structure of GPT4 has been leaked. (The tweet is now deleted and only exists on Archive, and the article in the second link has also been deleted). In early July, the detailed parameters, structure, training data, etc. of GPT 4 were unofficially leaked. # Parameter & layer Number of parameters 1.7 trillion/gPT3 (175 billion), which is about 10 times the number of parameters. (Later, it is said that the number of parameters was added through visual encoding and reached 2 trillion, which is the largest number among all as of July 23). Layer 120 (96 in GPT3) # MOE(Mixture of Experts) Uses 16 MOEs (mixture of experts) and has 100 billion parameters for each moe What is MOE? It refers to the creation of optimized parts (experts) for each field, and the appropriate and uses only that part. MOE uses a gating network to determine which expert is the most appropriate by using the metadata of the text (words used/context of the text, etc.) and assigns it to the part. Using this efficient method, each forward pass inference (generating one token) uses only about 28 billion parameters and about 560 TFLOPs. This compares to the full 1.8T of parameters and about 3,700 TFLOPs (3,700,000,000,000,000,000) of computation for a pure dense model without MOE. 64x fewer parameters 7x fewer calculations # Training Using 25000 a100S (GPU name) for about 100 days. 21,500,000,000,000,000,000,000,000 calculations per second. However, the actual learning efficiency is about 30%. This is due to the fact that there are too many errors during training, and it is necessary to start from the beginning again. # Dataset They trained with 13 trillion words (tokens), to be exact). That's about 100 million books, which is about 10 times the number of books in the world. But there's a catch: they trained on the same data twice, so we actually have to split it in half. Some of the data we actually trained on included 4 million books / 80 million scientific papers / all sources on GitHub Almost all of the world's training data can be converted to a txt file, so it has learned every field of human endeavor: science, philosophy, math, medicine, law, etc. Because of this learning, it is said to be able to answer any question without any limitations. After this learning, they used Scale ai to fine-tune it internally. # Pricing Assuming that $1 is spent to run an a100S GPU for an hour, the total cost of training is said to be around $800 million. # Encoding There are encoders that handle text image audio (using whisper) and interact with each other. By utilizing these, they were able to add more parameters to GPT4's 1.8 trillion parameters and fine-tune them to about 2 trillion tokens. They have also used these encoders to screenshots, YouTube videos, and # Speculative Decoding This method uses a small, fast model to think about the output in advance, and then feeds it to a large Oracle model at once: if it's right, keep going; if it's not, recalculate. It has been speculated that the recent decline in quality of GPT-4 is simply due to a lower cutoff for the oracle model to accept the initially fed information. # GPUs We run inference on a cluster of 128 GPUs. These clusters are located in different data centers around the world. # Batch Size We can't train too much data at once, so we divide it into batches. When training, we train one batch at a time. One batch contains 60 million tokens. Each expert in the MOE mentioned at the beginning learns 7 million out of 60 million. reference: https://archive.is/2RQ8X#selection- 202 e-mail : duwon570@gmail.com