Overview and mechanics of large language models
Large language models are artificial intelligence systems built on deep neural networks that process inputs and generate text by predicting subsequent tokens. Their primary characteristics include scale reaching tens or hundreds of billions of parameters, transformer architectures capable of analyzing entire textual contexts simultaneously, and extensive training on billions of sentences using both supervised and unsupervised learning. Once deployed, their internal parameters generally remain static, relying on in-context examples for temporary adaptation.
Core architecture and parameter scale
Large language models are artificial intelligence systems that can process and generate text autonomously [1]. At their foundation, these systems operate as neural network models that have billions of parameters, called weights, that contain the model's knowledge and process inputs to make predictions [2]. Model capacity is characterized by its parameter count, with large models containing approximately 100 billion parameters [3]. Building and fitting a model of this magnitude requires trillions of data points and is computationally expensive [3].
These systems are built on sequence-to-sequence transformer architectures [4]. Unlike sequential approaches that process words one by one, the transformer architecture simultaneously analyzes all parts of a text [4]. This structural design enables sequence-to-sequence models to excel at processing long texts, linking distant information across conversations or technical documents to produce precise answers [5]. Rather than processing individual words alone, these systems analyze complex sequences to capture overall meaning, subtle contexts, and linguistic nuances [6].
Training methodology and corpus diversity
Large language models are trained using two complementary approaches: supervised learning and unsupervised learning, which maximize their capacity to analyze and generate text [7]. Unsupervised learning does not require labeled data, allowing the model to explore large volumes of text to discover hidden structures and identify semantic relationships [8]. Through this unsupervised exploration, the model learns recurring patterns, implicit grammatical rules, and the contextualization of sentences and concepts [9].
Supervised learning complements this process by utilizing labeled data where each input is associated with an expected output [10]. The model learns to associate these inputs with the correct outputs by adjusting its internal parameters to reduce prediction errors [10]. Through this approach, the model acquires precise knowledge about specific tasks, such as text classification or named entity recognition [11]. To establish broad linguistic competence, models are trained on billions of sentences drawn from a variety of sources, such as news articles, online forums, technical documentation, and scientific studies [12].
Operational mechanics and reasoning constraints
During operation, many popular large language models work by predicting the next word, or token, given some natural language input [13]. When a user submits a query or text, the model applies its predictive ability to generate the most likely sequence based on the context provided [14]. To generate coherent and informative text, the model analyzes each word, studies grammatical and semantic relationships, and selects the most suitable terms [15].
Because these machine-learning models typically use only natural language to process information and answer queries, performing tasks that require numerical or symbolic reasoning can be difficult for them [16]. Furthermore, the mechanisms underlying many of the observed cognitive and social behaviors of language models remain unclear [17]. Human-like performance can then be mistaken for understanding, which obscures the norms, biases, and structures of power that models reproduce [17]. In specialized applications, fundamental challenges persist because models struggle with causal inference and raise urgent ethical concerns regarding privacy and misinformation [18].
In-context adaptation and weight stability
During training, the model adapts its weights to learn new information contained in its training data [19]. However, once an LLM has been deployed, its weights are static and cannot be permanently updated anymore [19]. Once a fully trained model has been deployed, its internal system is static and cannot permanently adapt itself to new knowledge [20].
To learn new tasks without permanent weight updates, models utilize in-context learning, in which a trained model learns a new task by seeing a few examples [21]. These demonstration examples guide the model's responses, but the knowledge disappears before the next conversation [21]. Model performance is also sensitive to prompt construction, as more detailed prompts lead to better responses [22]. In enterprise environments, using both a short description prompt along with a meta prompt of company legal or compliance policies enables the language model to create compliant responses [23].
Key facts
- Large language models are artificial intelligence systems that can process and generate text autonomously [1].
- These systems are neural network models that have billions of parameters, called weights, that contain model knowledge and process inputs to make predictions [2].
- Large models feature around 100 billion parameters, requiring trillions of data points and expensive computation to build [3].
- Sequence-to-sequence transformer architectures analyze all parts of a text simultaneously rather than processing words one by one [4].
- Sequence-to-sequence models excel at processing long texts by linking distant information across conversations or technical documents [5].
- Model training combines unsupervised learning on unlabeled text to discover hidden structures and semantic relationships with supervised learning on labeled data to reduce prediction errors [8][10].
- Unsupervised learning allows models to learn recurring patterns and grammatical rules, while supervised learning enables models to acquire precise knowledge for specific tasks like text classification or named entity recognition [9][11].
- Training corpora comprise billions of sentences drawn from news articles, online forums, technical documentation, and scientific studies [12].
- Predictive text generation functions by predicting the most likely next word or token based on grammatical and semantic relationships within the provided context [13][14][15].
- Relying primarily on natural language makes it difficult for language models to execute tasks requiring numerical or symbolic reasoning [16].
- The mechanisms underlying many observed cognitive and social behaviors remain unclear, meaning human-like performance can be mistaken for understanding [17].
- Models struggle with causal inference and raise urgent ethical concerns regarding privacy and misinformation [18].
- Once deployed, a model's weights are static and cannot be permanently updated, preventing permanent adaptation to new knowledge [19][20].
- Models use in-context learning to learn tasks from a few examples, but that knowledge disappears before the next conversation [21].
Sources
-
What is a large language model (LLM)? about.gitlab.com
- [1]
LLMs are artificial intelligence (AI) systems that can process and generate text autonomously.
- [4]
The best-known of these structures is the architecture of sequence-to-sequence models (transformers). This architecture has revolutionized NLP with its ability to simultaneously analyze all parts of a text, unlike sequential approaches that process words one by one.
- [5]
Sequence-to-sequence models excel at processing long texts. For example, in a conversation or a detailed technical document, they are able to link distant information in the text to produce precise and well-reasoned answers.
- [6]
These systems do not just process individual words; they analyze complex sequences to capture the overall meaning, subtle contexts, and linguistic nuances.
- [7]
LLMs are trained using two complementary approaches: supervised learning and unsupervised learning. These two approaches to machine learning maximize their ability to analyze and generate text.
- [8]
Unsupervised learning (or machine learning), on the other hand, does not require labeled data. The model explores large volumes of text to discover hidden structures and identify semantic relationships.
- [9]
The model is therefore able to learn recurring patterns, implicit grammatical rules in the text, and contextualization of sentences and concepts.
- [10]
Supervised learning relies on labeled data, where each input is associated with an expected output. The model learns to associate these inputs with the correct outputs by adjusting its internal parameters to reduce prediction errors.
- [11]
Through this approach, the model acquires precise knowledge about specific tasks, such as text classification or named entity recognition.
- [12]
LLMs are trained on billions of sentences from a variety of sources, such as news articles, online forums, technical documentation, scientific studies, and more.
- [14]
When the user submits a text, query, or question, an LLM uses its predictive ability to generate the most likely sequence, based on the context provided.
- [15]
The model analyzes each word, studies grammatical and semantic relationships, and then selects the most suitable terms to produce a coherent and informative text.
- [1]
-
Teaching large language models how to absorb new knowledge | MIT CSAIL www.csail.mit.edu
- [2]
LLMs are neural network models that have billions of parameters, called weights, that contain the model’s knowledge and process inputs to make predictions.
- [19]
During training, the model adapts these weights to learn new information contained in its training data. But once it is deployed, the weights are static and can’t be permanently updated anymore.
- [20]
Once a fully trained LLM has been deployed, its “brain” is static and can’t permanently adapt itself to new knowledge.
- [21]
However, LLMs are very good at a process called in-context learning, in which a trained model learns a new task by seeing a few examples. These examples guide the model’s responses, but the knowledge disappears before the next conversation.
- [2]
-
Building Better Answers With AI Agents: SAP’s Walter Sun sloanreview.mit.edu
- [3]
Now we’re looking at like 100 billion parameters, so that’s considered large. ... There’s a lot of free flexibility. So obviously, you need trillions and trillions of data points to fit that, to build that model. It’s expensive computation-wise.
- [22]
realizing the more detailed your prompt is, the better the response.
- [23]
So using both the short description, which is called the prompt, along with the meta prompt of whatever the company’s legal policy or compliance policies are, you can create a very nice response using a language model.
- [3]
-
Technique improves the reasoning capabilities of large language models | MIT CSAIL www.csail.mit.edu
- [13]
Many popular large language models work by predicting the next word, or token, given some natural language input.
- [16]
These machine-learning models typically use only natural language to process information and answer queries, which can make it difficult for them to perform tasks that require numerical or symbolic reasoning.
- [13]
-
Mapping the Emerging Social Science of Large Language Models arxiv.org
- [17]
The review also identifies a gap in current understanding: the mechanisms underlying many of LLMs' observed cognitive and social behaviors remain unclear. Human-like performance can then be mistaken for understanding, obscuring the norms, biases, and structures of power that LLMs reproduce.
- [17]
-
Large language models for computational social science - Nature Human Behaviour www.nature.com
- [18]
Yet fundamental challenges persist. LLMs struggle with causal inference and raise urgent ethical concerns about privacy and misinformation, and their ability to reproduce social complexity remains uneven and unproven.
- [18]