The concept of “understanding” is inherently subjective, especially when evaluating GPT-series models, which are often considered black boxes. Different people may hold varying perspectives on what constitutes understanding. While some argue that GPT-4 lacks true comprehension, it’s worth questioning how we measure understanding in humans. What level of performance or test scores can definitively distinguish between a solid grasp of concepts and mere conjecture?

I contend that framing the debate around whether these models “understand” is misleading.

Photo by Ben White on Unsplash

Let’s start by discussing the reasoning process. The most effective approach is to view it from the perspective of general learning. This process is analogous to how humans acquire knowledge through reading, frequent reflection, memorization, and interaction.

The reasoning ability of language models can be understood as a reflection of rationality in cognition, or as a data-centric learning problem, rooted in continuous learning from large-scale datasets. Within these human-annotated datasets, cognitive logic and causality from a human perspective are evident. The design of these models aims to enable them to learn both implicit and explicit correlations from token-level representations (commonly understood as word-level, though it’s more nuanced than just individual words). The process of learning natural language tokens is akin to learning images at the pixel level. For instance, just as one can generate images by understanding the distribution of pixels (e.g., mouths are located at the bottom of faces, symmetrical eyes, etc.), in language processing, a “pixel” corresponds to a token. The distribution of a sentence (comprising multiple tokens) conveys information in a way that mirrors human recognition of patterns and characteristics.

Features or representations provided by a good DNN model should encode meaning related to the current task (in this case, classification). With this representation, similar data items are closer to each other (right side); the original non-linear problem is now linearly separable, and therefore easier to solve. (referenced from Deep Representation Learning: What and Why! )

Thus, the central issue is representation — specifically, how much valuable information has been learned from the inherently noisy and sparse datasets. How can we efficiently extract and represent knowledge during the learning process?

Professionally but directly, the illustration of learning and representation leads to the question of how effectively the learning process can optimize probability, regarding whether language models can understand. This is why the reasoning ability of a language model is theoretically limited by the original rationality present in the dataset. It is important to remember that a language model is not a human-like being. People often mistakenly treat these advanced models as if they possess human qualities. However, they are more accurately described as complex functions designed to compress information by maximizing the representation of the balance between every detail and microstructure in the data, regardless of the extent of human knowledge embedded within it.

Instead of asking whether these language models can understand things, we may need to focus more on how effectively they perform specific tasks in a learning-and-inference fashion. Humans do not design the world, and machine logic is not entirely a human creation.