In fact, both incremental and emergent developments are already present in today’s AI systems. The mentioned incremental direction is perhaps obvious, since it is generally typical that newer and newer models and architectures build on the experience gained from previous ones and are consciously and purposefully developed and refined. This has been the case for GPT models in their evolutionary path, but it also includes the fine-tuning of individual models. The latter involves further training of the base model on a smaller, specific data set. The main goal here is to improve the model’s performance on a given task while preserving the general capabilities that it has previously acquired.

Fine-tuning is the process of “transforming” models with large, but general capabilities into a more specialized one. The aim is to make the model more efficient in a very narrow domain by using specialized training data. Several versions of the same model can be created using this method, each of which will be excellent at a particular narrow task. Examples include question-answering (QA), sentiment analysis, and even determining the public policy content of a text.

Examples of the emergent capabilities of large models are also easy to find. In the field of robotics, autonomous vehicles and robots can develop and refine complex movement patterns that are the result of autonomous learning. For example, robots from Boston Dynamics can learn and adapt to different terrains without having to be pre-programmed for each of their movements. Such emergent behavior allows robots to move efficiently in new situations.

A similar phenomenon can be observed in neural network-based architectures, for example in the field of image generation. Generative Adversarial Networks (GANs) can easily generate images that have never been seen before. Such systems consist of two networks: the generator and the discriminator. The generator generates new synthetic data, such as images, while the discriminator tries to distinguish the generated images from real ones. The generator and the discriminator compete: the generator tries to create more and more realistic images, while the discriminator tries harder to recognize and separate them from real images. Through these interactions, the generator can create increasingly realistic images, a form of emergent behavior. The “This Person Does Not Exist” website was perhaps the first example of this to receive much media attention. Since then, of course, there have been many more advanced solutions.

To understand these seemingly “random” properties, one more concept needs to be clarified. Nonlinear interactions are interactions in a system where the relationship between the elements is not straight. This means that small changes in the inputs to a system can have a disproportionately large effect on its outputs, and these effects cannot be predicted simply from the inputs.

Nonlinear systems therefore often exhibit complex behavior, chaotic dynamics, and emergent properties that cannot be predicted simply from the behavior of individual elements. This is in fact an indication of the strong emergence already mentioned in the previous section.

This kind of relationship between the initial and subsequent state of a system has long been known, and its understanding is important for studying, for example, the future number of individuals in a population.

Such interactions are also fundamental to chaos theory. Chaos theory is a branch of mathematics and physics that studies the behavior of deterministic systems, where the long-term behavior is unpredictable and seemingly random, even though the rules of the system are well-defined. Chaos theory basically studies the nonlinear behavior of dynamical systems, where the sensitivity to initial conditions (butterfly effect) makes the output highly sensitive to small changes in the inputs. To put it very simply, the theory is that the behavior of a system can change very much for very small changes in the initial condition. So much so, in fact, that the minimal initial change does not seem to be justified at all. The concept of nonlinear interaction refers to this, the change in output is not directly proportional to the change in input.

At this point, the question rightly arises as to how the three key concepts mentioned so far, nonlinear interactions, emergent behavior, and chaos theory are related, and of course why this is important for the future of AI research.

Nonlinear interactions play a fundamental role in the functioning of neural networks and learning processes. These interactions ensure that the network can recognize complex patterns and perform various tasks such as image and sound recognition, natural language processing, and predictive modeling. In fact, they are the ones that ensure the resilience even of today’s LLMs. This, moreover, has mainly to do with the activation functions used in such networks. The same interactions are the basis of chaos theory, where changes in initial parameters can amplify or cancel each other out, seemingly in a completely unpredictable way.

Similarly, emergent behavior is the result of such interactions. In other words, what chaos theory approaches at an abstract level takes the form of novel capabilities and “behaviors” in neural networks.

All of this relates back to the possibility of AGI being reached at any moment. The predictions that appear regarding the achievement of AGI are, in most cases, based on the assumption that AGI will be a planned step in development. It is usually in this light that experts also try to estimate when this step will be reached, considering current trends. However, emergent behavior can completely overturn these expectations at any time.

Of course, it is important to note that there is no guarantee that emergent properties will appear, nor that they will not appear in a system as complex as today’s Foundational Models or LLMs. Equally, it is undoubtedly true that seemingly random emergent properties may be completely unnoticed, beneficial, or even detrimental.

The main question is: how prepared are we really for the emergence of such properties? The development and application of AGI is an international issue that requires global regulatory coordination. Looking around the world, we see that different regulations and legislation in different countries can create difficulties in the global application of AGI systems. And the lack of a single international regulatory framework can be a serious barrier to innovation and the safe use of such technology​.

István ÜVEGES is a researcher in Computer Linguistics at MONTANA Knowledge Management Ltd. and a researcher at the HUN-REN Centre for Social Sciences, Political and Legal Text Mining and Artificial Intelligence Laboratory (poltextLAB). His main interests include practical applications of Automation, Artificial Intelligence (Machine Learning), Legal Language (legalese) studies and the Plain Language Movement.