Intro

The age of AI has officially begun. Stock markets rallied. Energy consumption rises. Policy makers scurry to keep up regulations while companies seemingly slap the term AI on every new product, inevitably reinforcing the hype. Maybe the same could be argued for the news. And yet another article on AI goes online. With all this the need for a better understanding of neural networks becomes obvious. Although much of the conversation around artificial intelligence is centered on where this technology is headed, it might just be useful to look backwards. AI history is longer than most expect and gives relevant insights to better understand modern neural networks. 

Photo (top-right): ©Penn Today Photo (bottom-left): ©Elias Rudigier

Early footsteps of AI

In retrospect AI history is almost as long as computer research itself. The groundwork for AI was laid by early computer scientists in the 1940s and 50s. The most famous among those researchers is Alan Turing. The British mathematician might be best known for being portrayed by Benedict Cumberbatch in the movie The Imitation Game and the invention Turing Test. The test that was designed to determine whether a machine can imitate human intelligence well enough to trick a human into thinking it is talking to another person. For Turing, this was the ultimate intelligence measure.


Dartmouth AI Conference (1956) - Photo: ©Margaret Minsky

With the research of Turing and others in mind, a group of young scientists around John McCarthy gathered at Dartmouth in the summer of 1956. Their goal was to get a deeper understanding of intelligence so that it subsequently can be replicated by a machine. Those meetings can be described as the birth of artificial intelligence. It was also McCarthy who first used and came up with the term artificial intelligence. He notably said in hindsight that none of the researchers particularly liked the term, but they had to name it something. Out of these meetings at Dartmouth the main figures of early AI research emerged. One of them was Marvin Minsky who founded the MIT AI Lab soon after. The new field of AI had officially begun and started off quite optimistic. The researchers expected to be able to replicate human intelligence in twenty years and thereby quickly pass the Turing Test. We now know that they embarked on a much more complicated journey that has been marked by ups and downs. And the Turing Test has only been passed by recent chatbots.

Subsymbolic vs symbolic AI

Over the next decades, two forms of AI research developed: Symbolic and subsymbolic AI. They split AI researchers into two camps and showed different approaches to replicate intelligence. It sparked competition and delivered varying results.

Symbolic AI focuses on the processing of symbols or concepts rather than numerical data. What does that mean? It means that symbolic AI has strict rules that it applies. It relies on carefully crafted guidelines rather than relying on big datasets. This form of AI is very transparent and allows users and developers to quickly see how it came up with the result. A famous example for a symbolic AI product is ELIZA. ELIZA was the first chatbot ever created. But it didn’t work with deep learning algorithms like modern Large Language Models do, rather it responded to certain keywords that had preprogrammed responses. The chatbot was designed by Joseph Weizenbaum in 1966 and is supposed to act like a real therapist. Down below is the original version of ELIZA translated into modern code. As you will notice, the chatbot is still quite limited and doesn’t know how to respond to scenarios which aren’t mentioned in the given keywords.

Here Katharina Schell from the Austrian Press Agency reminisces about Eliza and the first chatbots. Schell studied the impact of AI on journalism and society as a whole in Oxford as part of the Alfred-Geiringer-scholarship. She points out that even in early AI human interaction was a driving factor in the development of applications. The ability for humans to interact with smart machines was always of the utmost interest to AI researchers.

Following the decades after the fields' creation in Dartmouth, symbolic AI like Eliza was all the craze. Nowadays, symbolic AI is known as GOFAI – good old-fashioned – AI in the computer science community. The reason for that is the development of subsymbolic AI. This development happened in the shadows of GOFAI- systems and forms the basis of modern neural networks.
In the audio clip below Professor Andrea E. Rizzoli, who is the director IDSIA of one of Europe’s leading AI labs, explains why symbolic AI was favoured for so long.

Multilayered Networks and the start of ConvNets

Subsymbolic AI is the counterpart of symbolic AI and was technologically quite insignificant for some time meaning it wasn’t on the radar for a lot of researchers. Most of what we now think is AI falls under this category. Subsymbolic AI gave birth to machine learning and deep learning which most of our modern AI algorithms incorporate. But let’s start at the beginning!
The invention of the Perceptron, which Dr. Rizzoli mentioned, was quite groundbreaking. The Perceptron is one of the earliest examples of subsymbolic AI. What makes it so special is that it is modeled after our own brain. Hence the name neural networks. Frank Rosenblatt was the mind behind this first neural network.

This entire model is based on a human output neuron. Output neurons in our brain get input signals through dendrites and then deliver an output through the Axon. This subsequently transmits a signal to for example muscle cells which act upon this signal.

In essence, the Perceptron is an algorithm that classifies an input signal into two groups, either activate or not activate. The single layer Perceptron that Roseblatt invented has basically 3 parts like the human neuron. There is the input signal then there are the weights and biases, which are responsible for calculating the output. This animation shows all the different parts.

Roettger, Diana. (2012). Reconstruction and Visualization of Neuronal Pathways with Applications in Neuroscience.

Human Neuron Picture: ©Röttger Diana (2012).

To further understand how this single-layer neural network works, have a look at the following example:
You want to go swimming. Now let a perceptron make this decision for you. Three factors influence your decision — the weather, the company, and the proximity of the location. In the first instance it will be important wether the input (green bubbles) is active (=1) or inactive (=0).

For context, the weather is either good (=1) ; or not good (=0). It’s the same with the company. There can be company (=1) or no company (=0). The pool can be close or far away.

Now, the active or inactive inputs will be multiplied by the predetermined weights (numbers on lines). These weights are given to reflect the importance of each input. The weather is more important for swimming than company or proximity so the weights for weather is 4 instead of 2. In order for the output to be yes (=1), the sum of all inputs multiplied by weights needs to be greater than the set bias (blue bubble).

The bias (threshold) in this example is -5. If the total weighted input plus the bias doesn’t reach or exceed the activation threshold, the output will be no (=0) — meaning the perceptron decides not to go swimming. If the total exceeds 5 it will be yes (=1).

You can click on the green input bubbles to see the difference in the output.

In order for the Perceptron to make correct decisions and to configurate the weights and bias correctly in the first place, the Perceptron is trained with a learning algorithm.

It is also notable that the "output" of a biological neuron is much richer and more context-dependent! What is significant though is the basic principle which Frank Rosenblatt augmented for computational use. But there are not only output neurons in our brain. There are also billions of other neurons that are hidden and help to make an output decision. So Rosenblatt quickly went from the previous single-layer Perceptron to a multilayered network. This means that he introduced another layer between the input and output. The hidden layer now contains the weights and biases that are responsible for the binary output. This was the precise moment when neural networks became deep, meaning that multiple layers were implemented.

Although this multilayered Perceptron was theorized in the 50s, it took a long time until the first functional multilayered network was developed in 1975. During this period backpropagation was invented. Backpropagation is an algorithm used to train neural networks by adjusting exactly the weights biases that are in the hidden layers. This form of adjusting is still in use today!
But before backpropagation became useful, the developers encountered several problems like the  “Vanishing Gradient Descent”. This is a problem which AI researchers love telling you about. Vanishing gradient descent refers to the problem during training deep neural networks that occured with the use of backpropagation. It's like trying to pass a message down a long line of people, but each person whispers so quietly that the message gets lost before it reaches the start.

A man who was crucial to the development of backpropagation was Yann LeCun. LeCun was a young researcher who was inspired by Rosenblatt’s Perceptron and dedicated his whole life to AI research. He was also the father of Convolutional Neural Networks or ConvNets, a type of neural network that is primarily used for language processing or computer vision tasks.
While these researchers focused on subsymbolic systems like ConvNets, the wider field preferred the good old symbolic systems. Due to described hurdles like “Vanishing Gradient Descent”, critical voices raised skepticism. One of neural networks’ biggest critics was Marvin Minsky, who at one point called this research a waste of money. But as time went on, it got the better of him.

Forecast of a comeback

Oh, and then came the 80s. Synthesizers, shoulder pads, Walkmans, and the first home computers hit the scene. The Cold War loomed but so did a new kind of digital future.  Even though the period from the mid-80s to 90s is described as an AI winter with little progress there were signs that the ice was starting to melt. Maybe it was a innovative spirit that spilled over and slowly started to take hold. There were more real-life applications of AI which started to appear in everyday life. Symbolic AI proved itself useful for tasks like spam recognition in email inboxes. Even some subsymbolic systems were implemented. The previously mentioned LeCun made a successful ConvNet called LeNet that achieved commercial success in post offices, and from the 90s onward it was used to sort out postal codes. In hindsight, people often said that LeCun carried the subsymbolic AI torch through the Dark Ages.

And another big improvement happened in the 90s. You guessed it! Or maybe not but the “Vanishing Gradient Descent” was solved by Jürgen Schmidhuber and Sepp Hochreiter. Jürgen Schmidhuber is a colleague of Andrea E. Rizzoli at the IDSIA and has been a renowned figure in the AI-world ever since.

Yet again this advancement wasn’t revolutionary from the start, but it took some powerful chips to fully embrace the improvement. Symbolic systems were still too far ahead at the time, and one of their greatest triumphs came in 1996, when the best chess player was beaten by a computer. More specifically, when IBM’s Deep Blue defeated Garry Kasparov, the long-standing world chess champion.

Chip advancements and deep learning revolution

In the 2000s and beginning of the 2010s something major happened. The neural networks started to slowly rise up from the ashes. This reiteration represents a strong characteristic of AI history. Numerous times in AI history problems/challenges arose because of a gap between theory and practical implementation. What pulled neural networks out of the ashes this time were advancements in computer chips, more specifically GPUs (graphic processing units) and the combination of different discoveries like Schmidhuber’s and a wider data availability.
The field of machine learning slowly started to establish itself. It became noticeable that these improvements enabled algorithms to get better and better in learning and reasoning. Since AI scientists have always been a competitive bunch, it was only a matter of time before competitions on comparing AI systems emerged. The most famous among those was the ImageNet competition in which researchers competed in developing intelligent systems. Whoever the best image recognition system won.  

In 2012 AlexNet won this competition and marked the beginning of deep learning dominance in computer vision. Before that neural networks had not been dominant in the ImageNet competition. Deep learning is a subset of machine learning and describes the progressively layered neural networks. This means the previously described hidden layers got more extensive – deeper. AlexNet winning this competition also marks the point when markets took notice of this technological evolution. Companies like Google, Apple or Meta started pouring billions into their own AI research. 

The next big step for AI came in 2017 following Google researchers' landmark discovery of transformer architecture. Now the road for modern Chatbots or Large Language Models (LLMs) was paved. This kind of architecture for neural networks is incredibly good at paying attention to where it matters. Through tokenization of the input of words modern chatbots can successfully predict the correct output. In previous multilayered architecture every neuron "attends" every neuron from the previous layer, while in transformer architecture every neuron "attends" every other neuron with a self-attention mechanism. These discoveries ultimately were responsible for the creation of LLMs until, in late 2022 when OpenAI released ChatGPT and the public got a taste of artifical intelligence.

Current state of affairs

All this raises the question of where we are now in AI research. In order to get  a clearer picture of the current state of AI, it might be helpful to get different opinions of people who professionally work in the or around the field of artificial intelligence.

  • Ferdinand Doblhammer

    is an artist that works primarily with machine learning
    algorithms. He also teaches machine learning at the University of Applied Arts Vienna.

  • Katharina Schell

    is Deputy Editor-in-Chief of the Austrian Press Agency and is the representative the editorial team in the APA Group's “Trusted AI” unit. Her work focuses on digital innovation, AI in journalism and communication.

  • Andrea E. Rizzoli

    is the director of IDSIA and professor at SUPSI (University of Applied Sciences of Southern Switzerland). He holds a M.Sc. in Control Engineering (1989) and a Ph.D. in Control Engineering and Informatics (1993).

  • Michael Pucher

    is a Senior Researcher at the Austrian Research Institute for Artificial Intelligence. He also gives numerous computer science courses at the TU Graz.

    • One of the biggest hurdels AI researchers are currently facing is data acquisition and implementation. Without the right data, even the most sophisticated models can’t learn, adapt, or deliver reliable results. This problem has led to the creation of numerous companies which specialize in collecting and preparing data for training neural networks. As Dr. Michael Pucher said in the interview, this has also led to unconventional ways to solve the data shortage.

    A cautious look into the future

    The growing impact of AI makes us think seriously about how we should manage this powerful technology in the future. While it's hard to predict what lies ahead, studying the history of AI shows us that new advancements often build on previous ones. One possible next step could be neurosymbolic AI—a type of AI that combines two major approaches: symbolic AI (which uses rules and logic) and subsymbolic AI (like neural networks that learn from data).

    So, after contemplating the past, it might just be best to end with Andrea E. Rizzoli taking a cautious a look into the future.

    Credits

    Sources Pictures:
    Minsky, M. (1956). Dartmouth Conference 1956
    Brockmeier, E. K. (2021). The world’s first general purpose computer turns 75. Penn Today. Verfügbar unter https://penntoday.upenn.edu/news/worlds-first-general-purpose-computer-turns-75
    Röttger, D. (2012). Reconstruction and Visualization of Neuronal Pathways with Applications in Neuroscience. Dissertation, Universität Koblenz-Landau, Koblenz.
    ©Elias Rudigier

    Sources Videos, Audios & Animations:
    © Elias Rudigier

    Sources Text:
    SmythOS. (2024). Symbolic AI vs. Machine Learning: A Comprehensive Guide. Verfügbar unter https://smythos.com/ai-agents/ai-tutorials/symbolic-ai-vs-machine-learning/
    Wikipedia. (2025). Joseph Weizenbaum. Verfügbar unter https://de.wikipedia.org/wiki/Joseph_Weizenbaum
    Mitchell, Melanie (2020): Artificial intelligence. A guide for thinking humans. Published in paperback. London: Pelican an imprint of Penguin Books (A Pelican book).
    First Principles of Computer Vision. (2022). Perceptron | Neural Networks [Videobeitrag]. Verfügbar unter https://www.youtube.com/watch?v=OFbnpY_k7js
    3Blue1Brown. (2018). Aber was ist ein neuronales Netz? | Kapitel 1, Deep Learning [Videobeitrag]. Verfügbar unter https://www.youtube.com/watch?v=aircAruvnKk