I realised the Interface Theory of Perception ^[1] goes beyond us, reaches to everything we build.

There is an unnoticed selection we impose on our technology, selection of an alignment with our interfaces. We are testing our technology based of off our perceptual interfaces. Just because they are ‘working’ (which itself is a function that we are attributing) we cannot say that they are veridical. We evaluate our technology by whether it satisfies our interfaces not by whether it is veridical.

Here are couple of numbers that abide a rule: ${0,2,4,6,?}$. If you say the next number (in place of $?$)is $8$, you would be right. The next, $10$, you would be right. But my series do not abide your $+2$ scheme, it is just enough that the next number to be larger than before. If you cannot test this, you are wrong but it works. Science obviously is a more meticuluos and clever craft, but it may not that we do not test it but we simply can’t. It comes down to the same simple idea, you cannot prove things, you can disprove them. Thus, if something works (sadly) it does not prove the theory underneath.

Everything we discover and build are made out off our interfaces, and through them they are our extensions. One of our most glorious discovery, neural networks are not immune to it either. Whatever we train our models with it is impossible to evade Interface Theory of Perception.

The training and test data are both susceptible to ITP. The data we feed to them are a direct product of our interfaces. The geometry we embed is an interface. Euclidean space which has an incredible adaptive power, is our construct. Relativity breaks the idea of an Euclidean space, the space itself seems to bend and deform. The colors we see are a small subset of the wavelengths. We are supplying RGB or black and white pixel values to our models. Even the discreteness is an assumption that our interfaces lean on. Thus the struggle of AI with ITP becomes twofold, on top of ours, the perceptual limitations of the model.

The loss functions are also susceptible to ITP, they are actually the inducer of it. The loss function of any form acts as an evolutionary pressure. The internal state of the model is a parallel to genome and the predictions are parallel to the phenotype. In that regard, loss functions induce guided mutations to internal state, armed with the knowledge of gradients, select which internal state survives. It does not let the model try other internal states which at the moment are not the best to take but can open up different adaptations. The error is existential from the perspective of the internal state. The survival is parallel to an organism adapting and surviving w.r.t. given error.

The survival of the internal state solely is determined by the predictions. ITP argues that you don’t have to truly comprehend the underlying fabric of the reality, finding clever shortcuts and patterns that serve your adaptation is enough and encouraged. If they collide, evolution rewards adaptive behavior. There is nothing different when it comes to training. This encouragement to bend reality to fit needs are incentivising our models to take shortcuts or match patterns too. The same pressure that rids us of veridical perception, trains them. This pressure adds to their own perceptual limitations as a driving force. Now with the weakened pressure to be veridical, the perception can diverge more freely to meet the need to ‘adapt’.

We managed to get intelligent despite ITP. It raises the question “Does not ITP prevent becoming intelligent then?”. If ITP tends to produce intelligence (as we know it), we should be seeing it everywhere. For example, dragonflies are among the best predators on Earth. They have a hunting success rate of up to 95%, higher then the king of the jungle (25%) or great whale sharks (50%). This is called predictive interception . When you think about it, when you are hunting or being hunted down, it is not a decision you make. You don’t sit down and think where you should maneuver next, you just go for it. It may not require intelligence. Just like experienced firefighters had to make rapid decisions on the fire ground ^[2] there is not time think it through. Maybe pattern matching is encouraged by ITP. ITP itself may even be one of the mechanisms to be able to match patterns. Is it because it is very hard to develop intelligence or is it not that vital? Dragonfly example shows clearly that leverage against preys or predators does wonders. Why then we don’t see the engine that identifies leverage (* intelligence*) everywhere? The scarcity of intelligence implies that us becoming intelligent despite barriers ITP poses us, is not an easy feat and is not encouraged as well. Whatever the reason, it is also imposed on training neural networks.

We don’t yet know, what other underlying pressures we had, maybe there is a signal that we cannot reproduce yet that prioritises or emphasises intelligence. This just means evolutionary interventions (loss functions or generally error minimisation or reward maximisation to put the pressure) may not prioritise or effectively select for veridical perception. We are intelligent but not veridical. Our intelligence enables us to know that we are not veridical. Maybe contrarily, because our perceptions are not veridical, we had to become intelligent. This deserves its own post 😉.

Take Alpha Zero , the neural network which learns playing from scratch by playing against itself. AlphaZero internally predicts 2 things through a CNN, it emits a vector $p$, probabilities of moves and a value $v$ the probability of the current player to win. CNN’s are hierarchical feature/pattern recognizers. The deeper you go, the more complex and hierarchical patterns it can extract, but they lack searching capability. AlphaZero obtains this capability through the embedded Monte-Carlo Tree Search . Through the search, it accumulates necessary experience to learn patterns to look for. So by design, through seeing certain amounts of possible plays from a point in the game, it tries to extract what is important and what leads a victory. It effectively is training its situational awareness through experiences. At the end, it manages to best every human player. The caveat though is that it still searches while playing. It is performing a MCTS and in each attempt, it evaluates that state with its CNN. It would be interesting if we removed the MCTS from AlphaZero and let it play just by its CNN to see how much it relies on pattern recognition.

So, fundamentally, if the search capability is not embedded in the model’s inner workings, we cannot expect neural networks to reason by sitting down and evaluating every possible set of actions. It lacks the machinery, thus we don’t train them for it. We rather train them to identify a pattern and then move on with the set of actions that the identified pattern puts on the table. Go is not a time-critical game and does not have time-pressure as a fire does, thus there is room for searching through possible plays. Regardless of the task at hand, pure analytical reasoning cannot be enforced if the machinery to explore is not in place, the model has to deal with it through pattern recognition. I wonder how much of what we think as hard problems can be solved like that.

We do take intelligence partly to be able to see the patterns. So maybe they are not alternatives but parts of the same spectrum. Pattern matching is required for intelligence to be able to extract pattern to learn and exploit them.

Even though models seem to nail the image generation tasks, close inspections to the training data shows that they cannot fully capture the full diversity of the distribution . With a diverse set of queries, only a small subset of the training data seems to be incorporated to generations. In other words, when you ask the AI to draw elephants in all sorts of different circumstances, all the elephants it generated closely resembles a small subset of the training data. It only draws them from specific angles for example. It may be the case that some cases in the training data are easier to extract elephantness from. It shows however that what it truly means to be an elephant even for us, cannot be extracted effectively. Looking at the results, it may not be a necessity either. The ‘evolutionary’ pressure around the training was not enough or even maybe preventing the model to truly comprehend an elephant but rather a specific adaptive perception that is enough to satisfy the loss function. Maybe nobody ever complains about structurally similar elephants…

We distinguish elephants with its gray colored and bald skin. If we would see a mammoth for example, we would immediately think something is wrong or it is not totally the same as the elephant we know. In fact, the color and the hair are our own constructs. For bacteria, a hair can be mistaken with a flagellum . The hair of the mammoth can be mind bending for a bacterium, the mammoth appears to not use its ‘flagellum’ to move despite its abundance.

Our perception is guided by the range and frequency , what we have seen before and how frequently; and the functionality (to us) of what perceive. What makes us think that neural networks are immune to this? They are trained on them and possibly being exposed to new ones in their turn.

The major question is whether a neural network truly needs to comprehend anything to do the job. They inherit not only our blindspots but potentially introducing their own. Treating their outputs as veridical is the same problem that ITP warns us about, and it is two-fold. AlphaZero managed to do the job but not because ITP didn’t handicap its perceptions, rather the MCTS enabled it to discover alternatives. Even though the CNN was aggressively optimising for adaptation, MCTS broke it through the locked in path forced by the evolutionary pressure. In case models equipped with explicit search capabilities (or in AlphaZero’s case, it is mandatory), even if ITP blinds them, search may be freeing them. Even though the search is subject to the same perceptual frame, it has a potential to include alternative experiences. It may show our model a single black swan to let it disprove its “all swans are white” hypothesis. Search can saturate the experience space and enable ‘better’ pattern matching. Thus, the result can be considered as more ‘thought through’. In case the search is not explicit and forced, since the behavior is driven by adaptive pressures, the model tends to skip searching. In those cases, the performance is determined by where query at hand lands within the trainig distribution. Near the boundaries of the training distribution, the behavior will start to break down, become undefined. “Interpolation to the rescue” is not an antidote, it didn’t work for australian jewel beetles, what makes us be so sure that fast-forwarding millions of years of evolution will do the job?

How do we know exactly if our model is trying to mate with a beer bottle or not?

I transformed the ITP paper into a podcast by the way. I liked the content a lot and I recommend you check it out.