Day 8 - Robotics with neuromorphic brains - Yulia Sandamirskaya, Catherine Schuman, Guido De croon, Paul Verschure

 

today's authors: Eleni Nisioti, Muhammad Aitsam

Writing today's blog after lunch

Today's session kicked off with Yulia Sandamiskaya saying that robotics has not been as major a theme as it deserves in the lectures of the conference. She followed up with a bold statement: "All neuroscientists should do robotics".  This is motivated by the idea that, by taking into account the dynamics of robots that closely model reality, we can create neuroscientific models that are more true to the biology they try to represent.

We, then, talked about a thought experiment from the work of Braitenberg (a major read according to many in the room) that exemplifies how you can reproduce real-like behavior with a minimal set of mechanisms (Yulia called this a physicist's approach)

Picture a robot with two wheels that can turn left and right (like a tank), two motors controlling them and two sensors on its head that only receive the amount of light. There is a source of light on the left of the robot.

 

The simplest brain you can think of is two connections from the left sensor to the left motor and from the right sensor to the right motor. But this robot would turn right when its left sensor sees light, thus moving away from the light. If you instead connect the wires in an x-shape, it will move towards the light, which is a much more sensible behavior for a biological organism. This is just one of the 10 designs that this book studied to create different biological-inspired mechanisms. The major point is that both embodiment and environment matter for the design of these behaviors.

People from the audience mentioned Grey Walter's tortoises from the 80s as a similar-spirited example.

But these robots could not switch between different behaviours, which is something you want to do if you want to model cognition. This brings us to Yulia's question: why should neuroscientists care robotics?

She started with an anecdote: A mathematician, a biologist and a physicist want to understand how horses run fast in order to bet at races more successfully. The mathematician studied the statistics of horses to build predictive models. The biologist studied their body to understand how running works. The physicist spent a lot of time on calculation and came back saying: I have the formula but it is true only for spherical horses in a vacuum.

The point is: that neuroscientists coming from a physics background tend to make oversimplifications that someone studying the real world could not afford.

She, then, reminded us of a distinction made yesterday by Rodolphe: computation versus control. The former is about input-output relationships while control puts more emphasis on the feedback. Yulia wants to talk about how you can go from embodiment to condition by using feedback control.

What insights can you get about neurons once you think in terms of robots? One is that the sensory input is always there, so you need to decide why and where you need to do something. So deciding what to attend to becomes important, while non-embodied approaches often assume that a supervisor sets the tempo.

She then described how the winner-takes-all architecture can model how neurons choose which action to take:  There is an area of neurons each one being sensitive to some signals (for example some neurons sees blue and some red). Then neurons are connected in such a way so that the ones that are more similar positively interact with others while the dissimilar ones inhibit their neighbors. This will lead to a bump on a single neuron, thus making the agent go for one option (such as a red apple).

There are equations that describe the dynamics of this architecture, studied by Amari. They show that the decisions correspond to attractors.

She, then, mentioned the theory of intentionality (agents have ideas for actions that have not happened yet) and the concept of a perceptual action (I take the actions that make my representations match the real world). The intention is represented in a population of neurons.

 But how do I move out of an attractor once I have satisfied my intention? This is called conditional satisfaction. For each action I know what to anticipate and, once I'm done, something inhibit my behaviour.

She, then, mentioned that agents encode sequences of actions that fall into two types: fixed habits and rule-based sequences for adaptation. Learning is important in robotics to enable this adaptation.

The talk ended with some discussion on the usefulness of thinking in terms of attractors once you adopt a more dynamic perspective that thinks about noise and transitions between different attractors.

The next speaker was Guido de Croon from the Delft University of Technology who started by saying that, in robotics, we often design the connections of the brain ourselves but he is more interested in how artificial evolution can solve tasks with minimal cognition.

He continued with a critique of the mainstream approach in robotics: people designing robots tend to anthropomorphize robots, equipping them with the ability to form maps, use the GPS and plan their navigation. They do not think about how simple robots can leverage their morphology to solve tasks such as avoiding walls.

This raised some reactions from the crowd that thought that are many exceptions to this approach. Croon said that this is very refreshing because usually, he faces the opposite reaction in mainstream venues.

He, then, moved to drones. The classical approach of putting a lot of hardware on them for navigation won't work due to weight and memory constraints. Instead, he uses drones that weigh only 16 grams. It has two motors to stir its wings.

He then explained how this tiny robot can estimate distances in order to land successfully. Imagine the robot flying and then starting to move downwards. Then as it goes down items beneath it look bigger and bigger: the flow field is expanding. Using this information, you can compute the divergence constant which is equal to velocity over height. Thus, you can compute distance by knowing the divergence constant and the velocity.

Yet there was a problem with Guido's model: once the drone reached close to the ground it would become unstable and never land. He realized and proved mathematically that this is bound to happen once the distance get's close to 0 and you are using a simple P-controller that cannot handle infinity. How do flies do it?

They just detect the arising instability and use it to detect distance, as the point at which the instability arises is directed related to the remaining distance.

He then said that he prefers using neuromorphic hardware with spiking neurons, as it enables processing high-dimensional data on small platforms.

As an example of learning, he mentioned bees that learn to associate flower colors and shapes to distances and use them to navigate. This is not an innate behavior. Evolution pre-wires many of the connections but there are significant variations among individuals that necessitate learning during lifetime. 

How can we implement such learning mechanisms with simple robots?

He said that a major challenge is the reality gap: there is a difference between simulation and reality and normally you cannot afford to train on reality due to computational restrictions. 

There are different solutions to this:

• domain randomization, where we add noise during training. This makes you robust but also sub-optimal in case the environment is not noisy.
• having an adaptive mechanism can make you both robust and optimal
• or you can have a control mechanism. How do you separate this from the above? If the synaptic weights change then we talk about adaptation/learning, otherwise its control/calibration
• self-supervised learning, which is Guido's preferred approach as it is much more sample efficient than reinforcement learning

We return from the break with a more conceptual theme: Catherine (Katie) Schuman from the University of Tennessee will talk to us about her experience of designing robots with neuromorphic hardware.

 

Catherine began by making a distinction between a monolithic and a modular approach.

Monolithic versus modular design

The first robot they build followed a monolithic approach. This means that they had a Spiking Neural Network where they defined the inputs (LIDAR) and outputs (motor). The robot had to navigate in unfamiliar environments to avoid obstacles and they evolved the SNN with artificial evolution.

They then wanted to switch to more challenging tasks, that required exploration and targeting. The first caveat was that the robot had to navigate to orange traffic cones so it ended up following people wearing the typical orange shorts of University of Tenneesee. 

The next challenge was that evolution could not find solutions that solve both problems. So they switched to a modular architecture: some sensory modules, followed by processing modules (eg for object recognition, and planning) and then modules to control the actuators.

The general idea of the modular approach is that a human with expert knowledge thinks of how to separate the hardware into modules. 

She then drew a table with cons and pros of the two approaches and prompted the crowd to fill it: 

                                      

Some points were that monolithic is easier to implement but is hard to interpret.

A point raised by people with machine learning backgrounds is that there is an intermediate solution: you can have a modular architecture that emerges without a human designing for it. Also, we may need to differentiate between two instead of one axes: you can have a modular/monolithic approach in terms of both the optimization process and in terms of the final solution. What is more, a modular architecture does not necessarily lead to modular function.

This sparked some discussion on why people in this field are still using traditional methods like OpenCV and not deep learning. Is it inertia or do these methods for learning modularity not work?

Another point was that your choices regarding modules depend a lot on the hardware: it usually has different modules that can cause trouble when they are heterogeneous.

In terms of resource efficiency, monolithic networks can be more efficient because you can reuse components. For examples, an input neuron can be useful for computation when it is not receiving input. This raised the point that compilers for neuromorphic hardware are important: they could optimize the reuse of components.

Catherine concluded by saying that there is a spectrum between modular and monolithic solutions and that we need to keep practising both. She also said that we should focus more on the hierarchical structures that biological evolution has found.

The last speaker for today was  Paul Verschure from the Donders Institute for Brain, Cognition and Behaviour who started with a conceptual diagram of brain organization that he has been working on for the last 30 years:

                                      

 From bottom to go we go to increased levels of abstraction: closest to the world there is:

• the reactive layer that implements reflexes to environmental inputs that do not require computation
• the adaptive layer that is about shaping the actions and implementing the classical conditioning we will focus on
• the contextual layer that implements higher-level reasoning such as planning and long-term memory
  

The figure has another dimension: from left to right, all rows go from states of the world to states of the self to acting based on these states.

The difference between the orange and the green is that in the latter states are interpreted. For example, a thermostat that senes a change in this environmnet belongs to orange.

He then gave a detailed description of classical conditioning, the mechanism used to adapt on the green level.

Discovered by Pavlov in the 30s, this framework explains how animals learn to associate signals. What Pavlov observed was that, when a dog was presented with food and a sound went off very close to the feeding time, the dog learned to associate feeding with that sound and salivated, even in the absence of food, when that sound was heard.

Illustration of Pavlov's experiment

Food presence is the unconditional stimulus and the sound is the conditional one. Classical conditioning explains how the dog is conditioned to react to the uncoditional stimulus.

Paul calls this action counterfactual because it does not correspond to something in the sensory input of the agent but to its anticipation of the future. We can also call it predictive.

He then explained how a network can implement this conditioning that you can read about in his paper from 2001 "A real-time model of the cerebellar circuitry underlying classical conditioning: A combined simulation and robotics study".

Paul's network for implementing classical conditioning

Conditioning requires that the conditional and unconditional stimulus need to be well-timed.

He then talked about an experiment that surprised him: a moving robot needs to follow a turn without bumping on the wall. In this case, you can see the visual input as the unconditional stimulus and the collision as the conditional one. In order to learn how to not collide the robot needs to turn when it sees the unconditional stimulus.

They saw that the robot spiked both as a reflex (for the conditional input) and the unconditional one. They hypothesized that the second bump is not necessary but then saw that even when they replace the reflex with an acquired response, the unconditional bump still appears.

This made the form the hypothesis that the controller is responding to the error it expects and not to the signal itself. In this way it is not throwing away the feedback controller and can adjust quickly if something changes in the situation.

We then moved back to the first diagram and talked a bit more about the red, higher-level area. He said that simple organisms may not have this area, but organisms that appeared after the Cambrian Explosion do. 

In the higher layer decisions are not based on signals from the environment but on a model that the brain has built of its environment. This model is in the hippocampus which gives information to the prefrontal cortex. The thalamus is a critical hub that connects the reactive to the adaptive layer.

He wrapped up that the roadmap for neumorphic engineering that wants to understand how biological brains work is to build the system that implements all components of this schematic (so the missing upper layer).

"Was this enough robotics?" Julia asked the audience at the end. Perhaps the session deviated a bit from its initial intention. But in any case, we expect that robotics will keep acquiring importance in future CCNW years.

*****