Day 6 - Commercializing Neuromorphic-X - Kynan Eng and Jim Lewis

today's authors: Muhammad Aitsam & Eleni Nisioti, edits by Tobi Delbruck

Waking up to our first Saturday at Capocaccia and moving to a session different from others: commercializing our work. 

We need to set aside our academic glasses and look at the world around us as a society of humans and an overall ecosystem that we want to help with our products. 

Florian Engert set the stage by saying that commercialization often has two over-arching objectives: 

• Making useful products 
• Convincing people that we are making useful products by creating false needs or impressions of them. One example of this would be optimizing the search algorithms of websites such as YouTube or maximizing user clicks
  

Florian felt the need to clarify that we need to keep this lecture solely considering the first. This is an important message of our community, as the landing page of our workshop exemplifies, quoting:  "The mission of the CapoCaccia Workshops for Neuromorphic Intelligence is to understand the principles of biological intelligence and apply this knowledge in technologies, for the good of all mankind".

The first speaker was Kynan Eng, a co-founder and CEO at iniVation. IniVation sprang up at the Institute of Neuroinformatics (INI) in Zurich, where Tobi Delbrück was working with Kynan and others, on neuromorphic hardware.

The innovation that drove the creation of the company was a vision sensor that the lab developed, the Dynamic Vision Sensor (DVS), and gave birth to the event camera, one of the most emblematic neuromorphic hardware. Differently from classical frame-based cameras, each pixel in an event camera generates spike events that capture intensity changes continuously in time. Events are asynchronous and sparse and the benefits that come with this design are high pixel bandwidth and wide dynamic range, which enables capturing good quality images under noisy conditions while keeping latency low.

Kynan then shared his experience of moving from discovering this design to commercializing it. A main message in all our discussions is that it is often not easy to anticipate which changes will make the product more successful. For example, once they had the thing on chip, Tobi realized that adding a USB interface and on-chip digital bias generator made a big difference for its use. You'd think that it does not make a big change but it actually made people understand the function better. The takeaway message is that packaging is important.

Kynan said that putting something in the box and having someone buy it felt very satisfying. What mattered was not the amount of money he got but the fact that he created something of value to someone, a direct feedback that many of us in science may be missing.

They made an incubator called inLabs from which two spin-off companies appeared: iniVation and SynSense, both in mid 2010s. IniVation got investment from Samsung and Huawei (and we will see later why this was a good choice).

This process happened a bit outside of the start-up scene we are very familiar with. They did not directly look for investors, the fundamental reason being that they were not sure that they were solving a problem in the real world.

The audience then initiated a discussion on an aspect that many found important: we often tend to think about the end-user with this hardware but ensuring that the researcher developing solutions on it will not have a hard time. With many products of the past people in the workshop felt that it was difficult to prototype ideas due to lack of appropriate software and support. 

Christian Pele, in the audience, did not feel represented by this opinion: he thought that the actual problem of development on this hardware is the lack of a clear application. If we had that, then solutions would be found despite the difficulties.

The intuition of Kynan and Rodney Douglas was that we underestimate the challenge of adding documentation and support to a product. Documentation appears about 6 to 7 years after you make the chip and no one funds you for doing it.

Back to the speaker to talk about the applications of the chip: it is not so much consumer electronics that will benefit (although it may have some uses on devices like future spectacles), it is industrial applications in large engineering companies that mostly make use of it. A successful application is its use for monitoring trains. There is a start-up that realized its potential in providing accurate localization that will complement the one provided by the GPS, which leads to an uncertainty of about 100 meters.  By using vision based localization the accuracy is greatly improved so that the train maintenance can locate the rail faults much more quickly and cheaply.

Reducing this uncertainty has two major benefits in terms of cost: you can pack more trains in the network and you can reduce the cost of repairing the train lines. About 1.4 billions a year are spent on finding problems on the line and this camera enables locating them automatically without requiring engineering to manually go through these 100 meters.

Then Chiara Bartolozzi said that she feels she does not have a large incentive to introduce the event camera in her robotics projects since she would not have libraries like OpenCV and does not have a benefit to expect.

Tobi then shared his experience that there is another field that immediately saw the potential of the event camera: computer vision, a field that is very close to the industry. Many papers were published at CVPR (with people often misunderstanding the technology and its specs) with the event camera. The point is that opening up to other disciplines can help you discover benefits that you would not see otherwise.

Then Tobi mentioned a point that was bound to be repeated in the talk: what matters the most in these kinds of ventures is identifying potential customers, and understanding how they convert from potential to actual customers (this journey is termed the "sales funnel"), prioritize among them and plan according to your priorities.

We then moved back to the discussion of which application would drive the key camera. Jim Lewis emphasized that the key is discovering a problem that only the event camera can solve. Simply trying to improve upon existing solutions will not get you far.

Giacomo then shared the confession of Davide who is using the event camera on a drone and cannot justify to himself why: it takes a lot of energy and latency is not very important as the drone inertial latency is still about 30ms which is slow enough that frame cameras can address it. 

However this situation may still only apply on the well-controlled environments of arena / drone racing flight where lighting is optimized, and might not be true for real world environments like forests, building, and under mixtures of sunlight and shadow. 

This raised some comments by Yulia Sandamirskaya that it is important, to be honest about our motives when using a certain technology. 

Andre van Schaik  said that, in his paper using an event-based camera for Space Domain Awareness (SDA), the use of this technology was not critical, but that event cameras provide additional information that frame cameras cannot, e.g. about tumbling and glinting, both of which are important to SDA customers.

Kynan then explained that he does not see the DVS sensor as a camera for smartphones. This was just how they used it and it might not even be attractive for smartphones. But this does not explain why Sony and Omnivision (two of the biggest 3 CMOS Image Sensor (CIS) companies are investing in development of HVS (Hybrid Vision Sensors), if they do not see its potential for future mass production.

Then Lewis emphasized that it is important to recognize which one of our potential customers will be the ones to succeed. This anecdote shows that they can be challenging: one of inivation's customers wanted to use the camera for high-speed laser scanning. The application looked ideal because they managed to improve performance compared to existing solutions. Yet they hit a wall when tried to sell their solution: despite having a cheaper solution, they faced a lazy crowd unwilling to abandon its existing expensive solutions since people were already paying for them (this is a rather niche market).

We then talked into an illuminating discussion on the role of Venture Capitalists (VCs). Lewis said that they were very reluctant to raise money from VCs because in his experience they overvalue their value as advisors. He said that companies should insist that the only thing they want is the money, not the advice. He also said that this is the least pleasant part of his job to talk to VCs. His main problem is that they just care about the money and do not contribute towards improving the product. 

He also said that the scenery of VCs is slowly going through a broken phase. 15 years ago it would have been THE place to raise money. Today there is another option: strategic funding by customers of the technologies. He finds this way much more useful, as these people have expert knowledge and are invested into improving the product.

For people who want to read a bit deeper into VCs and their relationship with engineers, Tobi shared a well-known Spectrum article: Tredennick, N., and B. Shimamoto. “An Engineer’s View of Venture Capitalists.” IEEE Spectrum 38, no. 9 (October 1, 2001): 67–70. 

What about EU or other public or national science funds? Kynan said that he prefers to only use those funds at the very initial phase, like the first year. Such funds are very useful and the DVS and DAVIS cameras would not have existed without the CAVIAR, VISUALISE, and SEEBETTER EU projects, but it's better for companies to not use them as crutches and actually discover your market potential.  Jim Lewis said that these projects are very dilutive of the money and tend to distract from the startups goal of developing a product for a particular market.  Jim and Kynan both pointed out repeatedly that most companies start with a goal but end up pivoting to something completely different. Tobi adds here that a famous example is Intel, which started as a memory company and were only brought to processors with the famous development of a calculator chip (which incidentally was done by Federico Faggin, the inventor of the silicon gate FET and a founder of Synaptics, which is one of the first silicon valley "neuromorphic" or at least neural network companies, which itself pivoted from NNs to being a worldwide leader in touch interface/touchpad devices).

Andrea Chiba then shared a point: everything is hard if you are bad at it. This means that maybe hiring the right people for the right job will make the whole venture easier. For example, hiring business specialists with MBAs can help you navigate the challenge of identifying the right customer.

Kynan said that he has a strong opinion on this: at the beginning you can only afford to start with a few people. And he thinks it is easier to train an engineer to do good business rather than the reverse. He thinks that the challenges people face are due to the problem being inherently diffciult and not a bad choice of people.

Coming back from the break, we have Tim Gardner, one of the founding employees of Neuralink. Timothy introduced himself as a systems neuroscientist and physicist and said that Neuralink grew from 7 founders to 400 hundred people and that it has spent about 600M$ to date (100M from Musk and another 500M from subsequent investors).

He talked about the first success of Neuralink: inserting an implant in the M1 motor cortex of a quadriplegic patient with damage in his spinal cord and enabling them to play real time video games. 

Tim's main message is that we should think about the long-term. He disagrees with the vision of Elon Musk: humans are in danger of being replaced by AI, so let's equip them with brain implants to make them as good. While it is natural to try to increase the market of devices like Neuralink,we could discover better reasons in the future.

Neuralink was not so much aiming at revolutionizing brain-computer interfaces but at making a self-contained, easy-to-use product. Because recording over multiple channels is very costly, using neuromorphic chip design ideas was important and the chip design lead at NeuraLink was neuromorphic engineer Paul Merolla from Kwabena Boahen's lab and later IBM Almaden team of Dharmendra Modha which developed TrueNorth and now NorthPole. NeuraLink's electrode interface chip uses event-based detection and all is done digitally. 

Another potential future use case is for treating severe depression by implanting electrodes in multiple brain regions and stimulating them. Unsurprisingly these kinds of malfunctions are very specific to the individual, which makes finding the precise spatiotemporal signature of the person very important.

Timothy then asked the audience how they think neuromorphic-X will affect the next 100 years. People mentioned energy efficiency, replacing silicon, and robots being used more widely.

Then people wondered how you best approach funding sources. How do you attach to the first node in a community? Lewis said that he would not worry about that: the community is not risk-averse and if you have a good idea they will even find you. Tobi quoted something he learned from Terry Sejnowski (who told Tobi he learned it from his interactions with Washington): "If you want advice ask for money. if you want money ask for advice".

Rodney asked:  "why would an engineer shift in another direction once they are on the path that they now know has potential"? Timothy replied that for many engineers it is the first phase of the venture that is most exciting for them. After it, the engineering process crystallizes into more standard industrial development and production, which is not their expertise or interest.

The floor then switched back to Kynan who took us on a walkthrough of how political developments affect commercialization.

Neuromorphic computing started during a period when China was very unstable, so it mostly grew in the US. China instead developed its communications industry and this turned out to be problematic: when the whole situation with spying concerns and the US becoming increasingly closed to China, people initially thought it will pass. But actually, the temperature is going up. Today you can either invest in China or the US.

This has made their work even more challenging: every time there is an announcement from the US government suppliers give in to panic and want to ensure that all laws are followed. Lewis said that there is no middle path: you just stay away from China, period, the situation is too volatile to justify any engagement.

The audience then shared stories of companies being forced to close their markets by the government (e.g. the Manchester start-up case) and universities not accepting students from certain countries.

A game changer was the introduction of entity lists: the US government holds lists of companies getting supplies from US and enforces rules on them regardless of where the companies are based.

The last topic of discussion was patents. Interestingly, patents are a way to navigate around closed markets. You just need to register your patent in China. This is rather a loophole that won't probably last long.

Are patents of any use? Tobi said that he must have filed about 50 patents and only two of them were licensed; one was the bump circuit patent the resulted in about 20k$ and the other was the crucial basic DVS pixel patent that paid the university about $2M license income.

Lewis said that in the industry it's a bit different: about only one-third of them prove useful and the others are to add to a stack for either investment, acquisition, defense or trading.

Then Tobi wanted to know about the benefits of exclusive versus non-exclusive licensing. The former means that you are the only one allowed to develop the technology. Tobi said that PhD students who form a startup demand an exclusive with the justification that they cannot raise money without it. Lewis agreed that doing this may make you lazy and work against you. For the industry exclusive makes sense though cause you want to lock your customer base.

We closed a discussion with Tobi sharing his experience with Samsung: was it a good idea in the end? Tobi spent an entire CCNW preparing a proposal for the Neuromorphic Processor Project which ultimately resulted within Samsung in their first Neural Processing Unit (NPU). (they use it for DNN inference for example for photo enhancement, video processing, face ID, and even recognizing your fingerprint (which does not require NPU) requires processing because it updates model each time you unlock). Preparing a proposal that both were happy with was at first challenging because Tobi and Shih-Chii wanted SNNs and they wanted standard ANNs (conventional artificial neural networks). In the end, the first generation Samsung NPU used weight sparsity.  Tobi's impression from the whole relationship was that Samsung funded this venture because they wanted to get the ideas and that getting the industry involved was good for the project and good for the participants to ground them much better in reality of mass production silicon applications and requirements.

Kynan closed the discussion saying that Samsung is certainly an interesting company. It has risen to about 25% of the South Korean market and cares much about secrecy. Rumors say it has offices that don't exist on the map where you enter without a laptop and phone and swim in secret pools.

While it sounds intriguing, we cannot be jealous when looking at our own current offices:

Photo credit: Yves Fregnac

After a day packed with discussions, lots of concrete project work, a good sport break, and a fun workshop dinner at a beautiful, the evening took us on more adventures. An entire busload  hit the bars in Alghero, bouncing from one place to another, while the others headed back to the hotel to sleep. Finally when Tobi woke at about 6am he spotted the clubbers who had just returned and headed out for  refreshing swim before breakfast and bed.

Whether we were out partying or resting up, we all ended the day with some good stories to share!

A glimpse of today's sporting activities.