Summary: There’s a three way technology race to bring faster, easier, cheaper, and smarter AI. High Performance Computing is available today but so are new commercial versions of actual Quantum computers and Neuromorphic Spiking Neural Nets. These two new entrants are going to revolutionize AI and deep learning starting now.
AI and Deep Learning has a Problem - Three Actually.
Time: The amount of time needed to train a deep net like a CNN or an RNN can be weeks. This doesn’t count the weeks or months in defining the problem and the iterative successes and failures in programming deep nets before they reach the required performance thresholds.
Cost: Weeks of continuous compute time on hundreds of GPUs is expensive. Renting 800 GPUs from Amazon’s cloud computing service for just a week would cost around $120,000 at list price. This doesn’t begin to account for the manpower costs. Spinning up an AI project can mean months or a year or more of the highest cost talent.
Data: In many cases the unavailability of labeled data in sufficient quantity simply makes the project a non-starter. There are lots of good ideas out there that go unexplored since it’s clear that the training data just isn’t going to be available at an affordable price.
So where we have made good advancements in the commercial realm mostly involving image processing or text and speech recognition, as often as not these startups have exploited the work of Google, IBM, Microsoft and others that have made many good image and speech models available by API.
A Three-Way Race for the Future of AI
If you’re following the field you’ll see that we’ve sprinted out ahead using CNNs and RNNs but that progress beyond these applications is only now emerging. The next wave of progress will come from Generative Adversarial Nets (GANs) and Reinforcement Learning, with some help thrown in from Question Answering Machines (QAMs) like Watson. There’s a good summary of this in our recent article “The Three Ages of AI – Figuring Out Where We Are”.
Here’s the most frequently expressed vision of how we move forward. Use what we know; that’s increasingly complex deep neural nets with different architectures from the now common CNNs and RNNs. Just make them run faster.
Actually though the future may be quite different. What we see shaping up is a three-way race for the future of AI based on completely different technologies. Those are:
One of these, high performance computing is the major focus of what we’re seeing today. There’s a flat out race among chip makers plus some less likely non-hardware folks like Google to build chips designed to accelerate deep learning. The other two, neuromorphic, also known as spiking NNs, plus quantum computing always seemed that they were years away. The fact is however that there are commercial neuromorphic chips and also quantum computers in use today in operational machine learning roles.
Depending on how you like your metaphor this is either the tip of the iceberg or the camel’s nose under the tent. Hot or cold, both of these new technologies are going to disrupt what looked like a straight path to artificial intelligence, but disrupt in a good way.
High Performance Computing (HPC)
The path that everyone has been paying most attention to is high performance computing. Stick to the Deep Neural Net architectures that we know, just make them faster and easier to access.
Basically that’s meant two things: better general purpose environments like TensorFlow, and greater utilization of GPUs and FPGAs in larger and larger data centers, with the promise of even more specialized chips not too far away.
The new business model in AI is ‘open source’. During the first six months of 2016, literally just 12 months ago, essentially every major player in AI made their AI platforms open source. These are all competitors with enormous investments in data centers, cloud services, and the IP of AI. The strategy behind open source is simple. He with the most users (platform adopters) wins.
While Intel, Nvidia, and other traditional chip makers were rushing to capitalize on the new demand for GPUs, others like Google and Microsoft were swimming in completely new waters by developing proprietary chips of their own that make their own deep learning platforms a little faster or a little more desirable than others.
Google certainly threw a good punch with TensorFlow as its powerful, general purpose solution combined with their newly announced proprietary chips, the TPU (Tensor Processing Unit).
Microsoft has been touting its use of non-proprietary FPGAs and just released a major 2.0 upgrade of its Cognitive Toolkit (CNTK). CNTK offers a Java API for direct integration for Spark among others. It supports code written in Keras, a major competitor and essentially a front end for TensorFlow making it easy for users to migrate away from Google. CNTK is reported to be somewhat faster and more accurate than TensorFlow and also provides Python APIs.
Spark integration will continue to be an important driver. Yahoo has already brought TensorFlow to Spark. Spark’s main commercial provider Databricks now has its own open source package to integrate deep learning and Spark.
The key drivers here address at least two of the three impediments to progress. These improvements will make it faster and easier to program for more reliably good results, and faster chips in particular should make the raw machine compute time shorter.
The question, much like the limits of Moore’s Law, is just how far will these improvements take us. They’re available today and they will keep on moving forward. Will they be sufficient to break us out into GANs and Reinforcement Learning? Probably yes, at least insofar as we know how to use these deep learning architectures today.
Neuromorphic Computing (NC) or Spiking Neural Nets (SNNs)
Neuromorphic or Spiking NNs are on the pathway to strong AI and are based on several observations about the way that brains actually work that is significantly different from the way we’ve designed our deep neural nets so far.
To start with, researchers can observe that in a brain not all neurons fire every time. Neurons send selective signals down the chain and it appears that the data is actually encoded in some way in the spikes of potential in that signal. Actually these signals consist of a train of spikes so research proceeds on whether the information is encoded in the amplitude, the frequency, or the latency between the spikes in the train, or perhaps all three.
In our existing deep neural nets all neurons fire each time according to the relatively simple activation functions that are sigmoid, or increasingly ReLU.
Neuromorphic computing has already demonstrated several dramatic improvements over our current deep learning NNs.
So changing this basic architecture can solve all three of the fundamental problems facing deep learning today.
Most importantly, you can buy and utilize a Neuromorphic Spiking NN system today. This is not a technology that’s far in the future.
BrainChip Holdings (Aliso Viejo, CA) has already rolled out a commercial security monitoring system at one of Las Vegas’ largest casinos and has announced other applications about to be delivered. In Las Vegas, its function is to visually and automatically detect dealer errors by monitoring the video streams from standard surveillance cameras. It learns the rules of the game entirely by observation.
BrainChip is a publically traded company on the Australian Exchange (ASX: BRN) and claims significant IP patent protection of its SNN technology. It’s rolling out a series of its own gambling-monitoring products and pursuing licensing agreements for its IP.
Yes there are many improvements yet to come but SNNs are a commercial reality and option in developing AI today.
So Quantum represents still a third path forward to strong AI and overcomes the speed and cost issues.
How Will This All Shake Out?
The fact is that both Neuromorphic and Quantum are laying out competitive roadmaps for getting to deep learning and even newer versions of artificial intelligence faster and perhaps easier.
My personal sense is that with both Quantum and Neuromorphic we are at a point in time much like 2007, the year Google’s Big Table became open source Hadoop. At first we didn’t quite know what to do with it, but three years later Hadoop had largely taken over advances in data science. I think starting today, the next three years are going to be amazing.
For more understanding of these new technologies try these recent articles:
About the author: Bill Vorhies is Editorial Director for Data Science Central and has practiced as a data scientist and commercial predictive modeler since 2001. He can be reached at: