Subscribe to DSC Newsletter

Summary:  There have been several stories over the last several months around the theme that AI is about to hit a wall.  That the rapid improvements we’ve experienced and the benefits we’ve accrued can’t continue at the current pace.  It’s worth taking a look at these arguments to see if we should be adjusting our plans and expectations.

 

There have been several stories over the last several months around the theme that AI is about to hit a wall.  That the rapid improvements we’ve experienced and the benefits we’ve accrued can’t continue at the current pace.  It’s worth taking a look at these arguments to see if we should be adjusting our plans and expectations.

These concerns center around two issues:

  • The over concentration of investment in AI in just a few cities and how will that impact growth.
  • The amount of compute required for our most advanced applications appears to be rising seven times faster than before implying some limit on the cost/benefit to pursuing the most compute-hungry advances.

Over Concentrated Investment

It’s certainly true that investment in AI is extremely concentrated in just a few cities.  This chart from the National Venture Capital Association shows that over 80% of investment went to just six metros.  By the way, there’s a sharp break in the distribution below this level with the next best metro receiving only 2.4% of total investment (Chicago).

 

So what’s the concern?  The folks who have written about this see this as an over concentration of talent that increases labor costs in these areas making some types on new AI investment uneconomic.

It seems true that the concentration of AI talent in say San Francisco leads to the most new ideas in AI being generated there.  But is this really a black hole with gravity so extreme that good ideas can’t escape to be executed elsewhere?

Worriers about this concentration see these good ideas escaping to overseas low cost locations.  I’m not so sure. 

There is a pattern that the largest concentration of VC funded AI startups and established AI companies will draw in the most volume of new talent at the bottom.  New graduates and young people will naturally gravitate to metros where there are the most opportunities.

Personally I think this concern overlooks that as these metros become more dense with opportunity they also become massively more expensive to live in and that the quality of life drops off quickly (San Francisco, I’m looking at you).

As young professionals gain experience and also reach the life stage where they want to marry, buy homes, and start families they will naturally migrate to less expensive geographies.  This may mean they will gravitate to the outskirts of these metros with lower cost of living where AI businesses will naturally follow to capture talent.  It may also mean a willingness to relocate over longer distances to good quality of life locations like Austin, Seattle, Washington, Atlanta, or Miami where investment will follow talent.

Growth in Training Compute for the Most Advanced Applications is Now Growing 7X Faster than Before

This area of concern was first raised by the AI research lab OpenAI in 2018 and recently updated with new data.  Their findings are that the amount of computational power needed to train the largest AI models has doubled every 3.4 months since 2012.

They see history divided into two distinct eras with the new era beginning in 2012 now growing 7X faster than the previous era.

 

This phenomenon has led some leading industry experts including Jerome Pesenti, the VP of Artificial Intelligence at Facebook to observe in a recent Wired article that there may be cost/benefit limits to what can be accomplished at the extreme limits of improving AI models.

Pesenti says that while scaling up is a worthwhile solution to the most difficult challenges like deep fake image detection and AI moderation of fake news and online abuse that the rate of progress is not sustainable.

“If you look at top experiments, each year the cost is going up 10-fold. Right now, an experiment might be in seven figures, but it’s not going to go to nine or ten figures, it’s not possible, nobody can afford that.  It means that at some point we're going to hit the wall. In many ways we already have.”

Other advancements that are at risk would be many of the ‘next big things’ in AI including cutting edge applications that require increased look forwards or look backs to properly classify.  These would include many navigation apps in next gen computer vision, and contextually intelligent NLP assistants that can relate our schedules and actions to outside events.

While I wouldn’t want to have to do without these improvements, it’s interesting what’s been included in the OpenAI chart.  For example, the most compute-hungry model is actually a reinforcement learning game playing app AlphaGoZero.  Personally I’d like to see research continue in Deep Learning RL models but the commercial applications are not as compelling as say BERT, the new NLP language encoder from Google.

Another issue to keep in mind is that this is an issue that impacts incremental gains in existing capabilities.  Pesenti is right to wonder aloud if it isn’t time to apply cost/benefit criteria to these.  This so-called wall does not prevent us in any way from continuing the rapid application of existing AI/ML capabilities in business and harvesting the benefits currently underway.

In a sense this is the flying-car syndrome.  We’ve come so far so fast that if we don’t get deep fake detection or fake news automatic moderation right away we’ll wonder where’s our flying car.

It’s also worth noting that lots of research is being conducted in how to make the training process more efficient.  For example, just last week researchers at Rice University announced the results of their new MACH technique (Merged Average Classifiers via Hashing).  They report “training times about 7-10 times faster, and …memory footprints are 2-4 times smaller’ than previous large-scale deep learning techniques.

As for these walls, fortunately smart folks are working on both ends of this problem.  We may need to bring some cost/benefit discipline to our research but if the market need is there, my bet is that we’ll find a way.

 

Other articles by Bill Vorhies

 

About the author:  Bill is Contributing Editor for Data Science Central.  Bill is also President & Chief Data Scientist at Data-Magnum and has practiced as a data scientist since 2001.  His articles have been read more than 2 million times.

He can be reached at:

[email protected] or [email protected]

Views: 1340

Tags: AI, concentrated, cost, growth, hits, investment, limits, the, to, train, More…wall

Comment

You need to be a member of Data Science Central to add comments!

Join Data Science Central

Comment by Christopher H Chang on December 22, 2019 at 10:11am

Hi @Vincent, certainly it's possible to get VC funding outside of one of the concentrated metro areas the author mentioned. It's just less likely I'm guessing, and the bar is higher. So even if the cost of living is so high that you'll have to rough it for awhile in one of the top cities, is one more likely to get their idea off the ground living immersed in that culture, or heading out to a second tier location? The thesis after all was that AI is heading for a wall because a new generation will disregard the concentrated geographic areas of progress and prioritize quality of life, thereby seeding new areas. That's just not a thesis I've seen actually borne out. Amazon didn't choose NYC for a 2nd HQ because of a lack of incentives elsewhere; that location served the business needs best. Same for most AI startups I wager.

Comment by Vincent Granville on December 21, 2019 at 2:44pm

@Christopher: You can access Silicon Valley VC money even if located outside of the Bay Area, I almost got funded by one of them even though my company was in L.A. (eventually we got funded by an L.A.-based VC firm). Or work on AI projects that can be self-funded, like what I do with cryptography-related stuff out of my home in the Seattle area.  

Comment by Christopher H Chang on December 21, 2019 at 11:15am

And culture is a big contributor too (maybe the biggest). If you're immersed in an entrepreneurial culture, you're more likely to consider yourself "one of the tribe" and figure out how take steps by interacting with folks of that mindset every day. Replicating culture is not trivial, and probably a big reason why Silicon Valley can't just be replicated anywhere.

Comment by Christopher H Chang on December 21, 2019 at 11:08am

"But is this really a black hole with gravity so extreme that good ideas can’t escape to be executed elsewhere?"

The problem is not the flow of information/ideas, it's the flow of money. Opportunity is the confluence of money, talent, and an idea (and good execution of course). Ideas are everywhere, and talent may well be mobile, but access to VCs and investors is going to be a lot better in those metros until distance interaction tech (video conf, remote meeting) gets better and more trusted. 

Comment by Jeefri A. Moka on December 16, 2019 at 10:33pm
Nooo

Videos

  • Add Videos
  • View All

© 2020   Data Science Central ®   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service