.

- Fixed point strategies can approximate infinite depth.
- The methods are easy to train/implement.
- This essential set of tools can model and analyze a wide range of DS problems.

**Fixed point theory**, which first developed about 60 years ago, is directly connected to limits and traditional control and optimization [1]. These methods are ideal for finding solutions to a broad range of phenomena that crop up in large-scale optimization and highly structured data problems. They work for problems formulated as minimization problems, or more general forms like Nash equilibria [no term] or nonlinear operator equations.

Compared to traditional models, fixed point methods are in their infancy. However, there’s a lot of research suggesting that these algorithms may be the future of data science.

**How do Fixed Point Methods Work?**

Fixed point theory works in a similar way to optimization and is related to the idea of a limit in calculus: at some point in the process, you get “close enough” to a solution: one that’s good enough for your purposes. When it’s not possible to find an exact solution, or an exact answer isn’t needed, a fixed-point algorithm can give an approximate answer.

As **a simple example,** your company might want to create a model for the maximum amount of money a U.S. citizen is willing to spend on new household gadgets per year. An exact solution would depend on many factors, including the fickle nature of consumerism, changing tastes and the effect of climate change on purchasing decisions. It would be difficult (if not impossible) to find an exact solution ($561.23? $981.65?). But there’s going to be a cap, or a limit, which the amount spent tends towards: possibly $570 per annum, possibly $1,000.

You *could* attempt to find the solution to a large-scale optimization problem like this one with traditional methods—if you have the computer resources to take on the challenge. In some cases, even the most powerful computer may not be able to hand the computations, which is where fixed point theory steps in.

**Advantages of Fixed-Point Methods**

Fixed point methods have **several major advantages** over traditional methods. They create a more efficient framework for implicit depth without requiring more memory or increasing the computational costs of training [2]. On the algorithmic front, fixed point strategies include powerful convergence principles which simplify the design and analysis of iterative methods. In addition, block-coordinate or block-iterative strategies reduce an iteration’s computational load and memory requirements [3].

Google research scientist Zelda Mariet and MIT professor Suvrit Sra approached the problem of maximum-likelihood estimation [no term] by comparing performance of the EM algorithm against a novel fixed-point iteration [4]. When the authors compared performance on both synthetic and real-world data, they found that their fixed-point method gave shorter runtimes when handling large matrices and training sets. The fixed-point approach also ran “remarkably faster” for a range of ground set sizes and number of samples. Not only was it *faster* than the EM algorithm, but it was also remarkably simple to implement.

**The Future of Deep Learning?**

One of the major problems with the creation of deep learning models is that *the deeper and more expressible a model becomes, the more memory is required.* In a practical sense, the amount of computer memory is limited by model depth. A workaround is implicit depth methods, but these come with the burden of more computational cost to train networks. At a certain point, some problems simply become too complex to be solve using traditional methods. As we go on to the future, models are destined to become more complex, which means we must find better ways to arrive at solutions.

When finding an exact solution isn’t possible because of computational limits, many problems can be formulated in terms of fixed-point optimization schemes. These schemes, applied to standard models, guarantee the convergence of a solution to the fixed point limit.

**References:**

Image: mikemacmarketing / photo on flickr, CC BY 2.0 , via Wikimedia Commons

[1] Fixed Point Theory and Algorithms for Sciences and Engineering

[2] Fixed Point Networks: Implicit Depth Models with Jacobian-Free Back...

[3] Fixed Point Strategies in Data Science

[4] Fixed-point algorithms for learning determinantal point processes

- 10 Scrum benefits and how to realize them
- 5 proven patterns for resilient software architecture design
- Microsoft's acquisition of Nuance clears antitrust hurdle
- The right to disconnect vs. America's always-on culture
- The enterprise advantages of automated data collection
- Today's blockchain use cases and industry applications
- Salesforce Industries rolls out banking AI compliance tools
- DataRobot integrates AI modeling tools with Snowflake
- M&A wave rising in hot AI market
- AWS analytics helps nonprofit fight climate change

Posted 17 June 2021

© 2021 TechTarget, Inc. Powered by

Badges | Report an Issue | Privacy Policy | Terms of Service

**Most Popular Content on DSC**

To not miss this type of content in the future, subscribe to our newsletter.

- Book: Applied Stochastic Processes
- Long-range Correlations in Time Series: Modeling, Testing, Case Study
- How to Automatically Determine the Number of Clusters in your Data
- New Machine Learning Cheat Sheet | Old one
- Confidence Intervals Without Pain - With Resampling
- Advanced Machine Learning with Basic Excel
- New Perspectives on Statistical Distributions and Deep Learning
- Fascinating New Results in the Theory of Randomness
- Fast Combinatorial Feature Selection

**Other popular resources**

- Comprehensive Repository of Data Science and ML Resources
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- 100 Data Science Interview Questions and Answers
- Cheat Sheets | Curated Articles | Search | Jobs | Courses
- Post a Blog | Forum Questions | Books | Salaries | News

**Archives:** 2008-2014 |
2015-2016 |
2017-2019 |
Book 1 |
Book 2 |
More

**Most popular articles**

- Free Book and Resources for DSC Members
- New Perspectives on Statistical Distributions and Deep Learning
- Time series, Growth Modeling and Data Science Wizardy
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- Comprehensive Repository of Data Science and ML Resources
- Advanced Machine Learning with Basic Excel
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- Selected Business Analytics, Data Science and ML articles
- How to Automatically Determine the Number of Clusters in your Data
- Fascinating New Results in the Theory of Randomness
- Hire a Data Scientist | Search DSC | Find a Job
- Post a Blog | Forum Questions

## You need to be a member of Data Science Central to add comments!

Join Data Science Central