While everyone talk about unusual extreme weather events (heat waves, cold spells, floods, droughts), very few, including scientists, have been able to make sound predictions for extreme events, be it weather or stock market extreme behavior, or any bubble. Here you will learn how to produce simple model-free confidence intervals for extreme events in Excel, how to generate (correlated) simulated stock market data and (uncorrelated) natural data such as air pollution index, understand why extreme events are predictable in one case but not in the other one, predict them, and learn about how to simulate ad-hoc data. Extreme events are a big issue for insurance companies and anti-terrorist agencies: predicting expected loss is critical to set up premiums correctly.

We propose a solution to predict extreme events. It is based on simulations, and very different from extreme value theory (a statistical technique), which so far has proved useless. This article was first proposed as a DSC challenge of the week, but we believe it is important enough to make it a featured article and tutorial of its own.

Here, in our Excel spreadsheet (click on the link to download), we perform simulations to predict extreme values, with model-free confidence intervals, using a very simple methodology that does not involve any statistical knowledge or statistical models. We generated two types of data:

- Stock market simulations, where the value at time k depends on the value at time k-1 (random walk simulations)
- Air index simulations, where the value at time k is independent from value at time k-1, and unbounded

In case #1, predicting extreme values is impossible (there is lack of convergence), while in case #2, it is easy, as illustrated in the charts below. The simulations in case #2 try to reproduce a phenomenon where each new iteration can generate a value larger than all previously observed values. The record computed over the past k observations **is denoted as R(k)**. For each of the two examples, we produced 10 simulations (time series) each with 10,240 data points, then computed the confidence interval for the highest value R(k) observed over the last k iterations (for various values of k), based on these 10 time series.

**Data Generation**

To generate the stock market data, we used basic random walks, see Excel spreadsheet. It produces values that are NOT (statistically speaking) independent. For the air pollution index A(k) at time k, we first generate a random deviate U on [0, 1], then set A(k) to A(k) = - ln(U). The A(k) values are independent. The function used here, in this case -ln(U), must be chosen to provide a good fit with actual, observed data.

Even though our simulations clearly show that stock market highs are unpredictable, it is still possible to find patterns to make money in the stock market (insider trading is your best bet, more on this later, especially on how to use big data to beat insider traders).

**Results**

In the stock market example, we are unable to make predictions. We could start with a positive gain, then stay in negative territory forever, or in positive territory for ever, or oscillate forever.

In the case of predicting air pollution, extreme values are rather rare, rarer than for a Gaussian distribution if we use A(k) = -ln(U), and this makes predicting extreme events easy.

**DSC Resources**

- Career: Training | Books | Cheat Sheet | Apprenticeship | Certification | Salary Surveys | Jobs
- Knowledge: Research | Competitions | Webinars | Our Book | Members Only | Search DSC
- Buzz: Business News | Announcements | Events | RSS Feeds
- Misc: Top Links | Code Snippets | External Resources | Best Blogs | Subscribe | For Bloggers

**Additional Reading**

- Data Scientist Reveals his Growth Hacking Techniques
- 10 Modern Statistical Concepts Discovered by Data Scientists
- Top data science keywords on DSC
- 4 easy steps to becoming a data scientist
- 13 New Trends in Big Data and Data Science
- 22 tips for better data science
- Data Science Compared to 16 Analytic Disciplines
- How to detect spurious correlations, and how to find the real ones
- 17 short tutorials all data scientists should read (and practice)
- 10 types of data scientists
- 66 job interview questions for data scientists
- High versus low-level data science

Follow us on Twitter: @DataScienceCtrl | @AnalyticBridge

© 2019 Data Science Central ® Powered by

Badges | Report an Issue | Privacy Policy | Terms of Service

**Most Popular Content on DSC**

To not miss this type of content in the future, subscribe to our newsletter.

**Technical**

- Free Books and Resources for DSC Members
- Learn Machine Learning Coding Basics in a weekend
- New Machine Learning Cheat Sheet | Old one
- Advanced Machine Learning with Basic Excel
- 12 Algorithms Every Data Scientist Should Know
- Hitchhiker's Guide to Data Science, Machine Learning, R, Python
- Visualizations: Comparing Tableau, SPSS, R, Excel, Matlab, JS, Pyth...
- How to Automatically Determine the Number of Clusters in your Data
- New Perspectives on Statistical Distributions and Deep Learning
- Fascinating New Results in the Theory of Randomness
- Long-range Correlations in Time Series: Modeling, Testing, Case Study
- Fast Combinatorial Feature Selection with New Definition of Predict...
- 10 types of regressions. Which one to use?
- 40 Techniques Used by Data Scientists
- 15 Deep Learning Tutorials
- R: a survival guide to data science with R

**Non Technical**

- Advanced Analytic Platforms - Incumbents Fall - Challengers Rise
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- How to Become a Data Scientist - On your own
- 16 analytic disciplines compared to data science
- Six categories of Data Scientists
- 21 data science systems used by Amazon to operate its business
- 24 Uses of Statistical Modeling
- 33 unusual problems that can be solved with data science
- 22 Differences Between Junior and Senior Data Scientists
- Why You Should be a Data Science Generalist - and How to Become One
- Becoming a Billionaire Data Scientist vs Struggling to Get a $100k Job
- Why do people with no experience want to become data scientists?

**Articles from top bloggers**

- Kirk Borne | Stephanie Glen | Vincent Granville
- Ajit Jaokar | Ronald van Loon | Bernard Marr
- Steve Miller | Bill Schmarzo | Bill Vorhies

**Other popular resources**

- Comprehensive Repository of Data Science and ML Resources
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- 100 Data Science Interview Questions and Answers
- Cheat Sheets | Curated Articles | Search | Jobs | Courses
- Post a Blog | Forum Questions | Books | Salaries | News

**Archives**: 2008-2014 | 2015-2016 | 2017-2019 | Book 1 | Book 2 | More

**Most popular articles**

- Free Book and Resources for DSC Members
- New Perspectives on Statistical Distributions and Deep Learning
- Time series, Growth Modeling and Data Science Wizardy
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- Comprehensive Repository of Data Science and ML Resources
- Advanced Machine Learning with Basic Excel
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- Selected Business Analytics, Data Science and ML articles
- How to Automatically Determine the Number of Clusters in your Data
- Fascinating New Results in the Theory of Randomness
- Hire a Data Scientist | Search DSC | Find a Job
- Post a Blog | Forum Questions

## You need to be a member of Data Science Central to add comments!

Join Data Science Central