How do you know if an outlier is the result of a data glitch, or a real data point -- indeed maybe not an outlier. Difficult question to answer, but the chart below shows that in some cases, the outlier is not an error.
In this example, you could argue that we are not using the right metrics: comparing health expenditures in US (twice above average among developed countries) when US salary (after tax) is twice above average among developed countries, lead to a bias. When corrected for this salary bias, US might not be an outlier anymore in the above chart.
Also, is life expectancy the right metric to use? What if a large group of people die very young because of gang membership, and another group (the majority) dies pretty old? What would be interesting to see is the impact over time, in US, of increased health expenditures on life expectancy, after eliminating people dying from gun shots or car accidents. Note that a more stressful life (typical in US) can cause early death despite higher health expenditures.
Note the massive impact of the USA dot (outlier) on R^2 (at the bottom right corner) - making it much smaller than it should be (R^2 = 0.51). But R^2 is a bad metric, sensitive to outliers, and should not be used. Use this metric instead, to measure quality of fit. Indeed, the entire black curve going through the cloud, is bended too much towards the South-East, because of this outlier.