Income Distribution and Healthcare Spending Analysis¶

1. Project Overview¶

This analysis explores the relationship between income levels and healthcare spending using a large synthetic healthcare dataset.

The objective is to understand how healthcare expenditures vary across income groups and identify patterns that may reflect differences in access to care, healthcare utilization, or preventive health behavior.

The dataset contains over 1.2 million patient records, making it suitable for distributed processing using Apache Spark.

1a. Business Questions¶

Healthcare spending varies significantly across populations, but the relationship between income level and healthcare utilization is not always straightforward.

Key questions explored in this analysis:

1. How is income distributed across the population?
2. Do higher income groups spend more on healthcare?
3. At what income level does healthcare spending peak?
4. Are there demographic differences in healthcare expenses?

Understanding these patterns can help inform:

• healthcare policy planning
• insurance risk modeling
• population health management

2. Dataset Overview¶

• Total observations: ~1.2M
• Median income: ~\$32K
• Mean income: ~\$48K

Insight:
The dataset shows a strong right-skewed distribution.

2a. Big Data Considerations¶

• Why Spark was used instead of Pandas?
• Example explanation:
  • The dataset contains over 1M records. Apache Spark enables distributed processing and scalable analytics compared to single-node Pandas workflows.

2b. Analytical Insight¶

Healthcare spending does not increase linearly with income. While spending rises rapidly between the \$15K – \$70K income brackets, it plateaus and eventually declines among higher-income populations.

This pattern may suggest that middle-income populations experience the highest healthcare utilization, whereas higher-income individuals benefit from preventive care and better health management.

2c. Hypothesis for Further Analysis¶

Potential factors influencing the pattern:

• Insurance coverage differences
• Preventive care access
• Age distribution across income brackets
• Chronic disease prevalence

3. Data Engineering (Spark)¶

Data Processing Pipeline¶

The analysis follows a distributed data processing workflow using Apache Spark.

Pipeline steps:

1. Data ingestion from Snowflake dataset
2. Schema inspection and data validation
3. Distributed aggregation using Spark SQL
4. Feature engineering (income brackets)
5. Statistical analysis
6. Visualization using Pandas + Matplotlib

Spark was used for large-scale data processing, while smaller aggregated datasets were converted to Pandas for visualization.

This hybrid workflow enables scalable processing while maintaining flexible analysis capabilities.

4. Income Distribution Analysis¶

Interpretation:

The Y-axis shows frequency (count) — the number of records/rows in the dataset that fall within each income bin. The 1e6 multiplier at the top means the values are in millions, so:

• The tallest bar (~1.17) = approximately 1,170,000 records with income between $0–$100K
• The second bar (~0.19) = approximately 190,000 records with income between $100K–$200K
• All remaining bars are nearly flat = very few records with income above $200K

What the shape tells you

• This is a classic right-skewed (positive skew) distribution, which is very typical for income data:

• The vast majority of your population earns under \$100k

• A smaller but notable group earns \$100k – \$200k

• A tiny minority (outliers) earn up to \$1,000,000

Note: there's a huge difference in mean and median (or 50%). This is because mean was bias towards outliers (which present in this case).

Log transformation of x-axis¶

• The long right tail with near-zero bars suggests possible outliers or data quality issues — we may want to investigate those high-income records. Since the outliers are valid but distorting the viz, trying log-scaling the x-axis for a clearer picture.

• The log-scaling will compress the long right tail so we can actually see the distribution of high earners instead of flat near-zero bars.

• Interpretation:

  • The x-axis is now logarithmic, so the spacing between ticks is no longer equal — notice how 10,000 to 100,000 takes the same horizontal space as 100,000 to 1,000,000. This is intentional and reveals hidden structure.

  • In the previous plot, everything above \$200K looked like a flat zero line. Now we can clearly see:

    • There are still meaningful counts of people earning \$200K – \$500K
    • The distribution gradually tapers off rather than falling off a cliff
    • The data has a much smoother, more gradual decline across all income levels

• Interpretation:

  • From reading the bars, we observe:

  | Income Range | Approx. Count | Interpretation | |---------------|---------------|----------------| | ~$5K–$15K | ~410,000 | Largest group — very low earners | | ~$15K–$40K | ~305,000 | Second largest group | | ~$40K–$70K | ~200,000 | Middle-low earners | | ~$70K–$100K | ~150,000 | Approaching middle class | | ~$100K–$200K | ~100,000 | Upper-middle earners | | ~$200K–$500K | ~50,000 | High earners | | ~$500K+ | ~20,000–25,000 | Very high earners / outliers |

Insight:
The income distribution is strongly right-skewed, with the majority of individuals earning under $40K.

⚠️ Data Note:
The $500K+ category contains extreme outliers that may distort mean income statistics.

This shape — a straight declining line on a log x-axis — is actually close to a Pareto distribution, which is the classic mathematical model for income and wealth data in real-world populations. It basically confirms: the fewer people, the higher the income.

Dataset Snapshot¶

• Total records: ~1,235,000
• Median income: ~$32,000
• Mean income: ~ \$48,000
• Highest income bracket: \$500K+
• Lowest income bracket: \$5K

5. Gender Analysis¶

Summary stats a quick lense¶

From summary stats at a quick lense, there's not much difference in earnings between genders. Now we can take a look of viz, then do some statistics to confirm the observation.

Viz distribution¶

The box plot shows you median, spread, and outliers per gender at a glance. It seems that there's not much difference in median. Also notice lots of outliers for both genders

Statistical test: is the difference significant?¶

Use Mann-Whitney U (not a t-test) because income is skewed, not normally distributed.

With p-value above, there's no statistically significant difference in income between 2 genders

6. Income vs Healthcare Spending¶

This is now a relationship between two continuous variables, so the approach is different from the gender comparison

Summary stats¶

Viz¶

7. Income Distribution Summary¶

Key Finding: Healthcare Expenses RISE with Income (up to a point)

Important Patterns to Notice

Insight 1 — Healthcare spending rises sharply with income initially
Lower-income populations (<$40K) show much lower median healthcare expenses, likely reflecting:

• Reduced access to care
• Delayed or deferred treatments
• Affordability constraints

Insight 2 - The big jump from <$15K to \$15–40K

The median nearly triples between these two groups. The lowest earners spend dramatically less on healthcare — likely due to inability to afford care, not because they need it less.

Insight 3 — Peak spending occurs in middle-income groups (\$40K –\$100K)
Healthcare expenses increase dramatically and plateau around \$175K, suggesting:

• Higher healthcare utilization once financial barriers decrease
• Greater access to medical services

Insight 4 - Plateau and slight decline after \$70K

Healthcare expenses don't keep rising indefinitely. After \$70–100K, the median flattens then slightly drops — suggesting:

• high earners may be more efficient or healthier overall.

Insight 5 — Spending declines in higher-income brackets
Income groups above $200K show lower median healthcare expenses, which may reflect:

• Better preventive care
• Healthier lifestyles
• Superior insurance coverage

Insight 6 - Massive outliers in every bracket

Every income group has dots reaching \$1M – \$10M in healthcare expenses. These likely represent:

• Catastrophic illnesses (cancer, transplants)
• Long-term chronic conditions
• These occur across all income levels, not just the wealthy

Insight 7 - The Spread (Box Size) Tells a Story Too

• <$15K — very tight box → low earners consistently spend little
• \$40–100K — widest boxes → middle income has the most variability in healthcare spending
• \$200K+ — boxes tighten again → high earners are more consistent in their spending

Bottom Line

Income and healthcare expenses are positively correlated up to ~ \$70–100K, then level off. The biggest barrier is at the bottom — the <$15K group spends far less, likely reflecting access and affordability constraints rather than better health.

8. Final Takeaways¶

Although low-income populations represent the largest group in the dataset, their healthcare spending remains relatively low.

The majority of healthcare expenditures appear concentrated in middle-income brackets ($40K–$100K), where healthcare utilization peaks.

This finding may reflect increased healthcare access through employer-provided insurance and higher treatment utilization among middle-income individuals.

Jupyter Notebook Environment¶

Tools and libraries used in this project:

• Python 3.12
• Apache Spark (PySpark)
• Pandas
• NumPy
• Matplotlib / Seaborn
• Jupyter Notebook

Infrastructure:

• Local Spark environment
• Dataset sourced from Snowflake Marketplace