{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e35c934",
   "metadata": {},
   "source": [
    "# Predicting Heart Disease\n",
    "\n",
    "The World Health Organization (WHO) estimates that 17.9 million people die every year because of cardiovascular diseases (CVDs).\n",
    "\n",
    "There are multiple risk factors that could contribute to CVD in an individual such as unhealthy diet, lack of physical activity or mental illnesses. Being able to identify these risk factors in individuals early on could help prevent a lot of premature deaths.\n",
    "\n",
    "In this project, we will use the [Kaggle dataset](https://www.kaggle.com/datasets/fedesoriano/heart-failure-prediction) and build a K-Nearest Neighbors classifier to accurately predict the likelihood of a patient having a heart disease in the future. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7a6956c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6089c814",
   "metadata": {},
   "source": [
    "## EDA: Descriptive Statistics\n",
    "\n",
    "We will start with exploring our dataset. As per the source, each patient has the following information collected about them:\n",
    "\n",
    "\n",
    "1. `Age`: age of the patient [years]\n",
    "2. `Sex`: sex of the patient [M: Male, F: Female]\n",
    "3. `ChestPainType`: chest pain type [TA: Typical Angina, ATA: Atypical Angina, NAP: Non-Anginal Pain, ASY: Asymptomatic]\n",
    "4. `RestingBP`: resting blood pressure [mm Hg]\n",
    "5. `Cholesterol`: serum cholesterol [mm/dl]\n",
    "6. `FastingBS`: fasting blood sugar [1: if FastingBS > 120 mg/dl, 0: otherwise]\n",
    "7. `RestingECG`: resting electrocardiogram results [Normal: Normal, ST: having ST-T wave abnormality (T wave inversions and/or ST elevation or depression of > 0.05 mV), LVH: showing probable or definite left ventricular hypertrophy by Estes' criteria]\n",
    "8. `MaxHR`: maximum heart rate achieved [Numeric value between 60 and 202]\n",
    "9. `ExerciseAngina`: exercise-induced angina [Y: Yes, N: No]\n",
    "10. `Oldpeak`: oldpeak = ST [Numeric value measured in depression]\n",
    "11. `ST_Slope`: the slope of the peak exercise ST segment [Up: upsloping, Flat: flat, Down: downsloping]\n",
    "12. `HeartDisease`: output class [1: heart disease, 0: Normal]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d12aa5ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "#load dataset\n",
    "df = pd.read_csv(\"heart_disease_prediction.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bc74cce6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPainType</th>\n",
       "      <th>RestingBP</th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>FastingBS</th>\n",
       "      <th>RestingECG</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>ExerciseAngina</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>ST_Slope</th>\n",
       "      <th>HeartDisease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40</td>\n",
       "      <td>M</td>\n",
       "      <td>ATA</td>\n",
       "      <td>140</td>\n",
       "      <td>289</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>172</td>\n",
       "      <td>N</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Up</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49</td>\n",
       "      <td>F</td>\n",
       "      <td>NAP</td>\n",
       "      <td>160</td>\n",
       "      <td>180</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>156</td>\n",
       "      <td>N</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37</td>\n",
       "      <td>M</td>\n",
       "      <td>ATA</td>\n",
       "      <td>130</td>\n",
       "      <td>283</td>\n",
       "      <td>0</td>\n",
       "      <td>ST</td>\n",
       "      <td>98</td>\n",
       "      <td>N</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Up</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>48</td>\n",
       "      <td>F</td>\n",
       "      <td>ASY</td>\n",
       "      <td>138</td>\n",
       "      <td>214</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>108</td>\n",
       "      <td>Y</td>\n",
       "      <td>1.5</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>54</td>\n",
       "      <td>M</td>\n",
       "      <td>NAP</td>\n",
       "      <td>150</td>\n",
       "      <td>195</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>122</td>\n",
       "      <td>N</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Up</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age Sex ChestPainType  RestingBP  Cholesterol  FastingBS RestingECG  MaxHR  \\\n",
       "0   40   M           ATA        140          289          0     Normal    172   \n",
       "1   49   F           NAP        160          180          0     Normal    156   \n",
       "2   37   M           ATA        130          283          0         ST     98   \n",
       "3   48   F           ASY        138          214          0     Normal    108   \n",
       "4   54   M           NAP        150          195          0     Normal    122   \n",
       "\n",
       "  ExerciseAngina  Oldpeak ST_Slope  HeartDisease  \n",
       "0              N      0.0       Up             0  \n",
       "1              N      1.0     Flat             1  \n",
       "2              N      0.0       Up             0  \n",
       "3              Y      1.5     Flat             1  \n",
       "4              N      0.0       Up             0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9a59e51",
   "metadata": {},
   "source": [
    "The dataset seems to contain both numerical and categorical features. Let's look at the datatype for each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "344a302b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Age                 int64\n",
      "Sex                object\n",
      "ChestPainType      object\n",
      "RestingBP           int64\n",
      "Cholesterol         int64\n",
      "FastingBS           int64\n",
      "RestingECG         object\n",
      "MaxHR               int64\n",
      "ExerciseAngina     object\n",
      "Oldpeak           float64\n",
      "ST_Slope           object\n",
      "HeartDisease        int64\n",
      "dtype: object\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "int64      6\n",
       "object     5\n",
       "float64    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df.dtypes)\n",
    "df.dtypes.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aab3ff7b",
   "metadata": {},
   "source": [
    "`7` features in total are numerical while `5` are categorical. However, two of the numerical features, `FastingBS` and `HeartDisease` are categorical as well. \n",
    "\n",
    "We will focus on the numerical variables first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1bd0d2c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>RestingBP</th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>FastingBS</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>HeartDisease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>918.000000</td>\n",
       "      <td>918.000000</td>\n",
       "      <td>918.000000</td>\n",
       "      <td>918.000000</td>\n",
       "      <td>918.000000</td>\n",
       "      <td>918.000000</td>\n",
       "      <td>918.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>53.510893</td>\n",
       "      <td>132.396514</td>\n",
       "      <td>198.799564</td>\n",
       "      <td>0.233115</td>\n",
       "      <td>136.809368</td>\n",
       "      <td>0.887364</td>\n",
       "      <td>0.553377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.432617</td>\n",
       "      <td>18.514154</td>\n",
       "      <td>109.384145</td>\n",
       "      <td>0.423046</td>\n",
       "      <td>25.460334</td>\n",
       "      <td>1.066570</td>\n",
       "      <td>0.497414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>-2.600000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>47.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>173.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>54.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>223.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>138.000000</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>60.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>267.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>156.000000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>77.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>603.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Age   RestingBP  Cholesterol   FastingBS       MaxHR  \\\n",
       "count  918.000000  918.000000   918.000000  918.000000  918.000000   \n",
       "mean    53.510893  132.396514   198.799564    0.233115  136.809368   \n",
       "std      9.432617   18.514154   109.384145    0.423046   25.460334   \n",
       "min     28.000000    0.000000     0.000000    0.000000   60.000000   \n",
       "25%     47.000000  120.000000   173.250000    0.000000  120.000000   \n",
       "50%     54.000000  130.000000   223.000000    0.000000  138.000000   \n",
       "75%     60.000000  140.000000   267.000000    0.000000  156.000000   \n",
       "max     77.000000  200.000000   603.000000    1.000000  202.000000   \n",
       "\n",
       "          Oldpeak  HeartDisease  \n",
       "count  918.000000    918.000000  \n",
       "mean     0.887364      0.553377  \n",
       "std      1.066570      0.497414  \n",
       "min     -2.600000      0.000000  \n",
       "25%      0.000000      0.000000  \n",
       "50%      0.600000      1.000000  \n",
       "75%      1.500000      1.000000  \n",
       "max      6.200000      1.000000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbc1f5d4",
   "metadata": {},
   "source": [
    "From the table above, we can observe that:\n",
    "\n",
    "- The average age of patients is ~`53` years.\n",
    "- The median for `Cholesterol` is higher than its mean by roughly `25` mm/dl, indicating that it could be a left-skewed distribution with a possibility of outliers skewing the distribution.\n",
    "- `RestingBP` and `Cholesterol` have a minimum value of zero.\n",
    "- There don't seem to be any missing values in these columns. But we will have to confirm it across the entire dataset as well.\n",
    "\n",
    "`RestingBP` can't be `0`. And, as per the [American Heart Association](https://www.heart.org/en/health-topics/cholesterol/about-cholesterol/what-your-cholesterol-levels-mean), serum cholesterol is a composite of different measurements. So, it is unlikely that `Cholesterol` would be `0` as well. We will have to clean both of these up later.\n",
    "\n",
    "Next, we will look at the categorical variables. It would also be beneficial to look at how the target feature, `HeartDisease`, is related to those categories. Before that, let's quickly check if there are any missing values in the dataset or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5c436572",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Age               0\n",
       "Sex               0\n",
       "ChestPainType     0\n",
       "RestingBP         0\n",
       "Cholesterol       0\n",
       "FastingBS         0\n",
       "RestingECG        0\n",
       "MaxHR             0\n",
       "ExerciseAngina    0\n",
       "Oldpeak           0\n",
       "ST_Slope          0\n",
       "HeartDisease      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "847a31a8",
   "metadata": {},
   "source": [
    "There are no missing values in this dataset!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75988364",
   "metadata": {},
   "source": [
    "## EDA: Categorical Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47977c45",
   "metadata": {},
   "source": [
    "We identified that most of the categorical columns are all of dtype **object**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "304a72e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPainType</th>\n",
       "      <th>RestingECG</th>\n",
       "      <th>ExerciseAngina</th>\n",
       "      <th>ST_Slope</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>918</td>\n",
       "      <td>918</td>\n",
       "      <td>918</td>\n",
       "      <td>918</td>\n",
       "      <td>918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>M</td>\n",
       "      <td>ASY</td>\n",
       "      <td>Normal</td>\n",
       "      <td>N</td>\n",
       "      <td>Flat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>725</td>\n",
       "      <td>496</td>\n",
       "      <td>552</td>\n",
       "      <td>547</td>\n",
       "      <td>460</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Sex ChestPainType RestingECG ExerciseAngina ST_Slope\n",
       "count   918           918        918            918      918\n",
       "unique    2             4          3              2        3\n",
       "top       M           ASY     Normal              N     Flat\n",
       "freq    725           496        552            547      460"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe(include=['object'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfe32c99",
   "metadata": {},
   "source": [
    "We can confirm that those columns are indeed categorical given the number of unique values in each of them. But, we can't gather much else. Also, `FastingBS` and `HeartDisease` are categorical as well since they only contain binary values. We can confirm that quickly as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "99fc3fcd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0, 1], dtype=int64), array([0, 1], dtype=int64))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"FastingBS\"].unique(), df[\"HeartDisease\"].unique() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79253815",
   "metadata": {},
   "source": [
    "Let's start looking at the categories in more detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5af0a32f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 47316 characters]\n",
      "text/plain": [
       "<Figure size 1152x1080 with 7 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "categorical_cols = [\"Sex\", \"ChestPainType\", \"FastingBS\", \"RestingECG\", \"ExerciseAngina\", \"ST_Slope\", \"HeartDisease\"]\n",
    "\n",
    "fig = plt.figure(figsize=(16,15))\n",
    "\n",
    "for idx, col in enumerate(categorical_cols):\n",
    "    ax = plt.subplot(4, 2, idx+1)\n",
    "    sns.countplot(x=df[col], ax=ax)\n",
    "    # add data labels to each bar\n",
    "    for container in ax.containers:\n",
    "        ax.bar_label(container, label_type=\"center\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d97efb3",
   "metadata": {},
   "source": [
    "- The dataset is highly skewed towards male patients. There are `725` male patients and `193` female patients. This could potentially induce a bias in our model.\n",
    "- `496` patients had `ASY` (asymptotic) chest pain type.\n",
    "- `552` patients had a normal restin ECG.\n",
    "- `704` patients had blood sugar lower than `120` mg/dl\n",
    "\n",
    "Grouping these by `HeartDisease` will give us a better idea about the data distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a5f28e00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 52604 characters]\n",
      "text/plain": [
       "<Figure size 1152x1080 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(16,15))\n",
    "\n",
    "for idx, col in enumerate(categorical_cols[:-1]):\n",
    "    ax = plt.subplot(4, 2, idx+1)\n",
    "    # group by HeartDisease\n",
    "    sns.countplot(x=df[col], hue=df[\"HeartDisease\"], ax=ax)\n",
    "    # add data labels to each bar\n",
    "    for container in ax.containers:\n",
    "        ax.bar_label(container, label_type=\"center\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b44da4c",
   "metadata": {},
   "source": [
    "- We can further notice how skewed the dataset is towards male patients. Only `50` female patients in the dataset have been diagnosed with heart disease.\n",
    "- A significant number of patients, `392`, diagnosed with heart disease have asymptomatic (ASY) chest pain. While chest pain could be a relevant feature for our model, asymptomatic implies that those patients who had a heart disease did not have chest pain as a symptom. \n",
    "- A high number (`170`) of patients with blood sugar greater than 120 mg/dl were diagnosed with heart disease in relation to those who were not diagnosed as such.\n",
    "- Out of all patients who had an exercise-induced angina, `316` were diagnosed with a heart disease.\n",
    "- Out of all patients with a flat ST slope, `381` were diagnosed with a heart disease.\n",
    "\n",
    "Looking at the data distribution from the above plots, we can start to identify some features that could be relevant to us. We will clean up the dataset a bit first before narrowing down on our features."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb5c3c5",
   "metadata": {},
   "source": [
    "## Data Cleaning\n",
    "\n",
    "We identified that there are no missing values. However, as we noticed earlier, a couple of columns have 0 values which don't make sense.\n",
    "\n",
    "We will look at how many zero values `RestingBP` and `Cholesterol` contain and decide how to handle those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3ad28327",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPainType</th>\n",
       "      <th>RestingBP</th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>FastingBS</th>\n",
       "      <th>RestingECG</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>ExerciseAngina</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>ST_Slope</th>\n",
       "      <th>HeartDisease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>449</th>\n",
       "      <td>55</td>\n",
       "      <td>M</td>\n",
       "      <td>NAP</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>155</td>\n",
       "      <td>N</td>\n",
       "      <td>1.5</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Age Sex ChestPainType  RestingBP  Cholesterol  FastingBS RestingECG  \\\n",
       "449   55   M           NAP          0            0          0     Normal   \n",
       "\n",
       "     MaxHR ExerciseAngina  Oldpeak ST_Slope  HeartDisease  \n",
       "449    155              N      1.5     Flat             1  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df[\"RestingBP\"] == 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c4efa8b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex</th>\n",
       "      <th>ChestPainType</th>\n",
       "      <th>RestingBP</th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>FastingBS</th>\n",
       "      <th>RestingECG</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>ExerciseAngina</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>ST_Slope</th>\n",
       "      <th>HeartDisease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>293</th>\n",
       "      <td>65</td>\n",
       "      <td>M</td>\n",
       "      <td>ASY</td>\n",
       "      <td>115</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>93</td>\n",
       "      <td>Y</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>32</td>\n",
       "      <td>M</td>\n",
       "      <td>TA</td>\n",
       "      <td>95</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Normal</td>\n",
       "      <td>127</td>\n",
       "      <td>N</td>\n",
       "      <td>0.7</td>\n",
       "      <td>Up</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>295</th>\n",
       "      <td>61</td>\n",
       "      <td>M</td>\n",
       "      <td>ASY</td>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Normal</td>\n",
       "      <td>110</td>\n",
       "      <td>Y</td>\n",
       "      <td>1.5</td>\n",
       "      <td>Up</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>50</td>\n",
       "      <td>M</td>\n",
       "      <td>ASY</td>\n",
       "      <td>145</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Normal</td>\n",
       "      <td>139</td>\n",
       "      <td>Y</td>\n",
       "      <td>0.7</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>297</th>\n",
       "      <td>57</td>\n",
       "      <td>M</td>\n",
       "      <td>ASY</td>\n",
       "      <td>110</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ST</td>\n",
       "      <td>131</td>\n",
       "      <td>Y</td>\n",
       "      <td>1.4</td>\n",
       "      <td>Up</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>514</th>\n",
       "      <td>43</td>\n",
       "      <td>M</td>\n",
       "      <td>ASY</td>\n",
       "      <td>122</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Normal</td>\n",
       "      <td>120</td>\n",
       "      <td>N</td>\n",
       "      <td>0.5</td>\n",
       "      <td>Up</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>515</th>\n",
       "      <td>63</td>\n",
       "      <td>M</td>\n",
       "      <td>NAP</td>\n",
       "      <td>130</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ST</td>\n",
       "      <td>160</td>\n",
       "      <td>N</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Flat</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>518</th>\n",
       "      <td>48</td>\n",
       "      <td>M</td>\n",
       "      <td>NAP</td>\n",
       "      <td>102</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ST</td>\n",
       "      <td>110</td>\n",
       "      <td>Y</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Down</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>535</th>\n",
       "      <td>56</td>\n",
       "      <td>M</td>\n",
       "      <td>ASY</td>\n",
       "      <td>130</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>LVH</td>\n",
       "      <td>122</td>\n",
       "      <td>Y</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>536</th>\n",
       "      <td>62</td>\n",
       "      <td>M</td>\n",
       "      <td>NAP</td>\n",
       "      <td>133</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ST</td>\n",
       "      <td>119</td>\n",
       "      <td>Y</td>\n",
       "      <td>1.2</td>\n",
       "      <td>Flat</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>172 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Age Sex ChestPainType  RestingBP  Cholesterol  FastingBS RestingECG  \\\n",
       "293   65   M           ASY        115            0          0     Normal   \n",
       "294   32   M            TA         95            0          1     Normal   \n",
       "295   61   M           ASY        105            0          1     Normal   \n",
       "296   50   M           ASY        145            0          1     Normal   \n",
       "297   57   M           ASY        110            0          1         ST   \n",
       "..   ...  ..           ...        ...          ...        ...        ...   \n",
       "514   43   M           ASY        122            0          0     Normal   \n",
       "515   63   M           NAP        130            0          1         ST   \n",
       "518   48   M           NAP        102            0          1         ST   \n",
       "535   56   M           ASY        130            0          0        LVH   \n",
       "536   62   M           NAP        133            0          1         ST   \n",
       "\n",
       "     MaxHR ExerciseAngina  Oldpeak ST_Slope  HeartDisease  \n",
       "293     93              Y      0.0     Flat             1  \n",
       "294    127              N      0.7       Up             1  \n",
       "295    110              Y      1.5       Up             1  \n",
       "296    139              Y      0.7     Flat             1  \n",
       "297    131              Y      1.4       Up             1  \n",
       "..     ...            ...      ...      ...           ...  \n",
       "514    120              N      0.5       Up             1  \n",
       "515    160              N      3.0     Flat             0  \n",
       "518    110              Y      1.0     Down             1  \n",
       "535    122              Y      1.0     Flat             1  \n",
       "536    119              Y      1.2     Flat             1  \n",
       "\n",
       "[172 rows x 12 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df[\"Cholesterol\"] == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dbe3eca",
   "metadata": {},
   "source": [
    "`RestingBP` has only one zero value. We can remove that row from consideration. There are `172` zero values for `Cholesterol`. That's a relatively high number. We can't remove them all and replacing those values with the median might not be an ideal approach, but that's what we will go for now.\n",
    "\n",
    "To be more accurate, we will replace the zero values in `Cholesterol` in relation to `HeartDisease`. So, the 0 values in `Cholesterol` for patients who were diagnosed with a heart disease will be replaced by the median of the non-zero values for patients who were diagnosed with a heart disase. And we'll do the same for those who were not diagnosed with a heart disease."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "61c9e24a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = df.copy()\n",
    "\n",
    "# only keep non-zero values for RestingBP\n",
    "df_clean = df_clean[df_clean[\"RestingBP\"] != 0]\n",
    "\n",
    "heartdisease_mask = df_clean[\"HeartDisease\"]==0\n",
    "\n",
    "cholesterol_without_heartdisease = df_clean.loc[heartdisease_mask, \"Cholesterol\"]\n",
    "cholesterol_with_heartdisease = df_clean.loc[~heartdisease_mask, \"Cholesterol\"]\n",
    "\n",
    "df_clean.loc[heartdisease_mask, \"Cholesterol\"] = cholesterol_without_heartdisease.replace(to_replace = 0, value = cholesterol_without_heartdisease.median())\n",
    "df_clean.loc[~heartdisease_mask, \"Cholesterol\"] = cholesterol_with_heartdisease.replace(to_replace = 0, value = cholesterol_with_heartdisease.median())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6408d9a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>RestingBP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>917.000000</td>\n",
       "      <td>917.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>239.700109</td>\n",
       "      <td>132.540894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>54.352727</td>\n",
       "      <td>17.999749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>85.000000</td>\n",
       "      <td>80.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>214.000000</td>\n",
       "      <td>120.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>225.000000</td>\n",
       "      <td>130.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>267.000000</td>\n",
       "      <td>140.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>603.000000</td>\n",
       "      <td>200.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Cholesterol   RestingBP\n",
       "count   917.000000  917.000000\n",
       "mean    239.700109  132.540894\n",
       "std      54.352727   17.999749\n",
       "min      85.000000   80.000000\n",
       "25%     214.000000  120.000000\n",
       "50%     225.000000  130.000000\n",
       "75%     267.000000  140.000000\n",
       "max     603.000000  200.000000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean[[\"Cholesterol\", \"RestingBP\"]].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "512c073a",
   "metadata": {},
   "source": [
    "The minimum values for both have changed! There are no more zero values in either of those."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c71332d",
   "metadata": {},
   "source": [
    "## Feature Selection\n",
    "\n",
    "Thanks to our EDA and a general understanding of the features, we can identify some of the features that we could start with:\n",
    "\n",
    "- `Age`\n",
    "- `Sex`\n",
    "- `ChestPainType`\n",
    "- `Cholesterol`\n",
    "- `FastingBS`\n",
    "\n",
    "\n",
    "We will also identify how stronly the feature columns are correlated to the target colummn. That should help us narrow down on the features.\n",
    "\n",
    "In order to do that, we will first convert our categorical columns into dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "22829bb7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>RestingBP</th>\n",
       "      <th>Cholesterol</th>\n",
       "      <th>FastingBS</th>\n",
       "      <th>MaxHR</th>\n",
       "      <th>Oldpeak</th>\n",
       "      <th>HeartDisease</th>\n",
       "      <th>Sex_M</th>\n",
       "      <th>ChestPainType_ATA</th>\n",
       "      <th>ChestPainType_NAP</th>\n",
       "      <th>ChestPainType_TA</th>\n",
       "      <th>RestingECG_Normal</th>\n",
       "      <th>RestingECG_ST</th>\n",
       "      <th>ExerciseAngina_Y</th>\n",
       "      <th>ST_Slope_Flat</th>\n",
       "      <th>ST_Slope_Up</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40</td>\n",
       "      <td>140</td>\n",
       "      <td>289</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49</td>\n",
       "      <td>160</td>\n",
       "      <td>180</td>\n",
       "      <td>0</td>\n",
       "      <td>156</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37</td>\n",
       "      <td>130</td>\n",
       "      <td>283</td>\n",
       "      <td>0</td>\n",
       "      <td>98</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>48</td>\n",
       "      <td>138</td>\n",
       "      <td>214</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>54</td>\n",
       "      <td>150</td>\n",
       "      <td>195</td>\n",
       "      <td>0</td>\n",
       "      <td>122</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  RestingBP  Cholesterol  FastingBS  MaxHR  Oldpeak  HeartDisease  \\\n",
       "0   40        140          289          0    172      0.0             0   \n",
       "1   49        160          180          0    156      1.0             1   \n",
       "2   37        130          283          0     98      0.0             0   \n",
       "3   48        138          214          0    108      1.5             1   \n",
       "4   54        150          195          0    122      0.0             0   \n",
       "\n",
       "   Sex_M  ChestPainType_ATA  ChestPainType_NAP  ChestPainType_TA  \\\n",
       "0      1                  1                  0                 0   \n",
       "1      0                  0                  1                 0   \n",
       "2      1                  1                  0                 0   \n",
       "3      0                  0                  0                 0   \n",
       "4      1                  0                  1                 0   \n",
       "\n",
       "   RestingECG_Normal  RestingECG_ST  ExerciseAngina_Y  ST_Slope_Flat  \\\n",
       "0                  1              0                 0              0   \n",
       "1                  1              0                 0              1   \n",
       "2                  0              1                 0              0   \n",
       "3                  1              0                 1              1   \n",
       "4                  1              0                 0              0   \n",
       "\n",
       "   ST_Slope_Up  \n",
       "0            1  \n",
       "1            0  \n",
       "2            1  \n",
       "3            0  \n",
       "4            1  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean = pd.get_dummies(df_clean, drop_first=True)\n",
    "df_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97ccf85c",
   "metadata": {},
   "source": [
    "Now, we can find how they are correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e48e3b51",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 166676 characters]\n",
      "text/plain": [
       "<Figure size 864x576 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "correlations = abs(df_clean.corr())\n",
    "plt.figure(figsize=(12,8))\n",
    "sns.heatmap(correlations, annot=True, cmap=\"Blues\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "fbd10db2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 64772 characters]\n",
      "text/plain": [
       "<Figure size 864x576 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "sns.heatmap(correlations[correlations > 0.3], annot=True, cmap=\"Blues\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b81078d",
   "metadata": {},
   "source": [
    "From our correlation heatmap, we can identify the following features to be positively correlated (correlation coefficient greater than 0.3) to `HeartDisease`:\n",
    "\n",
    "- `Oldpeak`\n",
    "- `MaxHR`\n",
    "- `ChestPainType_ATA` \n",
    "- `ExerciseAngina_Y`\n",
    "- `ST_Slope_Flat`\n",
    "- `ST_Slope_Up`\n",
    "\n",
    "The correlation coefficient threshold was chosen arbitrarily. Surprisingly, `Cholesterol` is not strongly correlated to `HeartDisease`. We can consider ignoring the feature for now.\n",
    "\n",
    "Given everything we have attempted so far, we can narrow down our features to the following:\n",
    "\n",
    "- `Oldpeak`\n",
    "- `Sex_M`\n",
    "    - It has a relatively low value for the coefficient, but given what we observed in our EDA, let's also take it into account.\n",
    "- `ExerciseAngina_Y`\n",
    "- `ST_Slope_Flat`\n",
    "- `ST_Slope_Up`\n",
    "\n",
    "Time to create our model using these features!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "143bf6f0",
   "metadata": {},
   "source": [
    "## Building a Classifier with One Feature\n",
    "\n",
    "We will split our dataset into a training and a test set first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6b14956b",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_clean.drop([\"HeartDisease\"], axis=1)\n",
    "y = df_clean[\"HeartDisease\"]\n",
    "\n",
    "X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.15, random_state = 417)\n",
    "\n",
    "features = [\n",
    "    \"Oldpeak\",\n",
    "    \"Sex_M\",\n",
    "    \"ExerciseAngina_Y\",\n",
    "    \"ST_Slope_Flat\",\n",
    "    \"ST_Slope_Up\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2131229",
   "metadata": {},
   "source": [
    "We will start with creating a model for each of the features above and evaluate their performance using accuracy as a metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6ddcfc26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The k-NN classifier trained on Oldpeak and with k = 3 has an accuracy of 58.70%\n",
      "The k-NN classifier trained on Sex_M and with k = 3 has an accuracy of 61.59%\n",
      "The k-NN classifier trained on ExerciseAngina_Y and with k = 3 has an accuracy of 73.19%\n",
      "The k-NN classifier trained on ST_Slope_Flat and with k = 3 has an accuracy of 81.88%\n",
      "The k-NN classifier trained on ST_Slope_Up and with k = 3 has an accuracy of 55.07%\n"
     ]
    }
   ],
   "source": [
    "for feature in features:\n",
    "    knn = KNeighborsClassifier(n_neighbors = 3)\n",
    "    knn.fit(X_train[[feature]], y_train)\n",
    "    accuracy = knn.score(X_val[[feature]], y_val)\n",
    "    print(f\"The k-NN classifier trained on {feature} and with k = 3 has an accuracy of {accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76171c65",
   "metadata": {},
   "source": [
    "Our best forming model, with an accuracy of ~`82%`, was trained on the `ST_Slope_Flat` feature with `ExerciseAngina_Y` being a close second. These make sense given the data distributions we saw previously.\n",
    "\n",
    "We will train a model using all of these features next."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cacc6c3",
   "metadata": {},
   "source": [
    "## Building a Classifier with Multiple Features\n",
    "\n",
    "Before training on all of the above features, we need to normalize the data first. We will use scikit-learn's MinMaxScaler to scale the values between 0 and 1, and then train the model again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "cbbf5de2",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = MinMaxScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train[features])\n",
    "X_val_scaled = scaler.transform(X_val[features])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "96a81789",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 83.33\n"
     ]
    }
   ],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors = 3)\n",
    "knn.fit(X_train_scaled, y_train)\n",
    "accuracy = knn.score(X_val_scaled, y_val)\n",
    "print(f\"Accuracy: {accuracy*100:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d51e1f",
   "metadata": {},
   "source": [
    "The model's accuracy jumped to ~`83%`! That's not a significant improvement, but still a good start. Using all these features results in an improved model, but, let's see what parameters/hyperparameters might be optimal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30f4b42d",
   "metadata": {},
   "source": [
    "## Hyperparameter Optimization\n",
    "\n",
    "Let's prepare our data first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "531aaa84",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_clean.drop([\"HeartDisease\"], axis=1)\n",
    "y = df_clean[\"HeartDisease\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state = 417)\n",
    "\n",
    "features = [\n",
    "    \"Oldpeak\",\n",
    "    \"Sex_M\",\n",
    "    \"ExerciseAngina_Y\",\n",
    "    \"ST_Slope_Flat\",\n",
    "    \"ST_Slope_Up\"\n",
    "]\n",
    "\n",
    "scaler = MinMaxScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train[features])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "364195b9",
   "metadata": {},
   "source": [
    "We will use grid search to explore the following range of values for a couple of hyperparameters:\n",
    "\n",
    "- For `k` (`n_neighbors` in scikit-learn) we will use values in the range of 1 to 20.\n",
    "- We will use two distance metrics - `minkwoski` and `manhattan`.\n",
    "\n",
    "`minkwoski` is the default metric for KNeighborsClassifier in sklearn, so we don't expect `manhattan` to do better. But, let's see what happens!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8e1a3cdd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(estimator=KNeighborsClassifier(),\n",
       "             param_grid={'metric': ['minkowski', 'manhattan'],\n",
       "                         'n_neighbors': range(1, 20)},\n",
       "             scoring='accuracy')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_params = {\"n_neighbors\": range(1, 20),\n",
    "                \"metric\": [\"minkowski\", \"manhattan\"]\n",
    "              }\n",
    "\n",
    "knn = KNeighborsClassifier()\n",
    "knn_grid = GridSearchCV(knn, grid_params, scoring='accuracy')\n",
    "knn_grid.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2a22a9e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(83.43507030603806, {'metric': 'minkowski', 'n_neighbors': 19})"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_grid.best_score_*100, knn_grid.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e4334d3",
   "metadata": {},
   "source": [
    "Our best model had an accuracy of ~`83%` with `19` `n_neighbors` and `minkowski` as the distance metric. \n",
    "\n",
    "The accuracy of this model is only slightly better. Since `GridSearchCV` employs a cross-validation approach, it is reasonable to assume that this is a better estimate of how the model performs compared to our prior attempt.\n",
    "\n",
    "We will evaluate our model on our test set now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c42a3c",
   "metadata": {},
   "source": [
    "## Model Evaluation on Test Set\n",
    "\n",
    "We need to first normalize our test set similar to how we scaled our training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2086c66d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Model Accuracy on test set: 86.96\n"
     ]
    }
   ],
   "source": [
    "X_test_scaled = scaler.transform(X_test[features])\n",
    "predictions = knn_grid.best_estimator_.predict(X_test_scaled)\n",
    "accuracy = accuracy_score(y_test, predictions)\n",
    "print(f\" Model Accuracy on test set: {accuracy*100:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf618cba",
   "metadata": {},
   "source": [
    "Our model got an accuracy of ~`87`%. That's really good! This means that our model is likely to correctly predict whether a patient is at risk for a heart disease ~`87`% of the time.\n",
    "\n",
    "However, the accuracy being higher than the one before raises some flags.\n",
    "\n",
    "One explanation that could explain is to look at how the data is distributed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d05278ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distribution of patients by their sex in the entire dataset\n",
      "1    724\n",
      "0    193\n",
      "Name: Sex_M, dtype: int64\n",
      "\n",
      "Distribution of patients by their sex in the training dataset\n",
      "1    615\n",
      "0    164\n",
      "Name: Sex_M, dtype: int64\n",
      "\n",
      "Distribution of patients by their sex in the test dataset\n",
      "1    109\n",
      "0     29\n",
      "Name: Sex_M, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(\"Distribution of patients by their sex in the entire dataset\")\n",
    "print(X.Sex_M.value_counts())\n",
    "\n",
    "print(\"\\nDistribution of patients by their sex in the training dataset\")\n",
    "print(X_train.Sex_M.value_counts())\n",
    "\n",
    "print(\"\\nDistribution of patients by their sex in the test dataset\")\n",
    "print(X_test.Sex_M.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4818a3b",
   "metadata": {},
   "source": [
    "We used `Sex` as one of our features for training the model.\n",
    "\n",
    "- `X` had `724` males and `193` females.\n",
    "- `X_train` had `615` males and `164` females.\n",
    "- `X_test` had `109` males and `29` females.\n",
    "\n",
    "\n",
    "We can see that the above datasets have a significantly higher number of male patients than female ones. We briefly mentioned previously that this could present a bias because of this imbalance in our dataset and we can see it potentially impacts our model. If the test dataset doesn't have that many female patients and the model was trained on a dataset with more male patients, then it is understandable it has better accuracy on the test set. Of course, there could be other factors contributing to this discrepancy."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08f92bf4",
   "metadata": {},
   "source": [
    "## Summary and Next Steps\n",
    "\n",
    "Our final model was trained using the following features:\n",
    "\n",
    "- `Oldpeak`\n",
    "- `Sex_M`\n",
    "- `ExerciseAngina_Y`\n",
    "- `ST_Slope_Flat`\n",
    "- `ST_Slope_Up`\n",
    "\n",
    "and had a test set accuracy of `86.96`%. However, given the limitations of our data this accuracy might not be indicative of a well performing model.\n",
    "\n",
    "There are quite a few things we could try next to get better results:\n",
    "\n",
    "- Try out different features.\n",
    "- Expand the grid search parameters to identify more optimal hyperparameters.\n",
    "- Explore other algorithms that might perform better than k-NN.\n",
    "- Try and collect more data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
