ml-004 · Clean two columns and calculate F1 — Python

Quick tour · approximately 10 minutes

Problem statement

Open this practice in GitHub Codespaces runs in your browser; open this folder under practices/ after it starts

A small synthetic customer table (n = 300, generated in-script with a fixed seed) arrives with two messy columns. is_returning should be Boolean but is stored as inconsistent strings — “True”, “true”, “FALSE”, “False”, plus a few stray “yes” entries that cannot be mapped with confidence and must become missing. spend should be numeric but is stored as text, including exactly one value with a leading “\(" and exactly one literal "N/A". Clean both columns (recognized Boolean strings to true/false, "yes" to missing; strip the "\)”, turn “N/A” into missing), drop the rows left with any missing value and report how many were lost. Then make a seeded stratified 70/30 train/validation split on the binary churned label, fit a logistic regression of churned on spend and is_returning, predict on the validation set at the 0.5 probability threshold, and compute precision, recall, and F1 by hand from the four confusion counts (TP, FP, FN, TN) — no metric library calls. Python uses numpy/pandas with scikit-learn’s LogisticRegression; R uses base R with dplyr and glm(family = binomial).

Learning objectives

After completing this practice you should be able to:

  • normalize an inconsistent Boolean string column with a lowercase-then-map dictionary, sending unrecognized tokens to missing instead of guessing
  • clean a text numeric column (strip a “$” prefix, map “N/A” to missing) and let anything unexpected fail loudly with errors="raise"
  • write a seeded stratified 70/30 split by hand with a per-class permutation
  • fit LogisticRegression, take predict_proba, and threshold at 0.5
  • compute precision, recall, and F1 directly from TP/FP/FN — including the zero-denominator conventions — and explain why F1 can sit far below accuracy

Concepts and definitions

See concepts for the shared background. Python-specific notes: Series.map(dict) returns NaN for keys missing from the dict, which is exactly the “unrecognized token becomes missing” behavior we want; .astype("boolean") (nullable boolean dtype) keeps True/False/pd.NA in one column; pd.to_numeric(..., errors="raise") turns any leftover junk into an immediate exception rather than a silent NaN.

Input / output description

Item Description
Input One in-script table, 300 rows — customer_id (str), is_returning (str: “True”/“true”/“FALSE”/“False” + six “yes”), spend (str: “%.2f” numbers, one “$”-prefixed, one “N/A”), churned (int 0/1). Built with numpy.random.default_rng(7).
Output Printed diagnostics: rows dropped in cleaning, train/valid sizes with positive shares, the four confusion counts, and precision/recall/F1 at threshold 0.5. main() also returns the same numbers as a dict.

Before-coding checklist

My attempt

Work here. Run cells interactively; nothing below overwrites this section.

# Milestone 1: generate the messy table (default_rng(7), n=300; Boolean
# strings in mixed case + six "yes"; spend as text with one "$" and one "N/A")
# Milestone 2: clean both columns (lowercase-map the Booleans, strip "$",
# "N/A" -> missing), drop rows with missing values, report how many
# Milestone 3: seeded stratified 70/30 split by hand; check the positive
# share of train vs valid
# Milestone 4: fit LogisticRegression on spend + is_returning; predict_proba
# on valid and threshold at 0.5
# Milestone 5: count TP/FP/FN/TN, then precision, recall, F1 by hand
# (mind the zero-denominator cases)

Hint 1

For the Boolean column, think dictionary, not if-chains: raw.str.strip().str.lower().map({"true": True, "false": False}) — any token not in the dict (like “yes”) comes back missing automatically. For the split, “stratified” just means: do the shuffle-and-slice separately inside each class of churned, then concatenate the two train parts and the two valid parts.

Hint 2

Spend: raw.str.strip().str.lstrip("$"), then .mask(s == "N/A"), then pd.to_numeric(s, errors="raise"). Split: for each groupby("churned") group, rng.permutation(group.index.to_numpy()), take the first round(0.7 * len(idx)) indices for train. Metrics: with pred = (proba >= 0.5).astype(int), TP is ((pred == 1) & (y == 1)).sum() and so on; then precision = tp / (tp + fp), recall = tp / (tp + fn), f1 = 2 * precision * recall / (precision + recall) — each guarded so a zero denominator returns 0.0 instead of raising.

Complete solution

The official, tested script lives at solutions/solution.py. Run it with:

python practices/ml-004-quick-tour-clean-and-f1/solutions/solution.py

Core of the solution (data generation and printing omitted here):

def clean_boolean(raw):
    """'True'/'true' -> True, 'FALSE'/'False' -> False, else missing."""
    mapping = {"true": True, "false": False}
    return raw.str.strip().str.lower().map(mapping).astype("boolean")


def clean_spend(raw):
    """Strip a leading '$', turn 'N/A' into missing, parse to float."""
    s = raw.str.strip().str.lstrip("$")
    s = s.mask(s == "N/A")
    return pd.to_numeric(s, errors="raise")


def stratified_split(df, label_col="churned", train_frac=0.7, seed=7):
    rng = np.random.default_rng(seed)
    train_parts, valid_parts = [], []
    for _, group in df.groupby(label_col):
        idx = rng.permutation(group.index.to_numpy())
        n_train = int(round(train_frac * len(idx)))
        train_parts.append(df.loc[idx[:n_train]])
        valid_parts.append(df.loc[idx[n_train:]])
    return (pd.concat(train_parts).sort_index(),
            pd.concat(valid_parts).sort_index())


def precision_recall_f1(tp, fp, fn):
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = (2.0 * precision * recall / (precision + recall)
          if (precision + recall) > 0 else 0.0)
    return {"precision": precision, "recall": recall, "f1": f1}

With seed 7 the script reports: 7 rows dropped (six “yes” + one “N/A”), 293 clean rows, a 205/88 split with positive shares 0.312 / 0.318, confusion counts TP=8 FP=5 FN=20 TN=55, precision 0.6154, recall 0.2857, F1 0.3902 — while plain accuracy would be a flattering 0.716.

Line-by-line explanation

  • raw.str.strip().str.lower(): normalizes the token before the lookup so every casing of true/false hits the same dictionary key.
  • .map(mapping): tokens absent from the dict (“yes”) become NaN — the “don’t guess” rule falls out of the API for free.
  • .astype("boolean"): pandas’ nullable boolean dtype holds True/False/pd.NA in one column; plain bool cannot represent missing.
  • .str.lstrip("$") + .mask(s == "N/A"): handles the two known mess cases explicitly; pd.to_numeric(..., errors="raise") then guarantees any unknown token blows up instead of becoming a silent missing value.
  • dropna(subset=...) + a computed n_dropped: rows are removed, but the count is reported — cleaning should never be invisible.
  • The split loop: groupby(label_col) isolates each class, rng.permutation shuffles reproducibly, round(0.7 * len) slices, and the final sort_index() restores a stable order. Because slicing happens per class, both parts inherit the overall positive share.
  • predict_proba(...)[:, 1]: column 1 is P(churned = 1); proba >= 0.5 makes the hard calls, with >= deciding the exact-0.5 boundary.
  • confusion_counts: four vectorized boolean-AND sums — no library metric.
  • precision_recall_f1: each ratio guarded so TP+FP = 0 (no predicted positives) or a 0/0 F1 returns 0.0 by convention rather than raising.

Design decisions

  • Dictionary mapping over if/else chains. One mapping dict states the entire cleaning policy in one place, and its miss-behavior (missing, not a default) is exactly the conservative policy we want for “yes”.
  • errors="raise" on the numeric parse. The two known mess cases are handled explicitly first, so anything else is a surprise worth crashing on — quick-tour data should never train the habit of silent coercion.
  • A hand-written stratified split. train_test_split(stratify=...) would work, but writing the per-class permutation once makes the meaning of “stratified” concrete — and it is only six lines.
  • Metrics by hand. sklearn.metrics.f1_score is one import away; the point of the practice is knowing what it computes, including the zero-denominator conventions.
  • A deliberately modest F1 (0.39). Two weak features and a 0.5 threshold on ~31% positives make F1 sit far below accuracy (0.72) — the gap is the lesson, not a bug in the data.

Common mistakes

  • Mapping “yes” to True because it “obviously” means yes — fabricates a feature value from an unknown data-entry path; unrecognized tokens must become missing.
  • Using pd.to_numeric(raw, errors="coerce") on the raw column — the “$” value silently becomes missing and gets dropped, instead of being repaired.
  • Forgetting .str.lower() before the map — “True” maps but “true” becomes missing (or vice versa), quietly shrinking the table.
  • Splitting the whole table with one permutation — no stratification, so the validation positive share drifts and the metrics move with it.
  • Thresholding model.predict_proba(X)[:, 0] — that is P(class 0); the positive-class column is index 1.
  • Computing F1 as the plain average of precision and recall — it is the harmonic mean, which punishes imbalance between them.
  • Letting tp / (tp + fp) raise (or return NaN) when nothing is predicted positive instead of applying the precision-is-zero convention.

Alternative solution

The library route replaces most of the hand-rolled parts: sklearn.model_selection.train_test_split(df, test_size=0.3, stratify=df["churned"], random_state=7) for the split, and sklearn.metrics.precision_recall_fscore_support (or f1_score) for the metrics — useful as a cross-check that your hand computation matches. For the cleaning, an equally good idiom is raw.str.replace(r"^\$", "", regex=True) for the prefix and Series.replace({"N/A": np.nan}) for the sentinel; and F1 = 2*TP / (2*TP + FP + FN) is an algebraically identical one-liner that sidesteps the intermediate precision/recall divisions entirely.

Extensions and harder variants

  • Sweep the threshold from 0.1 to 0.9 and plot precision, recall, and F1 against it; find the F1-maximizing threshold on validation.
  • Add more mess: thousands separators (“1,204.50”), empty strings, and a “TRUE” with trailing whitespace — extend the cleaners without breaking the existing tests.
  • Replace the drop-rows policy with imputation (median spend, missing-as- category for the Boolean) and compare validation F1.
  • Compute a 95% bootstrap confidence interval for F1 by resampling the validation set; is 0.39 distinguishable from a coin-flip baseline?
  • Repeat the split with 20 different seeds and report the spread of F1 — how much of the number is signal vs split luck?

Review questions

Answer out loud before expanding.

“yes” violates the column’s True/False contract, so it signals a different data-entry path whose meaning is unverified — mapping it to True fabricates data. Practically, the six “yes” rows would survive cleaning with is_returning = True, n_dropped would fall from 7 to 1, and those rows would enter the split, the fit, and possibly the confusion counts — every downstream number changes on the strength of a guess.

It guarantees the train and validation positive shares match the overall share (0.312 vs 0.318 here) instead of drifting with the draw. F1 is built from TP, FP, FN — all positive-class quantities — so shifting the number of validation positives directly rescales recall’s denominator and moves F1; accuracy, dominated by the majority class, barely notices the same shift.

The 20 false negatives (with the 55 true negatives propping accuracy up). Accuracy counts TN as success, and TN is the biggest cell; F1 ignores TN entirely and is dragged down by recall = 8/28 ≈ 0.286. The lesson: on imbalanced data, accuracy mostly measures how easy the negatives are, while F1 measures how well you actually find and trust positives.

errors="coerce" turns every unparseable token into missing — the known “$54.20” case would be silently destroyed rather than repaired, and any future surprise (say “12,50”) would vanish the same way. Handling the two known cases explicitly and then raising on anything else repairs what can be repaired, drops only what is truly unknowable, and guarantees new mess announces itself instead of leaking into the model as missing values.

TP = FP = 0, so precision’s denominator is 0 → precision = 0 by convention; recall = 0/(0 + FN) = 0 (defined normally when FN > 0); their sum is 0, so F1 = 0 by the second guard. Returning 0 is right because “predicts nothing” is a legitimate — terrible — classifier that threshold sweeps and baseline comparisons must be able to score; crashing mid-sweep would conflate “worst possible score” with “undefined computation”.

Personal reflection

What was hard? What would you do differently? Date your entries.