ml-004 · Clean two columns and calculate F1 — R

Quick tour · approximately 10 minutes

Problem statement

Open this practice in GitHub Codespaces runs in your browser; open this folder under practices/ after it starts

A small synthetic customer table (n = 300, generated in-script with a fixed seed) arrives with two messy columns. is_returning should be Boolean but is stored as inconsistent strings — “True”, “true”, “FALSE”, “False”, plus a few stray “yes” entries that cannot be mapped with confidence and must become missing. spend should be numeric but is stored as text, including exactly one value with a leading “\(" and exactly one literal "N/A". Clean both columns (recognized Boolean strings to true/false, "yes" to missing; strip the "\)”, turn “N/A” into missing), drop the rows left with any missing value and report how many were lost. Then make a seeded stratified 70/30 train/validation split on the binary churned label, fit a logistic regression of churned on spend and is_returning, predict on the validation set at the 0.5 probability threshold, and compute precision, recall, and F1 by hand from the four confusion counts (TP, FP, FN, TN) — no metric library calls. Python uses numpy/pandas with scikit-learn’s LogisticRegression; R uses base R with dplyr and glm(family = binomial).

Learning objectives

After completing this practice you should be able to:

  • normalize an inconsistent Boolean string column with tolower(trimws(x)) plus explicit assignment, sending unrecognized tokens to NA instead of guessing
  • clean a text numeric column (strip a “$” prefix with sub(), map “N/A” to NA before as.numeric() so the conversion never warns)
  • write a seeded stratified 70/30 split by hand with a per-class sample()
  • fit glm(..., family = binomial()), get probabilities with predict(type = "response"), and threshold at 0.5
  • compute precision, recall, and F1 directly from TP/FP/FN — including the zero-denominator conventions — and explain why F1 can sit far below accuracy

Concepts and definitions

See concepts for the shared background. R-specific notes: rep(NA, n) is logical NA, so filling it with TRUE/FALSE keeps a clean logical vector; as.numeric() on a character vector emits a “NAs introduced by coercion” warning for unparseable tokens — the tidy pattern is to set the known sentinels to NA first so any remaining warning is a real surprise; predict.glm returns log-odds by default, so type = "response" is required to get probabilities.

Input / output description

Item Description
Input One in-script data.frame, 300 rows — customer_id (chr), is_returning (chr: “True”/“true”/“FALSE”/“False” + six “yes”), spend (chr: “%.2f” numbers, one “$”-prefixed, one “N/A”), churned (int 0/1). Built with set.seed(7).
Output Printed diagnostics: rows dropped in cleaning, train/valid sizes with positive shares, the four confusion counts, and precision/recall/F1 at threshold 0.5. main() also returns the same numbers invisibly as a list.

Before-coding checklist

My attempt

Work here. Run cells interactively; nothing below overwrites this section.

# Milestone 1: generate the messy table (set.seed(7), n=300; Boolean strings
# in mixed case + six "yes"; spend as text with one "$" and one "N/A")
# Milestone 2: clean both columns (tolower+map the Booleans, strip "$",
# "N/A" -> NA), drop rows with NA, report how many
# Milestone 3: seeded stratified 70/30 split by hand; check the positive
# share of train vs valid
# Milestone 4: fit glm(churned ~ spend + is_returning, family = binomial());
# predict(type = "response") on valid and threshold at 0.5
# Milestone 5: count TP/FP/FN/TN, then precision, recall, F1 by hand
# (mind the zero-denominator cases)

Hint 1

For the Boolean column, normalize first: low <- tolower(trimws(x)), start from out <- rep(NA, length(x)), and assign only the two tokens you recognize — everything else (like “yes”) stays NA by construction. For the split, “stratified” just means: sample() the row indices separately inside each class of churned, take 70% of each, and combine.

Hint 2

Spend: s <- trimws(x); s[s == "N/A"] <- NA; as.numeric(sub("^\\$", "", s)) — sentinel first, so as.numeric never warns. Split: for each level, rows <- which(df$churned == lvl), then sample(rows, round(0.7 * length(rows))) into the train index; validation is df[-train_idx, ]. Model: proba <- predict(model, newdata = valid, type = "response") (not the default link scale!), pred <- as.integer(proba >= 0.5). Metrics: tp <- sum(pred == 1 & y == 1) and so on; then precision = tp/(tp+fp), recall = tp/(tp+fn), f1 = 2*precision*recall/(precision+recall) — each guarded so a zero denominator returns 0 instead of NaN.

Complete solution

The official, tested script lives at solutions/solution.R. Run it with:

Rscript practices/ml-004-quick-tour-clean-and-f1/solutions/solution.R

Core of the solution (data generation and printing omitted here):

clean_boolean <- function(x) {
  low <- tolower(trimws(x))
  out <- rep(NA, length(x))
  out[low == "true"] <- TRUE
  out[low == "false"] <- FALSE
  out
}

clean_spend <- function(x) {
  s <- trimws(x)
  s[s == "N/A"] <- NA
  as.numeric(sub("^\\$", "", s))
}

stratified_split <- function(df, label_col = "churned",
                             train_frac = 0.7, seed = 7L) {
  set.seed(seed)
  train_idx <- integer(0)
  for (lvl in sort(unique(df[[label_col]]))) {
    rows <- which(df[[label_col]] == lvl)
    train_idx <- c(train_idx, sample(rows, round(train_frac * length(rows))))
  }
  train_idx <- sort(train_idx)
  list(train = df[train_idx, , drop = FALSE],
       valid = df[-train_idx, , drop = FALSE])
}

precision_recall_f1 <- function(tp, fp, fn) {
  precision <- if (tp + fp > 0) tp / (tp + fp) else 0
  recall <- if (tp + fn > 0) tp / (tp + fn) else 0
  f1 <- if (precision + recall > 0) {
    2 * precision * recall / (precision + recall)
  } else {
    0
  }
  list(precision = precision, recall = recall, f1 = f1)
}

With seed 7 the script reports: 7 rows dropped (six “yes” + one “N/A”), 293 clean rows, a 205/88 split with positive shares 0.415 / 0.420, confusion counts TP=14 FP=11 FN=23 TN=40, precision 0.5600, recall 0.3784, F1 0.4516 — while plain accuracy would be a flattering 0.614. (R’s RNG differs from numpy’s, so the exact figures differ from the Python page while every qualitative conclusion is identical.)

Line-by-line explanation

  • tolower(trimws(x)): normalizes each token before matching so every casing of true/false compares equal.
  • out <- rep(NA, length(x)) + two assignments: starting from all-NA and assigning only recognized tokens means “yes” stays NA without ever being mentioned — the “don’t guess” rule by construction. rep(NA, ...) is logical, so the vector stays logical after the TRUE/FALSE fills.
  • s[s == "N/A"] <- NA before as.numeric(sub("^\\$", "", s)): the known sentinel is removed first, so a coercion warning from as.numeric can only mean a genuinely unexpected token — warnings stay meaningful.
  • dplyr::filter(out, !is.na(is_returning), !is.na(spend)) plus a computed n_dropped: rows are removed, but the count is reported — cleaning should never be invisible.
  • The split loop: which(df[[label_col]] == lvl) isolates each class, sample() under set.seed(7) shuffles reproducibly, round(0.7 * n) sizes the train share, and df[-train_idx, ] is everything else. Because sampling happens per class, both parts inherit the overall positive share.
  • predict(model, newdata = valid, type = "response"): glm predictions default to the link (log-odds) scale; type = "response" gives probabilities, and proba >= 0.5 makes the hard calls.
  • confusion_counts: four vectorized sum(pred == a & y == b) counts — no metric package.
  • precision_recall_f1: each ratio guarded so TP+FP = 0 (no predicted positives) or a 0/0 F1 returns 0 by convention rather than NaN.

Design decisions

  • All-NA-then-assign over nested ifelse. One initialization plus two assignments states the cleaning policy in three lines, keeps the vector logical, and makes the miss-behavior (NA, not a default) structural.
  • Sentinel-before-coercion in clean_spend. Handling “N/A” first means as.numeric runs warning-free on good data, so any future warning is a real signal — quick-tour data should never train the habit of suppressWarnings().
  • A hand-written stratified split. rsample or caret would do it, but writing the per-class sample() once makes the meaning of “stratified” concrete — and it is only seven lines.
  • Metrics by hand. yardstick::f_meas is one call away; the point of the practice is knowing what it computes, including the zero-denominator conventions.
  • A deliberately modest F1 (0.45). Two weak features and a 0.5 threshold on ~42% positives keep F1 well below accuracy (0.61) — the gap is the lesson, not a bug in the data.

Common mistakes

  • Mapping “yes” to TRUE because it “obviously” means yes — fabricates a feature value from an unknown data-entry path; unrecognized tokens must become NA.
  • Calling as.numeric() on the raw column and ignoring the coercion warning — the “$” value silently becomes NA and gets dropped, instead of being repaired.
  • Using as.logical(x) for the cleaning — it happens to map “True”/“FALSE” variants but is exactly the silent-guess habit this practice warns against, and its rules surprise (“T” → TRUE).
  • Forgetting type = "response" in predict.glm — log-odds thresholded at 0.5 is a probability threshold of about 0.62, silently shifting every confusion count.
  • Splitting with one global sample() — no stratification, so the validation positive share drifts and the metrics move with it.
  • Computing F1 as the plain average of precision and recall — it is the harmonic mean, which punishes imbalance between them.
  • Letting tp / (tp + fp) produce NaN when nothing is predicted positive instead of applying the precision-is-zero convention.

Alternative solution

A tidyverse-flavored variant does the cleaning inside one mutate with dplyr::case_match(tolower(trimws(is_returning)), "true" ~ TRUE, "false" ~ FALSE, .default = NA) and readr::parse_number(spend, na = "N/A") (which strips “$” itself), and the counts with dplyr::count(valid, churned, pred). For the metrics, yardstick::precision/recall/f_meas provide a cross-check that your hand computation matches, and F1 = 2*TP / (2*TP + FP + FN) is an algebraically identical one-liner that sidesteps the intermediate divisions entirely. A table(pred, y) confusion matrix works too — but index it by names (tab["1", "1"]), since a class absent from the predictions drops its row.

Extensions and harder variants

  • Sweep the threshold from 0.1 to 0.9 and plot precision, recall, and F1 against it; find the F1-maximizing threshold on validation.
  • Add more mess: thousands separators (“1,204.50”), empty strings, and a “TRUE” with trailing whitespace — extend the cleaners without breaking the existing tests.
  • Replace the drop-rows policy with imputation (median spend, missing-as- category for the Boolean) and compare validation F1.
  • Compute a 95% bootstrap confidence interval for F1 by resampling the validation set; how wide is it at n = 88?
  • Repeat the split with 20 different seeds and report the spread of F1 — how much of the number is signal vs split luck?

Review questions

Answer out loud before expanding.

“yes” violates the column’s True/False contract, so it signals a different data-entry path whose meaning is unverified — mapping it to TRUE fabricates data. Practically, the six “yes” rows would survive cleaning with is_returning = TRUE, n_dropped would fall from 7 to 1, and those rows would enter the split, the fit, and possibly the confusion counts — every downstream number changes on the strength of a guess.

as.numeric coerces every unparseable token to NA with one generic warning, so “N/A” (expected) and a future “12,50” (a bug) become indistinguishable — and once you get used to the warning, you stop reading it. Removing the known sentinel first makes the happy path warning-free, so any warning that does appear is guaranteed to be new mess worth investigating. Same principle as Python’s errors="raise": handle the known, crash (or warn loudly) on the unknown.

The linear predictor — log-odds — not probabilities. Log-odds of 0.5 corresponds to a probability of plogis(0.5) ≈ 0.62, so thresholding the default output at 0.5 is really a ~0.62 probability cutoff: fewer predicted positives, inflated precision, depressed recall, and no error or warning anywhere. type = "response" applies the inverse link and returns probabilities in [0, 1].

The 23 false negatives (with the 40 true negatives propping accuracy up). Accuracy counts TN as success; F1 ignores TN entirely and is dragged down by recall = 14/37 ≈ 0.378. The lesson: accuracy mostly measures how easy the negatives are, while F1 measures how well you actually find and trust positives — report it (or precision and recall separately) whenever the positive class is the one you care about.

TP = FP = 0, so precision’s denominator is 0 → precision = 0 by convention; recall = 0/(0 + FN) = 0 (defined normally when FN > 0); their sum is 0, so F1 = 0 by the second guard. Returning 0 is right because “predicts nothing” is a legitimate — terrible — classifier that threshold sweeps and baseline comparisons must be able to score; a NaN would poison downstream summaries and conflate “worst possible score” with “undefined computation”.

Personal reflection

What was hard? What would you do differently? Date your entries.