Discriminative models directly model the posterior class probability p(y|x), making them well-suited for classification tasks. They estimate decision boundaries by learning what separates one class from another, without explicitly modeling how the data in each class is distributed. Logistic regression is the prototypical discriminative model: it learns the parameters θ that maximize the log-likelihood of the observed labels given the features. This direct approach to the task at hand often requires less data and can be more computationally efficient than modeling the full joint distribution.

Generative models take a different approach by modeling the full joint distribution p(x,y) through the factorization p(x,y) = p(x|y)p(y). They estimate the class-conditional likelihood p(x|y) and class prior p(y), then use Bayes rule to compute the posterior: p(y|x) = p(x|y)p(y) / p(x). This approach captures the underlying data generation process and allows for rich modeling of the features. Generative models can generate new synthetic samples, impute missing values, and perform other tasks beyond classification.

The key trade-off: discriminative models focus computational resources on the classification boundary, often achieving better performance with limited data. Generative models must model the entire feature distribution, which requires more samples but enables additional applications. Discriminative models make fewer assumptions and are more robust to model misspecification. Generative models leverage strong structural assumptions (e.g., Gaussian distributions) to achieve efficiency and interpretability when those assumptions hold.

Making the choice

Choose discriminative models (e.g., logistic regression, SVMs) when: you have abundant labeled data, computational efficiency is critical, features may have complex dependencies, the data distribution is unclear or non-standard. Choose generative models (e.g., Naive Bayes, GDA) when: labeled data is limited, you need to model the data distribution, you want interpretable probabilistic estimates, the feature distributions are well-understood or restricted (discrete, Gaussian), or you need to generate synthetic samples or handle missing data.

Practical workflow

Start with exploratory data analysis: visualize class distributions, check for Gaussian-like shapes, examine feature correlations. If the data appears well-structured and Gaussian, GDA is a strong candidate. If the relationships are complex or the data is high-dimensional and sparse (like text), Naive Bayes or logistic regression are better choices. In practice, implement both and evaluate via cross-validation. Combine both approaches in an ensemble: use the class probability from a discriminative model as a feature in a generative model, or vice versa. Modern systems often use generative models for feature engineering (embeddings) and discriminative models for final classification.

Gaussian Discriminant Analysis (GDA) is a generative classifier that models each class as a multivariate Gaussian distribution. The model assumes that given the class label y, the feature vector x is drawn from a normal distribution: p(x|y=k) = N(μₖ, Σ), where μₖ is the class-specific mean and Σ is the shared covariance matrix. This assumption is powerful because multivariate Gaussians are well-understood, admit closed-form MLEs, and produce linear decision boundaries—a powerful combination for interpretability and efficiency.

The multivariate Gaussian PDF is: p(x) = (1/(2π)^(n/2) |Σ|^(1/2)) exp(-1/2 (x-μ)ᵀ Σ⁻¹ (x-μ)). For a two-class problem with shared covariance, the decision boundary (where p(y=1|x) = p(y=0|x)) simplifies to a linear equation: θ₀ + θ₁x₁ + ... + θₙxₙ = 0, where θ = Σ⁻¹(μ₁ - μ₀). This linearity is striking: despite the Gaussian assumption being about the conditional distributions, the resulting boundary is a hyperplane.

GDA is particularly powerful when the Gaussian assumption is valid. Real-world data such as iris measurements, neural activations, and image features often approximate Gaussians. The shared covariance assumption (homoscedasticity) is restrictive but enables closed-form solutions. When covariances are allowed to differ per class, the boundary becomes quadratic, which can overfit on small datasets but sometimes better reflects reality. The linear boundary in the standard GDA case leads to a natural connection with logistic regression, which we explore later.

Geometric intuition

In the feature space, each class occupies a region centered around its mean μₖ, with spread determined by Σ. The inverse covariance Σ⁻¹ (the precision matrix) defines the metric—directions with low variance (high precision) have steep decision boundaries perpendicular to them. Points equidistant (in the Mahalanobis sense) from both means lie on the decision hyperplane. Features with high variance contribute less to the decision (lower precision), while correlated features are automatically deweighted by the Σ⁻¹ term.

Parameter significance

The prior φ = P(y=1) biases the decision boundary: if one class is much more common, the boundary shifts toward the minority class to account for the base rate. The difference (μ₁ - μ₀) determines the direction of the boundary. The covariance Σ controls its shape and orientation—high-variance directions allow points far from the mean to still have reasonable likelihood, affecting how far the boundary extends. Understanding these components helps diagnose when GDA is appropriate and how to interpret learned models in terms of feature importance and class structure.

Fitting GDA means estimating the parameters φ, μ₀, μ₁, and Σ via maximum likelihood estimation. Given m training examples {(x⁽ⁱ⁾, y⁽ⁱ⁾)}, the closed-form MLEs are elegant. The prior is: φ = (1/m) Σᵢ I(y⁽ⁱ⁾=1). The means are computed per-class: μ₁ = (Σᵢ:y⁽ⁱ⁾=1 x⁽ⁱ⁾) / (# examples in class 1), and similarly for μ₀. The shared covariance is: Σ = (1/m) Σᵢ (x⁽ⁱ⁾ - μ_{y⁽ⁱ⁾})(x⁽ⁱ⁾ - μ_{y⁽ⁱ⁾})ᵀ, pooling residuals from both classes.

These closed-form solutions are major advantages of GDA: no iterative optimization (unlike logistic regression), no hyperparameter tuning, and computational cost O(nm² + m³) dominated by the covariance matrix inversion. Training is fast even on large datasets. Making a prediction on a new point x amounts to: computing p(x|y=1) and p(x|y=0) from the Gaussian PDFs, multiplying by priors φ, and selecting argmax_y p(y|x). This simplicity makes GDA practical for real-time systems and suitable for situations where retraining must be fast.

The MLE is consistent and asymptotically efficient for data actually drawn from the assumed model. If the Gaussian assumption holds, GDA recovers the true parameters with enough data. However, if the true distribution differs from Gaussian, the estimates may be biased. A key observation: the MLE for Σ is biased (dividing by m instead of m-2), but this bias decreases with sample size and is often acceptable in practice. For small samples, using m-1 or m-2 as the divisor can reduce bias.

Implementation recipe

To fit GDA: (1) compute φ as the fraction of positive examples; (2) compute μ₀ and μ₁ as class means; (3) compute Σ as the pooled covariance; (4) invert Σ to get Σ⁻¹ (the precision matrix). For numerical stability, add a small regularization term λI to Σ before inverting, or use a robust inversion algorithm (LU decomposition, SVD). After fitting, compute the learned linear boundary θ = Σ⁻¹(μ₁ - μ₀) and bias θ₀ = (1/2) θᵀ(μ₁ + μ₀) + log(φ/(1-φ)). For new data, classify as y=1 if θᵀx + θ₀ > 0, else y=0.

Robustness considerations: if Σ is singular or nearly singular, the inversion fails. Solutions: add regularization λI (Σ_reg = Σ + λI), use pseudoinverse, or add a small amount of noise to the data. If classes have very different covariances, the shared-Σ assumption is violated; consider using separate covariances (QDA, Quadratic Discriminant Analysis) at the cost of more parameters. Regularization and robustness are practical concerns that make the difference between working and broken code.

Both GDA and logistic regression produce linear decision boundaries, but via different paths. GDA models p(x|y) and p(y), then uses Bayes rule to compute p(y|x). Logistic regression directly models p(y|x) as σ(θᵀx) where σ is the sigmoid function. A remarkable fact: if the class-conditional densities p(x|y=0) and p(x|y=1) are Gaussian with shared covariance Σ, then the posterior p(y=1|x) is logistic. That is, p(y=1|x) can be written as σ(θ₀ + θ₁x₁ + ... + θₙxₙ) for some θ values determined by μ₀, μ₁, Σ, and φ.

This connection runs deep: the linear logit log(p(y=1|x) / p(y=0|x)) is exactly θᵀx + θ₀ for both models. With infinite data and correct model assumptions, GDA and logistic regression converge to the same decision boundary. However, with finite data, they differ. GDA is a parametric model that makes stronger assumptions about the feature distribution. Logistic regression makes weaker assumptions, only assuming linearity of the decision boundary. This fundamental difference shapes their empirical performance.

GDA excels when: the Gaussian assumption is accurate and sample size is small (≤ 100 examples per class), because it leverages the structure encoded in the Gaussian model. When the Gaussian assumption is violated, logistic regression typically outperforms GDA because it doesn't waste modeling capacity on incorrect distributional assumptions. GDA also requires estimating O(n²) parameters (the covariance matrix), while logistic regression estimates O(n). For high-dimensional data with limited samples, logistic regression generalizes better.

Empirical comparison

On data where p(x|y) is actually Gaussian: GDA wins with small samples (fewer parameters to learn, structure helps), similar performance with large samples (enough data to overcome the assumption). On data where p(x|y) is not Gaussian: logistic regression typically wins, especially when features have heavy tails, multimodal distributions, or are discrete. In practice, logistic regression is often the safer default because the Gaussian assumption is restrictive. However, GDA remains useful for small-data regimes, interpretable probabilistic modeling, and domains where the Gaussian assumption is justified by theory or prior work.

When to use each

Use GDA: small datasets (< 500 examples), clearly Gaussian-like features, need interpretable probabilistic model, fast training is critical, or you want to model the feature distribution. Use logistic regression: large datasets, high-dimensional features, non-Gaussian distributions expected, robustness to outliers is important, or when parsimony (fewer parameters) is valued. In modern practice, logistic regression is the default for most classification problems due to its robustness and effectiveness. GDA is most valuable in specialized domains (e.g., bioinformatics) where Gaussian assumptions are reasonable and data is limited.

Naive Bayes is a probabilistic classifier that applies Bayes rule to compute p(y|x). The key simplifying assumption is conditional independence: given the class y, all features x₁, x₂, ..., xₙ are independent. Formally, p(x₁, x₂, ..., xₙ | y) = ∏ᵢ p(xᵢ | y). While this "naive" assumption is rarely true in real data (words in text are correlated, pixels in images are correlated), it leads to a simple, scalable model that often performs surprisingly well. The independence assumption reduces the parameter count from O(2ⁿ) (one parameter per possible feature combination) to O(n) (one parameter per feature per class).

The classifier computes the posterior as: p(y|x) ∝ p(y) ∏ᵢ p(xᵢ|y). To classify a new instance, compute this posterior for each class and select argmax_y. The model naturally handles missing features (just omit them from the product) and scales to high-dimensional data. For text classification, each xᵢ might be the count or presence of word i, p(xᵢ|y) is estimated from training data, and the algorithm learns which words are indicative of each class.

Despite the conditional independence assumption being violated in nearly all applications, Naive Bayes achieves strong empirical performance. Theoretically, this is understood through multiple lenses: the model is a good approximation under certain conditions, the parameters compensate for the independence violation, and the 0/1 loss function is forgiving (only the sign of the log-posterior matters, not its magnitude). Practically, this robustness is one of Naive Bayes' greatest strengths—it works well across diverse domains despite its naive assumptions.

Variants for different data

Gaussian Naive Bayes assumes each feature xᵢ|y is Gaussian: p(xᵢ|y) = N(μᵢᵧ, σᵢᵧ²). This is suitable for continuous features like measurements, image pixels, or normalized counts. Multinomial Naive Bayes models word counts: p(x|y) ∝ ∏ᵢ φᵢ|ᵧˣⁱ where φᵢ|ᵧ is the probability of word i in class y. This is the standard for bag-of-words text classification. Bernoulli Naive Bayes treats features as binary: present or absent in the document. Each variant has its use case, and the choice should match the data type and generation model.

The zero-frequency problem is critical in Naive Bayes. Suppose a word like "discombobulate" never appears in the training data for the "positive" class. The naive empirical estimate is p(word="discombobulate" | y=1) = 0. When predicting on a new document containing this word, the entire posterior p(y=1|x) becomes zero because it contains a factor of zero. This is problematic: the model fails catastrophically on any document with an unseen word, even if other words strongly support a class. In text classification, unseen words are inevitable due to the zipfian distribution of natural language.

Laplace smoothing (add-one smoothing) fixes this by adding a count of one to every word in every class, as if each word appeared at least once. The smoothed probability is: p(xᵢ|y) = (count(xᵢ, y) + 1) / (count(y) + |V|), where count(y) is the total word count in class y and |V| is the vocabulary size. This ensures all probabilities are positive, prevents zero probabilities, and allows the model to handle unseen words gracefully. The denominator includes |V| to ensure probabilities sum to one: Σᵢ (count(xᵢ,y) + 1) / (count(y) + |V|) = 1.

While add-one smoothing introduces bias (it biases estimates toward 0.5 for each feature), it dramatically improves generalization. A generalized version, add-α smoothing, uses: p(xᵢ|y) = (count(xᵢ, y) + α) / (count(y) + α|V|). When α=1, this is Laplace smoothing; when α → 0, we approach empirical counts; when α increases, probabilities become more uniform. In practice, α is often tuned via cross-validation. For most text datasets, α=1 (Laplace) works well, though some domains benefit from α < 1 (less smoothing when data is abundant).

Impact on classification

Without smoothing, Naive Bayes on text fails: unseen words cause zero posteriors. With smoothing, the model gracefully handles new vocabulary. The smoothed probability of an unseen word is small (1 / (count(y) + |V|)) but nonzero, allowing documents with rare words to still be classified based on other evidence. This is subtle but powerful: Laplace smoothing makes the difference between a classifier that crashes on new data and one that robustly generalizes. Empirically, text classification with Naive Bayes typically shows 5-10% accuracy improvement with smoothing on out-of-domain test sets.

Two event models formalize how text is generated, each defining p(x|y) differently. The multivariate Bernoulli model treats each position in the vocabulary as a binary random variable: xᵢ ∈ {0, 1} indicates whether word i appears in the document. The likelihood is: p(x|y) = ∏ᵢ (φᵢ|ᵧ)^(xᵢ) (1 - φᵢ|ᵧ)^(1-xᵢ), where φᵢ|ᵧ = P(word i appears | y). This model ignores word frequency: a document with one occurrence of "excellent" and one with ten occurrences look the same. For short documents or when word presence/absence matters more than frequency, Bernoulli Naive Bayes is suitable.

The multinomial model treats text as a bag of words with counts. The document is generated by drawing words one at a time from a multinomial distribution. Given document length d and class y, the likelihood is: p(x|y) ∝ ∏ᵢ (φᵢ|ᵧ)^(xᵢ), where xᵢ is the count of word i and ∑ᵢ xᵢ = d. The multinomial model captures word frequency: seeing "excellent" ten times versus once is meaningful. For longer documents and tasks where frequency matters (e.g., distinguishing formal vs. casual writing), multinomial Naive Bayes typically outperforms Bernoulli.

Empirically, multinomial Naive Bayes achieves better performance on standard text classification benchmarks. The frequency information is informative: spam often repeats certain keywords, positive reviews emphasize good words repeatedly, and topic classification benefits from word frequency. However, Bernoulli is simpler (fewer parameters to estimate), sometimes preferred for sparse text, and occasionally competitive when combined with feature engineering. The choice should be guided by data characteristics: use Bernoulli for binary word presence, multinomial for word counts. Both benefit from Laplace smoothing and feature selection.

Feature selection and engineering

Not all words are useful. Stopwords (the, a, an, is) appear in all documents and provide little discriminative signal. Rare words might be noise or class-specific jargon. Information gain or mutual information scores measure how much each word reduces uncertainty about the class: IG(word) = ∑_y P(y) [∑_{w∈{0,1}} P(w|y) log(P(w|y)/P(w))]. Selecting the top k words (typically 1000-10000) by information gain reduces noise, speeds up training, and often improves generalization. This feature selection step is practical in production systems where computational efficiency and model interpretability are important.

Spam filtering is the canonical application of Naive Bayes. Emails are documents, features are words, and classes are {spam, ham}. The classifier learns that certain keywords (pharmaceutical terms, money-related phrases, urgency language) are strongly indicative of spam: p(word | spam) >> p(word | ham). Similarly, legitimate words (recipient's name, official company domains, common greetings) appear more in ham: p(word | ham) >> p(word | spam). Laplace smoothing is essential: new spam variants use novel keywords, and the model must handle unseen words gracefully. Modern email filters still use Naive Bayes as a core component, often combined with other signals (sender reputation, authentication checks, deep learning features).

Sentiment analysis classifies text as positive, negative, or neutral. A review corpus is labeled, and Naive Bayes learns sentiment-indicative words. Positive reviews frequently contain "excellent," "love," "wonderful," while negative reviews contain "terrible," "waste," "disappointing." The learned word probabilities p(word | sentiment) directly reveal what the model considers. This interpretability is valuable: stakeholders understand why a review was classified as negative by pointing to specific words. More advanced systems combine Naive Bayes scores with syntactic and semantic features, but Naive Bayes remains a strong baseline for sentiment tasks.

Document classification assigns documents to predefined categories (news topics, support tickets, legal documents). Naive Bayes learns topic-specific vocabulary: sports documents contain "game," "team," "score," while business documents contain "revenue," "profit," "market." Because Naive Bayes handles multiclass naturally (not just binary), it directly extends to k classes by computing p(y=j | x) for each j and selecting argmax. The model scales linearly in both training set size and vocabulary, making it practical for large document collections.

Production systems and engineering

Naive Bayes is widely used in production because of its speed, simplicity, and interpretability. Training happens offline: scan the corpus, compute word frequencies per class, apply smoothing. Inference is O(d × |V|) where d is document length, enabling real-time classification. The probabilistic outputs p(y|x) naturally serve as confidence scores: high-confidence predictions can be trusted, while borderline cases may require human review. This is especially valuable in content moderation, where false positives (blocking legitimate content) are costly.

Ensemble methods combine Naive Bayes with other classifiers: a discriminative model (logistic regression, SVM) handles the decision boundary, while Naive Bayes provides robust probability estimates. Neural networks learn semantic embeddings that capture word similarity beyond bag-of-words, and Naive Bayes operates on top of these embeddings. The modularity of Naive Bayes—it factorizes into independent word models—allows easy integration into larger systems. Modern systems often use Naive Bayes not as the final classifier but as a feature engineer or baseline, establishing performance expectations and providing interpretable signals.

Course Materials

CS229 Main Lecture Notes — Coverage of generative learning algorithms including Gaussian Discriminant Analysis and Naive Bayes

Textbooks & Papers

Ng & Jordan — On Discriminative vs Generative Classifiers — Theoretical analysis comparing generative and discriminative models with formal asymptotic arguments

Bishop — Pattern Recognition and Machine Learning — Chapter 8 covers graphical models and generative approaches to modeling data