Feature Engineering: Building Better Models with Data

8 min read

⏱ 9 min read

Many model performance gains in production come from the data features you feed the model rather than from switching algorithms or running another round of hyperparameter search. Andrew Ng has emphasized this point under the “data-centric AI” banner; practitioners who’ve shipped enough models tend to arrive at the same conclusion independently. The feature matrix is where significant leverage often lies. The uncomfortable inversion: engineers often spend 80% of their time on modeling and 20% on data, when the ratio that may actually deliver better results is closer to the opposite.

Feature engineering sits at the intersection of domain knowledge, statistical intuition, and compute budgets. It’s not a purely creative act you either have a gift for or don’t. Treat it as a constrained optimization problem. You’re trying to maximize the signal available to your model while keeping training cost, inference latency, and pipeline complexity within acceptable bounds.

What Features Are Actually Doing

A feature isn’t just a column in your dataframe. It’s a transformation of the input space that can make the decision boundary easier for the model to find. That geometric interpretation matters. Take a binary classification problem where class A forms a ring around class B in two-dimensional space; no linear model finds a clean boundary on the raw x and y coordinates. Add a single engineered feature, r = x2 + y2, and the problem becomes linearly separable. One feature, derived from domain intuition about the structure of the data, can address what no amount of regularization tuning would have fixed.

Even in the deep learning era, this often matters. Tabular data remains prevalent in many production systems; research suggests tabular models represent a substantial portion of production use cases. Neural networks can learn representations, but they typically require more data and computational resources. For many production models running on structured data, hand-crafted features remain an important lever.

Maintain a working taxonomy of data features. Raw features come directly from the source with minimal transformation; a user’s age, a transaction amount, a sensor reading. Derived features are computed from one or more raw inputs: ratios, lag values, rolling statistics. Interaction features encode relationships between variables that the model might not discover efficiently on its own, particularly at scale. The third category is where most of the creative work happens, and also where computational debt often accumulates.

One practical note on model architecture: tree-based models can discover some interactions through splits, but they may do so inefficiently for high-order relationships. Linear models typically cannot capture interactions unless you provide them explicitly. Your feature strategy should reflect the model you’re actually using.

The Creativity Half: Techniques That Often Move the Needle

Domain-Driven Construction

A feature built from domain knowledge frequently outperforms an automated one because it may point at causal or near-causal structure rather than statistical association. Automated methods search a broad space; domain knowledge can narrow that space to the part that matters. A correlation between two variables and a causal relationship between them are not the same thing. A model trained on the former may fail differently than one trained on the latter.

In a churn prediction model, consider the difference between using days_since_last_login and average_session_frequency as separate raw features versus computing days_since_last_login / average_session_frequency. The ratio encodes behavioral decay; not just that someone hasn’t logged in recently, but that they’ve logged in recently relative to their own baseline. A user who normally logs in twice a year being absent for 90 days differs from a daily-active user being absent for the same period. Neither raw feature captures that; the derived one can.

Ask what a domain expert would look at first when assessing the outcome you’re predicting. That answer is your feature candidate list. Work that list before you reach for any automated generation tool.

Temporal and Contextual Features

Time-based features are often underutilized. Hour-of-day and day-of-week are obvious, but the encoding matters. Representing hour as an integer from 0 to 23 implies that hour 23 and hour 0 are far apart; they’re adjacent. Use sine and cosine transforms for cyclical variables: sin(2π * hour / 24) and cos(2π * hour / 24) preserve the circular structure and can help prevent the model from learning a spurious ordinal relationship.

For time-series contexts, lag features and rolling windows are standard tools, but the aggregation choice has consequences. Rolling means smooth noise but lag behind sharp changes; exponential weighted means (ewm in pandas) give more weight to recent observations and may respond faster. Medians are typically more robust to spikes but computationally heavier at scale. The right choice depends on whether your signal is gradual drift or punctuated events.

A feature that’s globally predictive may be less informative within a specific cohort. Purchase frequency might predict lifetime value across all users, but within your highest-value segment, variance in purchase frequency may be low and the feature may carry less discriminative power. Check conditional feature value before you lock in your feature set.

Interaction and Polynomial Features

When to generate interactions manually versus trusting the model depends largely on model depth. Linear models and shallow trees typically need explicit interactions; gradient boosted ensembles with sufficient depth may approximate many pairwise interactions through sequential splits. Deep neural networks can learn them through hidden layers. But “may approximate” isn’t “will find efficiently,” and there are interactions that even powerful models may miss when the signal is sparse relative to the noise floor.

The combinatorial explosion is real. With 50 features, you have 1,225 pairwise interactions. With 200 features, that’s 19,900. Before generating interaction features, run a mutual information screen or correlation filter to identify candidate pairs worth crossing. This narrows the search space to a tractable set and helps prevent flooding the model with noise.

The Hard Stop: Computational Reality

The features you can engineer are not necessarily the features you should deploy. Every feature added to a production pipeline carries a cost that compounds: training time, inference latency, memory footprint, and maintenance overhead. A feature that requires joining three upstream tables at inference time might add latency to your API response; this may be acceptable in a batch scoring job but problematic in a real-time recommendation system.

Feature debt accumulates quietly. Each feature is a dependency on upstream data schemas that can change, a monitoring surface for data drift, and an assumption about the world that may stop being true. Before adding any feature, ask one question: what does this cost me at inference time, and is the lift worth it?

The curse of dimensionality deserves a practical warning rather than a textbook mention. Sparse high-dimensional spaces can hurt distance-based models directly; k-nearest neighbors and SVMs may degrade as dimensions increase because distances become less informative. But the subtler effect is on overfitting surface area. More features give a model more ways to memorize the training set. Regularization can reduce the damage; it doesn’t eliminate it. Not adding the feature in the first place does.

The Science Half: Systematic Selection and Validation

Feature importance means different things depending on how you measure it. Mixing up the methods can produce suboptimal feature sets. Filter methods (mutual information, chi-square tests, Pearson correlation) are fast and model-agnostic. They evaluate each feature independently against the target, which means they may be blind to interaction effects. Use them for an initial pass to eliminate obvious dead weight, not as a final selection criterion.

Wrapper methods like recursive feature elimination (RFE) and forward selection can capture interactions because they evaluate feature subsets using the model itself. The tradeoff is computational cost; RFE with cross-validation on a large dataset is slow. Reserve wrapper methods for final candidate sets after filter methods have already reduced dimensionality.

Embedded methods (LASSO regularization, tree-based feature importance, attention weights in neural networks) sit between the two in cost and reliability. They’re convenient because importance scores emerge from the training process. But tree-based impurity importance can be biased toward high-cardinality features and may be unstable across runs on similar datasets. Permutation importance is often a more reliable post-hoc signal; it measures how much model performance degrades when a feature’s values are randomly shuffled. This directly tests predictive contribution rather than proxying it through split statistics.

The validation trap catches many practitioners. Evaluating features on the same split used for model selection introduces a subtle form of leakage; your feature selection has seen information from the validation set, even if your model hasn’t directly trained on it. The correct workflow wraps feature selection inside cross-validation folds: select features on the training fold, evaluate on the held-out fold, repeat. It’s more expensive; it’s also the way to get an unbiased estimate of whether your features generalize.

For feature pruning, use this decision rule: if removing a feature doesn’t degrade held-out model performance by more than your defined tolerance (say, 0.5% AUC for a system where you’ve defined that as acceptable variance), remove it. Pair this with a stability check. If a feature’s importance score varies widely across bootstrap samples of your training data, it may be picking up noise rather than signal and should be considered for removal regardless of its average importance rank.

Automation vs. Intuition: Where the Tools Fit

Automated feature engineering tools like Featuretools and H2O’s AutoML can be useful for exhaustive search over derived features and identifying interaction patterns that human intuition might miss. They’re also domain-blind, computationally intensive, and often produce features that are difficult to interpret; a ratio of a ratio of a lag is technically valid and practically challenging when a stakeholder asks why the model scored a particular customer the way it did.

Feature stores (Feast, Tecton, Hopsworks) solve a different problem. They’re governance infrastructure, not discovery tools. A feature store helps ensure that the days_since_last_login feature computed in training is computed identically at inference time, and that your fraud model and your churn model draw from the same canonical definition of “account age.” That consistency can help prevent training-serving skew; it doesn’t help you figure out which features to build in the first place.

Automation can generate candidates; domain knowledge and validation curate survivors. In a novel problem domain where no prior feature schema exists, automated methods have limited anchors. They’ll generate features, but without the validation rigor from the previous section, you have limited basis for trusting them.

A Framework for the Next Dataset You Open

When you sit down with a new dataset, the sequence matters. Start with domain questions, not data questions: what does a human expert look at first? Build that list into feature candidates before you touch the data. Then look at your raw features and ask which derived features encode the relationships your domain reasoning suggests; ratios, rates of change, deviations from cohort baselines.

Run a filter-method pass to eliminate features with near-zero mutual information against the target. Generate interaction candidates only from pairs that pass a correlation or MI threshold. Encode cyclical variables correctly. Build your validation pipeline so feature selection happens inside the folds. Measure permutation importance on your held-out set, not impurity importance on your training set.

Before you ship, price every feature at inference time. If a feature’s latency cost exceeds the performance lift it provides, it may not belong in the production pipeline regardless of how clever the construction was. The goal isn’t the most expressive feature set; it’s the most efficient one that meets your performance threshold.