When using AI to predict pay, context matters

ADP Research Institute built a simple model to show why context matters.

The ADP Research Institute (ADPRI) built an algorithm similar to those that power many AI-driven compensation benchmarks — but with one major difference: It predicts pay using far less context. ADPRI’s model — which doesn’t drive any ADP product — predicts pay from the characteristics of someone’s job title alone.

Amazingly, this model automatically learns patterns that capture economic realities about pay. But it also learns patterns that reflect little more than the model’s missing context.

In this report, we explain how our transformer works to estimate pay. Then we use the model to show why employers who use AI-driven compensation benchmarking tools must create an environment where compensation experts and AI experts combine their strengths.

Architecture of a transformer built by ADP Research Institute that predicts wage percentiles from job titles. Model not used in any ADP product.

Transformers: Insights in disguise

Algorithms called large language models, or LLMs, drive ground-breaking systems such as OpenAI’s ChatGPT. Part of what powers LLMs are machine-learning algorithms called transformers. Given a user’s input text, these transformers output a response that — most of the time — looks like it was written by a human. Transformers do this by learning complex patterns of association between words and their context.

ADPRI’s model uses transformers, as do the algorithms that drive many compensation benchmarking tools on the market. Yet instead of predicting a sequence of human-readable text, our model predicts the percentile of a worker’s pay based only on the characteristics of their job title. A job title’s characteristics include the words it contains, the meaning of those words as used in job titles, and the relative positions of those words in the title.

Because we predict pay based on the characteristics of a job title rather than the job title itself, the transformer can predict pay for alternative titles for the same job, or for jobs with similar titles, even if the title never appeared in the training data.

The trouble comes when the user asks the transformer to predict pay for titles too far outside the scope of the data we used to train it.

How the strength of transformers reveals a potential weakness

Because ADPRI’s transformer draws on the meaning and ordering of words in job titles, it can predict pay for almost any title imaginable.

For example, it estimates that the middle 50 percent of workers with the ludicrous and fictional job title professional swan fighter will earn between $28.62 and $34.73 an hour, while an apprentice wizard commands only $16.36 to $21.40 an hour.

These fictitious job titles use words that exist in the titles we used to train the transformer, but in combinations outside the scope of the training data. Even more amazing, the transformer makes plausible-seeming predictions about pay for job titles that contain words that never appear in the training data.

One such word: squirrel. If your dog could read the transformer’s output, they might demand an hourly wage between $30.43 and $35.97 as a senior squirrel chaser. We offer more hilarious examples in an appendix, but for the rest of this case study, we’ll stick to less-whimsical job titles.

No one fights swans for a living in the United States, and few if any workers battle Balrogs for a paycheck. The idea that someone could earn nearly $70,000 per year chasing squirrels seems doubtful, no matter how senior they are. Yet our toy model’s flexibility allows users to engage in such flights of fancy.

In quantitative analysis, the term extrapolation describes this practice of making predictions outside the scope of training data. Silly job titles make the risks of extrapolation obvious. Yet the most problematic extrapolations also are the most subtle. For example, if a user-entered job title includes a word that exists in the model’s training data but outside of the word’s original context — such as outside of the industry where it’s well defined — the results can be just as misleading as the fictitious job titles.

Below, we use ADPRI’s title-to-pay transformer to illustrate a few of these subtle but problematic extrapolations.