One of the defining features of Salary Confidential is the ability to run differentiated peer group surveys within a single poll. A poll represents the question you want to understand, while individual survey peer groups define who is answering it.
In this guide, we're looking at how to maximize the impact of what you can learn using our two-tier structure.
Polls ask a specific question. Surveys define who is answering
Your poll might be about creative directors with 15 year experience working in NYC and LA. One peer group may be specifically peers who work at agencies. Another peer group may be working in-house.
Each survey peer group is narrow by design. Each peer group receives its own response form, so when responses come in we know that form A corresponds to one peer group, while form B (which asked the same poll questions) corresponds to another. Both A and B generate a peer-group level results report with strictly the respondents of that peer group; the poll level generates a report that rolls up all respondents of A and B together.
Precision starts before the first response
A high-quality Salary Confidential survey is mostly designed before anyone responds.
If you're able to:
- identify truly comparable companies
- understand how titles differ but responsibilities align
- distinguish between tracks that look similar on paper but differ in practice
you're likely to run a strong survey, even with a small number of respondents.
That said, even the best-designed peer group still contains unknowns.
You won't know, for example:
- whether one respondent led a critical project and received a one-off bonus
- whether another negotiated unusually well at the time of hire
- whether equity outcomes reflect timing rather than role
Those unknowns don't invalidate the data, but they do shape how it should be interpreted.
What you're seeing is a real snapshot of real people, not a controlled experiment.
When further subsetting by peer group becomes unnecessary
A useful test when designing a survey peer group is this:
If I don't separate this characteristic out among all my respondents, will I lose something important when I read the results report at the poll level?
Take this example:
"How much do customer acquisition specialists with my same kind of experience and scope make?"
That's a solid question for a poll.
Ask yourself:
- Does location matter as a piece of context where you want to isolate top metro areas versus smaller cities?
- Do you care to track educational background, such as peers with an MBA versus those without one?
If location doesn't matter, there is no need to create separate peer groups for large metros and smaller cities. If educational background doesn't appear to influence outcomes in your industry, there is no need to create distinct peer groups around that dimension.
Or perhaps, in the first place, you're only asking folks from small cities and no one from large metros; and only folks with a specific educational background that matches yours, and not folks with a different educational background. In which case, you can put all such people in one peer group because you don't need to filter results by the dimension of city size or education background: all your participants were one dimensional according to a single inclusion criteria.
Running a larger peer group that merges contexts is perfectly reasonable. The question is not whether differences exist, but whether separating those contexts would meaningfully change how you interpret the results.
Building peer groups: understanding what Salary Confidential always collects, can optionally collect, and why we sometimes don’t show what we collect
Data we always collect but may not show if results become de-anonymizing
A limited number of peer-related dimensions are structurally collected in every response. These include:
- organization size
- respondent years of experience
Because these fields are always present, you generally do not need to create separate peer groups for them. You can often interpret responses using those dimensions directly in the results.
However, we do not guarantee that all collected dimensions will always be shown, and this should inform how you define your peer groups.
Some contextual attributes — including organization size — may be withheld when they cannot be safely represented. This happens when one response is too isolated relative to others in the peer group and cannot plausibly blend with nearby results. In those cases, the dimension becomes potentially de-anonymizing and is suppressed.
In practice, this depends on both the size of your peer group and how spread out respondents are across organization sizes:
- If your peer group is small and respondents span very different organization sizes, some results may become isolated, and organization size may not be shown
- If respondents are relatively similar in size, the dimension is more likely to be preserved
- If you intentionally include a wide range of organization sizes, each size range should be represented by multiple responses to avoid isolated singletons
If organization size is an important lens for your analysis, you have two reliable strategies:
-
keep peer groups relatively homogeneous in organization size
-
or ensure enough responses at each size level so that no single response stands alone
-
Read more about how we handle organization size and when it may be withheld
Data we can collect per response, but present at the peer group level to prevent de-anonymization
Further along this spectrum, there are attributes that you can ask about, but that we deliberately do not expose at the individual response level because they carry a higher de-anonymization risk. This differs from organization size: instead of conditionally showing or hiding it, we always separate it from individual response rows.
This is the case for location.
Location is an optional field, which means respondents may skip it even if you request it. But even when it is provided, we do not display it alongside individual responses. Instead, results are presented in two parts:
- individual compensation records
- and, separately, a summary of locations shared within the peer group (for example: “Some respondents in this group are located in: [Location1, Location2, ...]”)
This is not because location is unimportant. Quite the opposite — it can be highly relevant. But some dimensions are simply too identifying to be safely attached to individual responses.
For example, if you invited Alex and know Alex is the only respondent in Kansas, displaying “location = Kansas” on a response would effectively reveal their identity.
You can guarantee that a dimension remains visible by making it part of the peer group definition
For this reason, if a dimension is essential to your analysis, the safest way to preserve it is to define it as part of your peer group.
For example:
- Survey peer group 1: "Heads of Product, B2C fintech, Silicon Valley"
- Survey peer group 2: "Heads of Product, B2C fintech, NYC"
In this model, location is no longer an attribute of individual responses. It becomes a property of the group itself.
Because each peer group requires a minimum of four responses, any such characteristic is always shared across multiple respondents. This preserves anonymity while ensuring that the context remains clearly visible in results.
In practice, this means that any characteristic you care about can be reliably tracked, as long as it is used as an inclusion criterion for a peer group rather than attached to individual responses.
You're not optimizing for statistical significance
It's important to be explicit about what Salary Confidential surveys are not trying to do, and in fact, unless you're a professional pollster with a really good command of sampling, resolutely cannot do.
If you were attempting to prove statistical significance, you would need to:
- sample against a very carefully understanding of the reality of the field (ie, be a pollster)
- account for multiple forms of bias
- likely expand sample sizes to a far larger size
- test whether observed differences persist under scrutiny (control multiple samples)
That is not the goal here. Instead, the goal is to answer highly specific personal questions:
- What do people like me seem to be making?
- Where do I fall in that range?
- Is my situation obviously out of line, or plausibly within it?
For those questions, precision comes from asking highly specific peer groups who closely match your own situation.
The result is, by definition, anecdotal. But it is highly relevant anecdotal evidence. You're not trying to prove the universal truth of compensation for a role in a given market. You're trying to gather real-life examples of compensation in a well-understood context that closely matches your own.
Choosing peers: relevance over convenience
One of the temptations in peer surveys is relying on convenience peers: people you already know or people who are easy to reach. It's not necessarily wrong, but it can limit the usefulness of the results.
Salary Confidential is designed to help you move past that limitation.
It's normal to feel awkward asking people about compensation. We've been highly conditioned to treat these as conversations that aren't supposed to happen. It's also normal to hesitate before asking strangers why they should help you.
The premise of Salary Confidential is simple:
- you can't see how any individual answers
- you won't know whether they responded at all
- participation flows both ways
This distance makes the conversation less awkward and allows you to focus on who is best positioned to provide relevant insight, not who feels easiest to ask.
When choosing peers, ask:
Who actually does comparable work? Who operates in a similar context? Who is likely to have information that would meaningfully inform my understanding?
Connection to you should be secondary.
Tight peer groups increase anonymity
Even when submissions are anonymous, the data itself still tells a story. That’s the goal of course: the results should reveal something useful. But when the sample is tightly matched, the story the data tells can only be about the group as a whole, not about the individual source of any specific result. In looser groups, the results themselves may begin to hint at who they came from, even if the submissions remain technically anonymous. (This is another way to say that a homogeneous group has more structural k-anonymity, which is an area of research we took a lot of inspiration from in the design of the Salary Confidential platform)
In the context of Salary Confidential’s mechanics, this means that when respondents all occupy very similar roles and contexts, it becomes much harder for the requester to play the parlor game of attributing any particular result to a specific individual among those they invited.
Imagine you invite eight peers from companies similar to yours, all with comparable roles and scope. One of them might worry:
"I think I'm making much less than everyone else."
"I had a huge bonus year and I'm going to stand out."
Even if an outlier appears in the results, it becomes difficult to trace it back to any one person. Because everyone in the peer group operates in a similar context, any result could plausibly belong to anyone.
Now compare that with a looser peer group. Suppose a sample contains two VPs and two team leads. If two very large numbers appear and two smaller ones appear, it becomes tempting to assume the large numbers belong to the VPs and the smaller numbers to the team leads.
That assumption may be wrong. But it is an easy one for the requester to make, because the structure of the peer group itself suggests the interpretation.
A tightly defined peer group creates a natural blur around individual outcomes. When respondents understand that the peer group is very tightly matched, psychological safety increases. Even when outliers appear, the similarity of the peers makes attribution harder and protects respondents.
Tight peer groups increase participation
A narrow peer group is not only useful analytically. It is also attractive to respondents.
When a survey is tightly focused, people immediately recognize themselves in it. As they read the invitation in the outreach message, they can think:
"Yes, this is clearly about people very similar to me."
That moment of recognition matters. It makes the request feel relevant rather than generic and increases curiosity about the other side of the exchange: access to the final report.
The closer a survey feels to a respondent's own situation, the more compelling the results become and the more likely people are to participate.
Designing focused peer groups isn't just about analytical clarity. It is also about making participation feel worthwhile.