The government is promising to focus less on race, religion, sexual orientation and disability as it overhauls its equality policy.
Minister Liz Truss will say the debate on making society more equal has been dominated by “fashion” and not “facts”.
And, she will add, discrimination laws should also take into account people’s social background and “character”.
But a race equality think tank accused Ms Truss of wanting to “whitewash” British history.
The government is also launching an Equality Data Programme to gather information on people’s backgrounds, social mobility and inequality between regions.
Speaking at the Centre for Policy Studies think tank, Ms Truss will say: “To make our society more equal, we need the equality debate to be led by facts, not by fashion.
“Time and time again, we see politicians making their own evidence-free judgements.”
She will also say discussion “has been dominated by a small number of unrepresentative voices, and by those who believe people are defined by their protected characteristic and not by their individual character”.
And she will urge “right-thinking people” to “fight for fairness”.
The Equalities and Human Rights Commission lists protected characteristics – over which it is illegal to discriminate – as age, disability, gender reassignment, marriage and civil partnership, pregnancy and maternity, race, religion or belief, sex, and sexual orientation.
Ms Truss will argue that data based solely on these characteristics is not fit for purpose when it comes to setting equality policy.
The equalities minister will say: “Underlying this [approach] is the soft bigotry of low expectations, where people from certain backgrounds are never expected or considered able enough to reach high standards.
“This diminishes individual humanity and dignity, because when you choose on the basis of protected characteristics, you end up excluding people.”
Ms Truss will add that it is “appalling” that pregnant women suffer discrimination at work”, that women may “be encouraged to dress in a certain way to get ahead”, and that “some employers overlook the capabilities of people with disabilities”.
Debates on equality must “rooted” in “real concerns people face”, she will say, adding: “It is our duty to deliver, because if right-thinking people do not lead the fight for fairness, then it will be led by those whose ideas do not work.”
Halima Begum, chief executive of the Runnymede Trust, a racial equality think tank, said: “Liz Truss’s attempts to ‘overhaul’ the equalities work in the UK is nothing short of a whitewashing of British history and its relationship with race.”
She also said:” It is time that equalities ministers in this government are held accountable for their words.”
The government announced on Wednesday that its report into racial inequality will be delayed until next year, citing problems caused by the coronavirus pandemic.
It is looking at health, education and criminal justice, but also “wider inequalities” such as issues faced by working-class white boys.