Reddit mentions of Education and Race from Empire to Brexit

You're right to not trust Ferguson. He's the media face of popular perceptions of the British Empire. Big publishers like Routledge have their own guides to the British Empire/the fall of the British Empire that are good. There's tons of historians who have been writing critically about Empire, race, and Britain since the 1980s. Have a look for things written by Peter Fryer, Colin Holmes, Shompa Lahiri, Rozina Visram, Mary Chamberlain, Panikos Panayi, Bill Schwarz, Avtah Brah, Barbara Bush, Diane Frost, Kathleen Paul, Wendy Webster. David Olusoga's writes great public history books on the topic that are backed up by solid research and the views of people mentioned above; his documentaries on Forgotten Black British History and Britain's Forgotten Slave Owners are also fantastic and available on iplayer and youtube. I've recently picked this up and it's about the similar points that I discuss in my comment above https://www.amazon.co.uk/Education-Empire-Brexit-Sally-Tomlinson/dp/1447345843