The white legacy is more than bricks and mortars