I highly recommend P.E. Moskowitz's book "The Case Against Free Speech." The view you express is actually, historically, not what free speech represented but, rather, part of a well-documented propaganda effort funded by the Koch brothers starting in the late 1970s to hijack the concept of free speech to be about protecting hateful ideas. The idea of absolute free speech did not gain traction in this country until the 1990s, and then 8t was a conservative idea.