I'm pretty sure that Jesus could not have hurt the Pharisees any worse than by asking then, "have you never read the scriptures?" That was their job. They were among the very few who were literate. They were the teachers, nee the perfecters, of the law. All they did, day in and day out, was read the scriptures and argue about their meaning in daily life.
When Jesus asks them, "have you never read the scriptures," he says openly what he's been veiling in parable all this time, "y'all don't have a clue." It hurts to have your worldview challenged like that. Especially as it relates to one's religion, to be accused of being so ignorant of the basics as to have never even read the scriptures, that's about the worst challenge I can think of.
No wonder the Pharisees, realizing he's talking about them (clueless even here), want to arrest Jesus immediately.
As an Episcopal priest, I have two sacred texts: the Bible and the Book of Common Prayer. It assumed, rightfully, that I've read, marked, learned, and inwardly digested both of them. Despite some holes in the narrative of the Old Testament that I've failed, as yet, to inwardly digest, I can honestly say that I take those texts very seriously, and when it comes to the BCP, when I break a rule, I do so knowing the tradition in the rubric and the reasons for my changes.
What comes to mind this morning, is that challenge. What could Jesus look at in my life and ask, "have you never read or heard or lived my message?" That, I'm certain, would sting just as much as it did for the Pharisees.