John 8:4-5 ASV and John 8:7 ASV

This conversation with the teachers of the law and the Pharisees is remarkably interesting. How is it that Jesus says this one little phrase about if you're without sin you can cast the first stone that causes all of them to leave? What were they thinking that made them follow Jesus' bidding? After all, they didn't even think he was who he said he was. Some thought maybe he was a prophet, but even that was questionable in their minds. I don't think they listened and obeyed merely Jesus' words because Jesus said it, there must be something even in the Old Testament that in their minds brought conviction.
I've read some who say that they didn't bring the man and both of them must be brought forth according to the law...that is true (Deut. 22:22), but I don't think that was entirely it.
It has to do something with them being convicted about sinning, not missing an aspect of the law.
Didn't the religious leaders always bring people up for punishment with the notion that they themselves were not perfect and they were just fulfilling the law of Moses that commanded them to take care of evil within Israel? Maybe I'm just giving them the benefit of the doubt. I understand that Jesus constantly talked about their appearing to be clean, but on the inside they are not. Was it just Jesus' saying "well, you're not perfect," that brought them down?
I think Jesus' words of "who is without sin" in this circumstance is not merely some general conviction about perfection (although the greek word is this: one without error) that brought conviction, but the circumstance that brought conviction. They were convicted of their own adulterous acts (not physically -though it certainly could include, but more spiritually).
This type of spiritual adultery was restated over and over through the prophets (and used by James in the New Testament). They had become so much lovers of the law and not God, that when confronted with their spiritual adultery there was such conviction. Conviction enough to leave the law at this point altogether. They didn't say..."wait, we'll go get the man to and the witnesses"...that wasn't what convicted them, that they missed some part of the law.
They were convicted of their adultery and the notion of stoning this woman only concluded that God would respond to them as they respond to her (Ezek. 16:38-40).
Jesus tells her to leave her life of "sin" as alluding to adultery, so the saying of "without sin" is alluding to adultery as well. What is our spiritual adultery?