This teaching is so radical that His shocked Apostles say “If such is the case of a man with his wife, it is not expedient to marry” (Mt. 19:10). And Jesus doesn't disagree, instead using the opportunity to call those who can handle it to the ideal of celibacy (Mt. 19:11-12). The reaction of the Apostles makes no sense if Jesus is just saying that the pre-Hillel Jewish divorce laws apply. Nor does Jesus' repudiation of the Mosaic exception.
Whether you argue that Matthew's Gospel contains an exception allowing divorce and remarriage in the case of adultery, infidelity, or any other instance, you're going to run into a huge Gospel harmonization problem. Here's what Luke 16:18 says on marriage:
That's it. There's nothing before or after this verse to mitigate its punch: it's an absolute prohibition against divorce and remarriage. We see this absolute prohibition in Mark 10:11-12 as well.
To hold to the adultery exception is to hold that Jesus taught one thing (divorce and remarriage is okay in some cases) to the Jewish readers of Matthew's Gospel, while teaching a contradictory thing (divorce and remarriage is never okay) to the Gentile readers of Mark's Gospel. No faithful Christian can hold to such an incoherent position, since it amounts to claiming that either Matthew or Mark and Luke are presenting a false teaching.
Even leaving aside the impossibility of harmonizing the “infidelity exception” with Mark and Luke's Gospel, how can one harmonize it with the rest of Matthew's own Gospel? In Matthew 19:6, right before the verse in question, Jesus says that the spouses “are no longer two but one. What therefore God has joined together, let not man put asunder.”
5. The Greek Term in Question Actually Refers to Unlawful Marriages
The word porneia literally means something like fornication, but is used throughout Scripture (both in the New Testament and the Septuagint, the Greek translation of the Old Testament) to refer to marriages contrary to Levitical law, particularly ones involving incest.
The Book of Leviticus created stricter rules governing incest than what was observed the pagans: for example, your in-laws (Lev. 18:8, 15, 16), your aunt through marriage (Lev. 18:14), or the daughter or sister of a woman that you've been sexually involved with (Lev. 18:17-18), even though you're not actually blood relatives with any of these people. All of this was based upon the idea that, in marriage, the two become one flesh: so your in-laws become, in a real way, a part of your family.
The Gentile pagans didn't have these rules, and the Jews were often scandalized and disgusted by Gentiles' marriages. We can get a sense of this from 1 Corinthians 5:1, in which St. Paul says, “It is actually reported that there is immorality [πορνεία, porneia]among you, and of a kind that is not found even among pagans; for a man is living with his father’s wife.” When Paul wants to say that this sexual relationship is wrong, he does so by saying that it's even worse than the porneia practiced by the pagans.
At the Council of Jerusalem (Acts 15:20, 15:29, and 21:25), these Levitical anti-incest laws were left in place, as one of four parts of the Levitical law that remained in place on the early Christians (the other three involved food sacrificed to idols, food with the blood in it, and meat that had been strangled, all of which come from Lev. 17:8-14; the porneia rules come from Lev. 18:6-18).