He said it early, the way people do when they're still showing each other around their own insides: physical touch is my love language. You filed it. Believed it. Then a pattern started, and somewhere in the middle of it you stopped being able to tell whether you were meeting a real need or negotiating a request that keeps arriving wearing a different name.
This question gets asked far more than the internet admits, and the honest answer is not a side. It's a distinction most people were never taught to make.
What genuine touch-hunger actually looks like
Some people are simply under-touched. Not neglected in some dramatic way, just under-touched, the way anyone is who has gone months without a hand landing on them for no reason. There's a real, commonly used term for it: skin hunger. It doesn't announce itself as desire. It shows up as someone leaning slightly into a hand on their back, or asking to hold yours in the car for no destination, or missing being held far more than they miss what comes after. Genuine touch-hunger takes the non-sexual yes and stays in it. It doesn't need it to lead anywhere to count.
What the coded version actually looks like
The other pattern is real too, and it's not subtle once you know what you're looking for. Touch only registers as love if it ends in sex. Affection warms up after, not before. A "not tonight" doesn't get received as information, it gets received as a verdict on the whole relationship, and the questions that follow aren't about you, they're about managing what your no did to him. That's not a love language. That's a demand wearing one.
The actual test
Watch what happens to the yes, not the no. Say yes to ten minutes of non-sexual touch, on the couch, hand on your back, nothing more asked of it, and watch what he does with that yes. Does he stay in it, or does he deflate, sulk, stop touching you at all once sex is off the table? Genuine touch-hunger receives the yes as enough. The coded version treats anything short of sex as a technicality it's still owed.
Start entirely alone, before you bring this to him. Touch Base®: thumb to forefinger, moved slow, two minutes, so you know in your own body what wanted, unhurried touch actually feels like before you're trying to read someone else's. It gives you a baseline that isn't shaped by his reaction either way.
You are allowed to want to be touched and still ask what the touch is actually for. Those two things were never in conflict.