I'm pretty sure the reason we feel matter isn't as simple as "it's interacting in some way".
I've seen an IIT prof say that it's due to the Pauli Exclusion Principle, which I think is not really true; it's a factor for sure, but it only becomes relevant at very short ranges (on a molecular level), i.e. electronic charge is much more important when you're touching an object. Photons don't have charge.
Photons interact with matter in ways that are highly dependent on their specific wavelength (above 300nm anything they can do will be quantised), and most of that can be interpreted as an increase in temperature, at least. Given that temperature pretty much ends up being some type of kinetic energy, that's kind of promising, but not really because even if it happens to be translational (i.e. stuff actually moving and not just vibrating or whatever), I'm not sure it'd have any macro scale overall direction. So... Temperature up, that's not very pleasant.
On the downside, you could actually denature proteins in the person. Or like, create radicals. Lots of not so nice things.