Do you get, or have you ever gotten, cold sores? (Because those are oral herpes, and in the absence of antibodies, HSV-1 (the usual culprit for cold sores) will jump to the genitals pretty happily. They can also jump from mouth to hand to genital, if done quickly enough, though the virus does not like to be dry, so if "several minutes" and "dry hands" are involved, that's not a likely vector at all.)

Do try to get them swabbed quickly, as HSV is sometimes hard to catch if it's stopped shedding contagious virus bits. The other way to identify it is to get a blood test at the first outbreak (which will be negative if its the very first outbreak, because the body doesn't have detectable numbers of antibodies to it yet), and then 2-3 months later (when the body will have made antibodies).

Good luck!