In this section (http://www.vaginapagina.com/index.php?title=Herpes_%28HSV-1/HSV-2%29#Who_is_at_risk_for_herpes.3F__How_do_you_get_it.3F) of the Vulvapedia, it mentions:

"HSV is transmitted through direct skin-to-skin contact via the exchange of saliva, semen, cervical fluid, or vesicle fluid from active lesions."

It seems unlikely that the virus would be able to survive on clothing for a significant period of time. The American Social Health Association (http://www.ashastd.org/herpes/herpes_learn_questions.cfm#3) supports this idea when it says:

"There are no documented cases of a person getting genital herpes from an inanimate object such as a toilet seat, bathtub, or towel. Herpes is a very fragile virus and does not live long on surfaces."

So unless you put the boxers on right after he took them off, it sounds like your risk for contracting the virus was extremely low.