Wanna wear one, wear it.
Just make sure you're wearing a fit-tested N95 and wear it for no more than 2 hours. Be very careful not to touch your hands to your eyes while you're wearing the mask...Actually, you'd be better off wearing sterile goggles, also.
And a reminder...when removing your PPE, use sterile technique, as all outer garments (including your mask and goggles) are contaminated.