Use Latex fonts: Choose Text. on the bottom right of xfig, click on the font name, and choose Latex fonts. Choose any of them.
Then, in the "text flags" menu, where it says "hidden=off", click and choose for the "special flag" option the value "special" and not "normal". (Is this like double negative? what is a normal special flag? what is a special special flag? what is a flag that is called special flag? a special flag flag?)
Then you can export an existing figure, there is no need to write text again. But you will need to export all your figures again.
The other thing is use "filled" instead of "pattern" in thick lines and shape filling. It may look the same, but it will prevent a type 3 font. This you need to do either before you draw, or after - choose edit and click on all the filled shapes, apply and done.