Modified Fourier Neural Network
Fourier Neural network is highly effective for dealing with problems that have sudden changes or there are different materials interfacing  when training physicsinformed neural networks to solve such problems, it involves working with sharp gradients, which is very hard to handle in neural networks. A modified version of these networks can actually enhance their performance even more!
The modified version, inspired by a technique in ^{1}, adds two transformation layers to project the Fourier features into a new learned feature space. These transformed features then update the hidden layers through elementwise multiplications, similar to the original technique in ^{1}.
This small tweak in the architecture has been shown to improve both the training convergence and accuracy. However, there's a tradeoff: training time per iteration might increase a bit.
References:

Wang, Sifan, Yujun Teng, and Paris Perdikaris. "Understanding and mitigating gradient flow pathologies in physicsinformed neural networks." (opens in a new tab)" SIAM Journal on Scientific Computing 43.5 (2021): A3055A3081.