Playing as Prussia, I have a couple of infantry lines gain 1 chevron experience as well as my General. After retraining, my general loses the gained chevron as well as 1 of the infantry lines. I had a full stack of 20 units and trained 10 more in that city to replace the occupying army that I use as my main army. What causes some units to lose experience? Retraining, recruiting.
When you add raw recruits into a group of veterans, you dilute their collective experience.