Gameplay
It's uncertain how this gameplay mechanic will work, but the name hints at the Wizard's lore, and also its functionality.
Most likely each spell cast will cause "instability" due to the dangerous and volatile nature of the spells the Wizard wields.
Hypothesis 1
Too much instability will hurt her or make her unable to perform certain tasks or perhaps modify how a spell is cast or what spell is cast (eg: casting Disintegrate will produce Spectral Blades instead). It could just be a power gauge inspired by the now defunct Mana Burst, dealing damage based upon how much/little control she has over the instability.
This theory has merit because most resource systems use one or another way of gaining a resource and using skills by spending it.
Hypothesis 2
An alternate form would be that the instability somehow helps the Wizard to cast spells better in some way. The more spells cast, the instability will actually help by causing ripples in time or similar effects.
The only problem with this is that it could be quite unbalanced if the Wizard would continuously gain power the more she did, especially in PvP.