Vagifem is very low dose and is considered safe, so safe that you can buy it without prescription under the name Gina. UK is very strict about what meds you can buy over the counter, that alone should put your mind at ease.
I use it every day and wouldn't think about having a break, in fact missing a day brings back symptoms, that's how bad VA can get if not treated appropriately.
Just wish GP's would all sing from the same hymm book, my GP says ok to use daily if needed and certainly no breaks in between.