AlanP, unless you've discovered some hitherto unknown way of pulsing your piezo, apart from doing it mechanically or electronically, then what other options are there? Sounds like you're a bit worried about taking the electronics route, but, in all fairness it probably is the easiest way in this case.
Action's P73 would do what you're looking for. Alternatively, a simple two-transistor oscillator built on a scrap of stripboard would do the job. You could probably shoehorn the whole caboodle on a board measuring not much more than 25mm square. Also, of course, the ubiquitous 555 timer chip might be worth looking into.
What do you have in mind if you want to do it mechanically? Some sort of servo-operated lash-up, maybe?
Hey, Terry's Cage. How you doing, amigo?
I like your way of thinking. The LED-in-series dodge would probably work. It depends on how much current the piezo unit draws, but should be okay if'n it's just a handful of mA. Don't forget - those el-cheapo wallwart transformers employ the same simple method when they're used for recharging batteries.
A flashing LED in series with a bog-standard LED probably wouldn't work. Flasher LEDs have a built-in circuit (crammed inside the LED itself), so once power is applied the non-flashing LED will stay on all the time anyway.
Like you, I've not put this into practice, but all the theorum says it's a no-go.