You can do it, but it's a little roundabout in execution! You're going to need two common events to make each skill work (I'll get to the second one later).
Name: Skill A
Trigger: None
@>Control Switches: [0001: Skill A Used] = ON
You could add a conditional branch so it doesn't keep trying to turn the switch on if it's already on, but that's not necessary. :P
Now, go to the skill you want to attach this to and find the "Common Event" dropdown. Select that common event (in this example, it would be "001: Skill A").
Go to the troops tab and go down to the "Battle Event" section on the bottom half of the boss battle troop.
Condition: Switch [0001: Skill A Used] is ON
Span: Moment
@>Text: Oh noooo!
@>Control Switches: [0001: Skill A Used] = OFF
Now, when you use this skill, you'll display the text (you can really do whatever you want; I just used text as an example) and then turn the switch off. You
must turn the switch off. I excluded that the first time and it looped indefinitely. xD
This will not work if all the enemies are defeated! That's just how the scripting processes battles - once the enemy is defeated, it stops everything else to start the victory sequence. You might be able to find or request a script edit to change it. I would be able to figure it out eventually, but it would take me forever and I don't have much free time until the weekend! And I'm not skilled enough to guarantee it wouldn't conflict with other scripts you may have. :x
Anyways, you need one more thing! Another common event.
Name: Turn off Skill A
Trigger: Parallel
Condition Switch: 0001: Skill A Used
@>Control Switches: [0001: Skill A Used] = OFF
If you don't do that, using it on a normal battle would turn the switch on and it would stay on until you met that boss. And parallel common events only execute on the map, so you don't have to worry about it turning off the switch during battle.
I tested it and it seems to work! Let me know if you have any problems!