You don't need a script for this. Here is what you do (I don't have access to RMXP right now, so my wording in the examples may be a bit off and I can't use screenshots). First, open your database. Create a new State (status effect, like 'poison', 'knockout', etc.). You can name this state anything (I'll use Self-Damage: On, just as an example name). Set the Rating to 0 (this will keep the status hidden when your character is affected by it). Also, set it up to Release at End of Battle.
Create a new Common Event. To set up the percentages, use Set Variable [001]: Player X's MaxHP. (I will just use Variable 001 as an example). Now, just below that in the Common Event, use another Set Variable to multiply Variable 001 by whatever percentage you want to lower his MaxHP by. For example, if you want the skill to lower MaxHP by 25%, multiply Variable 001 by 25. Go to the next line in the Common Event and now divide Variable 001 by 100. This will give you a percentage.
For example: MaxHP = 100
100 x 25 = 2500
2500 / 100 = 25
25 is 25% of 100.
Next line in the Common Event, set a Conditional Branch: Player X (your character) is Self-Damage: On inflicted (or whatever your state name is). Now in the Conditional Branch, use Change Attributes to decrease MaxHP by Variable 001. Now you can also set up whatever else you wanted the skill to do.
Example of your Common Event:
@Set Variable 001: Player X's MaxHP
@Set Variable 001 *= 25
@Set Variable 001 /= 100
@Conditional Branch: Player X is Self-Damage: On inflicted
<>Change Attribute: Player X MaxHP -= Variable 001
<>Whatever else the skill is supposed to do
@Else:
<>
End Branch
Back in the database, create your skill and have it call this Common Event.
Now, in the database go to Troops. This is where you set up all of your boss battles, and at the very bottom is an Event handler. Select any boss battle you want this effect to be in. In the Event handler, you can select Turn 1 + 0 in the Conditions drop-down box (top of the event handler in the middle). The drop-down box in the upper right should default to Battle, which is good. Select "Battle" if it does not. Now, use State Change to inflict your status effect on your character.
So now, every time your character enters into the battle, the game will secretly hit them with your Self-Damage: On status. When you use your skill, it will first calculate the percentage of MaxHP your character will lose. Then, only if your character is inflicted with your Self-Damage status will he lose his health.
I hope this helps. Let me know if anything is unclear.