So I just checked how this works on KP115, and from the looks of it there can only be a single countdown timer and they do not get removed automatically... However, here's what you could do using raw commands to achieve your goal:

1. Create a rule that is disabled for now:

$ kasa --host 192.168.xx.xx raw-command count_down add_rule '{"enable": 0, "delay": 5, "act": 0, "name": "test"}'
No --strip nor --bulb nor --plug given, discovering..
{'id': 'FFCC02F439FFEE7E1D8853935949AA4A'}

Note down the rule id or perform a query to list available rules, you will need this to modify the rule to activate it later.
If you get an error on already existing rule, you can either delete this by using delete_rule and passing the ID as payload, or re-use the timer by noting down the ID returned by get_rules (see the next step).

1. You can check that the rule exists:

kasa --host 192.168.xx.xx raw-command count_down get_rules
No --strip nor --bulb nor --plug given, discovering..
{'rule_list': [{'id': 'FFCC02F439FFEE7E1D8853935949AA4A', 'name': 'foo', 'enable': 0, 'delay': 5, 'act': 0, 'remain': 0}]}

1. Start the count down by setting the enable to 1, you can also adjust other settings if wanted:

kasa --host 192.168.xx.xx raw-command count_down edit_rule '{"id": "FFCC02F439FFEE7E1D8853935949AA4A", "enable": 1, "delay": 5, "act": 0, "name": "foo"}'

delay indicates in how many seconds the rule will be executed, act can be either 0 (turn off) or 1 (turn on).

The action will be run after the timer expires, and it will get disabled. You can re-run the same edit_rule command to re-enable it.