You should be able to setup a firewall, blocking all access to the SSH port. Then setup a VPN so that only you can access via SSH after making your VPN connection.
If you connect via a static IP, you can also create an ACL for the VPN connection just in case. You can set an ACL for the SSH port forward rule directly as well, but I don’t like that personally. I prefer keeping things behind the VPN.
What VPS are you using?
You should be able to setup a firewall, blocking all access to the SSH port. Then setup a VPN so that only you can access via SSH after making your VPN connection.
If you connect via a static IP, you can also create an ACL for the VPN connection just in case. You can set an ACL for the SSH port forward rule directly as well, but I don’t like that personally. I prefer keeping things behind the VPN.