I think the best way is to use the Service principal with the role of Virtual Machine Contributor.
It just lets the user manage the VM, but without access to the VM and also do not have the permission to access the Azure portal.
Then you can use this service principal to execute the Azure CLI, Azure PowerShell commands or the REST API to start/stop the Azure VM, it does not cost.