Yes you can start with Tableau Desktop. It is a very powerful tool in itself and can be purchased at the price lower than that of Tableau Server.
Desktop gives you both the option to create and publish your dashboards. These dashboards might get connected to flat files as data source or even a database or datawarehouse. Both works equally.
You can even schedule subscriptions where registered users will receive eMail with the snapshot of the data in the dashboard at the time specified by the developer.
As such, there are no downsides of using Tableau Desktop over Tableau Server. There are a few functionalities that are only enabled in Tableau Server but absence of them won't cause any hinderance to your services.
On the other hand, if your organisation deals in B2B models, Tableau server gives you enough Security and better collaboration features. It has good visualisations on mobile devices.