Has your Drupal site poor performance from the beginning or have you become victim of your own success and need to scale your online store or business? There are quick wins and deeply buried treasures to be found.
If your site is slow even if there is not much load on the site there are likely some low hanging fruits to collect and your site will be delivered much faster without digging deep. Maybe adjusting some configurations, enable aggregation or caching for some views is enough. It is also important to check that the host system is not the root cause of the slow performance (e.g. massively oversold shared hosts).
The next step depends on the previously identified bottlenecks. This could be as easy as upgrade to a bigger hosting plan (if the host is generally good and not oversold) or switch to a new hosting provider/service with known track record or deliver assets through a CDN. If you are on a dedicated server already there are also relatively easy steps to gain performance by tuning the web server, PHP and Database configurations and introduce relatively easy to setup caching solutions like Redis or Memcached.
From here on it may makes sense continue with doing some basic PHP profiling to rule out that there is any single part of your codebase causing all the trouble (to avoid wasting time on further optimizations). If that was fixed or nothing found it could make sense to go on with checking server metrics. Maybe the webserver or PHP-FPM config is eating too much memory, or the database is on the same server fighting with other processes for system resources due to high load and needs to be offloaded to it's own server. This is the stage where your Drupal sites and infrastructure get's checked to identify changes prioritized by impact relative to time investment.