Disable direct access via static IP address to my website

Edit: Corrected the answer so that it actually makes sense.

When a HTTP/HTTPS request is made to your web server, one of the headers passed is Host. If someone is using a browser and types http://www.example.com/ then you'll see www.example.com in the Host header. If they type http://1.2.3.4 then you'll see 1.2.3.4.

So what you can do is configure your web server only to serve up your website when the Host header is your domain name. What you'd also generally do is configure it to serve a static page (which could be blank, it could show an error, it could be any HTML) for people accessing the web server with any other hostname.

It depends on which web server software you're using, but for example: with Apache you'd configured this using the VirtualHost directive.