madebot

Tag: nginx

  • setup a redirect for ads.txt in nginx

    ads.txt is the file that is used by ad providers to determine what ads should be served on what sites. It is a form of verification and helps to prevent fraud.

    If you run your own ad waterfall, or use a third party ad provider you may have to frequently update your ads.txt file in order to keep up to date with the latest providers. Keeping ads.txt up to date ensures that you are paid for the most up to date list of advertisers and networks.

    I have just updated this file manually when a new file is released. On our blog we have a Mediavine plugin from our ad provider that does this automatically, so when it changed I would just update it on our main www site. The IAB ads.txt 1.02 standard allows you a single redirect, so now that our ad provider Mediavine supports this standard I thought I would finally automate it and stop adding it to our source code checkins on our main www site. In nginx, it’s a simple redirect or rewrite.

    You can add this location directive to your nginx server block config:

    location /ads.txt {            
return 301 https://adstxt.mediavine.com/sites/{YOUR-SITENAME}/ads.txt;
}

    Alternatively, you could add a rewrite rule:

    rewrite ^/ads.txt$ https://adstxt.mediavine.com/sites/{YOUR-SITENAME}/ads.txt permanent;

    Test the redirect by loading yourdomainname/ads.txt in a browser. Huzzah, yay for redirects.

    If you’re not editing your web config directly, you could also use cpanel, your wordpress redirections plugin, rankmath redirections, or other admin tool.

    If you want a list of more comprehensive steps to check that your ads.txt is working correctly, check out google’s admanager post.

  • blocking bad bots in nginx from xmlrpc.php

    After upgrading from PHP 7.2 to 7.4.x, I was taking a look at my graphs in new relic for my wordpress blog and found a high number of E_WARNINGs for //xmlrpc.php. It’s a result of bots hitting a non-existent URL at //xmlrpc.php – note the extra slash in there. The default nginx behavior is to merge double slashes, which normally is fine.

    In this case, I added a location block with //xmlrpc.php to block it but I noticed it was not working (hitting that particular URI still returned with a 405 Method Not Allowed instead of a 403 Forbidden). It turns out that I needed to add ‘merge_slashes off;’ and then it worked. Voila, lots of spurious errors removed from my logs (and from hitting PHP). Note that most of the xmlrpc functionality has been supplanted by the newer WordPress REST API, so you likely do not need it enabled.

    Error graph showing bot or erroneous access to //xmlrpc.php
    merge_slashes off;
location = //xmlrpc.php {
    deny all;
}

    If you want to use a plug-in, you could search for ‘Disable XML-RPC’ in the WordPress plug-ins repository or install WordFence. Buh-bye bad bots.