When it comes to managing WordPress sites, one thing we often overlook is the mischief that crawlers can wreak—especially within endpoints. One notorious example is wp-json, an easy target for malicious bots. 

Fortunately, we have strategies to help you manage and quell uninvited bot parties in your endpoints.

Understanding the problem with crawlers

While WordPress endpoints are designed to index content without the need to load the theme, some crawlers—like googlebot—prioritize viewing content in the frontend context. 

Unfortunately, even legitimate bots, like the one from archive.org, sometimes stray into wp-json endpoints.

Mischievous bots can cause significant slowdowns, especially when they bombard a site with complex, simultaneous requests. For example, abuse of the search facility of many wp-json routes can rapidly produce uncached requests, busying PHP threads and SQL queries.

Worse, repeated uncached requests can consume extensive WordPress resources, potentially leading to a DDoS attack.

Setting boundaries for bots

So how can WordPress CMS customers tell bots, “Hey, get out of my endpoints?” Try these proven methods. 

Asking politely: gentle ways for legit bots

Here are two straightforward methods to ask bots that obey Robots directives to leave endpoints alone:

• Robots.txt. Just add a disallow rule for /wp-json/ or paths thereof. Check this guide to modify the robots.txt file.

• Headers. Add headers for the whole wp-json endpoint or individual route for crawlers that prefer header directives to robots.txt. Here’s an example:

add_filter('rest_pre_serve_request', function ($served, $result, $request, $server) {
    $route = $request->get_route();
   
    $routesNeedingHeader = array(
      '/wp/', // all of WP
      '/wp/v2/posts/' // specific route
    );
   
    foreach ($routesNeedingHeader as $path) {
      if (strpos($route, $path) === 0) {
        header('X-Robots-Tag: noindex');
        header('Customheader: whatever');
        break;
      }
    }
   
    return $served;
}, 10, 4);

Being assertive: stringent measures for malicious bots

So, what if asking bots gets the door slammed in your face? Try actively denying them. 

• Set endpoints to auth-only. Ideal for WordPress websites that want restricted endpoint access for most visitors but allow them for authorized access (for example, via app passwords). We like Disable JSON API to specify which routes to restrict.

• Disable wp-json entirely. To completely turn off wp-json, there are several plugins available, including Disable XML-RPC API.

• Rate-Limit bots. By setting a strict rate-limit for entities with known user-agent strings, it’s quite possible to serve legitimate wp-json requests at scale without ever stretching the site’s resources. 

Hit the road, bots!

Although the issue of “crawlers in the endpoints” might not be visibly rampant across all platforms, it’s undeniable that these bots can misuse and abuse the numerous endpoints offered by WordPress, including those added by plugins. In the end, it can all add up to a considerable, cumulative waste of resources. 

If you’ve devised other ingenious ways to safeguard both users and infrastructure from the next malicious bot party, we’d love to hear from you.

Learn more about rest-apihere.