Setting up a WordPress.org repo proxy/clone

If you read this you know how the internet works. You will therefore know that CPU and data traffic costs money.

In order to help OSS/WordPress we love giving back, read all about it here: https://www.managedwphosting.nl/giving-back/
We also love sustainable efforts, https://www.managedwphosting.nl/groene-hosting/

We reduce serverload, CPU and traffic from the dot-org servers for all Core installs like so: We have a local WP Core on each Managed server. When it’s its turn to update we use rsync to sync the changed files to each client’s sites. If , for instance, we have 50 sites at a server, then we only need to fetch WP Core once and then sync the changed files. We do this for years now.

However… we still have plugins and themes from dot-org to deal with. We are not to set up a server farm or infra just like the good folks at Aspirepress. We “just” want to reduce serverload at the dot-org servers.

Therefore we need to “tell” the installs that there is another repo URL. We only want to get the zip’s from other infra.

Do do so, we need to “hijack” the downloads.wordpress.org domain like so: https://gist.github.com/ramonfincken/525c9f38097a5ee19421fa0b1d94e7ce
Hint: use this as must-use plugin.

/**
 * Filters the HTTP request arguments before the request is made.
 *
 * This function checks if the request URL is for WordPress updates or plugin/theme information.
 * If so, it replaces the default API URL with our URLs.
 * Cloned from wpe_use_wpe_api_services
 *
 * @param bool|array $c     Whether to preempt the request return. Default false.
 * @param array      $args  HTTP request arguments.
 * @param string     $url   The request URL.
 *
 * @return bool|array The response or false if not preempting the request.
 */
function klarned_override_api_services( $c, $args, $url ) {
	if ( strpos( $url, 'downloads.wordpress.org' ) !== false && apply_filters( 'klarned_override_api_downloads', true ) ) {
		// Replace the default API URL with WPE's URL.
		$url = str_replace( '://downloads.wordpress.org', '://dotorg-downloads-proxy.yourdomain.com', $url );
		return wp_remote_request( $url, $args );
	}

	return $c;
}
add_filter( 'pre_http_request', 'klarned_override_api_services', 10, 3 );

Next .. we need a mechanism to get the zipfile, cache/store it and proxy it. There are 2 ways. In all cases YOU are responsible for upkeep.

Way 1.

Setup nginx ( high available cluster) and use this base snippet. Or use any other proxysoftware.
Note: you need to take care of TLS and other security stuff. If you don’t know what TLS is, stop reading now.

proxy_cache_path /var/cache/nginx keys_zone=my_cache:10m inactive=1w levels=1:2 max_size=50m;

server {
    listen 80;
    proxy_cache my_cache;

    location ~ \.zip$ {
        proxy_set_header Host downloads.wordpress.org;
        proxy_pass https://downloads.wordpress.org;
        proxy_cache_key $scheme://$host$uri$is_args$query_string;
        proxy_cache_valid 200 10m;
        proxy_cache_bypass $arg_should_bypass_cache;
        proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
        proxy_cache_lock on;
        add_header X-Cache-Status $upstream_cache_status;
    }
}

The result, and yes this is HTTP IP based as a first test (look out for ‘X-Cache-status’):

Way 2.

Less sexy, but it works

Use a pull CDN to downloads.wordpress.org. Preferably one that only allows zipfiles and if you want also IP based sources. It can be very pricey if it is too open.
Keep in mind that you need to override the host header.