Focus on your mission, not your tech - Another Cup of Coffee (Posts about SEO)

Fixing Access Forbidden (403) issues after migrating WordPress to a static site

Anthony Lopez-Vito — Thu, 09 Feb 2023 01:15:57 GMT

After migrating a WordPress site to static files, Google Search Console may start sending you notifications about page indexing problems. Often the issue will be marked as ‘Blocked due to access forbidden (403)‘, a fairly common error that can have different causes, depending on how your website and server is set up.

This article specifically deals with the case when:

You previously had a working WordPress installation;
You migrate the site to static HTML files on an Apache web server;
Google Search Console starts complaining about page indexing problems.

If this does not apply to your situation, the proposed solution may not work but you will still find the explanation useful in diagnosing the problem.

The Forbidden 403 error

Being in the 4XX category of HTTP response codes, Access Forbidden (403) is a client-side error that may show in a message similar to the following combinations:

Access Forbidden (403)
403 Forbidden
HTTP 403
HTTP Error 403 – Forbidden
Forbidden: You don’t have permission to access [directory path] on this server

The Google Search Console notification email will be about page indexing problems with ‘Blocked due to access forbidden (403)‘ as one of the Top Issues.

A typical Access Forbidden (403) notification from the Google Search Console

Nevertheless, it being a client-side error doesn’t necessary isolate the problem to your browser. Many solutions online will advise you to clear your cache or refresh the page. This is easy to do and usually a sensible first step but if you found the issue via a Google Search Console notification after a WordPress migration, these recommendations won’t help.

The 403 response code essentially means that a web client, such as your browser, does not have permission to access the requested resource. The web server understands the request but can’t allow access due to file permissions settings or a server misconfiguration. When it comes to WordPress, the error will almost certainly be due to a misbehaving plugin, a corrupt .htaccess file or incorrect file permissions. However, if you’ve recently converted your WordPress installation to a static site, the error will most likely have a different cause.

To understand why the Access Forbidden error happens, it’s worth reviewing the differences between how the WordPress content management system (CMS) serves content and how a static site responds to web client requests.

How WordPress serves your browser requests

WordPress is a database-driven Content management system (CMS). Most of the content is stored in a database and URLs generally do not correspond to any files in the web server’s filesystem. Instead, URLs are external references to the database content. WordPress calls these references permalinks and its rewriting engine uses internal rules, specified in the permalink settings, to build permalinks dynamically. When a client requests a page from the site, WordPress takes care of serving the correct content. Included with the pages will be any XML-based RSS (Really Simple Syndication) feeds. WordPress will send the HTML page or an XML feed, depending on what the client requests.

For example, if you set the permalink settings to the Post name structure, WordPress will generate a HTML post when you browse to the following URL:
https://example.com/your-post/

The RSS feed for that post will be at:
https://example.com/your-post/feed/

RSS feeds are normally parsed by an RSS client rather than for displaying directly in the browser window so requesting the RSS feed resource will not send HTML. Instead WordPress will generate XML for the feed.

Serving static files

By definition, static sites have no way of dynamically generating content based on the browser request. You make a request to the web server at a given URL and if the resource is present with the correct access permissions, the web server will go ahead and serve the file. The URL for a static site will normally look something like:
https://example.com/your-post.html.

If you have an RSS feed, it might look something like:
https://example.com/feed.xml.

Notice that the request includes the path to the file, such as your-post.html or feed.xml. The file is not present in a WordPress permalink because, as mentioned previously, it is only a reference to the actual content stored in the database.

Web server directory index

Web servers such as Apache also have a Directory Index directive. This is a configuration that can set the server to automatically send a file when a client makes a request without a filename in the URL. The file known as the directory index and is normally named index.html.

Web servers such as Apache also have a Directory Index directive. In the early days of the web, you could browse to a folder in the web server filesystem and get a listing of all the files present. For security, most web hosts now disable this feature for most of their hosting services. The Directory Index directive is a configuration that can set the server to automatically send a file when a client request only includes the folder name in the URL. For most hosting services, the standard directory index files are normally index.html and index.php.

Say a client makes a request for the following:
https://example.com/docs/

If the directory index is set to index.html, the server would return:
http://example.com/docs/index.html.

The index.html renders as a web page in the browser. Directory index resources can also be set to other file types like index.txt or index.xml.

Generating static HTML files from a WordPress site

WordPress plugins such as Simply Static will crawl your site to generate static HTML file copies for the pages. Since WordPress includes RSS feeds, static XML copies will also be generated for these feeds. The tables below show typical WordPress permalinks and their equivalents after static HTML copies are generated.

Page type	WordPress permalink
Page content	`https://example.com/your-post/`
RSS feed	`https://example.com/your-post/feed/`

Page type	Static file URL
Page content	`https://example.com/your-post/index.html`
RSS feed	`https://example.com/your-post/feed/index.xml`

Once again notice that the static site specifies the filename in the path whereas the WordPress permalink, when set to the Post name structure, does not.

Why access is forbidden

We have all the puzzle pieces to understand why you would get the Access Forbidden (403) error and how you can fix the problem. After you migrate your WordPress site to static files, the old permalink paths to pages will still serve a web page because most hosting providers have index.html as a directory index resource.

You can request the URL in the WordPress post name permalink format:
https://example.com/your-post/

The static file generator would have created an index.html in this location:
https://example.com/your-post/index.html

The webserver sees the index.html in the filesystem and delivers it to the browser which can render the web page content. Human site visitors will be perfectly happy because they receive the web page resources they expect. However, Googlebot, Google’s web page crawler, will spider through your site including the RSS feed locations. Remember, the RSS feed folders will contain an index.xml. XML files are not normally a default directory index resource for most web hosts. Since there is no index.html file in the feed folder, the web server thinks it’s being asked to deliver a file listing. Again remember that file listings are disabled by most web hosts for security. Thus you get the error:

Access Forbidden (403)

You are forbidden by the web server to access that folder listing.

How to fix the Access Forbidden (403) error

Now that we know exactly why we get this error we attempt a fix. If your host runs Apache, the solution will likely be simple. Edit or create the .htaccess file in the root of your site and add index.xml to the list of directory index resources. For example, if it not already there, add the following line somewhere near the top of the file:

DirectoryIndex index.html index.php index.xml

The index.xml in the list will tell Apache to serve the XML file when Googlebot requests the RSS feed directory. While you’re doing this, inspect a few o the feed directories to make sure the index.xml files have the correct permissions (usually 775 for most server setups) and the correct ownership. The ownership settings should be:

user is the user account with root privileges on your web server.
group is usually www-data or apache but you may need to check this with your hosting provider.

You can test your changes by pointing your browser to a few feed directories to see if the server returns XML. Remember to leave out the index.xml file and specify the directory only. For example:
https://example.com/your-post/feed/

If your browser displays the RSS feed XML, you know the problem has been solved and can be reasonably confident that Googlebot won’t encounter the 403 error.

This may not work if you’re running a VPS server for your site where some more in-depth configuration may be needed, such as editing your Apache configuration files. Also, .htaccess is not used by other web servers such as NGINX. In these cases, your next port of call should be to contact your hosting provider. Tell them that you would like to add index.xml to the directory index resources so that the XML file is served when a visitor lands in a directory. Most good hosting companies will be able to guide you on your options or make the change on your behalf. After you’ve applied the fix, be sure to go to your Google Search Console to revalidate the affected page.

Photo by Lucian Alexe on Unsplash

Preserving SEO during a Drupal to WordPress migration

Anthony Lopez-Vito — Wed, 16 Mar 2016 00:28:53 GMT

This article is about preserving SEO during a Drupal to WordPress migration. It’s not about which platform is better. A web search will bring up many articles about the relative SEO merits of Drupal versus WordPress. I’ve found the best summary on that topic in a forum post by Drupal.org user and developer, John Birchall:

“The consensus that WordPress has better SEO is a guess…WordPress fans will guess one way, Drupal fans the other. In my opinion, WordPress SEO plugins work better for sites which are short on skill or time or money. If you want to give your clients hand-made quality, Drupal does rather a good job. This also goes for SEO. Nevertheless, lack of developer skill and client budget does sometimes mean we are better off simply ‘reheating’ and rolling out the wonderful prepackaged solutions available for WordPress.” (Slightly paraphrased for clarity.)

If you’re reading this post, I will assume that you’ve already decided to migrate your site to WordPress. Rather than re-hashing the Drupal versus WordPress SEO debate, I’ll focus on some technical details that affect your SEO efforts when migrating from Drupal to WordPress. First, let’s start with some basics.

Can I preserve my SEO during the migration?

The quick answer is, ‘Yes’. It is possible to preserve search engine optimisation during a migration. However, it can be easy to overlook the idea that you might end up with better SEO by revisiting your approach entirely. I’ve had site migrations where replicating optimisations from the Drupal version wouldn’t have made a significant impact in the business—or at the very least, where SEO was something that could be improved after the migration. If your site falls into this category, I would not recommend putting too much into preserving your Drupal optimisations. The money spent hiring me or anyone else may be better spent on improving the WordPress SEO after exporting your Drupal content.

There are some easy and low-cost approaches you can take such as:

Using or building an SEO-friendly and responsive WordPress theme.
Installing SEO WordPress plugins.
Creating higher quality content.
Improving page load times. WordPress is a lighter CMS so you should get this ‘out of the box’ but you can lose any gains if, for example, you replicate calls to third-party libraries.

Of course, you should consult an expert if you rely heavily on search traffic. Preserving SEO may be vital for your site but it’s worth making a conscious assessment about the investment needed.

What kind of work is involved in preserving SEO?

The underlying differences between Drupal and WordPress mean preserving SEO often takes the bulk of a migration budget. The amount of work needed depends on what you’re trying to preserve. To give an example from a past migration project where SEO was vital, tasks included:

Pre-migration auditing, mapping and analysing links on the Drupal site;
Developing an SEO-friendly custom theme;
Creating redirects for every URL that would change on WordPress;
Ensuring we preserve on-page optimization for all high-value pages;
Making further adjustments as directed by an SEO consultant after she analysed her webmaster tools.

Remember that SEO isn’t a one-time effort. The project needed multiple months of on-going work. Further, I was taking direction from the client’s SEO consultant who incurred her own separate fees. Thus the client allocated a sizeable budget for SEO alone. In his case it was worth the money because the Drupal site earned him many times that amount on revenue from search traffic. Minimising any negative impact on SEO after migrating to WordPress was a priority.

How much time and money you’ll need to allocate for your migration depends on a range of factors that are very specific to your site. You’ll only know after doing some in-depth analysis and documentation. Here’s an article on Search Engine Journal that I send to clients when they ask about the SEO impact of a CMS migration: How To Avoid SEO Disaster During a Website Redesign. It’s worth a read and afterwards you can think about how much SEO should play a part of your site migration project.

SEO considerations

What exactly will affect SEO during a typical Drupal to WordPress migration? Here are a few SEO considerations to keep in mind for your migration project. It’s certainly not an exhaustive list but paying attention to these points should cover the basics.

On-page SEO

A lot of the on-page SEO comes over as a natural side-effect of most content migration processes. For example, page titles, content keyword optimisation, sub-heading tags, image file names and internal linking are all embedded into the Drupal node content. These remain intact when the nodes are imported into WordPress, unless content cleaning or other manipulation is part of the project.

Other areas, such as URL structure, meta data and load speed will need work due to the differences of the two systems.

Off-site SEO

Off-site SEO will be unaffected by a CMS migration as long as you have the necessary redirects set up to catch URL changes for any backlinks. This is an effort that’s external to the site so not within the scope of this article.

Drupal aliases vs WordPress permalinks

URL structure is one of the SEO areas that will change after a CMS migration. A big part is because of some important differences between how Drupal and WordPress handle URLs. Drupal creates URLs using either the node ID or a URL alias stored in its database’s url_alias table. Take a look at some sample entries.

Drupal url_alias table

You’ll see an entry for node/28 associated with a URL alias projects/april. This means that under Drupal, the page with node ID 28 can be accessed using either http://example.com/node/28 or http://example.com/projects/april. Not shown is that content can have multiple aliases. The site could therefore also have another entry node/28 associated with URL alias projects-april and yet another with my-projects/april. You will always be able to find a page by appending the URL alias entry in the url_alias table to the site’s domain name.

WordPress has a different mechanism for creating URLs. Rather than relying on database entries, it generates rewrite rules and builds the permalink dynamically depending on your permalink settings. WordPress’ closest equivalent to the Drupal url_alias is its permalink slug stored as the post_name in the wp_posts table. Under my standard Drupal to WordPress migration, I migrate over the Drupal aliases by converting them into WordPress post name slugs. (See the table mapping in a separate article, Drupal to WordPress migration: posts table mapping.) If you have the Post name structure set in the WordPress Permalinks Settings, you’d then find the Project April page at either http://example.com/?p=28 or http://example.com/april/. Note that this would cause an SEO problem because the new WordPress link does not match the old Drupal link.

WordPress wp_posts table

We would have to do some extra work to solve this problem. For example, if the april node was a page, we could create and link to a parent Projects page to create the Drupal http://example.com/projects/april structure. Alternatively, if it was a post, we could create a projects category or tag and set the Category base or Tag base setting. We could also create .htaccess redirects or use one of the many WordPress redirection plugins. The exact solution would depend on your project’s requirements.

WordPress permalinks have more constraints

Aside from the differences in how URLs are stored and generated, you should also be aware of WordPress permalink constrains. These will have a big impact on the amount of work needed to preserve your Drupal SEO when migrating to WordPress. I’ll briefly summarise them below.

The Drupal URL alias has a 255 character limit but the WordPress the post_name field stores only 200 characters. Make sure to keep track of any truncated posts names and create a corresponding redirect.
Since there’s no WordPress equivalent of the Drupal url_alias table, you’ll need to decide which alias will be converted into the post name. You should create separate redirects for the other aliases.
WordPress slugs which create the permalink are essentially a URL-friendly version of the post title. WordPress automatically cleans up the post title to create slugs for new content but you may encounter problems when migrating Druapl aliases into slugs. Characters that would be valid within a Drupal alias are invalid as a WordPress slug. For example, you cannot have forward slashes, accented characters and quotes in your WordPress slug. You’ll need to clean up and keep track of any problem aliases, otherwise WordPress will not be able to serve the page.
As previously mentioned, under Drupal you will always be able to find a page by appending the URL alias entry to the site’s domain name. WordPress does not store any records of its URL structure. You will end up with broken links and a potential SEO nightmare if you change the Permalink structure settings without consideration.

Remember your Drupal taxonomies

It’s easy to forget Drupal’s powerful taxonomy system is capable of generating listings that attract valuable traffic. Use webmaster tools to audit your site traffic to pick up on any such listing pages. Unlike Drupal which allows for multiple vocabularies, WordPress only offers one set of categories and tags out of the box. Replicating your Drupal taxonomy listings in WordPress may take considerable development but a quick fix may be to manually replicate the important listings pages. Obviously the size of your site will dictate if this will be feasible.

Menu and breadcrumb navigation

Menu structure and breadcrumb navigation is often an important part of your SEO. Ensure that your selected WordPress theme both supports breadcrumbs (some don’t) and the same type of breadcrumbs e.g. hierarchy, attribute and history-based breadcrumbs.

Migrate metatags generated by Drupal metatag modules

Also remember that you may have installed modules that generate metatag information contributing to your SEO. Export these from the relevant Drupal table and import them into the WordPress database using the format for your preferred WordPress metatag plugin. For example, if you used the Drupal metatag module and would like to use the WordPress Add-Meta-Tags plugin, you’ll need to extract data from the metatags table and insert them into wp_postmeta table as a meta_key and meta_value pair. Keep in mind that some investigative work will be necessary to find out how your preferred modules and plugins store their data.

Search engine visibility

WordPress has a setting to discourage search engines from indexing site. It can be found in Settings > Reading > Search Engine Visibility. It’s normal to have this checked during development but make sure you disable it when launching.

Load speed

WordPress, in general, requires fewer hosting resources than Drupal. You could very well find that a slow site on an under-performing server running Drupal becomes more responsive under WordPress. Don’t take this for granted, however. Use the migration as an opportunity to make improvements. Thoroughly vet plugin candidates to make sure they’re not pulling in third-party scripts that will increase page load times. Also avoid bloated page-builder themes. In my experience, these can be terrible for load speed as they try to cram in every conceivable bit functionality, thereby drastically slowing down what could have been a nimble site.

Preserving SEO during a Drupal to WordPress migration can be lots of work

As may have gathered by now, the amount of work and budget needed will reply on many factors. It’s not possible to prescribe what will be necessary for your site without in-depth detail about your configuration and requirements. Nevertheless, I hopefully will have given you enough information to get started with planning an approach.

A little plug to keep the lights running. If you think all of this work is too much trouble, please consider hiring me for your Drupal to WordPress migration project.

Drupal to WordPress migration service

Any Drupal version · All content · Custom content types · SEO · Plugins

If you’re not sure how to make the appropriate changes or would simply like someone else to do the work, please request a quotation for my Drupal to WordPress migration service.