Focus on your mission, not your tech - Another Cup of Coffee (Posts about Sys Admin)

Fixing Access Forbidden (403) issues after migrating WordPress to a static site

Anthony Lopez-Vito — Thu, 09 Feb 2023 01:15:57 GMT

After migrating a WordPress site to static files, Google Search Console may start sending you notifications about page indexing problems. Often the issue will be marked as ‘Blocked due to access forbidden (403)‘, a fairly common error that can have different causes, depending on how your website and server is set up.

This article specifically deals with the case when:

You previously had a working WordPress installation;
You migrate the site to static HTML files on an Apache web server;
Google Search Console starts complaining about page indexing problems.

If this does not apply to your situation, the proposed solution may not work but you will still find the explanation useful in diagnosing the problem.

The Forbidden 403 error

Being in the 4XX category of HTTP response codes, Access Forbidden (403) is a client-side error that may show in a message similar to the following combinations:

Access Forbidden (403)
403 Forbidden
HTTP 403
HTTP Error 403 – Forbidden
Forbidden: You don’t have permission to access [directory path] on this server

The Google Search Console notification email will be about page indexing problems with ‘Blocked due to access forbidden (403)‘ as one of the Top Issues.

A typical Access Forbidden (403) notification from the Google Search Console

Nevertheless, it being a client-side error doesn’t necessary isolate the problem to your browser. Many solutions online will advise you to clear your cache or refresh the page. This is easy to do and usually a sensible first step but if you found the issue via a Google Search Console notification after a WordPress migration, these recommendations won’t help.

The 403 response code essentially means that a web client, such as your browser, does not have permission to access the requested resource. The web server understands the request but can’t allow access due to file permissions settings or a server misconfiguration. When it comes to WordPress, the error will almost certainly be due to a misbehaving plugin, a corrupt .htaccess file or incorrect file permissions. However, if you’ve recently converted your WordPress installation to a static site, the error will most likely have a different cause.

To understand why the Access Forbidden error happens, it’s worth reviewing the differences between how the WordPress content management system (CMS) serves content and how a static site responds to web client requests.

How WordPress serves your browser requests

WordPress is a database-driven Content management system (CMS). Most of the content is stored in a database and URLs generally do not correspond to any files in the web server’s filesystem. Instead, URLs are external references to the database content. WordPress calls these references permalinks and its rewriting engine uses internal rules, specified in the permalink settings, to build permalinks dynamically. When a client requests a page from the site, WordPress takes care of serving the correct content. Included with the pages will be any XML-based RSS (Really Simple Syndication) feeds. WordPress will send the HTML page or an XML feed, depending on what the client requests.

For example, if you set the permalink settings to the Post name structure, WordPress will generate a HTML post when you browse to the following URL:
https://example.com/your-post/

The RSS feed for that post will be at:
https://example.com/your-post/feed/

RSS feeds are normally parsed by an RSS client rather than for displaying directly in the browser window so requesting the RSS feed resource will not send HTML. Instead WordPress will generate XML for the feed.

Serving static files

By definition, static sites have no way of dynamically generating content based on the browser request. You make a request to the web server at a given URL and if the resource is present with the correct access permissions, the web server will go ahead and serve the file. The URL for a static site will normally look something like:
https://example.com/your-post.html.

If you have an RSS feed, it might look something like:
https://example.com/feed.xml.

Notice that the request includes the path to the file, such as your-post.html or feed.xml. The file is not present in a WordPress permalink because, as mentioned previously, it is only a reference to the actual content stored in the database.

Web server directory index

Web servers such as Apache also have a Directory Index directive. This is a configuration that can set the server to automatically send a file when a client makes a request without a filename in the URL. The file known as the directory index and is normally named index.html.

Web servers such as Apache also have a Directory Index directive. In the early days of the web, you could browse to a folder in the web server filesystem and get a listing of all the files present. For security, most web hosts now disable this feature for most of their hosting services. The Directory Index directive is a configuration that can set the server to automatically send a file when a client request only includes the folder name in the URL. For most hosting services, the standard directory index files are normally index.html and index.php.

Say a client makes a request for the following:
https://example.com/docs/

If the directory index is set to index.html, the server would return:
http://example.com/docs/index.html.

The index.html renders as a web page in the browser. Directory index resources can also be set to other file types like index.txt or index.xml.

Generating static HTML files from a WordPress site

WordPress plugins such as Simply Static will crawl your site to generate static HTML file copies for the pages. Since WordPress includes RSS feeds, static XML copies will also be generated for these feeds. The tables below show typical WordPress permalinks and their equivalents after static HTML copies are generated.

Page type	WordPress permalink
Page content	`https://example.com/your-post/`
RSS feed	`https://example.com/your-post/feed/`

Page type	Static file URL
Page content	`https://example.com/your-post/index.html`
RSS feed	`https://example.com/your-post/feed/index.xml`

Once again notice that the static site specifies the filename in the path whereas the WordPress permalink, when set to the Post name structure, does not.

Why access is forbidden

We have all the puzzle pieces to understand why you would get the Access Forbidden (403) error and how you can fix the problem. After you migrate your WordPress site to static files, the old permalink paths to pages will still serve a web page because most hosting providers have index.html as a directory index resource.

You can request the URL in the WordPress post name permalink format:
https://example.com/your-post/

The static file generator would have created an index.html in this location:
https://example.com/your-post/index.html

The webserver sees the index.html in the filesystem and delivers it to the browser which can render the web page content. Human site visitors will be perfectly happy because they receive the web page resources they expect. However, Googlebot, Google’s web page crawler, will spider through your site including the RSS feed locations. Remember, the RSS feed folders will contain an index.xml. XML files are not normally a default directory index resource for most web hosts. Since there is no index.html file in the feed folder, the web server thinks it’s being asked to deliver a file listing. Again remember that file listings are disabled by most web hosts for security. Thus you get the error:

Access Forbidden (403)

You are forbidden by the web server to access that folder listing.

How to fix the Access Forbidden (403) error

Now that we know exactly why we get this error we attempt a fix. If your host runs Apache, the solution will likely be simple. Edit or create the .htaccess file in the root of your site and add index.xml to the list of directory index resources. For example, if it not already there, add the following line somewhere near the top of the file:

DirectoryIndex index.html index.php index.xml

The index.xml in the list will tell Apache to serve the XML file when Googlebot requests the RSS feed directory. While you’re doing this, inspect a few o the feed directories to make sure the index.xml files have the correct permissions (usually 775 for most server setups) and the correct ownership. The ownership settings should be:

user is the user account with root privileges on your web server.
group is usually www-data or apache but you may need to check this with your hosting provider.

You can test your changes by pointing your browser to a few feed directories to see if the server returns XML. Remember to leave out the index.xml file and specify the directory only. For example:
https://example.com/your-post/feed/

If your browser displays the RSS feed XML, you know the problem has been solved and can be reasonably confident that Googlebot won’t encounter the 403 error.

This may not work if you’re running a VPS server for your site where some more in-depth configuration may be needed, such as editing your Apache configuration files. Also, .htaccess is not used by other web servers such as NGINX. In these cases, your next port of call should be to contact your hosting provider. Tell them that you would like to add index.xml to the directory index resources so that the XML file is served when a visitor lands in a directory. Most good hosting companies will be able to guide you on your options or make the change on your behalf. After you’ve applied the fix, be sure to go to your Google Search Console to revalidate the affected page.

Photo by Lucian Alexe on Unsplash

How to set up a MySQL database server on Ubuntu for Drupal to WordPress migrations

Anthony Lopez-Vito — Fri, 11 Jun 2021 15:44:52 GMT

Content Management System (CMS) migration projects involve moving data between databases with different schemas. Table names, field types and constraints often don’t match up, leading to a number of frustrating errors. This is especially the case with highly complex Drupal to WordPress migrations that use custom scripts to extract, transform and load the data. These projects can seem like you’re constantly hitting roadblocks throughout. I’ve found from experience that running the migration in an appropriate development environment can reduce a great deal of effort. In this guide, I will describe how to set up a MySQL database server on Ubuntu for Drupal to WordPress migrations.

Why MySQL and Ubuntu?

The first thing to address is why MySQL and Ubuntu? How about MariaDB? How about Arch Linux, Mac OS or Windows? Yes! Any platform that runs Drupal and WordPress will work for your migration environment so you can use whatever you prefer. Personally, I have a fondness for OpenBSD but it’s not a practical platform for a CMS migration. OpenBSD’s niche user-base means you’ll spend much longer installing necessary tools and troubleshooting errors.

There are all sorts of tutorials covering MySQL on Ubuntu. This means you’re more likely to quickly find a solution from a web search when you hit a problem. Furthermore, if you need a software utility or program to help you get the job done, it will probably be available through apt, dpkg, snap or tasksel. Use whatever you like but for now, MySQL on Ubuntu is my recommended platform for Drupal to WordPress migrations. I expect this will be the case for some time to come. These projects are complex and time-consuming enough without making the job more difficult.

Installing MySQL on Ubuntu for CMS migrations

There are many detailed tutorials for installing MySQL on Ubuntu. DigitalOcean’s How To Install MySQL on Ubuntu 20.04 is a good one and writing another won’t add much value. My guide will therefore only give a brief overview of the MySQL server installation steps. Instead, I will focus on the configuration areas specifically related avoiding problems on a CMS migration project.

You may wonder why the migration environment should be much different from a live server. Migration projects require you to do things that aren’t supported by the CMS platform. You’re therefore likely to encounter weird errors that aren’t normally found when running standard Drupal or WordPress.

Set up your Ubuntu LAMP migration platform

The main source of unusual errors is almost certainly because you’re migrating on a setup suited to a live website. Live server configurations are more restrictive than you need for a migration project. You can therefore save yourself a huge headache by rolling your own local migration environment. It might take a little longer to get started but you’ll save time by avoiding lots of unnecessary troubleshooting.

I must highlight that this will be a local migration environment and should not be accessible from the public internet. The normal security considerations with running a live content management system don’t apply when you’re working locally. By all means follow basic security measures mentioned the various tutorials for setting up Ubuntu and MySQL. Nevertheless, a highly secure setup is counterproductive for these projects and you can avoid trouble by being a little more permissive.

Go ahead and install Ubuntu Desktop. Since this will be a development environment, you’ll want the Desktop environment rather than the more lightweight server version. Of course, you’ll still need to install a web and database server. Follow these instructions for installing LAMP stack but skip the step of installing MariaDB. As mentioned above, we’ll be using MySQL.

WARNING: Installing MariaDB over MySQL or vice versa on Ubuntu 20.04 may lead to all sorts of problems starting up the database server with errors like the following:

Failed to start mysqld.service: Unit mysqld.service not found.

The last time I did this, none of solutions mentioned online for purging the installation worked. I spent most of a day trying to fix the problem. In the end, I realised it was quicker to start again and rebuild the machine from scratch. This is a big reason why I decided to stick with MySQL as standard for my projects.

Install MySQL

You can read a more detailed tutorial on installing MySQL on Ubuntu but here’s an overview.

Update the package index on your server: sudo apt update
Install MySQL server: sudo apt install mysql-server
Secure MySQL: sudo /usr/bin/mysql_secure_installation

The mysql_secure_installation script doesn’t cause problems for migrations so it’s worth running.

Create an admin user:

CREATE USER ‘user’@'localhost’ IDENTIFIED BY ‘userpassword’;
GRANT ALL PRIVILEGES ON *.* to ‘user’@'localhost’ WITH GRANT OPTION;
FLUSH PRIVILEGES;
quit;

Now create a user and database for your migration project:

CREATE USER ‘projectuser’@'localhost’ IDENTIFIED BY ‘password’;

CREATE DATABASE project_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci;
GRANT ALL ON project_db.* TO ‘projectuser’@'localhost’;

You can give this user more restrictive permissions but GRANT ALL avoids errors when running scripts and SQL queries on the database. Only the migration team should have access and you won’t need it after the migration so why not make your life easier?

Setting the MySQL server SQL mode

It’s possible that you won’t be familiar with SQL modes unless you’ve done some database administration work. For our purposes, SQL modes do two things:

change the types of queries you can run on your MySQL server;
change the validation checks when altering the data.

I’ve found that specific SQL modes need to be set for Drupal to WordPress projects. You may find that all sorts of strange errors appear if the correct modes aren’t set. The following sections show you two ways to set your MySQL server’s SQL mode.

Option 1: Setting the global sql_mode in the database

Login in to database as an admin user.
View the current sql-modes using SELECT @@GLOBAL.sql_mode; and make a copy if necessary.
Copy the current modes (add or delete modes as needed) and paste in next step.
Add ALLOW_INVALID_DATES and removes both NO_ZERO_DATE, NO_ZERO_IN_DATE by setting the sql-modes with
SET GLOBAL sql_mode = ‘STRICT_TRANS_TABLES,ALLOW_INVALID_DATES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION’; (WARNING: check the modes correspond with your setup.)
Restart server:
sudo systemctl start mysql

Option 2: setting the sql_mode in the MySQL configuration file

Locating the MySQL configuration file

The my.cnf configuration file isn’t always found in the same place. It’s specific to the Linux distribution and server configuration but can normally found in one of the following locations:

/etc/my.cnf
/etc/mysql/my.cnf
echo/my.cnf
[datadir]/my.cnf
~/.my.cnf

If you can’t find your MySQL configuration file, you can try running locate my.cnf or mysqladmin –help. The latter will show something like the following in the output:

Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf

Also keep in mind that it’s possible to use !include directives to include other option files and !includedir to search specific directories for option files. Under Ubuntu, there may be a file /etc/mysql/my.cnf with !includedir directives to search /etc/mysql/conf.d/ and /etc/mysql/mysql.conf.d/

Editing the MySQL configuration file

If the MySQL server finds more than one configuration file, it will load each one in turn. The values override each other and it can be difficult to know which takes priority. Furthermore, the –defaults-file parameter can also override all configurations. Keep things simple and have only one file and place it the directory that makes sense to you.

Before editing the my.cnf, first log in to MySQL with an administrator user and run the SELECT @@GLOBAL.sql_mode query to see the values used in your setup.

Running the SELECT @@GLOBAL.sql_mode query on MySQL Workbench

Next, open the configuration file, look for the section [mysqld] and edit the line starting with:
sql_mode = …

Add the line if it’s not there. Adjust the exact modes to match your project’s needs so take a look at the list of SQL modes to see which may apply. I’ve found the following works well:
sql_mode = “STRICT_TRANS_TABLES,ALLOW_INVALID_DATES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION”

Finally, restart the MySQL server. On Ubuntu this will probably be with

sudo systemctl start mysql

If you are logged in to the MySQL server, you may also need to disconnect your client and reconnect for the changes to take effect for your session.

Potential errors

Here are some potential errors that you may come across during a CMS migration project. I usually find them when running a Drupal to WordPress migration on a freshly built development environment.

mysqldump access denied when trying to dump tablespaces

Migrations involve dumping and importing databases and this process is straightforward on a mature development environment. However you may receive an ‘Access denied’ error out of the blue when dumping your MySQL database:

mysqldump: Error: ‘Access denied; you need (at least one of) the PROCESS privilege(s) for this operation’ when trying to dump tablespaces

If you see this, perhaps you were working on an environment that was a little too ‘mature’ (in other words, obsolete!) and you have recently upgraded your installation. The updates for MySQL 5.7.31 and MySQL 8.0.21 in July 2020 introduced an incompatible change that produces this error.

Read my separate article, How to fix the mysqldump access denied process privilege error, for more information and instructions on how you can try solving this.

ERROR 1067 (42000) Invalid default value

Drupal nodes store the date as a Unix timestamp in an int (e.g. 1623427200) field whereas WordPress stores dates as datetime (e.g. 2021-06-11 16:00:00). There may be a conversion error in your migration script or the source date could simply be zero for some reason. Normally your MySQL server mode will be set to NO_ZERO_DATE, NO_ZERO_IN_DATE so trying to insert a zero date will give you the error:

ERROR 1067 (42000) Invalid default value

You can fix this by replacing NO_ZERO_DATE, NO_ZERO_IN_DATE with set to ALLOW_INVALID_DATES in your global SQL mode.

Expression #1 of SELECT list is not in GROUP BY clause

You run an SQL query and get the rather cryptic error:

Expression #1 of SELECT list is not in GROUP BY clause and contains
nonaggregated column ‘database.table.pid’ which is not functionally dependent
on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by

What’s wrong? MySQL has a only_full_group_by mode which, when enabled, strictly applies ANSI SQL rules when using GROUP BY. Fix this by reworking your script or removing the ONLY_FULL_GROUP_BY SQL mode.

Error Code: 2013. Lost connection to MySQL server

This often happens when it an SQL query takes too long to return data. The connection between your MySQL client and database server times out so the connection gets dropped. For ideas on how to try solving this, read my separate article, How to fix Error Code 2013 Lost connection to MySQL server.

Conclusion

Setting up a MySQL database server for Drupal or WordPress is a familiar task for web developers and site administrators. However, CMS migrations have quirks that can cause obscure and baffling errors. In this guide I’ve shown you some little tricks that may save time and annoyance. While there’s no way to provide an exhaustive list of solutions to all the MySQL problems you’ll encounter, I hope to have pointed you in the right direction.

If you have a site migration project and would like to hire me, please ask for a quote for my consulting service.

CMS migration consulting

All content · Custom content types · SEO · Plugins

Migrating content from a site and need a specialist? Please contact me for a quotation. Whether you’re a media agency who needs a database expert or a site owner looking for advice, I’ll save you time and ensure accurate content exports.

Cover photo by Rubaitul Azad on Unsplash