Handy to know shizzle

Explore

Handy to know shizzle

Handy to know shizzle

apache + fcgid + suEXEC

Author:

/u/kyle0r⁠

Published: 2024-05-03

Motivations

To create a secure and performant shared vhost setup with apache.

If needed, vhosts can be provisioned with MySQL compatible MariaDB storage, which would qualify as LAMP hosting [

ref⁠

I implemented the first revision of this setup back in 2010 or perhaps earlier. This page represents the 2024 refresh and knowledge dump of the setup. I’ll likely create a container with this setup in the near future, so these notes will be handy.

In the [

performance⁠

] section it is possible to see that the “very-large” performance profile could serve a simple PHP8 script at ~400 RPS without any significant performance tuning or opcache. In contrast the default “small” performance profile prevents vhosts from overloading the system and mitigates the risk of DoS attacks on cgi scripts.

General implementation notes

Apache handles a request, vhost config invokes suEXEC [

doc⁠

] which spawns a CGI wrapper as a specific user:group. Not as safe as a jailed apache per vhost (or container per vhost) but a good compromise, much more secure than running mod_php.

Thanks to a dedicated Linux user:group being created per vhost, suEXEC can invoke vhost CGI applications with a dedicated user:group. This creates a logical permissions firewall between one vhost and the next.

Each vhost has its own PHPRC and PHP_INI_SCAN_DIR so PHP is configurable per vhost to the nth degree by the sysop.

Following InfoSec best practices of removing defaults and hardening config, this approach prevents PHP using the distribution/system default PHP config and modules. With this approach a PHP enabled vhost will have no PHP modules loaded by default. The sysop must selectively enable PHP modules per vhost based on the requirements of the running apps. The PHP config paths are owned by root, and by the vhost group specified in SuexecUserGroup who can only read, so the PHP config is not modifiable by the vhost user:group. i.e. PHP CGI apps cannot modify their own PHP config.

/var/www/{hostname}/php

/var/www/{hostname}/php/conf.d

dr-xr-x--- 3 root {vhost-group}

Each vhost has its own CGI wrapper, which creates process separation between vhosts. The wrapper can customise the ENV for the CGI application. Aspects like PHP_* variables. suEXEC clears the ENV when it spawns CGI processes except for a hardcoded env var whitelist.

Request and execution sequence

request arrives at apache

vhost lookup

apache calls FcgidWrapper via suEXEC using the vhost specified Linux user:group in SuexecUserGroup

suEXEC clears the env except for compile time defined env var whitelist [

see here⁠

]

this means the use of env vars in FcgidInitialEnv and FcgidCmdOptions is limited to the suEXEC env var whitelist

all other env vars will be discarded

The FcgidWrapper script/binary can set required env vars

for example: PHPRC, PHP_INI_SCAN_DIR, PHP_FCGI_CHILDREN, PHP_FCGI_MAX_REQUESTS can be dynamic: export PHPRC="$(readlink /var/www/000-vhost-usermaps/${USER})/php" export PHP_INI_SCAN_DIR="$PHPRC/conf.d" This example facilitates a vhost specific php configuration and module specification

FcgidWrapper calls exec to replace the process with the specified program e.g. php-cgi*

🔐 Apache AuthN and AuthZ

With apache configuration, I have found it is super critical to pay attention to “What am I putting behind an auth challenge?”. Consider the following statement from the apache docs [

citation⁠

Use <Location> to apply directives to content that lives outside the filesystem. For content that lives in the filesystem, use <Directory> and <Files>. An exception is <Location "/">, which is an easy way to apply a configuration to the entire server (editors note: or vhost).

💡 So, I try to remember: “Is the the thing I’m trying to place behind an auth challenge logical OR physical?” If its physical i.e. a path on the system then I probably want to use <Directory>, and if its more of a logical concept or alias then I probably want to use <Location>.

Pitfalls

I fell into the situation were I had configured the relevant auth directives, copied from a known working config. However requests that I expected to provide an auth challenge were being permitted without a challenge (auth not working). Why? I was trying to use a vhost DocoumentRoot path <Directory /var/www/.../user/htdocs> directive to apply auth directives to files outside of this physical path. This sounds obvious when I write it down... BUT the app I was trying to secure access to was being Include’ed into the vhost (it was outside the doc root), which included an Alias directive to specify the URL the app should use e.g. Alias /app1 /usr/share/app1. So /app1 was logically in the doc root but not physically in the doc root, so the <Directory> directive never matched for the /app1 requests and no auth challenge was sent in the response. 😡

Once upon a time, late at night... I went down an incorrect rabbit hole thinking that apache could not handle the auth for the CGI /app1. Either the CGI app or some form of proxy+auth would have to handle the auth. I got distracted by FcgidAuth* directives [

doc⁠

] and mod_authnz_fcgi [

doc⁠

]. While those are valid options they are orders of magnitude more complex that what I needed.

Solutions

To resolve this issue there are two solutions:

Use a <Location /app1> directive to specify the auth directives. I refer to this a logical location specification OR

Use a <Directory /usr/share/app1> directive to specify the auth directives. I refer to this a physical location specification

PHP and OPCode caching

OPcache improves PHP performance by storing pre-compiled script bytecode in shared memory, thereby removing the need for PHP to load and parse scripts on each request.

OPCode caching support is limited when running PHP under fcgid. From the docs:

The popular APC opcode cache for PHP cannot share a cache between PHP FastCGI processes unless PHP manages the child processes (which should be disable when using fgcid). Thus, the effectiveness of the cache is limited with mod_fcgid; concurrent PHP requests will use different opcode caches.

This means an alternative OPCode cache strategy is required when running PHP under fcgid (if you wish to achieve optimal OpCode cache usage).

Important conventions

{hostname} is a reference to the primary FQDN[

ref⁠

] aka hostname for a vhost e.g. blog.example.com.

{vhost-user} and {vhost-group} reference the vhost Linux username and group.

blog.example.com vhost hostname would be mapped to Linux user blog-example-com. Note the dots vs. dashes.

As a best practice Linux usernames should only contain characters defined in NAME_REGEX located in /etc/adduser.conf. Here is a [

link⁠

] to regulex explaining the regex from by Debian (bookworm) system.

Therefore, the vhost {hostname} should not be used as the Linux username because it contains dots. A simple replacement of the dots with dashes solves the issue but it means a {hostname} ←→ {vhost-user} mapping is required. I solve this with symlinks which you’ll discover in this documentation.

This means if we know the {vhost-user}, we can lookup or reference the {hostname}, and the path to the vhost files and configs. If we know the vhost {hostname}, we can lookup or reference the Linux {vhost-user} associated to a vhost.

Important paths

Defined in apache envvars and utilised in vhost configs BASE_PATH=/var/www and INCLUDE_PATH=/etc/apache2/sites-includes

This symlink facilitates referencing ${BASE_PATH}/fastcgi in vhost configs to specify the path to FcgidWrapper. /var/www/fastcgi → /etc/fastcgi

This symlink facilitates mapping and referencing {hostname} ←→ {vhost-user} /var/www/000-vhost-usermaps → /etc/ssh/chroot-usermaps

suEXEC doc root and location of CGI wrappers per vhost /etc/fastcgi/{hostname}

Location of fcgid performance profiles to be included in apache vhost configs /etc/fastcgi/000-apache2

This symlink is used in vhost config e.g. ${INCLUDE_PATH}/fastcgi/small.conf /etc/apache2/sites-includes/fastcgi → /etc/fastcgi/000-apache2

This config specifies suEXEC doc root and optionally userdir /etc/apache2/suexec/www-data

🔐 ensuring mod_php is disabled

mod_php is really only suitable for deployment on non-shared hosting (or hosting that is chroot’ed) because scripts run as the apache user e.g. www-data, which is very insecure on a multi-site / shared system. Any vhost can read/write any other vhost files.

Just in case mod_php is installed and activated, as a InfoSec best practice, the ZZZ-php-blocked-and-disabled default server-wide config ensures that any request for .php.* and .phtml is Require all denied.

🔐 This also protects the source code of any php files which are inadvertently located on a vhost where PHP is disabled.

In addition, if for some reason the php_module is loaded then php_admin_flag engine off will be enforced, which disables PHP.

a2dismod php*

a2enconf ZZZ-php-blocked-and-disabled

💡 vhosts that require PHP must explicitly include a PHP enabler config which reverses the Require all denied. See the sample vhost config for details [

here⁠

Include ${INCLUDE_PATH}/php_files_enabled.conf

PHP CGI default conf must be disabled

This sounds counter-intuitive but for the setup we are going for... the defaults are not suitable and conflict:

a2disconf php5* php6* php7*

# OPTIONAL: if you are paranoid you can take this a step further and rename the conf files

# # ls -1 /etc/apache2/conf-available/php*

/etc/apache2/conf-available/php7.4-cgi.conf.disabled

/etc/apache2/conf-available/php8.3-cgi.conf.disabled

💡 This is a really important point, I’ve wasted too much time over the years after dist-upgrades trying to trace what nuked the setup.

Packages

Install deb sury repo [
`info`⁠
]

💡 This next block assumes you require multiple concurrent PHP versions (one of the key features of deb sury). ⏩ Skip this block if you only need the latest PHP version from your distro release.

# The commands in this block have been tested on Debian 12 bookworm.

# YMMV[

1⁠

] on older releases or different distros.

# make sure everything is up-to-date

sudo apt update && up full-upgrade

# prerequisites, no-op if already installed

apt install curl software-properties-common ca-certificates lsb-release

# repo key

curl -sS https://packages.sury.org/php/apt.gpg | sudo tee /etc/apt/trusted.gpg.d/debsuryorg-archive.gpg 1>/dev/null

# deb sury source

sudo sh -c 'echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list'

apt update

Install packages and update alternatives

# 💡 customise the php-cgi verions to your needs

apt install apache2-suexec-custom libapache2-mod-fcgid php8.3-cgi php7.4-cgi

# if you have multiple versions of php installed - choose the default version

update-alternatives --config php

update-alternatives --config php-cgi

update-alternatives --config php-cgi-bin

# InfoSec best practice - purge mod-php in case it was automatically installed

aptitude -o Aptitude::Delete-Unused=1 -o Aptitude::Purge-Unused=1 purge libapache2-mod-php*

Versions reference

# Debian GNU/Linux 12 (bookworm) as of 2024-05-03

apache2 2.4.59-1~deb12u1

apache2-bin 2.4.59-1~deb12u1

apache2-data 2.4.59-1~deb12u1

apache2-suexec-custom 2.4.59-1~deb12u1

apache2-utils 2.4.59-1~deb12u1

libapache2-mod-fcgid 1:2.3.9-4

php8.3-cgi 8.3.6-1+0~20240424.32+debian12~1.gbp9a7ce5

php7.4-cgi 1:7.4.33-10+0~20240422.92+debian12~1.gbp6bc7ef

# if mysql compatible db required

mariadb-server 1:10.11.6-0+deb12u1

Config

apache2.service overrides

My systemd knowledge reference is

here⁠

in case you need a systemd refresher.

Add relevant overrides to the apache2.service. One can do this interactively with systemctl edit apache2.service OR with the following approach.

Secure default umask: This override sets a more secure default umask for apache2 so paths created by the apache2 user are not globally readable/browsable. This does not effect CGI applications, they must be handled separately [

see here⁠

# create 99-umask.conf drop in

printf '[Service]\nUMask=0027\n' > /etc/systemd/system/apache2.service.d/99-umask.conf

# make systemd aware of the change

systemctl daemon-reload

# verify the change

systemctl show apache2|grep -i mask

# if you wish to see all drop-ins on the system

systemd-delta --type=extended

# restart apache to pick up the change

systemctl restart apache2

/etc/apache2/envvars

export BASE_PATH=/var/www

export INCLUDE_PATH=/etc/apache2/sites-includes

# created this var back in Debian Lenny. Sqeeze added APACHE_LOG_DIR, could migrate at some point, for now its an alias

export LOG_PATH=$APACHE_LOG_DIR

# ip addresses

export IP_1=192.168.XXX.XXX

export IP_SECURE=$IP_1

export IP_HOSTED=$IP_1

/etc/apache2/suexec/www-data

Ensure the suEXEC doc root is set correctly/as desired

fcgid.conf

/etc/apache2/mods-available/fcgid.conf is customised and defaults disabled/changed. Here are some notable directives.

FcgidIPCDir /run/fcgid/sock # non-persistent storage for sockets

FcgidMaxRequestsPerProcess 0 # controlled by PHP_FCGI_MAX_REQUESTS in CGI wrappers

FcgidMinProcessesPerClass 0 # mitigate lingering CGI processes (once FcgidProcessLifeTime is reached?)

FcgidMaxProcessesPerClass 1 # safe default to mitigate DoS

FcgidFixPathinfo 1 # should match php.ini cgi.fix_pathinfo (for all vhosts on the server)

FcgidProcessLifeTime 600 # 10 minutes rather than the 1 hour default

Sample vhost user creation

# create the user

useradd --password "$pass" --comment "$hostname" --shell /usr/sbin/nologin --home-dir "/etc/ssh/chroot-usermaps/${user}/user" $user

# append the user to the relevant group

usermod --groups --append sftp-chroot $user

# $pass is a hashed+salted verison of the users password, in my vhost creation script I use pythons crypt.crypt method to hash+salt the password.

# we see here the symlink to the vhosts user path

# /etc/ssh/chroot-usermaps/${user}/user translates to /var/www/${hostname}/user.

# The per vhost symlink is created by my vhost creation script. ${user} symlinks to /var/www/${hostname}.

# The symlink command in my vhost creation looks like this

ln -s "/var/www/$hostname" "/var/www/000-vhost-usermaps/$user"

# I have a convenience symlink for when I want to use a /var/www path:

# /var/www/000-vhost-usermaps -> /etc/ssh/chroot-usermaps

Details on the sftp chroot setup for vhost users: [

here⁠

]

vhosts

A Linux user:group should be created per vhost

SuexecUserGroup should be defined

AddHandler is defined e.g. AddHandler fcgid-script .php

FcgidWrapper is defined FcgidWrapper ${BASE_PATH}/fastcgi/{hostname}/php8-small.sh .php

The doc root settings:

DocumentRoot "${BASE_PATH}/{hostname}/user/htdocs"

Options -Includes -IncludesNOEXEC -FollowSymLinks +MultiViews +ExecCGI +SymLinksIfOwnerMatch

AllowOverride FileInfo

...

</Directory>

vhost typical dir structure

{hostname} is replaced with the vhost primary FQDN/hostname name. {version} is replaced by the PHP version the vhost should use.

The .well-known symlink can be used to provision lets encrypt certificates when using HTTP-01 challenges [

info⁠

/var/www/{hostname}/

/var/www/{hostname}/php

/var/www/{hostname}/php/php.ini -> /etc/php/{version}/cgi/php.ini

/var/www/{hostname}/php/conf.d

/var/www/{hostname}/php/conf.d/10-mysqlnd.ini -> /etc/php/{version}/cgi/conf.d/10-mysqlnd.ini

/var/www/{hostname}/php/conf.d/20-ctype.ini -> /etc/php/{version}/cgi/conf.d/20-ctype.ini

/var/www/{hostname}/php/conf.d/20-iconv.ini -> /etc/php/{version}/cgi/conf.d/20-iconv.ini

/var/www/{hostname}/php/conf.d/20-mbstring.ini -> /etc/php/{version}/cgi/conf.d/20-mbstring.ini

/var/www/{hostname}/php/conf.d/20-mysqli.ini -> /etc/php/{version}/cgi/conf.d/20-mysqli.ini

/var/www/{hostname}/php/conf.d/overrides.ini

/var/www/{hostname}/user

/var/www/{hostname}/user/htdocs

/var/www/{hostname}/user/htdocs/index.htm

/var/www/{hostname}/user/htdocs/.well-known -> /var/www/000-default/user/htdocs/.well-known

/var/www/{hostname}/user/logs

/var/www/{hostname}/user/private

/var/www/{hostname}/user/private/php_session

/var/www/{hostname}/user/private/php_session/sess_tv7oqortt______i3kp7pmmk94

/var/www/{hostname}/user/private/php_session/sess_mi3k7ac2m______o9608mrgcuj

/var/www/{hostname}/user/private/php_upload

vhost file permissions

If we take a hostname example of blog.example.com and a linux user:group of blog-example-com

The document root dir “.” aka /var/www/blog.example.com/user/htdocs must be chown blog.example.com:www-data; chmod 0750 The non application/dynamic script files should be chown blog.example.com:www-data; chmod 0640 The application/dynamic (cgi) script files should be chown blog.example.com:blog.example.com; chmod 0640

🔐 The critical point is to ensure global/other read/write/exec/browse permissions are not permitted. The umask for apache and fcgid is documented elsewhere in this documentation [

here⁠

] and [

here⁠

drwxr-x--- 2 blog-example-com www-data 4096 May 1 19:04 .

-rw-r----- 1 blog-example-com www-data 13 May 1 19:04 index.htm

-rw-r----- 1 blog-example-com blog-example-com 13 Dec 19 2011 pi.php

Configuring PHP per vhost

Typically I use the following pattern:

# pick up the distribution default php.ini

/var/www/{hostname}/php/php.ini -> /etc/php/{version}/cgi/php.ini

# a file like this to override distribution defaults, i.e. specifics for the vhost

/var/www/{hostname}/php/conf.d/overrides.ini

# enabling PHP modules per vhost, example for mbstring

/var/www/{hostname}/php/conf.d/20-mbstring.ini -> /etc/php/{version}/cgi/conf.d/20-mbstring.ini

Sample vhost

Replace instances of {hostname} with the vhosts primary FQDN hostname. Replace instances of {vhost-user} and {vhost-group} with relevant Linux users for the vhost. apache env variables use the ${} syntax defined in /etc/apache2/envvars.

<IfModule mod_ssl.c> # require SSL

<IfModule suexec_module> # require suEXEC

<IfModule fcgid_module> # require fcgid

ServerName {hostname}:443

CustomLog "${LOG_PATH}/{hostname}/ssl_access.log" combined

ErrorLog "${LOG_PATH}/{hostname}/ssl_error.log"

#LogLevel trace6

DocumentRoot "${BASE_PATH}/{hostname}/user/htdocs"

Options -Includes -IncludesNOEXEC -FollowSymLinks +MultiViews +ExecCGI +SymLinksIfOwnerMatch

AllowOverride FileInfo

</Directory>

#############