Verified Commit 7fcc8f03 authored by Hephaestus Builder's avatar Hephaestus Builder

CHG: split docs/

CHG: update CloudFlare module
REM: docs
parent 9bdb3180
apnscp is comprised of several components. Any component under vendor/ is
copyright the respective author. All software that is modified under GPL
is available under
Code under different locations has different licenses:
- lib/modules
* Artistic 2.0
- lib/HTTP
- lib/Util/Process
- resources/playbooks
- build/bootstrap/apnscp-bootstrap-jdk
- build/bootstrap/apnscp-bootstrap-sdk
- apnscp root
* apnscp Proprietary License (refer to LICENSE)
All unlicensed content is assumed to be licensed under apnscp's proprietary license unless specified.
\ No newline at end of file
## Conventions
Multiple components of ApisCP are referenced with different markup.
* *[section]* => *option*: config.ini settings
* *Class\Subclass*: typically in changelogs, provides visibility of at least two levels corresponding to class
* *group*.*name*: configuration [scopes](docs/
* *service*,*value*: service configuration (e.g. siteinfo,email)
* *role*/*subrole*: [Bootstrapper]( tasks, correspond to `roles/` directory
* *module*: typically in changelogs (italicized outside), module name (`lib/modules`)
* *Word Word*: applications within ApisCP's front-end (`apps/`, see `lib/Template/` for name => directory mapping)
* *siteXX*: site identifier. See []( for information on determining this value
* *siteXX/path* (also *siteXX/fst/path*): refers to account root in /home/virtual/siteXX/fst/*path*
* *siteXX/shadow/path*: similar to account root, but refers to data-only layer in /home/virtual/siteXX/shadow/*path*
* *siteXX/info/path*: account metadata location, refers to /home/virtual/siteXX/info/*path*
# Debugging basics
ApisCP may emit debugging information when **debug mode** is enabled. You can enable debugging on a per-request basis, or globally using *cp.debug* [Scope](admin/ Backtraces (code pathways) can be enabled by setting *[core]* => *debug_backtrace_qualifier*. Backtraces give context around how an error occurred, making them invaluable for debugging.
`cpcmd scope:set cp.debug true` enables debugging mode globally.
`cpcmd scope:set cp.config core debug_backtrace_qualifier -1` enables backtraces for all reporting classes (fatal, error, warning. info, deprecated, and debug) as well as exceptions. Increasing verbosity levels inherit lower levels. **Backtraces are mandatory** when reporting issues.
The following table is only relevant when panel debugging is enabled.
| Level | Features |
| ----- | ------------------------------------------ |
| 0 | Disabled, no backtraces |
| 1 | Errors - error() |
| 2 | Warnings - warn() |
| 3 | Informative remarks - info() |
| 4 | Deprecated - deprecated_fn(), deprecated() |
| -1 | All above classes + debug() |
![UI Visibility](images/debugging-ex-visibility.png)
`env DEBUG=1 cpcmd common:whoami` executes the [whoami]( method in common module. This method simply returns the current username. A special module [test]( is available in debug mode to facilitate development. Only this request operates in debug mode ensuring appropriate isolation in a production environment.
# env DEBUG=1 cpcmd test:benchmark common_whoami
benchmark common_whoami
time: 0.01 sec (1000 rounds; 0.0051 ms each; 195429.32 per second)
# env DEBUG=0 cpcmd test:benchmark common_whoami
ERROR : test_benchmark: command does not exist
Reporter level: ERROR
ERROR: test_benchmark: command does not exist
When in debug mode, housekeeping and cron services are disabled as well as job runner. Housekeeping/cron tasks may not be invoked traditionally. Job runner invocation is covered under [Jobs](#jobs).
## Targeted frontend debug
`misc:debug-session(string $id, bool $mode = true)` enables debugging for a frontend session (DAV, UI, SOAP). Session identifier may be retrieved from a browser session by accessing the `` property. `session.debug` encodes whether debug mode is enabled.
# Enable debugging for active session aE2...JL9r in browser
env DEBUG=1 cpcmd misc:debug-session aE2uNGVkvpathoXqhofnsKDDXGNBJL9r true
## Log locations
ApisCP logs messages in a few places. Respective services use their preferred logging locations. This table summarizes common services and their log locations.
All locations are within /var/log unless noted. siteXX is shorthand for /home/virtual/siteXX/fst/. siteXX is the site ID identifier that can be resolved using `get_site_id`. "..." following siteXX is short-hand for /var/log thus *siteXX ... log* indicates /home/virtual/siteXX/fst/var/log/log. CP_ROOT is the panel home, typically either /usr/local/apnscp or /usr/local/apiscp. Words fully capitalized are symbolic.
| Service | Location | Remarks |
| ------------------------- | -------------------------- | ------------------------------------------------------------ |
| Apache | httpd/error_log | HTTP startup |
| Apache per-site | siteXX ... httpd/error_log | Per-site error logs, FPM connectivity |
| PHP-FPM | siteXX ... php-fpm/POOL | Per-site PHP errors, notices |
| Mail (**all** excl. auth) | maillog | SMTP prefixed "postfix". IMAP/POP3 "dovecot". Local delivery "maildrop". Excludes authentication. |
| Mail auth | secure | Rejections from invalid passwords via PAM |
| FTP | vsftpd.log | |
| MySQL | /var/lib/mysql/mysqld.log | |
| PostgreSQL | /var/lib/pgsql/X/data/log/ | Circular buffer by day-of-week. X is version major, 11, 10, etc. |
| SSH | secure | SSH login attempts, successes |
| crond | cron | Periodic services via Dev > Task Scheduler and /etc/cron.d |
| fail2ban (Rampart) | fail2ban.log | "Found" is log match. "Unban" automatic expiry. |
| ApisCP frontend | CP_ROOT/storage/logs/error_log | Same logging as Apache |
| ApisCP backend | CP_ROOT/storage/logs/start.log | Errors originating from backend |
| Passenger (launcher) | /.socket/passenger/logs | Launcher issues for Python, Ruby, and Node apps |
| Passenger (app) | APPROOT/log | Per-application messages. APPROOT is one directory down from document root. |
### Automated email reporting
ApisCP can be configured to forward a copy of unhandled errors (PHP notices/errors and exceptions) to an email address. Set [core] => bug_report in config.ini. This should be used by developers only, as it generates false positives that are encountered during typical operation.
cpcmd scope:set cp.config core bug_report
Given the volume generated, plus-address notation or a separate email address is recommended to facilitate mail filtering by your SMTP provider.
## Backend
You can start the backend broker from the command-line in the foreground. It handles elevation requests from the frontend via the [query()]( function in the API.
To enable debugging, run:
systemctl stop apiscp
cd /usr/local/apnscp/bin
env DEBUG=1 ./apnscpd -f restart
When debugging is active, the following tasks are disabled: cron, housekeeping, jobs.
## PHP-FPM status
FPM pools are grouped by site ID identifier and name. By default, one pool is created named after the primary domain. `php:pools()` lists active pools for a site.
`php:pool-status(string $pool = '')` provides the internal PHP-FPM pool status as reported by systemd's notify feature. These values are real-time metrics as seen by the pool manager.
`php:pool-info(string $pool = '')` reports service information from systemd. This command is equivalent to *systemctl show POOLNAME*. *StatusText* is the plaintext value of php:pool-status.
## Jobs
Laravel Horizon is used for jobs unless `has_low_memory` is enabled (via Bootstrapping or *[cron]* => *low_memory* is set in config.ini). Horizon can be manually launched using:
`./artisan horizon`
`misc:get-job-queue()` reports the pending work queue. Pending jobs may be processed using `./artisan queue:work`. An optional flag, --once, processes these jobs singularly.
# Disable job runner, housekeeping, and cron to prevent jobs from starting
cpcmd scope:set cp.debug true
systemctl restart apiscp
# Validate the queue is empty
cpcmd misc:get-job-queue
# Jobify a command
cpcmd misc:jobify 'common_whoami'
# Validate the queue has 1 job
cpcmd misc:get-job-queue
# Run first job in queue, enable verbose output
./artisan queue:work --once -vvv
# Validate queue is now empty
cpcmd misc:get-job-queue
*Jobs are unavailable when the panel is in debug mode unless Horizon has been manually started.*
## Web App installation
Web Apps engaged through the UI are dispatched to a job runner, and may also be installed using API commands. Each app maps to a module named after itself, and follows a common interface:
`NAME:install(string $hostname, string $path = '', array $options = [])`
| Web App | Module |
| --------- | --------- |
| Discourse | discourse |
| Drupal | drupal |
| Ghost | ghost |
| Joomla! | joomla |
| Laravel | laravel |
| Magento | magento |
| NextCloud | nextcloud |
| WordPress | wordpress |
Applications support both generalized options and specific options. The following are common options found in Web > Web Apps:
| Name | Type | Remarks |
| ---------- | ------ | ------------------------- |
| version | string | Version number |
| ssl | bool | Enable SSL |
| user | string | Optional username of |
| autoupdate | bool | Enable automatic updates |
*Jobs are unavailable when the panel is in debug mode unless Horizon has been manually started.*
## Command listing
`misc:list-commands(string $filter = '')` is a role-aware helper that displays available commands. Used in conjunction with [cpcmd](admin/, it provides a convenient interface to filter available commands.
# Show commands available to Appliance Administrator ("admin" username)
cpcmd misc:list-commands
# Show commands available to site1 Site Administrator
cpcmd -d site1 misc:list-commands
# Show commands available to site1 in the "ghost" module
cpcmd -d site1 misc:list-commands 'ghost:*'
# "l" is an alias and equivalent to the above command
cpcmd -d site1 misc:l 'ghost:*'
### Introspection
`misc:command-info(string $filter = '')` provides verbose information about the command, including its method signature and documentation. This can be used to explain what parameters an API command anticipates. Method usage is similar to `list-commands`:
# Show signature for ghost:install as Site Administrator
cpcmd -d site1 misc:command-info ghost:install
# Show command signature for all commands in "admin" module
cpcmd misc:command-info 'admin:*'
# "i" is an alias and equivalent ot the above command
cpcmd misc:i 'admin:*'
## User preferences
Preferences are stored in siteXX/info/USER. `common:load-preferences()` is a convenient interface to show these preferences.
`common:get-user-preferences(string $user)` allows for a Site Administrator access to a user's preferences.
`YAML_INLINE` is an environment variable that controls array folding depth. Increasing folding depth improves readability. The default value is *2*.
# Show Appliance Administrator's preferences
cpcmd common:load-preferences
# Show preferences for site1's Site Administrator
cpcmd -d site1 common:load-preferences
# Show preferences for user "foobar" on site1
cpcmd -d site1 -u foobar common:load-preferences
# The following command is equivalent
cpcmd -d site1 common:get-user-preferences foobar
# Use YAML_INLINE=n to expand collapsed fields
env YAML_INLINE=4 cpcmd -d site1 common:load-preferences
`common:purge-preferences()` will purge the active user's preferences. This may be useful to gauge API interaction or to reset a role to a clean state. `purge-preferences` may only be invoked in debug mode.
# Get current user preferences
cpcmd common:load-preferences
# Purge all preferences
env DEBUG=1 cpcmd common:purge-preferences
# Preferences for active role is now empty
cpcmd common:load-preferences
## Connecting to Redis
Redis manages caching and job queues over a UNIX domain socket. Database 1 is assigned to ApisCP, 2 to jobs, and 3 to rspamd (if utilized), with a decreasing priority assigned to each. Do not issue the `FLUSHALL` command as this will purge rspamd logical replication from PostgreSQL.
redis-cli -s /usr/local/apnscp/storage/run/redis.sock
# Show memory usage
info memory
# Show stored keys
keys *
## API bypasses
You may bypass API permissions by using a surrogate module. This allows for rapid prototyping of individual API methods which may otherwise be restricted. Surrogates are covered in detail in [](
<?php declare(strict_types=1);
class Dns_Module_Surrogate extends Dns_Module {
public function __construct() {
// ensure we always win permissions
$this->exportedFunctions = ['*' => PRIVILEGE_ALL];
public function t() {
return $this->_cron();
You can now interact with the _cron method from the command-line instead of being restricted by API accessibility rules.
env DEBUG=1 cpcmd dns:t
Be careful with this approach. All cross-module calls inherit the current role. This means that you may try calling an API method intended for PRIVILEGE_SITE, as PRIVILEGE_ADMIN. Unless permissions are overwritten with a surrogate, cross-module violations are still blocked.
\ No newline at end of file
# apnscp Firewall
apnscp utilizes [firewalld]( for its firewall. Rampart is a module that serves as a wrapper for [fail2ban](, a brute-force deterrent that blocks threats through firewalld. These two components act in tandem to keep your server secure while exercising some intelligence. Rampart is for ephemeral blocks that automatically expire after a fixed duration (see [network/setup-firewall]( whereas a separate firewalld permanent whitelist/blacklist is provided.
During installation, apnscp will detect the connected IP address and whitelist it to avoid triggering a block by fail2ban, for example if you forget your password multiple times. If your IP address changes or you setup apnscp from behind a proxy, then you can easily update the whitelist with `cpcmd`
cpcmd scope:set rampart.whitelist
To view active whitelists use [scope:get](
cpcmd scope:get rampart.whitelist
Whitelists may be IP address ( or CIDR hosts ( rampart.whitelist is an append-only operations. Edit /etc/fail2ban/jail.conf by hand to remove old IP addresses.
## Whitelisting access
apnscp restricts access to all ports except for well-known services (HTTP, FTP, mail, SSH) and optional services (CP, user daemons). A second whitelist, which allows access to blocked ports as well as overrides Rampart can be set using `cpcmd rampart:whitelist`:
cpcmd rampart:whitelist
These entries are permanent.
## Blacklisting
Likewise a blacklist exists to block addresses that are not blocked by Rampart's adaptive firewall.
cpcmd rampart:blacklist
Blacklists are lower priority than whitelist and Rampart blocks.
## Unbanning IP addresses
All IP addresses automatically unban from Rampart after a fixed duration. To manually unban an address from Rampart use cpcmd:
# Ban in recidive, which is a long-term ban > 1 week
cpcmd rampart:ban recidive
# Validate which jails is present in
cpcmd rampart:is-banned
# Unban from all jails
cpcmd rampart:unban
Permanent blacklist and whitelist entries can be removed with firewall-cmd
# Add to the permanent whitelist
cpcmd rampart:whitelist
# Show all whitelist entries
ipset list whitelist
# Remove from whitelist
cpcmd rampart:whitelist remove
{:#Account metadata}
: *Metadata* about an account that resides in *siteXX/info*. Examples of metadata include the primary domain, admin username, bandwidth allotment
: Amount of transfer (inbound and outbound) that an account registers over a period. Bandwidth is in bytes unless a *unit* prefix is specified.
{:#Billing invoice}
: an identifier that relates an account or set of accounts to a common billing entity. Billing invoices are stored in billing,invoice account metadata.
: An indivisible unit comprised of 8 bits.
: Direct server access accessed over terminal. SSH and embedded terminal (KVM) are used to access this mode.
: data about data. A common example of this is account metadata that lives in siteXX/info for each account.
: An expression that refers to /home/virtual/siteXX where siteXX is the *site identifier* of an account.
{:#Site identifier}
: internal marker for sites, which is defined on account creation. Site identifier index begins at 1 and the next largest value is assigned. Site identifiers do not change unless the account is deleted, then recreated. Examples: site12, site1, site999999999. Site identifiers may be determined by domain using `get_site_id` from the command-line.
{:#Site ID identifier}
: numeric component of the *Site identifier*.
: A measurement quantity. "Bytes" are the most common unit referenced.
\ No newline at end of file
# Installation
## Requirements
* 2 GB RAM
* 20 GB storage
* 1 CPU
* RHEL/CentOS 7.4+
* Best with Linode, DigitalOcean, Vultr, AWS, or CloudFlare for DNS
* Containers are not supported (Virtuozzo, OpenVZ)
## Bootstrapping
Install the bootstrap utility. This can be downloaded from the [GitHub]( repository.
curl | bash
This arms apnscp with a 15 day license (60 day for development releases). If you have purchased a license via [](, then the license may be provided at install by providing the token:
curl | bash -s - <api token>
Before the bootstrap utility kicks off [stage 2](, you are given an opportunity to pause to edit variables. This is optional, but will allow you to immediately configure SSL. These values can be changed later with `scope:set` in the panel.
![Bootstrap utility pause](images/edit-option.png)
## Server provision
[Bootstrapper]( is an idempotent tool to continuously update and correct your server. If things drift, Bootstrapper is designed to provide the **minimal set** of enforcing changes to make your server work. You will always have free rein of your server as long as it doesn't impede upon the responsibility of apnscp.
Bootstrapping takes between 60 - 120 minutes depending upon provider capacity. Generally lower figures indicate a less oversold provider, but times can change as clients are housed on the server. Once installed, be sure to log out of the server and log back in or load a new shell with the correct apnscp environment,
exec $SHELL -i
### Recommended configuration settings
Bootstrapper can run without any changes to `/root/apnscp-vars.yml`. The following changes are recommended to make setup seamless. These settings are configured when the bootstrap utility pauses:
- **apnscp_admin_email**: (email address) used to set the admin contact. Notified when apnscp is installed (FQDN required) as well as monthly maintenance notifications. This email address is also used as your Let's Encrypt admin contact.
- **ssl_hostnames**: (list or string) hostnames that resolve to this server that should be considered for Let's Encrypt SSL issuance.
- Examples:
- ['','','']
#### Optional settings
- **has_low_memory**: (true/false) disables auxiliary services for 2 GB instances. See [low-memory mode](#low-memory-mode) below.
- **user_daemons**: (true/false) opens up ports 40000-49999/tcp + udp on the server for accounts that want to run a service. If you're running strictly PHP/Node/Python/Ruby services, turn this off for added security.
- **mail_enabled**: (true/false) if using GMail or a third-party email provider disables IMAP/POP3 + ESMTPA. Mail can still originate from the server (PHP [mail()](, but blocks ingress.
- **passenger_enabled**: (true/false) disable building Passenger + accompanying Ruby/Python interpreters if running a purely PHP mix. Node/npm/yarn is still available, but can't serve websites.
- **mysqld_per_account_innodb**: (true/false) places tables + data in an aggregate InnoDB pool for higher performance or per account for resource enforcement. An account over quota can cause a cyclic crash in MySQL/MariaDB 5.0+ on recovery. **You have been warned**. Ensure [Argos]( is setup if enabled.
- **data_center_mode**: (true/false) ensure all resources that apnscp can account for are accounted. Also enables the pernicious bastard `mysqld_per_account_innodb`!
### Setting FQDN for SSL/Email
All servers should have a fully-qualified domain name ([FQDN]( Failure to have one will cause email to fail, including the installation notice. Moreover, Let's Encrypt will fail issuance. A FQDN must at least contain 1 dot:
- ✅
- ✅
- ✅
- ❌ centos-s-1vcpu-2gb-nyc1-01 (no period/dot)
Set your hostname with,
hostnamectl set-hostname MYHOSTNAME
Where *MYHOSTNAME* is your hostname for the machine. For consistency reasons, it is required that this hostname resolves to the IP address of the machine and vice-versa with [FCrDNS]( Check with your DNS provider to establish this relationship.
Once apnscp is setup it can be reconfigured anytime with,
scope:set net.hostname MYHOSTNAME
SSL will automatically reissue as well as impacted services restart.
### Low-memory mode
apnscp is designed to work on 2 GB+ machines, but can work on 1 GB machines with minor finagling. Enabling `has_low_memory` scrubs non-essential services. It converts the job daemon to a single worker; disables Passenger, including support for Python, Ruby, Node, and Meteor applications; and removes vscanner, which is a combination of mod_security + ClamAV to scrub uploads. This frees up ~700 MB of memory.
If no mail services are required, setting `mail_enabled` to *false* also disables Dovecot and haproxy freeing up an additional 60 MB. Setting `rspamd_enabled` to *false* (policy milter, outbound spam filtering, DKIM/ARC signing) will free a further 125 MB if rspamd is used for mail filtering (see `spamfilter` setting).
### Headless mode
apnscp can run in headless mode, that is to say without a front-end UI. This can further save on memory requirements and keep your site secure.
Set `panel_headless` to *true* to activate headless mode. In headless mode, you are limited to CLI helpers - `cpcmd`, `AddDomain`, `EditDomain`, `DeleteDomain`. Fear not though! Anything that can be done through the panel can be done from CLI as the API is 100% reflected.
This mode can be quickly toggled after setup using a [configuration scope](admin/ `cpcmd config:set apnscp.headless true`.
### Provisioning failures
In the event of failure, Bootstrapper can be easily restarted from the command-line,
cd /usr/local/apnscp/resources/playbooks
ansible-playbook bootstrap.yml
#### Network failures
Network connectivity is a common failure that can be caused by transient errors either in DNS resolution, which is unreliable by UDP protocol design, or network congestion. Network tasks use a built-in retry/wait algorithm to reduce the risk of failure - up to 3 retries with a 5 second wait in between attempts. Even this pragmatic approach to handling network interruptions can fail.
> TASK [software/rbenv : Add GEM_HOME pathing support] ***************************
2019-03-05 12:42:19,129 p=8833 u=root | FAILED - RETRYING: Add GEM_HOME pathing support (3 retries left).
2019-03-05 12:46:38,839 p=8833 u=root | FAILED - RETRYING: Add GEM_HOME pathing support (2 retries left).
2019-03-05 12:50:58,550 p=8833 u=root | FAILED - RETRYING: Add GEM_HOME pathing support (1 retries left).
2019-03-05 12:55:18,270 p=8833 u=root | fatal: [localhost]: FAILED! => ...
The above fragment failed to download a repository off GitHub indicating possible DNS issues with the resolver configured on the machine. Replace the configured resolver with a reliable public DNS resolver (Google, Level3, Cloudflare) by editing `/etc/resolv.conf`. Remove all instances of `nameserver` and change the DNS timeout which defaults at 5 seconds. When using multiple nameservers, `rotate` in the `options` directive will round-robin resolvers to distribute lookups across all nameservers.
As of v3.0.29, apnscp automatically replaces the machine's nameservers with CloudFlare, which consistently performs in the [top tier](!dns-resolvers) of reliability and speed. This feature may be disabled with `use_robust_dns`. Alternate nameservers may be defined via `dns_robust_nameservers `, a list (see *apnscp-internals.yml*).
##### List of public DNS servers
| Provider | Server |
| ---------- | ------------- |
| Google | |
| | |
| Cloudflare | |
| | |
| Level3 | |
| | |
| Quad9 | |
| Dyn | |
| | |
*Original /etc/resolv.conf*
# Generated by NetworkManager
*Revised /etc/resolv.conf*
# Generated by NetworkManager
options timeout:30 rotate
# After Bootstrap
Domains that have a properly qualified FQDN will receive notification once apnscp is installed. If not, the admin username/password/contact can be reconfigured at anytime using apnscp's API helper.
cpcmd auth_change_username NEWUSER
cpcmd auth_change_password NEWPASSWORD
cpcmd common_set_email NEW@EMAIL.COM
Setting all 3 will allow you to login to your new panel at http://IPADDRESS:2082/. If SSL has been setup, or you can trust a bespoke certificate, then use https://IPADDRESS:2083/. When logging in as admin, leave the domain field blank.
# Adding your first domain
## Within apnscp
After logging into the panel as admin, visit **Nexus**. Services can be reconfigured within Nexus or from the command-line.
## From command-line
`AddDomain` creates a site from command-line. Multiple parameters can be provided to alter the services assigned to an account. Nexus within the Administrative panel is a frontend for this utility. `admin_add_site` is the backend API [call]( for this command-line utility.
### Basic usage
AddDomain -c siteinfo, -c siteinfo,admin_user=myadmin
> Creates a new domain named ** with an administrative user *myadmin*. The email address defaults to []( and password is randomly generated.
## Editing domains
### From apnscp
Domains may be edited by clicking the _SELECT_ button in Nexus.
### From command-line
EditDomain is a helper to change account state without removing it. You can toggle services and make changes in-place in a non-destructive manner. More advanced usage is available in the blog post, [Working with CLI helpers](
**Rename domain**
A simple, common situation is to alter the primary domain of an account. Simply changing the domain attribute under the siteinfo service will accomplish this.
EditDomain -c siteinfo,
**Changing password**
Changing the password is another common operation:
EditDomain -c auth,tpasswd=newpasswd site12
## Logging into services
apnscp uses *username*@*domain* notation to log into all services. This allows for multiple domains to share the same username without conflict. The only restriction is that the primary account username *must be unique*.
Unless the domain is explicitly required, such as when logging into the control panel or accessing MySQL remotely use `@` or `#` to join the username + domain. For example, when logging into SSH as user `foo` on ``, all are acceptable variations of ssh:
ssh -l
ssh -l