Module management for a chrooted Ngnix

Context

I was roaming around when I found ngx-fancyindex on Github and thought it would be nice on my Nginx. As soon as it said, I remembered there is no third-party module installed, which means FancyIndex has to be installed, then enabled.

Unfortunately, the only way to add a new module is to re-compile the whole source and (partially) rebuild the chroot environment.

Tarball download

The first step is to download the lastest or whatever version you need. Check out Nginx official website to fetch the tarball download link.

SRC=/opt/nginx_workplace
cd $SRC
wget http://nginx.org/download/nginx-1.7.11.tar.gz
tar xzf nginx-1.7.11.tar.gz

Download the latest version of FancyIndex as well.

git clone https://github.com/aperezdc/ngx-fancyindex.git ngx-fancyindex

Since recompilation is settled, let's costumize Nginx and add more optional functionality.

Install requirements

pacman -S gperftools geoip

Compile and install

Before anything else, let's see with which flag the current Nginx has been compiled with.

CHRT_PATH=/my/chroot/path
$CHRT_PATH/usr/bin/nginx -V

nginx version: nginx/1.4.2
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-imap --with-imap_ssl_module --with-ipv6 --with-pcre-jit --with-file-aio --with-http_dav_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_realip_module --with-http_spdy_module --with-http_ssl_module --with-http_stub_status_module --add-module=/usr/lib/passenger/ext/nginx

Nota: You can see that I'm such a lazy-ass, I have not upgraded my Nginx version for awhile !

Below the list of modules and its brief description, based on modules currently activated and additionnal as a costumization. (see here and here for more information)

Google performance tools

gperftools need to access to '/proc/cpuinfo', so if you have not '/proc' mounted on your jail. Please do it as below

mkdir -p $CHRT_PATH/proc
mount -o bind /proc $CHRT_PATH/proc

./configure --prefix=/etc/nginx \
            --conf-path=/etc/nginx/nginx.conf \
            --sbin-path=/usr/bin/nginx \
            --pid-path=/var/run/nginx.pid \
            --lock-path=/var/lock/nginx.lock \
            --user=http \
            --group=http \
            --http-log-path=/var/log/nginx/access.log \
            --error-log-path=/var/log/nginx/error.log \
            --with-google_perftools_module \
            --with-http_ssl_module \
            --with-http_spdy_module \
            --with-http_realip_module \
            --with-http_geoip_module \
            --with-http_addition_module \
            --with-http_stub_status_module \
            --with-http_xslt_module \
            --with-http_image_filter_module \
            --with-http_sub_module \
            --with-http_dav_module \
            --with-http_flv_module \
            --with-http_mp4_module \
            --with-http_gunzip_module \
            --with-http_gzip_static_module \
            --with-ipv6 \
            --with-pcre-jit \
            --with-debug \
            --http-client-body-temp-path=/var/lib/nginx/client-body \
            --http-proxy-temp-path=/var/lib/nginx/proxy \
            --http-fastcgi-temp-path=/var/lib/nginx/fastcgi\
            --http-scgi-temp-path=/var/lib/nginx/scgi \
            --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
            --with-imap \
            --with-imap_ssl_module \
            --add-module=../ngx-fancyindex

Configuration summary
  + using system PCRE library
  + using system OpenSSL library
  + md5: using OpenSSL library
  + sha1: using OpenSSL library
  + using system zlib library

  nginx path prefix: "/etc/nginx"
  nginx binary file: "/usr/bin/nginx"
  nginx configuration prefix: "/etc/nginx"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/var/run/nginx.pid"
  nginx error log file: "/var/log/nginx/error.log"
  nginx http access log file: "/var/log/nginx/access.log"
  nginx http client request body temporary files: "/var/lib/nginx/client-body"
  nginx http proxy temporary files: "/var/lib/nginx/proxy"
  nginx http fastcgi temporary files: "/var/lib/nginx/fastcgi"
  nginx http uwsgi temporary files: "/var/lib/nginx/uwsgi"
  nginx http scgi temporary files: "/var/lib/nginx/scgi"

Let install it !

make
make install

Here is the result of the make install

make -f objs/Makefile install
make[1]: Entering directory '/opt/nginx/nginx-1.7.11'
test -d '/etc/nginx' || mkdir -p '/etc/nginx'
test -d '/usr/bin'              || mkdir -p '/usr/bin'
test ! -f '/usr/bin/nginx'              || mv '/usr/bin/nginx'                  '/usr/bin/nginx.old'
cp objs/nginx '/usr/bin/nginx'
test -d '/etc/nginx'            || mkdir -p '/etc/nginx'
cp conf/koi-win '/etc/nginx'
cp conf/koi-utf '/etc/nginx'
cp conf/win-utf '/etc/nginx'
test -f '/etc/nginx/mime.types'                 || cp conf/mime.types '/etc/nginx'
cp conf/mime.types '/etc/nginx/mime.types.default'
test -f '/etc/nginx/fastcgi_params'             || cp conf/fastcgi_params '/etc/nginx'
cp conf/fastcgi_params          '/etc/nginx/fastcgi_params.default'
test -f '/etc/nginx/fastcgi.conf'               || cp conf/fastcgi.conf '/etc/nginx'
cp conf/fastcgi.conf '/etc/nginx/fastcgi.conf.default'
test -f '/etc/nginx/uwsgi_params'               || cp conf/uwsgi_params '/etc/nginx'
cp conf/uwsgi_params            '/etc/nginx/uwsgi_params.default'
test -f '/etc/nginx/scgi_params'                || cp conf/scgi_params '/etc/nginx'
cp conf/scgi_params             '/etc/nginx/scgi_params.default'
test -f '/etc/nginx/nginx.conf'                 || cp conf/nginx.conf '/etc/nginx/nginx.conf'
cp conf/nginx.conf '/etc/nginx/nginx.conf.default'
test -d '/var/run'              || mkdir -p '/var/run'
test -d '/var/log/nginx' ||             mkdir -p '/var/log/nginx'
test -d '/etc/nginx/html'               || cp -R html '/etc/nginx'
test -d '/var/log/nginx' ||             mkdir -p '/var/log/nginx'
make[1]: Leaving directory '/opt/nginx/nginx-1.7.11'

Re-populate Chroot

Adding new modules means add new dependencies and its libraries, copy the additionnal libraries to $CHRT_PATH/usr/lib

ldd /usr/bin/nginx | grep /usr/lib | sed -sre 's/.+(\/usr\/lib\/\S+).+/\1/g'

/usr/lib/libpthread.so.0
/usr/lib/libcrypt.so.1
/usr/lib/libpcre.so.1
/usr/lib/libssl.so.1.0.0
/usr/lib/libcrypto.so.1.0.0
/usr/lib/libdl.so.2
/usr/lib/libz.so.1
/usr/lib/libxml2.so.2
/usr/lib/libxslt.so.1
/usr/lib/libexslt.so.0
/usr/lib/libgd.so.3
/usr/lib/libGeoIP.so.1
/usr/lib/libprofiler.so.0
/usr/lib/libc.so.6
/usr/lib/liblzma.so.5
/usr/lib/libm.so.6
/usr/lib/libgcrypt.so.20
/usr/lib/libgpg-error.so.0
/usr/lib/libjpeg.so.8
/usr/lib/libpng16.so.16
/usr/lib/libfreetype.so.6
/usr/lib/libfontconfig.so.1
/usr/lib/libXpm.so.4
/usr/lib/libX11.so.6
/usr/lib/libvpx.so.1
/usr/lib/libtiff.so.5
/usr/lib/libstdc++.so.6
/usr/lib/libgcc_s.so.1
/usr/lib/libbz2.so.1.0
/usr/lib/libharfbuzz.so.0
/usr/lib/libexpat.so.1
/usr/lib/libxcb.so.1
/usr/lib/libglib-2.0.so.0
/usr/lib/libgraphite2.so.3
/usr/lib/libXau.so.6
/usr/lib/libXdmcp.so.6

I think it's not a good idea to copy those librairies manually. I forgot to update libssl.so and libcrypto.so which leads me to This error.

I recommand you to run the following command to repopulate the /usr/lib/ of your jail.

cp $(ldd /usr/bin/nginx | grep /usr/lib | sed -sre 's/.+(\/usr\/lib\/\S+).+/\1/g') $CHRT_PATH/usr/lib

Production environment

If you are running on your production environment, make sur that all user are informed that the HTTP Server will be down for a moment.

Anyway, backup you old version of Nginx

cp -a $CHRT_PATH/usr/bin/nginx $CHRT_PATH/bin/nginx.old

You will not be able to hot-migrate Nginx whithout shuting down the running instance then copy your binary from /usr/bin as defined in --sbin-path flag to your jail.

systemctl stop nginx
cp /usr/bin/nginx $CHRT_PATH/bin/nginx
setcap cap_net_bind_service=+ep $CHRT_PATH/usr/bin/nginx
systemctl start nginx

It should work as rock !

$CHRT_PATH/usr/bin/nginx -V

nginx version: nginx/1.7.11
built by gcc 4.9.2 20150304 (prerelease) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --with-google_perftools_module --with-http_ssl_module --with-http_spdy_module --with-http_realip_module --with-http_geoip_module --with-http_addition_module --with-http_stub_status_module --with-http_xslt_module --with-http_image_filter_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-ipv6 --with-pcre-jit --with-debug --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-imap --with-imap_ssl_module --add-module=../ngx-fancyindex

Troubleshoting

Libssl/libcrypto

When I first started Nginx, I had a weird error related to SSL library.

chroot[16686]: /usr/bin/nginx: symbol lookup error: /usr/lib/libssl.so.1.0.0: undefined symbol: EVP_aes_128_cbc_hmac_sha256

After some research I found this post on Stackoverflow. So, I migrate libcrypto.so ... and it works !

Bind()/Permission denied

Got another error:

chroot[16750]: nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

Even with strace I found nothing more

socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
setsockopt(6, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
ioctl(6, FIONBIO, [1])                  = 0
bind(6, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
write(5, "2015/03/26 01:35:52 [emerg] 1839"..., 89) = 89
write(2, "nginx: [emerg] bind() to 0.0.0.0"..., 67nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)
) = 67
close(6)                                = 0
close(4)                                = 0
close(5)                                = 0
brk(0x755000)                           = 0x755000
exit_group(1)                           = ?
+++ exited with 1 +++

It's clearly a lower ports bind 'permission' issue, since Nginx run under http user. Nevertheless, the question is why the old version works fine while the freshly compiled one stuck at the binding step ?

Honnestly, I did not found the answer. However, This wiki gives me what I need.

I found the answer, it was right in front of me ...

setcap cap_net_bind_service=+ep $CHRT_PATH/usr/bin/nginx

Now let's Rock'n'Roll !

Update version 1.11.2

Let's go for another update, even if it's a little late as usual #FML.

Download the current last version of nginx from official repo

wget https://nginx.org/download/nginx-1.11.2.tar.gz
wget https://nginx.org/download/nginx-1.11.2.tar.gz.asc

# check the signature
gpg --verify nginx-1.11.2.tar.gz.asc nginx-1.11.2.tar.gz
gpg: Signature made Tue 05 Jul 2016 06:00:05 PM CEST using RSA key ID 520A9993A1C052F8
gpg: Good signature from "Maxim Dounin <mdounin@mdounin.ru>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: B0F4 2533 73F8 F6F5 10D4  2178 520A 9993 A1C0 52F8

If you get the following error gpg: Can't check signature: No public key please run this

gpg --keyserver keys.gnupg.net --recv-keys 520A9993A1C052F8

The ID 520A9993A1C052F8 is the id given by the gpg --verify command above.

Extract the source and starting working on it

tar xzf nginx-1.11.2.tar.gz
cd nginx-1.11.2

here is the configure option I used to build my own nginx binary

./configure --prefix=/etc/nginx \
            --conf-path=/etc/nginx/nginx.conf \
            --sbin-path=/usr/bin/nginx \
            --pid-path=/var/run/nginx.pid \
            --lock-path=/var/lock/nginx.lock \
            --user=http \
            --group=http \
            --http-log-path=/var/log/nginx/access.log \
            --error-log-path=/var/log/nginx/error.log \
            --with-http_v2_module \
            --with-google_perftools_module \
            --with-http_ssl_module \
            --with-http_realip_module \
            --with-http_geoip_module \
            --with-http_addition_module \
            --with-http_stub_status_module \
            --with-http_xslt_module \
            --with-http_image_filter_module \
            --with-http_sub_module \
            --with-http_dav_module \
            --with-http_flv_module \
            --with-http_mp4_module \
            --with-http_gunzip_module \
            --with-http_gzip_static_module \
            --with-ipv6 \
            --with-debug \
            --with-pcre-jit \
            --http-client-body-temp-path=/var/lib/nginx/client-body \
            --http-proxy-temp-path=/var/lib/nginx/proxy \
            --http-fastcgi-temp-path=/var/lib/nginx/fastcgi\
            --http-scgi-temp-path=/var/lib/nginx/scgi \
            --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
            --add-module=../modules/ngx-fancyindex

--with-http_spdy_module is no longuer exist, it has been replace by --with-http_v2_module. Actually it's the same tools and project (Google's project) the HTTP next-gen.

I also removed --with-imap and --with-imap_ssl_module: First of all there are deprecated now I should use --with-mail and --with-mail_ssl_module. Then, I have no attend to use it for the moment so no need to keep it in.

Configuration summary
  + using system PCRE library
  + using system OpenSSL library
  + using system zlib library

  nginx path prefix: "/etc/nginx"
  nginx binary file: "/usr/bin/nginx"
  nginx modules path: "/etc/nginx/modules"
  nginx configuration prefix: "/etc/nginx"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/var/run/nginx.pid"
  nginx error log file: "/var/log/nginx/error.log"
  nginx http access log file: "/var/log/nginx/access.log"
  nginx http client request body temporary files: "/var/lib/nginx/client-body"
  nginx http proxy temporary files: "/var/lib/nginx/proxy"
  nginx http fastcgi temporary files: "/var/lib/nginx/fastcgi"
  nginx http uwsgi temporary files: "/var/lib/nginx/uwsgi"
  nginx http scgi temporary files: "/var/lib/nginx/scgi"

let's build and install this

cp -a /usr/bin/nginx /usr/bin/nginx.old
make && make install
systemctl stop nginx
cp /usr/bin/nginx <chroot_path>/usr/bin/nginx
setcap cap_net_bind_service=+ep <chroot_path>/usr/bin/nginx
systemctl start nginx

we're done here for the moment !

AddModuleToNginx (last edited 2016-07-24 19:41:45 by kipsnak)