Upgrading SSL (TLS) on Kerio Workspace 2.1.4

Update 2022-03-10: There were a few glitches that needed ironing out: disabledAlgorithms property to allow more flexible TLS on SMTP out; and some extra setup for the Kerio Client, which I briefly touch upon at the end.

Problem statement

A friend of mine recently contacted me to ask whether I’d be willing to help out to modernize an installation of Kerio Workspace 2.1.4.

This software – as far as I can tell – was last released in 2014. As such, it doesn’t support TLSv1.2 (or better) out of the box.

And thus, the modern browsers start complaining quite a bit. I mean, this isn’t a pretty sight:

SSL Server Test result, courtesy of Qualys; ignore the self-signed cert for the purpose of this post.

This article shows what can be done about it. With minimal disruption, that is.

Basic recon

To avoid messing up the live system too much, I’ve installed 32bit Debian 11.2, to try and install the *.deb from the vendor.

Why 32bit? Because the install package targets that architecture and 64bit debian was very fussy about multiarch support for it. Uncooperative, even.

It wasn’t exactly a surprise that libpng12-0 dependency is no longer available. A little bit of fumbling around Debian archives and a three-liner¹ later: et voilà! We have contact:

$ apt install libpng12-0
$ dpkg -i ./kerio-workspace-2.1.4-2341-linux.deb
$ apt -f install

Kerio Workspace installs into /opt/kerio/workspace. Judging by the presence of jre and tomcat directories, it’s clearly a Tomcat webapp. ;)

Which version… you ask?

$ cd /opt/kerio/workspace
$ ./jre/bin/java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) Client VM (build 20.6-b01, mixed mode)

$ ./jre/bin/java -cp ./tomcat/lib/catalina.jar \
  org.apache.catalina.util.ServerInfo
Server version: Apache Tomcat/7.0.26
Server built:   Feb 17 2012 02:11:27
Server number:  7.0.26.0
OS Name:        Linux
OS Version:     5.10.0-11-686
Architecture:   i386
JVM Version:    1.6.0_31-b04
JVM Vendor:     Sun Microsystems Inc.

So there’s that.

Hard choices

There are several options how to proceed²:

1. Bump the protocol version in tomcat/conf/server.xml
2. Upgrade the Tomcat to newer release in the 7.X tree
3. Upgrade the JVM’s SSL support
4. Put a brand spanking new HTTP(s) proxy in front of the Kerio³

The first option did not work at all. Alas, the installed Tomcat and/or JRE only supports TLSv1 and nothing more.

I explored the second option for a while (bumping JRE to 1.8.0_321, Tomcat to 7.0.109), but was underwhelmed by the subtle incompatibilities in the logs.

The proxying would work, but seemed like a lot of work, given that this was running on an old openSUSE-13.2.

In the end I chose to upgrade the SSL support on the JVM.

Upgrading JVM on Kerio Workspace for great good

In order to not mess up the installation, I went for a few preliminaries:

cd /opt/kerio/workspace

# Get a dedicated log tailer script
cat > tail-logs.sh <<'EOF'
cd /opt/kerio/workspace
tail -f logs/{activity,config,debug,error,security,warning}.log \
  logs/kerio-workspace-application-stderr.log
EOF
chmod a+x tail-logs.sh

# Create .gitignore
cat > .gitignore <<'EOF'
logs
store
rendering
EOF

# Store everything but the user data and logs in git
git init
chmod go= .git
git add .
git commit -m "Initial"

The first rule of dealing with legacy software? Have a backup. The second rule? Have a rollback⁴.

Git might be imperfect (hello, file ownership), but it’s quick and easy.

With basic safeguards in place, I moved the old jre as jre.orig and symlinked it back as jre:

cd /opt/kerio/workspace
mv jre jre.orig
ln -s jre.orig jre

That’s prep for the easy rollback strategy in case I messed up.

Basic Java

Get jre-6u45-linux-i586.bin from Oracle’s archive and unpack it.

It’s a zip file with added bonus of doubly packed *.pack files that need to be jre1.6.0_45/bin/unpack200 if you don’t want the full goodness of the installer.

The installer-less way:

cd /opt/kerio/workspace
unzip /path/to/jre-6u45-linux-i586.bin
cd jre1.6.0_45/
for i in $(find -iname \*.pack); do
  ./bin/unpack200 "$i" "$(echo $i|sed 's/.pack$/.jar/')" && rm "$i"
done

Get the jce_policy-6.zip (unlimited non-crippled java crypto policy) from Oracle’s archive and install:

cd jre1.6.0_45/lib/security
unzip /path/to/jce_policy-6.zip
mv jce/*.jar .
rm -rf jce

Install BouncyCastle SSL library

This is stolen from How to use TLS 1.2 in Java 6 and adapted.

Download the Provider, ASN.1, TLS from Bouncy Castle latest releases, pay attention to the version, you want the JDK 1.5 - JDK 1.8.

Install bcprov-ext-jdk15to18-170.jar, bcprov-jdk15to18-170.jar, bctls-jdk15to18-170.jar, bcutil-jdk15to18-170.jar to jre1.6.0_45/lib/ext.

Tweak the providers in jre1.6.0_45/lib/security/java.security:

# [...]

# Original list:
# security.provider.1=sun.security.provider.Sun
# security.provider.2=sun.security.rsa.SunRsaSign
# security.provider.3=com.sun.net.ssl.internal.ssl.Provider
# security.provider.4=com.sun.crypto.provider.SunJCE
# security.provider.5=sun.security.jgss.SunProvider
# security.provider.6=com.sun.security.sasl.Provider
# security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI
# security.provider.8=sun.security.smartcardio.SunPCSC

# New list (bouncy castle):
# See: https://stackoverflow.com/a/61115302
security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider
security.provider.3=sun.security.provider.Sun
security.provider.4=sun.security.rsa.SunRsaSign
security.provider.5=com.sun.net.ssl.internal.ssl.Provider
security.provider.6=com.sun.crypto.provider.SunJCE
security.provider.7=sun.security.jgss.SunProvider
security.provider.8=com.sun.security.sasl.Provider
security.provider.9=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.10=sun.security.smartcardio.SunPCSC

# Here we are changing the default SSLSocketFactory implementation
ssl.SocketFactory.provider=org.bouncycastle.jsse.provider.SSLSocketFactoryImpl

# update 2022-03-10: re-enable TLSv1 and TLSv1.1, to allow more flexible SMTP out
jdk.tls.disabledAlgorithms=RC4, DES, MD5withRSA, DH keySize < 1024, \
  EC keySize < 224, 3DES_EDE_CBC, anon, NULL
jdk.certpath.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, \
  DSA keySize < 1024, EC keySize < 224


# [...]

Commit everything:

cd /opt/kerio/workspace
git add .
git commit -m "Brand spanking new JRE w/ BouncyCastle"

Switch to new JRE (and nothing happens)

Switching to the new JRE is as easy as:

cd /opt/kerio/workspace
ln -sf jre1.6.0_45 jre

But as you would find out (by restarting the Kerio), there’s no change. The SSL/TLS is still at old version, and even altering the tomcat/conf/server.xml to force a higher TLS version (e.g. SSLProtocol="TLSv1.1" in the <Connector>) does absolutely nothing good (server refuses to start).

Adjusting Tomcat

Here I have to be clear, I have cut some corners. Normally, if I had unlimited time and patience, I would rebuild the Tomcat Native Connector library (instead of going with the Bouncy Castle).

The problem with the Tomcat configuration is that it uses a native library (libtcnative-1.so) that links against supplied libktcrypto.so.0.9.8 and libktssl.so.0.9.8, which is IMO just a rebranded OpenSSL 0.9.8. It makes sense, but it’d be a headache to relink on an old system. No thank you.

Hence the Bouncy Castle library above. Maybe I should have explained that earlier?

Anyway —

The trouble with BouncyCastle is twofold:

1. It needs certificates in *.jks (java store, PKCS#12) format instead of PEM
2. It needs a slightly different config directives in tomcat/conf/server.xml

The certificate conversion is easy (sslcert/convert.sh):

#!/bin/bash
PATH=$PATH:/opt/kerio/workspace/jre/bin/
set -ex
cd /opt/kerio/workspace/sslcert || exit 1

convert() {
  test -f "$1.crt"
  test -f "$1.key"
  openssl pkcs12 -export -in "$1.crt" -inkey "$1.key" -out "$1.p12" \
    -name "certificate" -passout pass:foobar
  keytool -importkeystore -srckeystore "$1.p12" -srcstoretype pkcs12 \
    -destkeystore "$1.jks" -storepass foobar -srcstorepass foobar -noprompt
}

for i in *.key; do
  convert "$(basename "$i" .key)"
done

The config adjustment is also straightforward:

<?xml version='1.0' encoding='utf-8'?>
<Server port="4064" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <Service name="Catalina">
    <Connector
      port="4060" protocol="HTTP/1.1" scheme="https" secure="true" pollerSize="1024"
      URIEncoding="utf-8"
      compression="on"
      compressableMimeType="text/html,text/css,application/javascript,application/json-rpc"
      keystoreFile="/opt/kerio/workspace/sslcert/cert.jks" keystorePass="foobar"
      truststoreFile="/opt/kerio/workspace/sslcert/cert.jks" truststorePass="foobar"
      sslEnabledProtocols="TLSv1.3,TLSv1.2"
      sslProtocol="TLS"
      SSLEnabled="true"
      />
    <Connector
      port="${com.kerio.workspace.http.port}" protocol="HTTP/1.1" pollerSize="1024"
      redirectPort="${com.kerio.workspace.https.port}"
      URIEncoding="utf-8"
      compression="on"
      compressableMimeType="text/html,text/css,application/javascript,application/json-rpc"
      />
    <Connector
      port="${com.kerio.workspace.https.port}" protocol="HTTP/1.1" scheme="https"
      secure="true" pollerSize="1024"
      URIEncoding="utf-8"
      compression="on"
      compressableMimeType="text/html,text/css,application/javascript,application/json-rpc"
      keystoreFile="/opt/kerio/workspace/sslcert/cert.jks" keystorePass="foobar"
      truststoreFile="/opt/kerio/workspace/sslcert/cert.jks" truststorePass="foobar"
      sslEnabledProtocols="TLSv1.3,TLSv1.2"
      sslProtocol="TLS"
      SSLEnabled="true"
      />

    <Engine name="Catalina" defaultHost="localhost">
      <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">
      </Host>
    </Engine>
  </Service>
</Server>

And the finishing touch (of switching to the Bouncy Castle) is trivial:

cd /opt/kerio/workspace
mv tomcat/lib/libtcnative-1.so tomcat/lib/libtcnative-1.s_

Without libtcnative-1.so Tomcat falls back to Java-backed SSL. And that is configured to use Bouncy Castle.

And there you have it – profit (after service restart):

SSL Server Test result after⁵.

Don’t forget to git add .; git commit -m "Bouncy Castle installed.".

To revert this should be also super easy, barely an inconvenience:

1. Revert the tomcat/conf/server.xml change.
2. mv tomcat/lib/libtcnative-1.s_ tomcat/lib/libtcnative-1.so
3. ln -sf jre.orig jre

See? Easy rollback.

Client access woes

This is part of the 2022-03-10 update.

Unexpectedly, the Kerio Workspace win32 client binary is also linked against openssl 0.9.8. That means that this setup prevents it from connecting.

As a work around, I opted for putting HAProxy in front of the Kerio Workspace server (with an appropriate <Valve> to patch the remote IP through), which essentially renders a portion of this tweak moot.

And the resulting qualys ssl score a “B”, not “A”.

Still, I kept the tweak in place (and only HAProxied the normal https port) to make the access to the admin page more robust and avoid fighting Kerio’s implementation.

Closing words

I’m positively surprised how tweakable the Kerio Workspace install was, all thanks to proper layering (Java VM, Application Server, Application itself).

If this were a C binary, it would be a nightmare to update⁶. I would end up slapping an HTTP(s) proxy in front of it… and calling it a day.

Anyway, installing a Let’s Encrypt certificate for the legacy server was also a bit of fun. And that’s a topic for another post.