Upgrading Splunk? Save your certs!

Splunk 4.1 is out, and I am very pleased to find that the upgrade is quick and painless, and everything I have blogged about previously works the same way in the newest version. However, if you issued a certificate to your Splunk server from your AD-integrated CA (as discussed in this post) you’ll want to take a couple of extra steps during your upgrade.

It seems that the upgrade process replaces your certificate and key with a self-signed one. It’s easy enough to put yours back, as long as you were expecting this and kept a copy. If you didn’t, you’ll just want to repeat the steps in the post on using enterprise certificates to get back to business as usual. Sure, your CA still has a copy of the certificate you issued, but it doesn’t have the private key. If you followed my steps exactly to generate the original certificate and key pair, you’ve already upgraded your splunk> instance and are now finding out that you lost your enterprise cert, look in C:\OpenSSL\bin> on your workstation. You’ll probably find them there. Stop splunk>, skip down to step 7 of these instructions, and you’ll be fine. If you are reading this before you start the upgrade, we’re good to go.

Before you upgrade

Make sure you’ve read this information before proceeding, as well as the following:

Windows app and its inputs are disabled by default.
The Windows App was enabled by default in its app.conf file in versions 4.0-4.0.2. Starting in version 4.0.3, it is disabled in this file by default. Read on for important details:If you’re upgrading from 4.0-4.0.2 to 4.0.3 or later, the Windows App will be disabled, even if it was enabled in the version you’re upgrading from. If you’re doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default via the MSI and if you want to install it in a disabled state, you must specify this using the SPLUNK_APP msiexec command as described in "Install on Windows via the commandline".
Inputs in the Windows or *Nix apps are disabled by default in 4.1. If you’re using inputs that belong to the Windows or *Nix apps that shipped with 4.0.x and want to make sure they stay enabled after the upgrade, copy the inputs.conf file from $SPLUNK_HOME/etc/apps/<Windows_or_*Nix_app>/default and put it in$SPLUNK_HOME/etc/apps/<Windows_or_*Nix_app>/local .
Make sure you specify the same domain user.
When upgrading, you must explicitly specify the same domain user that you specified during first time install. If you do not specify the same user, Splunk will default to using the Local System User. If you accidentally specify the wrong user during your installation, use the instructions in these instructions to switch to the correct user before starting Splunk.
Don’t change the ports.
Changing the management port and/or the HTTP port when upgrading is not supported.
Back your files up.
Before you perform the upgrade, we strongly recommend that you back up all of your files, including Splunk configurations, data and binaries. Splunk does not provide a means of downgrading to previous versions; if you need to revert to an older Splunk release, just reinstall it.

Upgrading using the GUI installer

Stop Splunk either using the Windows Start menu option or by opening an administrative cmd prompt in C:\Program Files\Splunk\bin and typing
splunk stop [enter]
Copy your cert.pem and privkey.pem files from C:\Program Files\Splunk\share\splunk\certs and save them to safe place outside the install directory path for splunk>.
Download the new MSI file from the Splunk download page.
Double-click the MSI file. The Welcome panel is displayed. Follow the onscreen instructions to upgrade Splunk>. For information about each panel, refer to the installation instructions.
Splunk> will start up by default when you complete the installation.
Stop splunk> again by opening an administrative cmd prompt in C:\Program Files\Splunk\bin and typing
splunk stop [enter]
Copy your cert.pem and privkey.pem files from your safe place back to C:\Program Files\Splunk\share\splunk\certs, overwriting the existing files.
Start splunk> again by returning to the administrative cmd prompt in C:\Program Files\Splunk\bin and typing
splunk start [enter]
Logon to your splunk> instance, re-enable the Windows app, and call it a day.
Note: A log of the changes made to your configuration files during the upgrade is placed in $TEMP$.

And that, as we say, is that. Splunk> 4.1 is ready to rock and roll. I mistyped that last sentence as ready to rick and roll and for a brief moment, I was tempted to close this post with a rick-roll, but I realise that not everyone out there considers that an oldie but a goodie, so instead, I wanted to share this little insight into working at Splunk>. I’ve got to get myself into a gig like this.