Imagine you’re a Systems Administrator responsible for keeping your companies’ custom developed application up and running. It is a critical application responsible all the ordering and payments for your company and is the sole interface for your customers to buy your products.
Today that application went down for 4 hours. During that time your company lost 10 million dollars in sales. You have been called into the CIO’s office for a debriefing.
You walk in her office, she quickly asks you “How could this have been prevented”.
By using Splunk and specifically using Splunk with Universal forwarders to proactively monitor those critical applications.
How are you going to stay out in front of issues that may happen?
What about preventive fixes and DevOps?
Splunk is the answer to keeping you from having system crashes and pulling all nighters. Analyzing applications with Splunk can allow developers and administrators to test scenarios before going to production with applications. How did Nissan test their Website before their first ever Super Bowl commercial? Nissan used Splunk to thoroughly
What is Splunk?
Splunk is huge in the data center when it comes to analyzing log files and IT security. Splunk is an application that allows for machine data to be stored, indexed and visualized quickly. In the past log files were parsed and stored by writing custom scripts with regular expressions to make the files human readable. Splunk simplifies all that with setting up default parsers for many common and uncommon log files and letting users start visualizing their data with in the Splunk application.
Since Splunk setup is so easy to setup the popularity of Splunk has been going through enormous growth. Recently I attended an Big Data conference where they said 70% of companies are using Splunk in some fashion. Gartner placed Splunk in the Leader Magic Quadrant for 2016.
What is a Splunk Universal Forwarder
So how can you analyze application server log files while running the application in production?
Splunk has forwarders for sending data between different instances of Splunk. Using a forwarder allows to move log files from one machine to another without having to write custom batch scripts and clog up bandwidth. Let’s talk about how the Splunk forwarder is used in the data center.
FedEx as a Splunk Forwarder?
FedEx is amazing at moving packages. Here recently my cousin graduated from college and I wanted to send her the book The Obstacle is the Way (seriously check it out the book). Think of the book as the data and my cousin and myself as machines and FedEx as the forwarder. I was able to package up the book (data) and send it off to my cousin. The package was wrapped (encrypted) and the correct address (URL) was placed on it. The Splunk Universal Forwarder is like FedEx. It will deliver machine data to other instances of Splunk.
When installing universal forwarders, Splunk has two option to chose from depending on the use case.
What is a Splunk Light Forwarder
The first type of Splunk forwarder is the light or universal forwarder. Think of it as a lightweight or minimal installation. The light forwarder has minimal features and its main objective to move data from one machine to another. No analysis or indexing. It’s even limited in the data that will parse because it’s goal is to move data to an Splunk Indexer. Another thing missing with the light forwarder is the Web CLI so it’s strictly from the command line for this forwarder. Since the goal of the light forwarder is low impact not having a Web CLI isn’t a deal breaker. Why add features not needed if we are going to analyze the data else where.
What is a Splunk Heavy Forwarder
The second type of forwarder is called a heavy forwarder. Think of a full blown instance of Splunk. It’s similar to what we have running in local development environment. The only difference is what we choose to disable. Remember depending of the scenario we want to the option to have the lowest impact to the CPU of the machine we are hosting on. So the heavy forwarder allows for us to disable features we aren’t going use. Management of the heavy forwarder can be done through the Web CLI, which we have been using, or the command line like in the universal forwarder.
All the Splunk forwarders have build in enterprise features like encryption and compression. Encryption offers the ability to protect data in-flight and prevent unwanted reads of log files from packet capture. The compression option will vary on the amount off data that is duplicated and white space in the log file. So if you looking to calculate the compression just know it’s going to depend. data Both encryption and compression are opt in features and are not enabled by default.
Where are Universal Forwarders used?
Use Cases
- Application Servers
- Database Servers
- Networking Infrastructure
- Web Servers
- Internet of Things
- Continuous Integration and Testing
- Detecting Insider Threats
- Securing Networks
How to Install Splunk Universal Forwarder
Let’s look at how to setup a Splunk Universal Forwarder. Just like the full blown Splunk instance you have to pick the flavor of OS for the host machine. After getting the correct Splunk version you will run the default install unless you are the light version (which I recommend) it will all be down for the command line.
For example below are the steps for installing the light Forwarder for Ubuntu server:
- Download Specific version of Splunk for Windows, Mac or Linux Distributions – Download directly or use wget -url to download from command line.
- Install on Ubuntu Machine – Move downloaded package to Ubuntu /tmp directory. Once .tgz is in /tmp directory run dpkg -i splunk-verison-xxx.tgz. Command will kick off the installation of the Splunk Universal forwarder.
- Start up Splunk Forwarder – After running the dpkg command for installation move to Splunk directory cd /opt/splunkforwarder/bin. Next start up Splunk server ./splunk start.
- Set up forwarding machine on Ubuntu – Last configuration change is to ensure log files will be forwarder through port 9997. Port 9997 is default but it won’t hurt to run the following command ./splunk add forward-server hostmachineIP:9997.
- Configure receiving on Splunk instance – Finally now that the install is complete on the host machine you will need to configure Splunk to receive the log files from Ubuntu server. On the Splunk instance enable receiving from UI in settings –> receiving. Ensure that Splunk is listening on the default port of 9997.
Final Thoughts on Splunk Universal Forwarder
Splunk forwarding is the secret sauce for Splunking. It allows for data to be streamed in real time to the main Splunk instance with little performance concern on host machine. Installation for Splunk Universal Forwarders is a little tricky at first but once you get one installed the next one are simple.
If you aren’t using Splunk to monitor your mission critical application you are missing an opportunity for better insight and the ability to be more proactive vs. reactive. My advice is to start small with Splunk by picking a couple applications to monitor. Once you get the hang of it start adding new application. Trust me it will save you from future headaches and long nights.
Be sure to sign up for my newsletter for the latest Big Data tips.