This is part 2 of the Splunk Enterprise Fundamentals: Standalone Deployment, if you haven't ready Part 1 see here.
Step 5: Configure Splunk
If you've installed a desktop version of Linux on your Splunk server, enter localhost:8000. I have to make it easier for anyone a bit new to Linux, however you do not need to.
If not, open a browser on the same LAN and enter the [Splunk Server IP Address]:8000.
You will be prompted for the default credentials, which are displayed and then ask you to change them. Be cool, change the default password.
5.1 Configure Indexer:
Go to Settings > Forward and Receiving
Under Receiving data, click New
Finally, configure the port the Universal Forwarders will send traffic to. The default port is 9997, however, if in Step 1 you changed this, make sure you're consistent!
5.2 Install universal forwarder:
On a separate machine, I'm using my MacBook but you can use a VM running Linux, download universal forwarder (or UF) that's appropriate for your device.
If you are using a Windows UF Client, you will need to install a different Add-On such as Add-On for Microsoft Windows.
As I used a MacBook my instructions for installing follow this, see here for documentation on all options if you feel stuck.
On a Macbook, you can install via .dmg file and double click it, or run via the command line: dpkg -i splunk_package_name.deb. As well as a .tgz, again, choose what you like.
Next, we're going to configure our UF to call home to our deployment server. Remember, we're configuring a StandAlone deployment, this means our one machine is #AllTheThings.
To get the IP of your Splunk VM enter the following in terminal: ifconfig, if using a Windows install: ipconfig
Now, under /Applications/SplunkForwarder/bin, and enter the following: sudo ./splunk set deploy-poll [hostname or ip address]:8089 -auth admin:changeme
You may have changed the port in your design, so be consistent. Then, again for security, change the UF password: sudo ./splunk edit user admin -password [new password] -role admin -auth admin:changeme
Finally, restart the UF: sudo ./splunk restart
5.3 Configure Deployment Server:
Now we're going to check out our client in the Deployment Server. Under Settings > Forwarder Management, notice your client is now 'phoning home.' If not, just wait a couple minutes.
Let's get data into Splunk by configuring the Indexer information, this will be achieved via an app we're going to create called Send to Indexer.
Back on our Splunk web interface, top left, choose Apps > Manage Apps > Create New App. The following is taken directly from the Send to Indexer official documentation.
Name: Send to Indexer
Folder name: send_to_indexer
Version: 1.0.0
Visible: No
Author: [your name here]
Description: This app allows universal forwarders to receive the Indexer(s) information.
Template: barebones
Click Save, congratulations you have created a folder, let's make our app do something shall we... Send to Indexer is saved under $SPLUNK_HOME/etc/apps, open terminal and navigate there: cd /opt/splunk/etc/apps/send_to_indexer/
Review what folders we have: ls
You should have: bin, default, local, and metadata. Today we're going to ignore all of these except local, go into that one. cd ./local
Note: only make changes under local, all other changes will be written over.
Now, we likely have an app.conf file, you can check out what it's saying using cat app.conf but again, we're going to ignore that. Let's create an output.conf file: sudo nano ./output.conf
Enter the following, and save:
Congratulations - we've created and finished our first app! However, did you catch where we saved it? /opt/splunk/etc/apps/ contains the apps for the local Splunk machine, but we want to deploy this to the UFs. To do this we have two options:
(i) If we want to share this app with both the local machine, and the UFs, we use sudo cp -f /opt/splunk/etc/apps/send_to_indexer/ /opt/splunk/etc/deployment-apps/
(ii) If we want to only use this app with the UFs, we use sudo mv - f /opt/splunk/etc/apps/send_to_indexer/ /opt/splunk/etc/deployment-apps/
I'm going to use (ii), but feel free to do whichever you'd like. Once completed, because you've changed a file, you will need to restart Splunk for the changes to be picked up.
Remember, restarting Splunk can be done via terminal, but we're going to restart via the web interface now: click Settings > Server Controls, and click Restart Splunk. Once the popup confirms Splunk has been restarted, click Okay and sign back in.
Go to Settings > Forwarder Management now, there's a setting we need to configure for the App. In the Apps tab, you should see your beautiful App, select Edit.
Enable App should be checked, but you also want to check off Restart Splunkd, then click Save. Make sure you do this for every App and/or Add-On you install!
To summarise, we have:
- configured the Indexer to receive data on port 9997,
- configured the Deployment Server with one app and one client,
- created a app that tells the UFs what address the Indexer is found at.
What we haven't done yet is:
- install an Add-On to have the UF client actually send any data, and
- assign the App or Add-On to the UF client.
5.4 Install the first Add-On
In this example I'm using the *NIX Add-On, as I am using a MacOS as the client that has an UF installed. You can use any Add-On you'd prefer, but for ease I'd suggest following along for now.
Back in the Splunk web interface, top left, select Apps > Manage Apps, and Browse more apps.
This takes you to a limited view of Splunkbase, only Apps and Add-Ons that are compatible with your version of Splunk will appear here.
In the search bar type NIX and enter, Install: Splunk Add-on for Unix and Linux.
Again, this will install the Add-On for the local machine. Choose option (i) or (ii) above for /opt/splunk/etc/apps/Splunk_TA_nix/.
5.5 Get data in
You've now installed the NIX Add-On, however it doesn't actually do anything, because we've not told it to turn on any data; now we're going to do that.
Navigate to your newly moved folder: cd /opt/splunk/etc/deployment-apps/Splunk_TA_nix/default/
Remember, don't make any changes to the default folder files, these changes will just be overwritten by the Deployment Server. All we're doing is checking out the options available to turn on in the Add-On. You can also checkout the official documentation here Splunk Add-On for Unix and Linx.
Type: nano ./inputs.conf Scroll through the file, these are all the built-in data your Add-On has, a couple things to note:
SourceType: this option tells Splunk how to structure the events.
Index: this is the location Splunk will save the data, both raw and created events, you can customise this option but we're going to keep it as it is.
Disabled: to enable this option, we're going to create a file under local that will override this.
We're going to collect usersWithLoginPrivs, lastlog, and cpu of the UF Client, so now that we've decided, close this file.
Navigate to the local folder: cd ../local/
Edit/create the inputs.conf file here: sudo nano ./inputs.conf
Enter the following, then save and close the file:
Congratulations, you have:
Created an app
Installed an Add-On
Customised the Add-On to send data
Now it's time to deploy these!
5.6 Create Server Class
Back in the Splunk web interface, top right, select Settings > Forwarder Management, go to the middle tab Server Classes and click New Server Class, and name it 1337, or whatever you prefer.
Once you click Save you will come to a new window that displays the following buttons: Add Apps, Add Clients, Edit, or Documentation, let's focus on adding Apps.
Click Add Apps, your Apps* are displayed on the left, click them to assign them to our 1337 class and click Save.
Click Add Clients, this window is quite neat! There are three options: Include, Exclude, and Filter by Machine Time. By selecting:
Include: *
Filter by Machine Type: darwin-x86_64
I can add MacBook's to this Server Class as they connect to the Deployment Server, which is neat! That or searching for "*.domainName" etc, it's useful. Once you're happy, click Save.
Just for clarity, Apps and Clients can be added to multiple Server Classes, think of this as a group for similar requirements.
Step 6: Searching
Let's explore the data we collected - back in the Splunk web interface select Apps > Searching & Reporting.
Search: index="os"
Note: if you're still not seeing data try these troubleshooting steps:
On the Splunk UF:
cd /Applications/SplunkForwarder/bin
sudo ./splunk list forward-server
(The password here is the one you changed back in step 5.2)
If you do not have an Active Forwarder (Indexer), then verify settings in Forwarder Management, and the ownership of the /Applications/SplunkForwarder or /opt/SplunkForwarder/ folder.
If you need to change the ownership: sudo chown -R $user:admin /opt/SplunkForwarder/
Note: if you find this is the issue, let me know, I have a script to fix this each time the daemon starts up.
Step 7: Profit
Just kidding. Remember in my first edition I said Splunk is complex? Yep, it is. You can customise almost anything in Splunk – from the data indexed, down to the font and colours in your reports.
To keep things short, I will end here.
*I realise I've used Apps and Add-Ons interchangeably, Splunk does this, it's confusing but just go with the flow. This is the App we created to notify UFs what Indexer to use, and the Add-On we downloaded and customised for NIX.