diff --git a/README.md b/README.md index dec5fb2253bae61e9e820037c73d97a3f0e35cf8..9f4447ba0a5ae221eec6123ddf1c0919c9803121 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,50 @@ # elk-hole -## elasticsearch, logstash and kibana configuration for pi-hole visualization +## Pi-hole data visualization using Elasticsearch, Logstash and Kibana elk-hole provides the relevant files and configuration to easily visualize pi-holes/dnsmasq statistics via the popular elasticstack. -### show, search, filter and customize pi-hole statistics ... the elk way +### Show, search, filter and customize pi-hole statistics ... the elk way -### requirements: -## working installation of: -1. logstash (currently tested up to version "7.1.0") -2. elasticsearch (currently tested up to version with "7.1.0") -3. kibana (currently tested up to version "7.1.0") -4. filebeat on pi-hole (tested with "1.3.1" & "7.1.1") +### Requirements: +## Working Installation of: +1. [logstash](https://www.elastic.co/products/logstash) (currently tested up to version "7.3") +2. [elasticsearch](https://www.elastic.co/products/elasticsearch)(currently tested up to version with "7.3") +3. [kibana](https://www.elastic.co/products/kibana)(currently tested up to version "7.3") +4. [filebeat](https://www.elastic.co/products/beats/filebeat) on pi-hole (tested with "1.3.1", "7.1.1" & "7.3") --> installation of the elk stack - refer to https://www.elastic.co/ for details. +For official installation guides of the elk stack - refer to [Elastic](https://www.elastic.co/ for details) -this repo provides the relevant files and configuration for sending the pi-hole logs via filebeat directly to logstash/elasticsearch. We will then visualize the logs in kibana with a custom dashboard. +For a quick setup, check out [easyELK](https://github.com/josh-thurston/easyELK) + + +Elk-hole provides the relevant files and configuration for sending the pi-hole logs via filebeat directly to logstash/elasticsearch. We will then visualize the logs in kibana with a custom dashboard. The result will look like this:  -#### alternative: +#### Alternative:  # HOW TO USE ### LOGSTASH HOST + + +1. Download the files from Elk-hole repo +2. From the downloaded files, copy ```20-dns-syslog.conf``` and the ```/patterns``` folder to your logstash system. Place them both in ```/etc/logstash/conf.d/```. + +Your files should be like this: + +```/etc/logstash/conf.d/20-dns-syslog.conf``` + +```/etc/logstash/con.d/patterns/dns``` + +======= 1. copy ```/conf.d/20-dns-syslog.conf``` to your logstash folder (usually ```/etc/logstash/```) + If you have other files in this folder make sure to properly edit the input/output/filter sections to avoid matching our filebeat dns logs in these files which may be processed earlier. For testing purposes you can name your conf files like so: ``` @@ -37,6 +53,70 @@ If you have other files in this folder make sure to properly edit the input/outp /conf.d/40-other2.conf ``` + +This makes sure that ```/conf.d/20-dns-syslog.conf``` is processed at the beginning. + +2. Using vim or nano, open/edit ```20-dns-syslog.conf```. Scroll down to the Output section and change ```ELASTICSEARCHHOST:PORT``` to match your environment. If elasticsearch is running on the same system as logstash, then ```127.0.0.1:9200``` should work. +4. Restart logstash - ```systemctl restart logstash.service``` + +### PI-HOLE Host + +5. From the downloaded files, copy ```filebeat.yml``` to your ```/etc/filebeat/``` and copy ```99-pihole-log-facility.conf``` to ```/etc/dnsmasq.d/``` +6. Using vim or nano, open/edit the ```hosts:``` line and enter the IP address of the logstash system ```LOGSTASH IP:5141``` +7. Restart filebeat ```systemctl restart filebeat.service``` +8. *Important:* Restart pi-hole and ensure filebeat is sending logs to logstash before proceeding further. ```pihole restartdns``` +10. You can verify this filbert is running properly with the following two steps +11. ```service filebeat status``` The output should show a couple key message. Active: active (running) & Connection established +12. ```sudo filebeat test output``` should show: + +``` +Logstash: <Logstash IP>:5141... + Connection.. + Parse hosts... OK + Dns lookup... OK + Addresses: <Logstash IP> + Dial up... OK + TLS... WARN secure connection disabled + Talk to server... OK +``` + +The following steps on the Kibana Host will not work correctly if sending data to logstash is not successfull! + +### KIBANA HOST (CAN BE THE SAME AS LOGSTASH AND ELASTICSEARCH) + +13. Browse to the Kibana management interface using a web browser ```http://Kibana IP:5601``` +14. Go to Management --> Kibana --> Index Patterns and click Create the index pattern +14. Type ```logstash-syslog-dns*``` - It should find one index +15. Click next step and select ```@timezone``` +16. Create index pattern +17. Once the index is created, verify that 79 fields are listed +18. Click the curved arrows on the top right to refresh the index fields. This is important because this will not automatically happen. +19. Browse to Management --> Kibana --> Saved Objects +20. Select Import (You will repeat this step) +21. From the downloaded files, locate the ```json``` folder and import the following files depending on your software version (1.3.1 or 7.x) + +```elk-hole - vis.json``` + +```elk-hole - vis_enhanced.json``` + +```elk-hole - vis_enhanced_fix.json``` + +```elk-hole - dash.json``` + +```elk-hole - dash_enhanced.json``` + +*Note:* When you import these files, you could possibly see a message "Index Pattern Conflicts". This is ok. Below that message you may see one or two rows of data. On each row click on the drop down menu and select "logstash-syslog-dns*" + +20. Browse to Dev Tools (wrench on left navigation) +21. When Dev Tools comes up, there will be two columns when you are in the section with "Console" underlined. +21. Delete any existing data in the left column +22. From the downloaded files, locate ```logstash-syslog-dns-index.template_ELK7.x.json``` +23. Open that file in a text editor on your system +24. Copy the entire contents of the file +22. Paste the content of: ```logstash-syslog-dns-index.template_ELK7.x.json``` into kibanas dev tools console +23. Click the green triangle in the upper right of the pasted content (first line). Output should be: + +======= This makes sure that ```/conf.d/20-dns-syslog.conf``` is beeing processed at the beginning. 2. customize ```ELASTICSEARCHHOST:PORT``` in the output section at the bottom of the file @@ -72,11 +152,20 @@ it should say ```ok``` on every step. ```DELETE /_template/logstash-syslog-dns*``` 22. import the template: paste the content of: ```logstash-syslog-dns-index.template_ELK7.x.json``` into kibanas dev tools console 23. click the green triangle in the upper right of the pasted content (first line). Output should be: + ``` { "acknowledged" : true } ``` + +24. As a precaution restart the whole elk stack + +``` +systemctl restart logstash.service +systemctl restart elasticsearch.service +systemctl restart kibana.service +======= 24. as a precaution restart the whole elk stack ``` systemctl restart logstash