Compare commits

...

5 commits

Author SHA1 Message Date
renovate[bot]
1e6f92e06d
chore(deps): update module github.com/hugomods/icons/vendors/font-awesome to v0.6.10 2024-11-15 07:18:30 +00:00
ditatombot[bot]
45862a251c
Merge pull request #277 from ditatompel/renovate/github.com-hbstack-docs-0.x
Merge pull request #277

This merge action was created automatically.

Reviewed-by: ditatompel <ditatompel@users.noreply.github.com>
2024-11-15 07:17:52 +00:00
ditatombot[bot]
b354f99713
Merge pull request #278 from ditatompel/mikrotik-netflow
Merge pull request #278

This merge action was created automatically.

Reviewed-by: ditatompel <ditatompel@users.noreply.github.com>
2024-11-14 22:06:42 +00:00
d9ebc31a40
Added mikrotik-netflow-elk article 2024-11-15 05:05:20 +07:00
renovate[bot]
83ba15446b
chore(deps): update module github.com/hbstack/docs to v0.21.0 2024-11-14 04:50:57 +00:00
5 changed files with 337 additions and 4 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 129 KiB

View file

@ -0,0 +1,165 @@
---
title: Analisa Lalu Lintas Jaringan Router MikroTik Menggunakan Traffic Flow, Filebeat, ElasticSearch, dan Kibana
description: Konfigurasikan Filebeat dan MikroTik Traffic Flow untuk mengirim data NetFlow ke ElasticSearch.
summary: Panduan konfigurasi Traffic Flow MikroTik, mengaktifkan modul NetFlow pada Filebeat, dan menyiapkan dasbor Kibana.
date: 2024-11-11T13:26:00+07:00
lastmod:
draft: false
noindex: false
nav_weight: 1000
categories:
- TIL
tags:
- MikroTik
- NetFlow
- ElasticSearch
- Kibana
- Filebeat
images:
authors:
- ditatompel
---
Di pertengahan tahun 90-an, **Cisco** memperkenalkan fitur **NetFlow** pada
router yang diproduksinya. Fitur NetFlow ini menyediakan kemampuan untuk
mengumpulkan informasi _packet_ masuk maupun _packet_ keluar dari sebuah
_network interface_. Secara umum, setup NetFlow terdiri dari 3 komponen utama,
yaitu:
- **Flow Exporter**: Bertugas mengumpulkan _network packets_, kemudian
mengirimkan ke **Flow Collector**.
- **Flow Collector**: Bertugas menerima dan melakukan _preprocessing_ data
yang diterima dari **Flow Exporter**.
- **Aplikasi Analisis**: Aplikasi yang bertugas menganalisa data yang
diterima dari **Flow Collector**, dan biasanya memvisualisasikan data yang
diterima dalam bentuk grafik.
Fitur NetFlow ini kemudian diimplementasikan oleh banyak perusahaan yang
memproduksi _router_ dan _switch_ dengan nama yang berbeda. Sebagai contoh,
**Juniper Networks** menggunakan nama **J-Flow** sedangkan **MikroTik**
menggunakan nama **Traffic Flow**.
Kebetulan, saya menggunakan router **MikroTik RB450G**, dan di kesempatan kali
ini saya ingin berbagi informasi tentang cara menggunakan fitur Traffic Flow
pada router MikroTik, dan mengintegrasikan-nya dengan Filebeat, ElasticSearch
dan Kibana.
{{< youtube yHbH-oJX-Lg >}}
## Prasyarat
Sebelum memulai, Anda perlu menginstall dan memastikan bahwa **Filebeat**,
**ElasticSearch** dan **Kibana** berjalan dengan baik karena saya tidak akan
mengulas cara menginstall aplikasi-aplikasi tersebut disini. Silahkan mengikuti
proses installasi dan konfigurasi dari halaman dokumentasi terkait:
- [ElasticSearch][elasticsearch-install]
- [Kibana][kibana-install]
- [Filebeat][filebeat-install]
Sebagai tambahan informasi terkait artikel ini, saya menginstall Filebeat di
sebuah mesin Linux yang masih berada di satu jaringan dengan router MikroTik
saya. Sedangkan untuk ElasticSearch dan Kibana, saya menginstallnya di sebuah
VPS.
## Hak Akses Untuk Filebeat
Saya membuat 1 user baru dengan hak akses khusus yang nantinya digunakan oleh
Filebeat untuk mengirimkan data yang diproses dari router ke ElasticSearch.
Caranya, dari Dashboard Kibana, masuk ke **"Management"** > **"Security"** >
**"Roles"**. Buat sebuah role baru, dan beri nama yang untuk untuk role
tersebut. Di artikel ini saya menamainya dengan `filebeat_setup`.
Pada bagian **"Cluster privileges"** saya memberikan akses berikut:
- `monitor`
- `manage_ilm`
- `manage_index_templates`
- `manage_collector`
- `manage_ingest_pipelines`
- `manage_logstash_pipelines`
- `manage_ml`
- `manage_pipeline`
Kemudian pada bagian **"Index privileges"**, saya memberikan `all`
**Privileges** untuk semua **Incides**.
Masih dari halaman **"Management"** > **"Security"**, masuk ke halaman
**"Users"** dan buat sebuah user baru. Saya menamai user baru tersebut dengan
nama `custom_filebeat`. Pada bagian **"Privileges"**, saya memberikan role
`filebeat_setup` yang sudah kita tambahkan sebelumnya. Selain itu, saya
memberikan role berikut:
- `kibana_admin`
- `ingest_admin`
- `editor`
- `monitoring_user`
- `kibana_system`
## Filebeat NetFlow Module
Login ke mesin Linux yang sudah terinstall Filebeat dan sesuaikan konfigurasi
pada `/etc/filebeat/filebeat.yml`, terutama pada bagian `setup.kibana` dan
`output.elasticsearch`. Konfigurasi `filebeat.yml` saya kurang lebih sebagai
berikut:
```yml
setup.kibana:
host: "https://kibana.ditatompel.com:443"
output.elasticsearch:
hosts: ["https://elastic-ap-southeast1-ctb1.ditatompel.com:443"]
username: "custom_filebeat"
password: "MySuperSecretPasswordThatMayAppearsOnYourFreakinAICompletions"
ssl:
enabled: true
```
Kemudian edit `/etc/filebeat/modules.d/netflow.yml.disabled` dan ubah
`netflow_host` ke `0.0.0.0` supaya Filebeat dapat menerima data dari router
MikroTik. Kurang lebih konfigurasi module `netflow` saya seperti berikut:
```yml
- module: netflow
log:
enabled: true
var:
netflow_host: 0.0.0.0
netflow_port: 2055
internal_networks:
- private
```
Setelah itu, _enable_ `netflow` module dengan menjalankan perintah `sudo
filebeat modules enable filebeat`. Untuk melihat module yang tersedia, baik
yang aktif maupun tidak, gunakan perintah `sudo filebeat modules list`.
Kemudian jalankan perintah `sudo filebeat setup -e` untuk melakukan setup
_index template_ dan _dashboards_ pada Kibana.
Terakhir, restart filebeat service dengan menjalankan perintah `sudo systemctl
restart filebeat`.
## Konfigurasi Traffic Flow MikroTik
Login ke router MikroTik, Anda bisa menggunakan SSH atau **Winbox.** Di artikel
ini, saya menggunakan Winbox untuk mempermudah konfigurasi.
Masuk ke **"IP"** > **"Traffic Flow"**. Klik pada menu **"Targets"** dan
tambahkan target baru. Ubah **"Dst. Address"** ke alamat IP dimana Filebeat
server berjalan. Ubah **"Version"** ke **"IPFIX"** dan pastikan
**checkbox "Enabled"** tercentang. Kemudian tekan tombol **"Ok"**.
Pada menu **"Traffic Flow Settings"**, pilih **"Interfaces"** yang ingin
diproses dan pastikan pastikan **checkbox "Enabled"** tercentang.
## Kibana Dashboard
Untuk hasil NetFlow yang telah diproses, masuk ke **"Analytics"** >
**"Dashboards"**. Disana akan muncul banyak dashboard yang sudah tergenerate
secara otomatis oleh Filebeat. Cari dengan kata kunci _"netflow"_ dan silahkan
mengeksplor berbagai macam informasi yang sudah tersedia.
[elasticsearch-install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
[kibana-install]: https://www.elastic.co/guide/en/kibana/current/install.html
[filebeat-install]: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html

View file

@ -0,0 +1,164 @@
---
title: "Traffic Analysis on MikroTik Routers: A Study Using Traffic Flow, Filebeat, Elasticsearch, and Kibana"
description: Configure Filebeat and MikroTik Traffic Flow to send NetFlow data to ElasticSearch for real-time analysis.
summary: Step-by-step guides on adjusting Filebeat configuration, enabling NetFlow modules, and setting up Kibana dashboards.
date: 2024-11-11T13:26:00+07:00
lastmod:
draft: false
noindex: false
nav_weight: 1000
categories:
- TIL
tags:
- MikroTik
- NetFlow
- ElasticSearch
- Kibana
- Filebeat
images:
authors:
- ditatompel
---
In the mid-1990s, **Cisco** introduced the **NetFlow** feature on its routers.
This NetFlow feature provides the ability to collect information on incoming
and outgoing packets from a network interfaces. In general, the NetFlow setup
consists of these three main components:
- **Flow Exporter**: Responsible for collecting network packets and sending
them to the **Flow Collector**.
- **Flow Collector**: Responsible for receiving and preprocessing data sent
from the **Flow Exporter**.
- **Analysis Application**: An application that is responsible for analyzing
data received from the Flow Collector and usually visualizes it in the form
of graphs.
This NetFlow feature was then implemented by many companies that produce
routers and switches with different names. For example, **Juniper Networks**
uses the term **J-Flow**, while **MikroTik** uses the name **Traffic Flow**.
I'm currently running a **MikroTik RB450G** router for my home network, which
is why I'm excited to share my experience with using the Traffic Flow feature
on one.
{{< youtube yHbH-oJX-Lg >}}
## Prerequisites
Before starting, you need to install and ensure that **Filebeat**,
**ElasticSearch**, and **Kibana** are running properly. Please follow the
installation and configuration process from the related documentation pages:
- [ElasticSearch][elasticsearch-install]
- [Kibana][kibana-install]
- [Filebeat][filebeat-install]
As additional information related to this article, I installed Filebeat on a
Linux machine that is still connected to the same network as my MikroTik
router. For ElasticSearch and Kibana, I set them up on a Virtual Private Server
(VPS).
## Access Rights for Filebeat
I created a new user with special access rights that will be used by Filebeat
to send processed data from the router to ElasticSearch. To do this, you need
to:
1. Go to the **"Management"** > **"Security"** > **"Roles"** page in the
Kibana Dashboard.
2. Create a new role and give it a name for the role. In this article, I named
it `filebeat_setup`.
In the **"Cluster privileges"** section, I granted access to the following:
- `monitor`
- `manage_ilm`
- `manage_index_templates`
- `manage_collector`
- `manage_ingest_pipelines`
- `manage_logstash_pipelines`
- `manage_ml`
- `manage_pipeline`
And, in the **"Index privileges"** section, I granted **all** privileges for
all **indices**.
To complete the setup, follow these additional steps:
1. Go to the **"Management"** > **"Security"** page.
2. Create a new user and name it `custom_filebeat`.
3. In the **"Privileges"** section, assign the `filebeat_setup` role that we
created earlier.
4. Additionally, assign the following roles:
- `kibana_admin`
- `ingest_admin`
- `editor`
- `monitoring_user`
- `kibana_system`
## Filebeat NetFlow Module
Login to the Linux machine that has Filebeat installed and adjust the
configuration in `/etc/filebeat/filebeat.yml`, especially in the `setup.kibana`
and `output.elasticsearch` sections. My current `filebeat.yml` configuration is
as follows:
```yml
setup.kibana:
host: "https://kibana.ditatompel.com:443"
output.elasticsearch:
hosts: ["https://elastic-ap-southeast1-ctb1.ditatompel.com:443"]
username: "custom_filebeat"
password: "MySuperSecretPasswordThatMayAppearsOnYourFreakinAICompletions"
ssl:
enabled: true
```
Then edit `/etc/filebeat/modules.d/netflow.yml.disabled` and change
`netflow_host` to `0.0.0.0` so that Filebeat can receive data from the MikroTik
router. My current `netflow` module configuration is as follows:
```yml
- module: netflow
log:
enabled: true
var:
netflow_host: 0.0.0.0
netflow_port: 2055
internal_networks:
- private
```
After that, enable the `netflow` module by running `sudo filebeat modules
enable netflow` command. To see the available modules, both active and
inactive, use the command `sudo filebeat modules list`.
Then run the command `sudo filebeat setup -e` to set up the _index template_
and _dashboards_ on Kibana.
Finally, restart the Filebeat service by running the command `sudo systemctl
restart filebeat`.
## MikroTik Traffic Flow Configuration
Login to the MikroTik router, using either SSH or Winbox. In this article, I
use **Winbox** for easier configuration management.
Go to **"IP"** > **"Traffic Flow"**. Click on the **"Targets"** menu and add a
new target. Change the **"Dst. Address"** field to the IP address of your
Filebeat server. Set the **"Version"** to **"IPFIX"** and ensure the
**"Enabled" checkbox** is selected. Then, click the **"Ok"** button.
In the **"Traffic Flow Settings"** menu, select the interfaces you want to
process and verify that the **"Enabled" checkbox** remains checked.
## Kibana Dashboard
For processed NetFlow results, go to **"Analytics"** > **"Dashboards"**. You
will see a variety of automatically generated dashboards by Filebeat. Search
for keywords like _"netflow"_ and explore the dashboards.
[elasticsearch-install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
[kibana-install]: https://www.elastic.co/guide/en/kibana/current/install.html
[filebeat-install]: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html

8
go.mod
View file

@ -25,7 +25,7 @@ require (
github.com/hbstack/carousel v0.2.7 // indirect github.com/hbstack/carousel v0.2.7 // indirect
github.com/hbstack/code-block-panel v0.1.2 // indirect github.com/hbstack/code-block-panel v0.1.2 // indirect
github.com/hbstack/content-panel v0.10.2 // indirect github.com/hbstack/content-panel v0.10.2 // indirect
github.com/hbstack/docs v0.20.1 // indirect github.com/hbstack/docs v0.21.0 // indirect
github.com/hbstack/docs/modules/breadcrumb v0.1.10 // indirect github.com/hbstack/docs/modules/breadcrumb v0.1.10 // indirect
github.com/hbstack/docs/modules/doc-nav v0.3.7 // indirect github.com/hbstack/docs/modules/doc-nav v0.3.7 // indirect
github.com/hbstack/docs/modules/featured-image v0.2.8 // indirect github.com/hbstack/docs/modules/featured-image v0.2.8 // indirect
@ -71,9 +71,9 @@ require (
github.com/hugomods/hugopress v0.5.0 // indirect github.com/hugomods/hugopress v0.5.0 // indirect
github.com/hugomods/i18n-js v0.2.1 // indirect github.com/hugomods/i18n-js v0.2.1 // indirect
github.com/hugomods/icons v0.6.6 // indirect github.com/hugomods/icons v0.6.6 // indirect
github.com/hugomods/icons/vendors/bootstrap v0.5.8 // indirect github.com/hugomods/icons/vendors/bootstrap v0.5.9 // indirect
github.com/hugomods/icons/vendors/font-awesome v0.6.9 // indirect github.com/hugomods/icons/vendors/font-awesome v0.6.10 // indirect
github.com/hugomods/icons/vendors/simple-icons v1.0.59 // indirect github.com/hugomods/icons/vendors/simple-icons v1.0.60 // indirect
github.com/hugomods/idb-js v0.1.0 // indirect github.com/hugomods/idb-js v0.1.0 // indirect
github.com/hugomods/images v0.11.2 // indirect github.com/hugomods/images v0.11.2 // indirect
github.com/hugomods/katex v0.3.3 // indirect github.com/hugomods/katex v0.3.3 // indirect

4
go.sum
View file

@ -44,6 +44,7 @@ github.com/hbstack/content-panel v0.10.2 h1:rga9bbr2hiFd0Lzuq36cZQ43yvTWr2s9+VyG
github.com/hbstack/content-panel v0.10.2/go.mod h1:KwqGZkheSR7BfSBzuP13oAurMSaIh2atCeweNRFUUqA= github.com/hbstack/content-panel v0.10.2/go.mod h1:KwqGZkheSR7BfSBzuP13oAurMSaIh2atCeweNRFUUqA=
github.com/hbstack/docs v0.20.1 h1:v/ByBrh8lvHpZ2q5hYPt30SUwEGSHT9W64fsLs4irJQ= github.com/hbstack/docs v0.20.1 h1:v/ByBrh8lvHpZ2q5hYPt30SUwEGSHT9W64fsLs4irJQ=
github.com/hbstack/docs v0.20.1/go.mod h1:34Lnp91olEaYg2Ipbgx667U2EXz+aNRcPq7VUyYNook= github.com/hbstack/docs v0.20.1/go.mod h1:34Lnp91olEaYg2Ipbgx667U2EXz+aNRcPq7VUyYNook=
github.com/hbstack/docs v0.21.0/go.mod h1:ELqW8PDL68RHa8RwdmfyEB2smfP8nYN4ryhLmYptawU=
github.com/hbstack/docs/modules/breadcrumb v0.1.10 h1:Ryp68WkRdOxL2UlGNYtN5ZJUxbbHv7bWe0NdzDt2Obg= github.com/hbstack/docs/modules/breadcrumb v0.1.10 h1:Ryp68WkRdOxL2UlGNYtN5ZJUxbbHv7bWe0NdzDt2Obg=
github.com/hbstack/docs/modules/breadcrumb v0.1.10/go.mod h1:pTh43gl04dPERZZVDQ70KVU0j0Z4hmPK7GrxUfLw15s= github.com/hbstack/docs/modules/breadcrumb v0.1.10/go.mod h1:pTh43gl04dPERZZVDQ70KVU0j0Z4hmPK7GrxUfLw15s=
github.com/hbstack/docs/modules/doc-nav v0.3.7 h1:K0U0EAsYYsnOV4AGYMbbkMEJe2HrBSmuWipbpBg4V4A= github.com/hbstack/docs/modules/doc-nav v0.3.7 h1:K0U0EAsYYsnOV4AGYMbbkMEJe2HrBSmuWipbpBg4V4A=
@ -136,10 +137,13 @@ github.com/hugomods/icons v0.6.6 h1:gGlafcBDRP7sSID+tgLcWdog+s/QBj8DIfU+h9tZj1U=
github.com/hugomods/icons v0.6.6/go.mod h1:cIkSvK6W0q6N4U6n9KGz+QfRWQXAW0INd+1P31gPNGg= github.com/hugomods/icons v0.6.6/go.mod h1:cIkSvK6W0q6N4U6n9KGz+QfRWQXAW0INd+1P31gPNGg=
github.com/hugomods/icons/vendors/bootstrap v0.5.8 h1:6M+G4F58pt3GkiGFeCDjKonExVV8PbBAVhDS8YQT7oY= github.com/hugomods/icons/vendors/bootstrap v0.5.8 h1:6M+G4F58pt3GkiGFeCDjKonExVV8PbBAVhDS8YQT7oY=
github.com/hugomods/icons/vendors/bootstrap v0.5.8/go.mod h1:jla3IbQAByyz45AikUAw/c7h0wUMUn5Z2j4gh+zVDj4= github.com/hugomods/icons/vendors/bootstrap v0.5.8/go.mod h1:jla3IbQAByyz45AikUAw/c7h0wUMUn5Z2j4gh+zVDj4=
github.com/hugomods/icons/vendors/bootstrap v0.5.9/go.mod h1:DuB5N1o3GCUyvYFnEeBvJRBrCHqbMWEbjoEWVLULggw=
github.com/hugomods/icons/vendors/font-awesome v0.6.9 h1:ZpnyXeCOa9n7NBVURSRIZuGZykz80udHT8yxM+RE6NQ= github.com/hugomods/icons/vendors/font-awesome v0.6.9 h1:ZpnyXeCOa9n7NBVURSRIZuGZykz80udHT8yxM+RE6NQ=
github.com/hugomods/icons/vendors/font-awesome v0.6.9/go.mod h1:YPxdyI0WHuZfvwNeG8ADHXwnWmWIuu3PqiYGz8q9Skw= github.com/hugomods/icons/vendors/font-awesome v0.6.9/go.mod h1:YPxdyI0WHuZfvwNeG8ADHXwnWmWIuu3PqiYGz8q9Skw=
github.com/hugomods/icons/vendors/font-awesome v0.6.10/go.mod h1:3izAIdRkMg2O0N1RWB4Uw615VErx75b/nC/199BYV50=
github.com/hugomods/icons/vendors/simple-icons v1.0.59 h1:zpjSmGkhcG40fzEVL+xDQRrPf0vk+xfeWWrw/Gixz/s= github.com/hugomods/icons/vendors/simple-icons v1.0.59 h1:zpjSmGkhcG40fzEVL+xDQRrPf0vk+xfeWWrw/Gixz/s=
github.com/hugomods/icons/vendors/simple-icons v1.0.59/go.mod h1:orl9o8nQFUF8h4m0qS+ympS8btJTaTxx4ZsXHSvXzHw= github.com/hugomods/icons/vendors/simple-icons v1.0.59/go.mod h1:orl9o8nQFUF8h4m0qS+ympS8btJTaTxx4ZsXHSvXzHw=
github.com/hugomods/icons/vendors/simple-icons v1.0.60/go.mod h1:/skqfU1fJ+WnPV8nX3sP2yTBJc7PLMdciMp8qNXDKYU=
github.com/hugomods/idb-js v0.1.0 h1:mA6hEGtR0aFEdmq9zYMGpfSpCU6SMilx9RzC4uK3DqY= github.com/hugomods/idb-js v0.1.0 h1:mA6hEGtR0aFEdmq9zYMGpfSpCU6SMilx9RzC4uK3DqY=
github.com/hugomods/idb-js v0.1.0/go.mod h1:GPfnmr+BQpP4s6Epd4Yu+RlUog8q/Djw8qy8BZFWZ04= github.com/hugomods/idb-js v0.1.0/go.mod h1:GPfnmr+BQpP4s6Epd4Yu+RlUog8q/Djw8qy8BZFWZ04=
github.com/hugomods/images v0.11.2 h1:d7u3c/ePboeOGx8OI5eUnilpA79YvD7KJjUgqVMR83Y= github.com/hugomods/images v0.11.2 h1:d7u3c/ePboeOGx8OI5eUnilpA79YvD7KJjUgqVMR83Y=