UPDATE 2021-12-01
Elastic have deprecated custom Beat generation in v7.16. I’m leaving this article for posterity but you won’t be able to follow along in the near future and I wouldn’t suggest building a custom Beat now. Now on to the original post…
The Beats documentation, specifically the Beats Developer Guide, is not in great shape. If you want to write your own custom Beat and follow the docs, you won’t be able to create a new custom Beat. A lot of questions are being posted on the Elastic Discuss forum by people struggling to get started.
As well as the documentation being incorrect in places, the prerequisites are in different areas and some steps are missing altogether.
I have opened pull requests to fix some of the problems but they’re not getting any traction so it doesn’t appear that the situation will improve any time soon.
In the meantime, if you’re looking to create a custom Elasticsearch Beat, I’ll describe what you need to do to get the base code generated.
I have tested these steps on macOS (Catalina), CentOS (in a Vagrant VM) and an AWS EC2 instance running Amazon Linux 2. I can’t test the process on Windows, unfortunately.
Install exactly Go 1.13.10
The documentation states that 1.13.10 is the minimum version required but it’s actually the exact version required. Running 1.14 will result in errors during the code generation:
There is an open issue to add support for 1.14 but it’ll be a while before it makes it into master.
Simply follow the guide on the Golang website; not all package managers will have 1.13.10 available.
Your PATH needs to be updated after installing but we’ll do that in a later step.
Install gcc
gcc is required by the Beats build system. Depending on how you installed Go, you may not have it installed.
Your favourite package manager will be able to locate, download and install it for you.
Configure environment
Create the GOPATH environment variable. This is usually ~/go/.
export GOPATH=$HOME/go
Add the $GOPATH/bin directory to your $PATH. Some Go packages - including Mage, which we’ll install shortly - will add binaries to this folder and we want to be able to run them from the terminal. While we’re here, we’ll also add Go to the $PATH; I’m assuming it’s installed in the default /usr/local/go/.
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
Install Git
You will likely have this installed already but, for the sake of completeness, it is a required step.
sudo yum install git
Install Mage
This step isn’t mentioned in the documentation but it’s absolutely required.
Mage is used to build Beats. It’s also used to run the code generator that builds the base of our new Beat.
mkdir -p ${GOPATH}/src/github.com/magefile && cd $_
git clone https://github.com/magefile/mage
cd mage
go run bootstrap.go
Install Python 3
If you don’t have Python 3 installed, go ahead and install that. While there’s ongoing work to remove the dependency on Python, parts of the Beats platform still use it for various tasks.
Check which version you have as the default by running python --version. If that shows Python 2, also try python3 --version. If python3 isn’t installed, you’ll need to install it. You can use yum, homebrew, or apt for that.
sudo yum install python3
Create your path
You will need your own directory in your GOPATH. Assuming your GitHub username is beatsAuthor, you’d run:
mkdir ${GOPATH}/src/github.com/beatsauthor
Clone the Beats repository
Create a directory in your $GOPATH for the Beats repository:
mkdir ${GOPATH}/src/github.com/elastic && cd $_
git clone https://github.com/elastic/beats
cd beats
Create your Beat
This is another spot where the documentation is wrong; it tells you to cd into your beatsauthor directory and run the next command from there. Doing so will give you the following error:
No .go files marked with the mage build tag in this directory.
You need to stay in the $GOPATH/src/github.com/elastic/beats directory, then run mage GenerateCustomBeat:
[ec2-user@ip-111-111-111-111 beats]$ mage GenerateCustomBeat
Enter the beat name [examplebeat]: firstbeat
Enter your github name [your-github-name]: beatsauthor
Enter the beat path [github.com/beatsauthor/firstbeat]:
Enter your full name [Firstname Lastname]: Beats Author
Enter the beat type [beat]:
Enter the github.com/elastic/beats revision [master]:
go: creating new go.mod: module github.com/beatsauthor/firstbeat
go: finding github.com master
go: finding github.com/elastic/beats/v7 master
go: finding github.com/elastic/beats master
go: finding github.com/elastic master
go: finding golang.org/x/tools latest
...
...
=======================
Your custom beat is now available as /home/ec2-user/go/src/github.com/beatsauthor/firstbeat
=======================
Follow the prompts and the base code for your new Beat will be generated.
Generate dependencies
Now cd into your new firstbeat directory.
cd $GOPATH/src/github.com/beatsauthor/firstbeat
Another error with the documentation is that says to run make setup; doing so will error with the following message:
make: *** No rule to make target `setup'. Stop.
Instead, run:
make update
This generates the Beat config and fields files.
Build your Beat
Now you’re ready to build the default Beat:
mage build
Learn Go!
You’re now in a position to flesh out the Beat and set it to work. You can fetch data from anywhere you need to, modify it as required, and index it in Elasticsearch.
If you’re not familiar with Go, this is a great learning exercise. I hadn’t used it prior to writing my first Beat but there are a lot of resources available to help you get to grips with it.
One big frustration I had was serialising and deserialising JSON. A strongly typed language requires more groundwork than Python or Javascript. There are plenty of other reasons to really like Go but the main draw is likely the reason Go was chosen as the language for Beats; it allows you to write highly concurrent code.