Quick tip: Using Git with NiFi Registry in Docker

Apache NiFi is a great tool for handling data flows, however the flow development lifecycle has been slightly challenging.

The recent release of NiFi Registry, a sub-project to provide shared resources across instances of NiFi, initially provides the capability to manage versioned flows. As of version 0.2.0, NiFi Registry added support for persisting flow snapshots to Git, making it very compelling!

In this post, we’ll see how to set this up for use when developing NiFi flows in a dockerised environment.

There are a couple of tricks to get this working:

1. Using an external volume for the flow snapshots (initialised as a Git repo or NiFi Registry will fail to start)
2. Externalising the configuration for GitFlowPersistenceProvider

Setting up NiFi Registry in Docker

Pre-reqs

• A working Docker installation
• NiFi 1.5+ (necessary for NiFi Registry integration)

Steps

Note that operating system commands (*nix-based) appear as follows:
echo

Getting the container image

The Apache NiFi project publish NiFi Registry docker container images on Docker Hub. Let’s pull this:

docker pull apache/nifi-registry:0.2.0

Create a directory for the config

Subsequent steps assume that your current working directory is as follows:
mkdir nifi-registry; cd nifi-registry

Create the flow snapshots directory

This will be used as a volume by the container:
mkdir flow_storage; pushd flow_storage

It must be initialised before use otherwise NiFi Registry won’t start and the container will be terminated.

This can be initialised using either git init or by cloning an existing repo (see the admin guide). For simplicity, we’ll execute the git init command:

git init
popd

Git Flow Persistence configuration

The default nifi-registry.properties points to ./conf/providers.xml, so we will mount this file from outside the container.

You’ll need to create providers.xml with the following content in the current working directory (e.g. nifi-registry):
vi providers.xml

Paste this XML (and then save the file), which configures the GitFlowPersistenceProvider to use the externalised volume:

<providers>
    <flowPersistenceProvider>
        <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
        <property name="Flow Storage Directory">/flow_storage</property>
    </flowPersistenceProvider>
</providers>

There are more optional configuration options, such as the ability to push from NiFi – see the configuration for GitFlowPersistenceProvider in the admin guide.

Create the container

docker run --name nifi-registry -p 18080:18080 -d -v $PWD/flow_storage:/flow_storage -v $PWD/providers.xml:/opt/nifi-registry/nifi-registry-0.2.0/conf/providers.xml apache/nifi-registry:0.2.0

Create a bucket

NiFi uses buckets within the registry to store flows, so it make sense to create one first. This is achieved using the NiFi Registry administration UI, available on http://localhost:18080/nifi-registry.

Bucket creation is available under settings (see this section of the getting started guide).

Connect NiFi to the NiFi Registry

The documentation shows how to connect NiFi to the NiFi Registry.

If you are running both NiFi and NiFi Registry in docker containers, then you will need to use the IP address of the host machine and the exposed port number. The host IP address can be seen as the Gateway address in the output of docker inspect nifi-registry (e.g. 172.17.0.1).

Summary

If you’ve followed this quick tip through, then you should now have the ability to version flows, commit them to git and push them (external to NiFi).