30 July 2021

Here a little How To to create a Warp10 instance with a Zeppelin server. The following configuration is for Warp10-ci docker image. It means there is a read and write token already inserted : 'readTokenCI' and 'writeTokenCI'.

Warp10 and Zeppelin servers use the docker-host local mount point to store configurations and databases.

We need :

  • a docker-compose.yml configuration file to run both server

  • Warp10 configuration files

  • Zeppelin configuration files.

  • the Zeppelin plugin for both Warp10 and Zeppelin

At the end, you can try the configuration with the github: https://github.com/fasar/warp10-dockerfile/tree/W10Plugin-Zeppelin. Take care, the configuration is on the branch: W10Plugin-Zeppelin

Configuring the docker-compose

The docker-compose.yml file is the following:

version: '2'

services:
    warp10-alldb:
        container_name: warp10-alldb
        image: warp10io/warp10:2.8.1-ci
        environment:
            - WARP10_HEAP=6g
            - WARP10_HEAP_MAX=6g
        volumes:
            - ./warp10-rt:/data
    zeppelin:
        # WarpScript interpreter plugin should be configured at: warp10-alldb:9377
        container_name: zeppelin
        image: apache/zeppelin:0.8.2
        environment:
            # Configuration are : https://zeppelin.apache.org/docs/0.8.2/setup/operation/configuration.html
            - ZEPPELIN_ADDR=0.0.0.0
            - ZEPPELIN_PORT=8080
            - ZEPPELIN_LOG_DIR=/zepp-rt/logs
            - ZEPPELIN_NOTEBOOK_DIR=/zepp-rt/notebook
            - ZEPPELIN_CONF_DIR=/zepp-rt/conf
        volumes:
            - ./zepp-rt:/zepp-rt/
            - ./warp10-rt/warp10/lib:/zeppelin/interpreter/WarpScript

Get the Warp10 configuration files

You can get the configuration Warp10 files with :

DOCKERW_ID=$(docker run --rm -d warp10io/warp10:2.8.1-ci)
echo $DOCKERW_ID
# You can explore container with docker exec $DOCKERW_ID ls -l --color /
# Create mandatory files
mkdir -p ./warp10-rt/warp10/logs
touch ./warp10-rt/warp10/logs/warp10.log
touch ./warp10-rt/warp10/logs/warpscript.out
mkdir -p ./warp10-rt/warp10/leveldb
# Wait the time Warp10 configure himsef
sleep 20
# Copy the configuration of Warp10
docker cp -L $DOCKERW_ID:/opt/warp10/etc ./warp10-rt/warp10/
docker cp -L $DOCKERW_ID:/opt/warp10/lib ./warp10-rt/warp10/
docker cp -L $DOCKERW_ID:/opt/warp10/jars ./warp10-rt/warp10/
docker cp -L $DOCKERW_ID:/opt/warp10/macros ./warp10-rt/warp10/
docker cp -L $DOCKERW_ID:/opt/warp10/warpscripts ./warp10-rt/warp10/
# Copy the configuration of Sensision
mkdir warp10-rt/sensision/
docker cp -L $DOCKERW_ID:/opt/sensision/etc ./warp10-rt/sensision/
docker cp -L $DOCKERW_ID:/opt/sensision/scripts ./warp10-rt/sensision/
mkdir ./warp10-rt/sensision/{logs,metrics,queued,targets}
touch ./warp10-rt/sensision/logs/sensision.pid
touch ./warp10-rt/sensision/logs/sensision.log

# Stop and remove the running Warp10 docker container
docker stop $DOCKERW_ID

Edit the configuration file ./warp10-rt/warp10/etc/conf.d/80—​plugins.conf and add the following:

warp10.plugin.zeppelin=io.warp10.plugins.zeppelin.ZeppelinWarp10Plugin

Edit the configuration file `./warp10-rt/warp10/etc/conf.d/ ` and change the following:

egress.clients.expose = true

Get the Zeppeling plugin for both Zeppeling and Warp10

wget https://repo.maven.apache.org/maven2/io/warp10/warp10-plugin-zeppelin/1.0.4-uberjar/warp10-plugin-zeppelin-1.0.4-uberjar.jar
mv warp10-plugin-zeppelin-1.0.4-uberjar.jar  warp10-rt/warp10/lib/

Get the Zeppelin configuration files

DOCKERZ_ID=$(docker run --rm -d -v $PWD/warp10-rt/warp10/lib:/zeppelin/interpreter/WarpScript  apache/zeppelin:0.8.2)
echo $DOCKERZ_ID
# You can explore container with docker exec $DOCKERZ_ID ls -l --color /
# Copy the configuration of Zeppelin
sleep 4
mkdir -p ./zepp-rt/conf/
docker cp -L $DOCKERZ_ID:/zeppelin/conf ./zepp-rt
# Stop and remove the running Warp10 docker container
docker stop $DOCKERZ_ID

Docker magics !

docker-compose up

You will find the IP of the servers with the command line: docker network inspect test-w10_default

You can use WarpStudio with the warp10-alldb IP on port 8081.

You can use Zeppelin with the zeppelin IP on port 8080

Enjoy.

Configure Zeppelin server

Go on http://ZEPPELIN_DOCKER_IP:8080/#/interpreter and configure the interpreter WarpScript with:

  • Connect to existing process

  • Host: warp10-alldb

  • Port: 9377

In case of error

You can remove all the docker jobs with:

docker-compose down && docker container prune && docker network prune

Enjoy Warp10 + Zeppelin

Now you can enjoy with the first script:

%WarpScript

ZLEVELS

'%09' URLDECODE 'char.tab' STORE
'%0A' URLDECODE 'char.newline' STORE

// Create a List of string for the result (a table) with header "ts and val"
[ '#table ts' $char.tab 'value' ] '' JOIN
[ SWAP ]

[ [ 1000 14.2 ] $char.tab JOIN ] APPEND
[ [ 2000 42.12 ] $char.tab JOIN ] APPEND
[ [ 3000 33.10 ] $char.tab JOIN ] APPEND
[ [ 6000 20.7 ] $char.tab JOIN ] APPEND

$char.newline JOIN