Tutorial: Setup basic site

A site holds everything MARV knows about your datasets. It contains one or more collections of datasets that can be browsed and queried separately. Here you learn how to setup a basic site with one collection and a couple of example bags.

All code and configuration of this tutorial is included in the top-level tutorial folder of your MARV Robotics checkout.

Prerequisites

Installation

Initialize site

Let’s start by creating a directory that will hold bag files, called a scanroot, and one for our marv site:

$ mkdir site
$ mkdir site/scanroot

Then create the site configuration file site/marv.conf with the following content:

[marv]
collections = bags

[collection bags]
scanner = marv_robotics.bag:scan
scanroots =
    ./scanroot

There is one collection that uses marv robotics’ default bag scanner marv_robotics.bag.scan(), which is looking for bags in the scanroot we just created. We are ready to initialize the site – after activating the virtual environment marv is installed to, and simplifying marv’s log format as we are not interested in times:

$ source venv/bin/activate
(venv) $ export MARV_LOG_FORMAT='%(levelname).4s %(name)s %(message)s'
(venv) $ cd site
(venv:~/site) $ marv init
INFO marv.site Initialized from /home/zaphod/site/marv.conf

Note

Whenever you change your configuration, remember to stop marv serve (see below), rerun marv init, and then start marv serve again.

docker:

Tell container to run marv init and install all code in development mode.

$ MARV_INIT=1 DEVELOP=1 ./scripts/run-container site site/scanroot

Serve the site

The MARV backend is implemented using the asynchronous HTTP client/server framework aiohttp and is by default served using the Python WSGI HTTP Server Gunicorn. MARV internally wraps Gunicorn through the MARV cli server subcommand. If you followed the installation instructions, the page you are looking at is already being served by MARV. When run without any options MARV will try to find a marv.conf in the current directory. You can also point MARV to serve a specific site using:

CTRL-C
(venv) $ marv serve --config site/marv.conf
...
[2019-08-15 14:48:03 +0200] [8255] [INFO] Starting gunicorn 19.9.0
[2019-08-15 14:48:03 +0200] [8255] [INFO] Listening at: http://0.0.0.0:8000 (8255)
[2019-08-15 14:48:03 +0200] [8255] [INFO] Using worker: aiohttp.GunicornUVLoopWebWorker
[2019-08-15 14:48:03 +0200] [8258] [INFO] Booting worker with pid: 8258
[2019-08-15 14:48:03 +0200] [8259] [INFO] Booting worker with pid: 8259
[2019-08-15 14:48:03 +0200] [8260] [INFO] Booting worker with pid: 8260
[2019-08-15 14:48:03 +0200] [8261] [INFO] Booting worker with pid: 8261

You should see something like the above lines and MARV should be accessible via http://localhost:8000. In case you are running inside a container, make sure you forwarded the correct port.

Note

In the course of this tutorial we’ll keep changing the configuration. For these changes to take effect, marv serve has to be stopped (CTRL-C) and the site reinitialized with marv init.

docker:

Restart container.

$ MARV_INIT=1 DEVELOP=1 ./scripts/run-container site site/scanroot
CTRL-C
$ MARV_INIT=1 DEVELOP=1 ./scripts/run-container site site/scanroot

Create user account

MARV requires you to be signed-in to see anything worthwhile. In order to be able to sign-in, you need to create a user first. Let’s make him an admin, so he can also discard datasets from marv. From now on we can continue in the terminal we used to initialize the marv site previously:

(venv:~/site) $ marv user add zaphod
Password:
Repeat for confirmation:
(venv:~/site) $ marv group adduser zaphod admin

After creating the user, you should be able to sign-in using his credentials and be presented with an yet empty listing of the bags collection.

docker: Run commands inside container, after entering it with ./scripts/enter-container.

$ ./scripts/enter-container
$ marv user add zaphod
Password:
Repeat for confirmation:
$ marv group adduser zaphod admin

Populate scanroot

Let’s give marv two bag files from the mit stata center data set:

(venv:~/site) $ cd scanroot
(venv:~/site/scanroot) $ curl -O http://infinity.csail.mit.edu/data/2011/2011-01-24-06-18-27.bag
(venv:~/site/scanroot) $ curl -O http://infinity.csail.mit.edu/data/2011/2011-01-25-06-29-26.bag
(venv:~/site/scanroot) $ cd -
(venv:~/site) $

After scanning for datasets they will appear in the bag collection’s listing:

(venv:~/site) $ marv scan
INFO marv.collection.bags added <Dataset qmflhjcp6j3hsq7e56xzktf3yq 2011-01-24-06-18-27>
INFO marv.collection.bags added <Dataset vmgpndaq6frctewzwyqsrukg2y 2011-01-25-06-29-26>

The dataset ids, or short set ids, are generated randomly – you will see different ones. If not, watch out for the Heart of Gold. Now, reload the browser. The listing should contain the two datasets. Visiting a detail view, there is no information yet, but after logging in it’s already possible to comment and tag the datasets.

docker: Run commands inside container, after entering it with ./scripts/enter-container.

Add and run basic nodes

MARV ships with some default nodes. Let’s run these:

(venv:~/site) $ marv run --collection=bags
INFO marv.run qmflhjcp6j.meta_table.dwz4xbykdt.default (meta_table) started
INFO marv.run qmflhjcp6j.summary_keyval.dwz4xbykdt.default (summary_keyval) started
INFO marv.run qmflhjcp6j.summary_keyval.dwz4xbykdt.default finished
INFO marv.run qmflhjcp6j.meta_table.dwz4xbykdt.default finished
INFO marv.run vmgpndaq6f.meta_table.dwz4xbykdt.default (meta_table) started
INFO marv.run vmgpndaq6f.summary_keyval.dwz4xbykdt.default (summary_keyval) started
INFO marv.run vmgpndaq6f.summary_keyval.dwz4xbykdt.default finished
INFO marv.run vmgpndaq6f.meta_table.dwz4xbykdt.default finished

By now the detail summary section is providing minimal information.

Given that we are dealing with bag files, it makes sense to add nodes that extract and display bag meta information:

[marv]
collections = bags

[collection bags]
scanner = marv_robotics.bag:scan
scanroots =
    ./scanroot

nodes =
    marv_nodes:dataset
    marv_robotics.bag:bagmeta
    marv_robotics.detail:bagmeta_table
    marv_robotics.detail:connections_section

detail_summary_widgets =
    bagmeta_table

detail_sections =
    connections_section

(venv:~/site) $ marv run --collection=bags
INFO marv.run qmflhjcp6j.bagmeta_table.gahvdc4vpg.default (bagmeta_table) started
INFO marv.run qmflhjcp6j.connections_section.yjrewalqzc.default (connections_section) started
INFO marv.run qmflhjcp6j.bagmeta.dwz4xbykdt.default (bagmeta) started
INFO marv.run qmflhjcp6j.bagmeta.dwz4xbykdt.default finished
INFO marv.run qmflhjcp6j.connections_section.yjrewalqzc.default finished
INFO marv.run qmflhjcp6j.bagmeta_table.gahvdc4vpg.default finished
INFO marv.run vmgpndaq6f.bagmeta_table.gahvdc4vpg.default (bagmeta_table) started
INFO marv.run vmgpndaq6f.connections_section.yjrewalqzc.default (connections_section) started
INFO marv.run vmgpndaq6f.bagmeta.dwz4xbykdt.default (bagmeta) started
INFO marv.run vmgpndaq6f.bagmeta.dwz4xbykdt.default finished
INFO marv.run vmgpndaq6f.connections_section.yjrewalqzc.default finished
INFO marv.run vmgpndaq6f.bagmeta_table.gahvdc4vpg.default finished

Reload your browser and check the result.

docker: Run commands inside container, after entering it with ./scripts/enter-container.

Summary

You initialized a marv site with one collection that looks for bag files in a scanroot directory. You setup Gunicorn to serve your site for development purposes, created a user account for sign-in to the web application, populated the scanroot with some bag files and configured and ran nodes to display meta information about these bag files.

Familiarize yourself a bit with the web frontend (http://localhost:8000). We intend it to be self-explanatory. Please let us know if you have questions.

Now your are ready to write your first nodes Tutorial: Write your own nodes.