Tutorial: Setup basic site
A site holds everything MARV knows about your datasets. It contains one or more collections of datasets that can be browsed and queried separately. Here you learn how to setup a basic site with one collection and a couple of example bags.
All code and configuration of this tutorial is included in the top-level tutorial
folder of your MARV Robotics checkout.
Prerequisites
Initialize site
Let’s start by creating a directory that will hold bag files, called a scanroot, and one for our marv site:
$ mkdir site
$ mkdir site/scanroot
Then create the site configuration file site/marv.conf
with the following content:
[marv]
collections = bags
[collection bags]
scanner = marv_robotics.bag:scan
scanroots =
./scanroot
There is one collection that uses marv robotics’ default bag scanner marv_robotics.bag.scan()
, which is looking for bags in the scanroot we just created. We are ready to initialize the site – after activating the virtual environment marv is installed to, and simplifying marv’s log format as we are not interested in times:
$ source venv/bin/activate
(venv) $ export MARV_LOG_FORMAT='%(levelname).4s %(name)s %(message)s'
(venv) $ cd site
(venv:~/site) $ marv init
INFO marv.site Initialized from /home/zaphod/site/marv.conf
Note
Whenever you change your configuration, remember to stop marv serve
(see
below), rerun marv init
, and then start marv serve
again.
docker:
Tell container to run marv init
and install all code in development mode.
$ MARV_INIT=1 DEVELOP=1 ./scripts/run-container site site/scanroot
Serve the site
The MARV backend is implemented using the asynchronous HTTP client/server framework aiohttp and is by default served using the Python WSGI HTTP Server Gunicorn. MARV internally wraps Gunicorn through the MARV cli server
subcommand. If you followed the installation instructions, the page you are looking at is already being served by MARV. When run without any options MARV will try to find a marv.conf
in the current directory. You can also point MARV to serve a specific site using:
CTRL-C
(venv) $ marv serve --config site/marv.conf
...
[2019-08-15 14:48:03 +0200] [8255] [INFO] Starting gunicorn 19.9.0
[2019-08-15 14:48:03 +0200] [8255] [INFO] Listening at: http://0.0.0.0:8000 (8255)
[2019-08-15 14:48:03 +0200] [8255] [INFO] Using worker: aiohttp.GunicornUVLoopWebWorker
[2019-08-15 14:48:03 +0200] [8258] [INFO] Booting worker with pid: 8258
[2019-08-15 14:48:03 +0200] [8259] [INFO] Booting worker with pid: 8259
[2019-08-15 14:48:03 +0200] [8260] [INFO] Booting worker with pid: 8260
[2019-08-15 14:48:03 +0200] [8261] [INFO] Booting worker with pid: 8261
You should see something like the above lines and MARV should be accessible via http://localhost:8000. In case you are running inside a container, make sure you forwarded the correct port.
Note
In the course of this tutorial we’ll keep changing the configuration. For these changes to take effect, marv serve
has to be stopped (CTRL-C) and the site reinitialized with marv init
.
docker:
Restart container.
$ MARV_INIT=1 DEVELOP=1 ./scripts/run-container site site/scanroot
CTRL-C
$ MARV_INIT=1 DEVELOP=1 ./scripts/run-container site site/scanroot
Create user account
MARV requires you to be signed-in to see anything worthwhile. In order to be able to sign-in, you need to create a user first. Let’s make him an admin, so he can also discard datasets from marv. From now on we can continue in the terminal we used to initialize the marv site previously:
(venv:~/site) $ marv user add zaphod
Password:
Repeat for confirmation:
(venv:~/site) $ marv group adduser zaphod admin
After creating the user, you should be able to sign-in using his credentials and be presented with an yet empty listing of the bags collection.
docker: Run commands inside container, after entering it with ./scripts/enter-container
.
$ ./scripts/enter-container
$ marv user add zaphod
Password:
Repeat for confirmation:
$ marv group adduser zaphod admin
Populate scanroot
Let’s give marv two bag files from the mit stata center data set:
(venv:~/site) $ cd scanroot
(venv:~/site/scanroot) $ curl -O http://infinity.csail.mit.edu/data/2011/2011-01-24-06-18-27.bag
(venv:~/site/scanroot) $ curl -O http://infinity.csail.mit.edu/data/2011/2011-01-25-06-29-26.bag
(venv:~/site/scanroot) $ cd -
(venv:~/site) $
After scanning for datasets they will appear in the bag collection’s listing:
(venv:~/site) $ marv scan
INFO marv.collection.bags added <Dataset qmflhjcp6j3hsq7e56xzktf3yq 2011-01-24-06-18-27>
INFO marv.collection.bags added <Dataset vmgpndaq6frctewzwyqsrukg2y 2011-01-25-06-29-26>
The dataset ids, or short set ids, are generated randomly – you will see different ones. If not, watch out for the Heart of Gold. Now, reload the browser. The listing should contain the two datasets. Visiting a detail view, there is no information yet, but after logging in it’s already possible to comment and tag the datasets.
docker: Run commands inside container, after entering it with ./scripts/enter-container
.
Add and run basic nodes
MARV ships with some default nodes. Let’s run these:
(venv:~/site) $ marv run --collection=bags
INFO marv.run qmflhjcp6j.meta_table.dwz4xbykdt.default (meta_table) started
INFO marv.run qmflhjcp6j.summary_keyval.dwz4xbykdt.default (summary_keyval) started
INFO marv.run qmflhjcp6j.summary_keyval.dwz4xbykdt.default finished
INFO marv.run qmflhjcp6j.meta_table.dwz4xbykdt.default finished
INFO marv.run vmgpndaq6f.meta_table.dwz4xbykdt.default (meta_table) started
INFO marv.run vmgpndaq6f.summary_keyval.dwz4xbykdt.default (summary_keyval) started
INFO marv.run vmgpndaq6f.summary_keyval.dwz4xbykdt.default finished
INFO marv.run vmgpndaq6f.meta_table.dwz4xbykdt.default finished
By now the detail summary section is providing minimal information.
Given that we are dealing with bag files, it makes sense to add nodes that extract and display bag meta information:
[marv]
collections = bags
[collection bags]
scanner = marv_robotics.bag:scan
scanroots =
./scanroot
nodes =
marv_nodes:dataset
marv_robotics.bag:bagmeta
marv_robotics.detail:bagmeta_table
marv_robotics.detail:connections_section
detail_summary_widgets =
bagmeta_table
detail_sections =
connections_section
(venv:~/site) $ marv run --collection=bags
INFO marv.run qmflhjcp6j.bagmeta_table.gahvdc4vpg.default (bagmeta_table) started
INFO marv.run qmflhjcp6j.connections_section.yjrewalqzc.default (connections_section) started
INFO marv.run qmflhjcp6j.bagmeta.dwz4xbykdt.default (bagmeta) started
INFO marv.run qmflhjcp6j.bagmeta.dwz4xbykdt.default finished
INFO marv.run qmflhjcp6j.connections_section.yjrewalqzc.default finished
INFO marv.run qmflhjcp6j.bagmeta_table.gahvdc4vpg.default finished
INFO marv.run vmgpndaq6f.bagmeta_table.gahvdc4vpg.default (bagmeta_table) started
INFO marv.run vmgpndaq6f.connections_section.yjrewalqzc.default (connections_section) started
INFO marv.run vmgpndaq6f.bagmeta.dwz4xbykdt.default (bagmeta) started
INFO marv.run vmgpndaq6f.bagmeta.dwz4xbykdt.default finished
INFO marv.run vmgpndaq6f.connections_section.yjrewalqzc.default finished
INFO marv.run vmgpndaq6f.bagmeta_table.gahvdc4vpg.default finished
Reload your browser and check the result.
docker: Run commands inside container, after entering it with ./scripts/enter-container
.
Summary
You initialized a marv site with one collection that looks for bag files in a scanroot directory. You setup Gunicorn to serve your site for development purposes, created a user account for sign-in to the web application, populated the scanroot with some bag files and configured and ran nodes to display meta information about these bag files.
Familiarize yourself a bit with the web frontend (http://localhost:8000). We intend it to be self-explanatory. Please let us know if you have questions.
Now your are ready to write your first nodes Tutorial: Write your own nodes.