Extend developer guide

[Woseseltops]

It's still unclear, but there's a chance @rem0g will work on this repository for a bit, which is a good excuse to extend our developer guide https://github.com/Signbank/Global-signbank/wiki/Developer-guide .

@Jetske , when you were getting started, what were things you struggled with that are not in this guide?

[Jetske]

Well that's hard to say as I had no experience with anything more than static websites. So I guess just getting used to Django MVT structure and how to work with a database and migrations, and user permissions were very confusing (changing group/user permissions in admin vs. on the manage datasets page).

[susanodd]

Semi-related, Manage Video Storage is deployed:

https://signbank.cls.ru.nl/datasets/checks/5

(Related as it allows to see how the videos are stored in the file system.)

There's a button to it on the Manage Datasets page for the specific dataset.

[Woseseltops]

Okay, I've reorganized and extended https://github.com/Signbank/Global-signbank/wiki/Developer-guide . I think what's missing is what you do if you want to start with an empty database. @susanodd , can you explain this to me, so I can add it to the page?

[susanodd]

Do we assume that they will get this from you? Or will it be the one that @vanlummelhuizen is making? As far as I know his isn't ready yet.

• It should run just by changing these settings for the "vacuum-empty" one:

DEFAULT_DATASET = 'TestDatasetMicha'
DEFAULT_DATASET_ACRONYM = 'tstMH'
DEFAULT_DATASET_LANGUAGE_ID = 1
DEFAULT_DATASET_PK = 1

• The empty database only has the tstMH dataset. So the settings need to be modified to that dataset in order to run the tests. Some of the code makes use of those settings, since there always needs to be at least one dataset.

• You need to have an account for yourself (the admin can be yourself, with staff or superuser permission set in the admin.

• the secret key needs to be in the server specific settings!

• The name of the dataset can be changed in the setting, but it also needs to be changed using the admin. The settings need to match an actual dataset. This will also cause the folders that make use of the dataset name to be changed.

• It is necessary to make at least one dataset public or people not logged in can't do anything at all.

• Before running the tests, you need to run the virtual environment command:

python bin/develop.py create_test_db

• all the necessary folders need to exist with the correct permissions (as can hopefully be seen in the ansible)

• pages can be hidden by unselecting "publish" for the page in the admin. The pages shown there should exist because that is all the urls that are shown in the menus. You can set group permissions for the pages. Some urls do not have any corresponding page in the admin. They are either linked to by other pages, or used for implementing internal data requests, or are part of the API, or are just hidden. It is advised not to randomly delete anything. PyCharm catches things at compile time and basically the server just breaks if any urls are missing. Only the "about" pages have text in the admin; the others have templates. Only modify the "about" pages in the admin. These are multilingual.

• it is easy to use Signbank locally for development using PyCharm, using a local copy of the database (empty if needed). This also works on MacOS and the local "run server" can even run in iCloud.

How to generate an empty database (roughly, probably needs to be rewritten)

If you are going to send a "vacuum-empty" database, then you could add an account for the recipient to it before you send it. Or we could just generate a new empty database after they have an account already on Signbank.

There are three commands for generating an empty database like the one I made. The data is removed except for that necessary to run Signbank. Then the datasets are removed, except for that one. Then the users are removed, except for us developers.
Those commands can be changed to keep different users. It did not work to remove users in the Admin. It is done in the virtual environment in a command. There are too many related objects that ended up causing problems in the Admin.

It was too complicated to preserve one dataset without emptying it as well. That is because when you delete data, it records the delete operations in the database. So it ended up that they just needed to be "vacuumed" as is done for tests. Not do delete operations on all the glosses. (The delete operations cause signals to save data about deletion.)

[susanodd]

@Woseseltops this one is obsolete:

https://github.com/Signbank/Global-signbank/wiki/How-to-batch-upload-videos-for-signbank.science.ru.nl

Now it's either via the Manage Datasets -> Manage Media page or using the API, all by uploading a zip file.

I added a couple of lines to the start of that page. It ought to be rewritten because it won't actually work as described. There didn't use to be as many permission checks. There is also a container now.

The zip file has the same structure as is described in that wiki page.

For the API, a second zip structure is also available, that the videos are just the gloss ID.mp4

The videos are not converted and assumed to be in mp4, as per request of UvA. (Video conversion was sometimes failing, so it was turned off.) The original Management command checked the format. (That can be restored if needed.)