-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#1270: Signbank empty database. #1316
base: master
Are you sure you want to change the base?
Conversation
Modified permissions. Command to empty tables, command to remove users except for developers.
Put permissions on database back to what they were.
@Woseseltops this is on signbank-test. If you look at the admin you can see what the contents is. I need to make a migration to remove the extra interface languages. All of the Field Choices and Handshapes are in the database. It won't work without them. Or at least some. (They can't be empty.) I suspect Django will make a migration as soon as we update the model translation languages settings. We probably need some basic data in tstMH. Just because it's not possible to test everything otherwise. |
@vanlummelhuizen can we make a conditional migration to remove the model translations that are not needed? (I write this in general although this is for the VGT site. #1270 ) |
@vanlummelhuizen I needed to alter the permissions in the Dataset Select page when I started using the empty signbank. It did not want to display anything. Then I went ahead and changed them elsewhere to be the same. But then the tests didn't work, so I changed the ones in the tests (manage dataset tests). (Although I had to remove the checks on "messages" returned because it wasn't showing any messages. So it merely checks the status of the operation now.) Perhaps you can check out the permissions? [SEE CODE, ALL TESTS WORK NOW] I removed users in a command, and I left more tables in the database (create_empty_db), based on create_test_db but it didn't work with the users removed, since I couldn't login anymore, so I left the development team, plus the future user of the VGT site.) I removed datasets by hand. It didn't work to remove users by hand. (Some error about a content type on some users, internal to Guardian.) The removal of the datasets will be needed to be added to the create_empty_db command. Maybe we just leave the test dataset? Plus there likely need to be some "start" data in the database because it's not possible to check that everything works without any glosses. It would be good if we had some "representative data" in the test dataset. I made the one gloss I created in tstMH be public. |
I left the Handshape, Field Choice, Semantic Field, and Derivation History choice tables in the database. |
[CODE DETAILS RELATED TO PERMISSIONS, THIS SHOWS DETAILS, SKIP THIS] In the
results in this kind of permission:
NOT the permissions that look like "change_dataset". So there are different looking permissions floating around. |
[SKIP THIS COMMENT, SCROLL DOWN TO NEXT, READ CODE TO SEE FIXES NEEDED] I have no idea how to get it to work. However, it is possible to create a gloss in the interface. I don't know if the error that the test does not work is related to permissions or not. URGH!!!!!! [DESCRIPTION OF BUG AND FIX]
So a totally different permission than in other parts of the code. Now I have no idea how to consistently code the permissions tests. I'll make it so the code respects the tests, since the test actually catch broken code. So above, "change_permission" on the dataset was needed, but here, "change_dataset" permission was needed. |
ALL OF THE TESTS ARE SUCCESSFUL !!! |
@Woseseltops if you browse the changes to files, most are permissions. I have no idea WHY two different kinds of permissions are needed. |
Fixed permissions that were causing to fail.
Many of the changes here are also in #1282, although not completely identical. They differ on this: you have When I dive directly into the database ( sqlite> select * from auth_permission where codename like '%permission%';
id|content_type_id|codename|name
1|1|add_permission|Can add permission
2|1|change_permission|Can change permission
3|1|delete_permission|Can delete permission
142|44|add_userobjectpermission|Can add user object permission
143|44|change_userobjectpermission|Can change user object permission
144|44|delete_userobjectpermission|Can delete user object permission
145|45|add_groupobjectpermission|Can add group object permission
146|45|change_groupobjectpermission|Can change group object permission
147|45|delete_groupobjectpermission|Can delete group object permission
221|1|view_permission|Can view permission
282|45|view_groupobjectpermission|Can view group object permission
283|44|view_userobjectpermission|Can view user object permission Then see what ContentType these refer to: sqlite> select * from django_content_type where id in (1,44,45);
id|app_label|model
1|auth|permission
44|guardian|userobjectpermission
45|guardian|groupobjectpermission As you can see none of the permissions with permission in the codename refers to Dataset. Conclusion: The codename So, I think all |
@vanlummelhuizen Yes, it looks like that. But some of the code does not run at all unless the kind with On the initial startup, it would not show any datasets at all in the Select Dataset page. I had to change it to the |
@susanodd I can only imagine that this is coincidental, probably because the current user is a Permissions with |
@Woseseltops can you check that there isn't anything signbank-specific to Radboud in the empty database? I found one instance of "site" that is the url of signbank. (That is inside the database, not a setting.) If you use the new Amazon-tool, can it be with the empty database to check it out? |
I still don't know why the select dataset page was not working. I need to figure out if too many things were deleted to make it empty, causing it to not work. |
What about the Manage Dataset page? In the empty database, I noticed that it had made me owner of something. I had removed all the owner users of NGT, which is empty but not deleted. |
Okay, so I switched to this branch with a "normal" database. I'll set the permissions back to what they were. |
I set the permissions back, although they now match the other issue about the permissions. As before all the tests using the empty database pass. |
@Woseseltops since you have experience with a totally different signbank branch, ASL, perhaps you can decide what to do for VGT. Once they start modifying the settings, it ought to remain possible to pull from the master branch. Also the migrations for the model translations will need to be conditional, and also on master. In the empty database (signbank-test) the About pages have content in the Pages model. |
I added more tables to the KEEP list as discovered by @vanlummelhuizen. (For some reason this did not affect the tests on the normal database, but they need to be explicitly in the KEEP list when creating an empty database.) Then performed the steps above. (#1316 (comment)) All of the tests work now. The empty database is on signbank-test. The name of the dataset can be changed to something generic. I changed the server specific to have dataset 1 tstMH be the default. And of course migrating Chinese, Arabic, and Hebrew out. (#1320) @Woseseltops communicate however you did this before with the empty database. I like the method of @vanlummelhuizen much better of course. But I'm not able to develop that myself, I think it should be tested on the Amazon temporary servers. |
It is here: /vol/tensusers/mhulsbosch/signbank/signbank.db
I don't think so. The handling of pages not in a urls.py somewhere is done by PageFallbackMiddleware in https://github.com/Signbank/Global-signbank/blob/master/signbank/pages/middleware.py: when the normal flow results in a 404 it tries to find the page in Pages.
This is a good question. I assumed editing early migrations is possible. These migrations are already in the Global-Signbank instance so they will not be applied again there. Also, I tried to make sure the end result is not different from the database we already have in terms of structure/schema. If you find anything in my edits that makes you think otherwise, please let me know.
Good point. I will look into that. |
I added fixtures for Handshape, SemanticField and DerivationHistory. Again, it is on branch make_installable.
Would be cool to see a fresh install on AWS this way. I first need to find out how that works. |
@vanlummelhuizen I just put your signbank.db (empty as on tensusers) onto signbank-dev. I revised the settings in the server_specific.py file (for the dev server only) as shown above for the dataset settings. I'm not sure how I can login now? It's on signbank-dev. Check it out! |
You need to create a super user first: https://docs.djangoproject.com/en/5.0/intro/tutorial02/#creating-an-admin-user |
Okay, quite a lot of tests fail. (!!) Many are that the Dataset does not exist. In the pull-down menus, lots of pages show up in the pull-downs in spite of an anonymous user not being able to see them. (The extra group permissions you set in the admin for the pages, hides them from the menus if the user is not logged in.) |
@vanlummelhuizen could you have a look at the settings files? Global-signbank/signbank/video/models.py Line 17 in a18988c
Those things aren't defined in base. Here is a setting that is not related to anything: Global-signbank/signbank/settings/base.py Line 199 in a18988c
|
Summary of a conversation with @susanodd :
Optional functionality: also have this tool/command create a page with copyright info |
Do you mean what is done in make_installable?
When creating a database from scratch as in make_installable, a default dataset is created: "Your dataset", YDS. See https://github.com/Signbank/Global-signbank/blob/master/signbank/dictionary/migrations/0027_auto_20180803_1340.py But it would be best if the tests are creating a test-dataset themselves, independent of the contents of the (test-)database. |
I did some testing and one problem was a typo in the DEFAULT_DATASET setting of the server_specific.py file I gave earlier. It should be: DEFAULT_DATASET = 'Your dataset' (lower 'd' of dataset) |
Yes, some of the older tests use NGT. (Kind of dangerous.) Some do create a new dataset. I agree that this needs to be for all. I an work on that. @vanlummelhuizen you're creating a default language and a sign language? |
To solve quite some tests I add |
That will also solve many problems in the templates, since all the little annotation dictionaries map the 2-char to a string. |
There is one test that still fails: Global-signbank/signbank/dictionary/tests.py Line 623 in 103da9c
I think this is because the test expects something that will not happen: the test sends a form (stage 2) with changes that were pretended to be extracted from a CSV in the stage before (stage 1). However, the thing that is tested ( Global-signbank/signbank/dictionary/tests.py Line 728 in 103da9c
views.py the text 'No changes were found.' is only used in stage 1, not stage 2. See Global-signbank/signbank/dictionary/views.py Lines 1499 to 1502 in 103da9c
Global-signbank/signbank/dictionary/views.py Line 1489 in 103da9c
@susanodd Do you agree and if yes, do you have an idea to solve this? |
@vanlummelhuizen @Woseseltops I copied the empty database I had created onto my tensusers account.
I needed to put the animation code on signbank-test because the CORS isn't set up properly on signbank-susan and there is no ajax lookahead. (Some bug it thinks that it is on a different site.) |
@vanlummelhuizen that's weird about the test. I will need to check it out. For some reason it works on the normal databases. |
The test is testing whether there is one translation language. So probably something else is going on. Global-signbank/signbank/dictionary/tests.py Lines 698 to 704 in a18988c
Global-signbank/signbank/dictionary/tests.py Lines 723 to 728 in a18988c
Yes, something needs to be modified for the tests because it assumes the dataset is NGT and has two languages. (That's not generic enough.) [BLAH BLAH] It could even be that the template area for messages has become hidden or behind other stuff. I could have modified something for dark mode, or else for showing the scroll bar and not noticed. (Those are things that can hide the messages, also the "affix". I have no idea if that is relevant here, since it's a virtual template. That's why the tests just look to see if the string is contained, since they don't parse the template.) The tests also assume the messages are in English. So if your own account that is running the test has a different language set up, that will affect the language of the messages. That the template has the text "No changes were found" is different than that the test also has this. Some of the CSV routines have a bail-out. So in the gloss update, if a column is not processed by the code, say the user tries to update the senses (this is not available, only creation is available), then the code skips those columns. The routines for the API that do constraint checks or need to check existence of field choices, also report back about this when the operation cannot be performed. This is also translated. We don't have any tests for the API. But the constraint tests are needed. |
I made changes so that creating translation fields is done dynamically: 18a97e2 so that it also works with the following language settings LANGUAGES = (
('en', gettext('English')),
('nl', gettext('Dutch')),
('zh-hans', gettext('Chinese')),
('he', gettext('Hebrew')),
('ar', gettext('Arabic')),
)
LANGUAGES_LANGUAGE_CODE_3CHAR = (
('en', 'eng'),
('nl', 'nld'),
('zh-hans', 'zho'),
('he', 'heb'),
('ar', 'ara'),
)
INTERFACE_LANGUAGE_SHORT_NAMES = ['EN','NL','官话','עִברִית','عربى']
MODELTRANSLATION_LANGUAGES = ['en','nl','zh-hans',
'he', 'ar'
] All tests pass except |
On a normal test database, even including the migrations for animations, this is what that test prints:
That is the first branch of the code. Not the branch that looks for No changes found. Can you put the newest empty database on your tensusers again? Is this set?
I tried monkeying with the various settings to try to force it to get an error. If you set the following to a constant "1" instead, then it gives the above error as you are getting
So somehow this must be wrong. |
@vanlummelhuizen Okay, remove the "else" clause of that test, but keep the "if" clause.
Remove the "else" where it looks for "No changes found." You're right, even if there is only one translation language (I tried ASL), it won't work for that. I suspect that is a remnant of old code that did additional tests on the input file. Once upon a time, it kept checking the dataset column again and again because it was allowed to modify multiple datasets. |
Done.
Yes, it is taken from
I did the following: diff --git a/signbank/dictionary/tests.py b/signbank/dictionary/tests.py
index e84517d7..64de0252 100644
--- a/signbank/dictionary/tests.py
+++ b/signbank/dictionary/tests.py
@@ -699,9 +699,6 @@ class ImportExportTests(TestCase):
if count_dataset_translation_languages > 1:
print('More than one translation language, attempt to update a lemma translation')
self.assertContains(response, 'Attempt to update Lemma translations')
- else:
- print('Only one translation language, no changes found')
- self.assertContains(response, 'No changes were found.')
# Prepare form data for linking to SEVERAL EXISTING LemmaIdgloss + LemmaIdglossTranslations
@@ -723,9 +720,6 @@ class ImportExportTests(TestCase):
if count_dataset_translation_languages > 1:
print('More than one translation language, attempt to update a lemma translation')
self.assertContains(response, 'Attempt to update Lemma translations')
- else:
- print('Only one translation language, no changes found')
- self.assertContains(response, 'No changes were found.')
def test_Import_csv_new_gloss_for_lemma(self): and now there is no failure. But then again, I just removed the failing test. |
@vanlummelhuizen it fails because it thinks the number of translation languages is not 2. |
@Woseseltops see here The empty database is available as described there. This is not a pull request per se. This does not need to be merged, so you don't need to review the code. |
Modified permissions. Command to empty tables, command to remove users except for developers.
[ORIGINAL, SKIP THIS]
The empty database is created with the same name as the test database, it does not overwrite the original database.
Then to use the new empty database instead, you need to copy it to overwrite the real signbank.db (on signbank-test this was done.) It is intended that other developers (the VGT-ers will start out with the empty database).
Then a new test database need to be made (the create_test_db throws away more table contents.)
This is a TEST DATABASE
This is on signbank-test with an EMPTY database. (Well, nearly empty, the interface translations are still there..)
ALL TESTS SUCCEED!!
I was able to create a gloss in tstMH.
NEXT
Remove interface languages. This will require a migration, so this branch won't be compatible with other branches.
The settings for this server will need to change the model translation settings.
I'll see if this can be a script so it can be used optionally.
Some of the permissions probably need to revert to what they were. It was a problem at first because there was no data. The Dataset pages did not show any of the datasets and the original permissions did not work.
Specific to VGT, see if they want to retain the VGT dataset glosses.
This would facilitate sharing in the future since the gloss IDs would be retained.
One of the issues of @ocrasborn was to add extra relations to other sites (#93)
[READ THIS, UPDATE PER 02-09-2024]
After the new permissions (#1281, #1282), the empty database has zero permissions to choose from in the Admin.
All the "change_dataset, view_dataset, etc have disappeared from the permissions. It's not possible to grant users permissions in the admin because there are no permissions.
Does the "vacuum" command do this? Or are too many tables erased, but not shown in the log of tables erased?