- I was focusing only on building an enricher that converts the gitlab raw data to enriched items that can be fed to Prosoul.
- The approach for making this enricher was simple, fetch all the raw items and filter them into a complex nested-dictionary data structure and fill it using the extracted metric values. Later, using this data structure, generate the enriched items. I framed and shaped this approach as we have a definite format of the data item which can be consumed by Prosoul.
- The data format consumed by Prosoul has some important fields. vchrombie/gsoc#7
{
...
...
"metric_id": "issues.numberCreatedIssues",
"metric_name": "Number of Created Issues",
"project": "cms-mobile",
"datetime": "2020-05-27T00:00:00+00:00",
"metric_es_value": 1,
...
}
- So, I started to frame the best format of the data structure which suits the use case. I experimented a lot and came up with an initial version of it.
{
'cms-mobile': {
'created_issue': {
'2020-05-27T00:00:00+00:00': 1,
...
},
'closed_issue': {
'2020-06-27T00:00:00+00:00': 1,
...
}
}
}
- Using this, I made the initial working model of the enricher and submitted it for review. Valerio pointed out some mistakes, there were many faults in the implementation and the data structure. 👾 vchrombie/gsoc#6, PR 👉 chaoss/grimoirelab-elk#892
- So, I spent some more time on making the workflow more clear and less prone to mistakes. I revamped the data structure and finally, after a lot of trials, made another which suits the requirements for the enricher.
{
'project': 'cms-mobile',
'data': {
'created_issue': {
'2020-05-27T00:00:00+00:00': 1,
'2020-08-27T00:00:00+00:00': 2,
...
},
'closed_issue': {
'2020-05-27T00:00:00+00:00': 1,
'2020-08-27T00:00:00+00:00': 5,
...
}
}
}
- I was successfully able to extract the metrics (initial metrics were the number of issues created/opened per day, number of issues closed per day) and built this data structure with all the values. 🎊
- The next challenge was with generating the enriched items which had other fields too like
metric_id,metric_name,metric_description, etc. As these were based on the metric we are generating at that moment, so I made use of the metric id which we used in building the data structure (created_issue,closed_issue). - After a few hours of debugging and fixing the loopholes, the enricher was working perfectly fine and the items (only issues by then) were generated according to the format. After Valerio's suggestion, I tried using them with Prosoul. I made a simple QM with only two metrics (
issues.numberCreatedIssues,issues.numberClosedIssues) and performed an assessment. It was a hit in the first attempt, I was able to complete the assessment and generate reports. The dashboard had some problem, I'll look into it. 📊
- Well, I had sometime before the weekend. So, I tried adding support for merge request related metrics too. I finished that too and I had to refactor the whole code to remove the redundant pieces. vchrombie/gsoc#8
- I also made a dashboard with some visualizations in it so that it would be easy to analyze the items when I'm trying on different projects. 📈
- I'm almost done with this enricher. I'll be working on the gitqm and pipermailqm enrichers in the coming week. 🕺