TL;DR
Nordvision agreed to fund a programmer’s exchange between DR in Denmark, Rúv in Iceland and Yle in Finland. The exchange started off with me going to Iceland for two weeks. We collaborated on a meta-data project for Drupal called Yild by preparing for migration of Rúv’s terms to a Yild-compatible format and by writing a new Spotify provider module for it. We feel a lot of good came out of the project and are eager to continue it when someone from Iceland comes to Yle for a similar exchange in Q1/2017.
The project
Three nordic broadcasting companies: Svenska Yle, DR and RÚV have received funding from Nordvision to undertake a two week developers’ exchange between the companies. This text documents my experiences and findings from working with RÚV for two weeks.
Where it all started
The project first started in Amsterdam in 2014, when we met Helgi Páll Þórisson from RÚV at Drupalcon, a big convention for enthusiasts and professionals of the content management system Drupal. Both Icelandic RÚV and Finnish Yle were already using Drupal and we have the unique common ground of both being a publicly funded public service company with the same challenges and demands on the web platform.
We discussed the merits of Drupal and the unique problems we were facing. Quite soon the idea of collaborating was born. Coincidentally, there is already an organisation for such collaboration: Nordvision, which gathers representatives from all Nordic public service companies around the same table in a partnership.
Later, we would run into Helgi several times at various Drupal events and our visits to other Nordic public service companies. The idea of working together started to grow and pretty soon a developer exchange was proposed: One developer from Yle would work in Iceland at RÚV for a couple of weeks and later on, someone from RÚV would come work with Yle in Finland. Danish DR also expressed interest in being part of this program and time tables and other circumstances permitting, a similar exchange involving DR is planned for the future.
The exchange
After the summer of 2016 things started coming together: Nordvision approved the request for funding and pretty soon a time table was drafted: I would go to RÚV for two weeks in November 2016. At Yle and RÚV we started planning the details of the exchange.
We decided the developer exchange would focus on metadata, since Yle already uses a tagging system for Drupal that connects tags to an official source, such as Wikidata. This module or plugin for Drupal is an open source project called “Yild” (for Yle integrator for linked data). We agreed that our main focus would be to determine whether the same system could be used at RÚV and if so, to start implementing it.
The team
RÚV has a team of highly skilled web developers wrestling with a multitude of technologies, such as Swift, Python, PHP, to name a few. The team is cooped up in a far away corner of the building, fairly well isolated from the rest of the workforce. This is typical and unsurprisingly the same arrangement we use at Yle: Developers need to be isolated from outside disturbances and any interaction should ideally be channeled through one intermediary, a team leader or a scrum master.
What’s impressive is that two of the developers, Hannes and Jón, had only been at RÚV for a month when the exchange started and they were already highly productive. Hannes showed me the Apple TV app he was working on and told me the first version was already in production! Most of all, what stayed with me, was how fast RÚV had managed to make their new employees productive. Something many other companies could take note of.
Volunteer work
One thing that sets our organisations apart is that many developers at RÚV seem to be involved in some kind of extracurricular and in many cases volunteer activities:
Jón runs an non-profit initiative at the web address Koder.is to teach young kids programming. This is done with a Raspberry Pi computer that is able to run the popular game Minecraft in such a way that it can be programmatically manipulated using Python. This means the kids can learn to build houses by programming logic instead of the traditional approach of walking (in the game) to a spot and placing a building block there. I was impressed by how visual this process of learning the basic tools of programming is.
Gunnar is deeply involved in an initiative first introduced by BBC to give 9000 Icelandic children their own one-circuit-board computer, the so called Micro:bit. This collaboration between Microsoft and BBC gives the children an app that runs on their mobile. Using Bluetooth, they can first design the program in the app and then transfer it to the Micro:bit for running. I started Googling the Micro:bit and apparently there have been some Workshops at Kiasma, our modern museum, in Finland to introduce it to the public. In Iceland, this project has already evolved into an Icelandic web page developer by RÚV that contains dozens of exercises for those who want to learn how to program. I’m hoping to see similar initiatives in Finland as well. Maybe Yle could look into something similar or even collaborate with RÚV?
Team management at RÚV
The first thing that catches your eye at the web developers’ office at RÚV is the gigantic kanban board in the middle of the room. Almost all the sticky notes are in the “ready for testing” or “done” column.
Like most software companies these days, the chosen methodology is a sort of customised agile model with daily meetings and a desire to achieve synergy through shared problems and tasks. Whenever I told the team about a problem I was facing during a daily meeting, someone was always eager to offer a solution: “Icelandic always has problems with character encoding because of our many special characters”, Gísli Þórmar Snæbjörnsson offered helpfully.
Whenever one team-member recognised another member was working on something they had previous knowledge about, they were quick to offer assistance. I believe one big reason, perhaps the biggest, for implementing agile in a team is to move from working as individuals to working as a team and the Icelandic developers’ team was clearly a team. I often saw three or four people flock around the same screen, speaking enthusiastically about some new idea of problem they were having. In contrast, at Yle I usually see people keep to their own screen and I think this is something we could learn from the Icelandic team: to be more brave in seeking out other people’s ideas and thoughts.
Kick-off
We started the project with a kick-off meeting the very first day. Everyone on the web-team attended and it was decided I would primarily work with Jón, Hilmar and Helgi. These were all people with prior Drupal experience or, in Jón’s case, new employees who needed to familiarize themselves with the platform.
As part of the Kick-off meeting I demonstrated the Yild module to the whole team. Yle has been using the module for tagging content for several years and it is this experience and expertise that puts us in a position to help RÚV take their tagging to the next level.
Real structured data opens up exciting possibilities: For one, Google will start to know what the content is about, which will help the search engine ranking. Using a common vocabulary, there is also the possibility of linking content from different sites. If Svenska Yle writes about the Eurovision Song Contest, we could link to RÚV’s content about the same subject.
Code review
The next day the Drupal developers gathered for a more technical meeting: The code review and presentation of the Yild module. I explained the structure of the project and walked the team through the programming patterns in use.
Yild is a multi-part module where a core module provides the basic functionality. The core then calls upon so called provider modules to fetch tag suggestions and details from various external sources. Each external service has its own provider: Wikidata, Geonames, Wikipedia, Musicbrainz, etc.
As a result of the collaboration with Rúv, a stable version of Yild was published. You can read more about it and test it yourself at drupal.org.
Writing your own provider module is a relatively trivial task if they have a well-documented api to fetch data from. As a result from the code review, Hilmas became enthusiastic about writing a provider module for Spotify since that was something he needed for another project. By making Hilmar a co-maintainer for the Yild project, it would also be possible for him to contribute to the Yild project itself with a new component that added functionality: the ability to tag articles and other content with any artist, song or album that is on Spotify, complete with l descriptions and album covers.
Jón would in turn work on a provider for tagging with Icelandic politicians (an idea that was later scrapped and replaced by a provider for looking up Icelandic companies).
Helgi and I would work on researching how to best migrate from the current tag system on RÚV’s Drupal site. It was suggested that a migration tool for trying to match existing tags against a preferred provider would be useful for the whole Drupal community, so that would be my main focus for the next weeks, along with trying to match RÚVs tags against Wikidata to find out how many I could actually match.
The Spotify Provider
Hilmar was quick to finish his first draft of the Spotify provider module. After committing it to the version control repository at drupal.org, I read the code, tested the module and offered my comments and insights.
In just a few days, Hilmar went from not having seen the project before to contributing a provider module that was added to drupal.org for the benefit of the whole world. This felt like a big win for the project. Two Nordic public service companies had actually worked together on the same code and then released it as open source.
In the meantime, Jón discovered a bug that broke an important part of the functionality of Yild on cleanly installed systems. He quickly fixed the bug, made a patch, which was added to drupal.org and credited to him. We now had two Icelandic web developers from RÚV with official contributions to a Drupal project started by Finnish Yle.
Greynir
One thing to keep in mind is that the quality of your structured data can only be as good as the data you get from the provider. There was always a concern that Wikidata wouldn’t be good enough in Icelandic to actually be used with RÚV’s content.
During one of our first meetings, Jón mentioned there was a very exciting open source initiative for cataloging and structuring content on Icelandic news sites called Greynir. Greynir is an NLP (Natural Language Processing) parser for the Icelandic language. It indexes several sites and breaks the content into a tree structure, distilling the content into specific terms, such as names of people, places, job titles etc.
Greynir is developed by Vilhjálmur Þorsteinsson, a veteran programmer and enthusiast of deep learning and artificial intelligence. Jón set up a meeting to sit down with mr Þorsteinsson to discuss the possibility of turning Greynir into an API for use with Yild.
One way we use Yild at Yle is to fetch term suggestions based on the content. This requires some kind of analysis service that reads the text and offers suggestions on terms that are relevant for the content. There is currently no such service for Icelandic that we know of, but Greynir is very close and with a small amount of work might become such a service. This would help immensely with the tagging of Content at RÚV.
During the meeting mr. Þorsteinsson said it would be quite easy to give Greynir an Api structure and it’s an angle we are excited to move forward with.
Read more about Greynir and try it out yourself at their website.
Migration tools
The biggest obstacle for implementing Yild on RÚV’s site was always migrating from the current, unverified tags to something actually connected to Wikidata.
I set out to write a migration tool that would go through all 2000 tags used on RÚV’s website and try to match them with their Wikidata counterparts. After some challenges with character encoding – of which I had been duly warned – and a lot of translation help from my fellow team members and Google translate the tool was finally ready. Out of more than 2000 terms approximately half could be matched against Wikidata. The rest of the terms were saved in a report delivered to Helgi for further inspection.
The way to move forward is now to analyse the migration data and see if it’s good enough to work with. The plan is to run the migration script on RÚV’s test servers and then on the production servers, after which moving to use Yild is simply a matter of enabling the module and teaching the editors to use it.
One particularly amusing problem was when I tried to match the Icelandic word for heavy rock, Þungarokk and kept getting a totally unrelated word from Wikidata that means “roofing sheet” (the sheets of corrugated steel used as roofing on some houses). The word was bárujárn. After a while I asked a team member if he had any idea why this might happen and he laughingly told me it was actually a term used by metalheads as a synonym for heavy rock since back in the beginning of the musical genre they thought the music sounded like hitting sheets of corrugated steel with a hammer. Sometimes the code works perfectly and it’s just the interpretation of the data that is wrong.
The way forward
Having completed the first part of the collaboration project, I feel it’s important not to lose momentum. It’s easy to pat ourselves on the back and congratulate each other on a job well done, but the reality is we’re not done yet.
In the near future, possibly in January, a developer from RÚV is scheduled to join our developers at Yle for two weeks. During that time we will work more on the metadata issues and hopefully come a step closer to implementing Yild on RÚV’s website. We will also likely work together on other issues to benefit from the knowledge of the Icelandic web developers.
In February RÚV is hosting Drupalcamp Northern Lights 2017 in Reykjavik. This will be another opportunity to meet and discuss the project. Hopefully by then, we will be on track to enable the Yild module at RÚV and start tagging in a common way.
There is another Nordvision project focused on API collaboration between the companies. During my stay I demonstrated Yle’s many APIs and walked some RÚV developers through the documentation. As it turns out, RÚV is eager and excited to bring their content to a similar API structure and I feel there is a great opportunity for collaboration around this issue as well. Both in imparting our knowledge of API design, but also ensuring our APIs are built with a similar structure so that we can share information and data more easily in the future.
Above all else though, I feel it’s important to keep the wheels in motion. There is a lot of potential and we’ve come a long way already. By producing something tangible, we stand a chance of bringing even more Nordic public service companies and specifically the developers together in collaboration.
