Fandom Developers Wiki
Register
Advertisement

Jmalyszko Jmalyszko 24 February 2023
0

How to maintain order in the versioning and lifecycle policies of your S3 buckets

I know that topics such as cloud storage may not seem exciting, but I find S3 to be just as cool as it can be. With S3, you can store as much data as you need, and add an analytics layer on top to create a fully operational Data Lake. This has proven to be very useful for the Data Engineering team at Fandom.

Still, as you accumulate numerous datasets and pipelines, things can become complicated and expensive. S3 provides options for object versioning and lifecycle policies, but if you do not pay attention to these things, at some stage you may find yourself in a situation where you have no idea what data actually still resides on your S3 and what is generating your higher and higher costs on AWS bills. Not to mention losing your data due to…

Read Full Post
Mech.wikia Mech.wikia 3 October 2022
0

Replacing memcached proxy - a bumpy road from Twemproxy to Mcrouter

This is an example of how to replace a critical component with zero downtime, how and why projects grow in size, and where risks can come from.

At Fandom, we make heavy use of memcached as our primary caching solution for backends. We’ve been using Twemproxy as a layer of abstraction between our backends and memcached, as it provides us with consistent caching and other useful features. Twemproxy was serving us well but recently we’ve decided to replace it with Mcrouter. In this blog post, I’ll describe the project I took on - migrating our biggest service, the MediaWiki-based Community Platform, from Twemproxy to Mcrouter. Although executed successfully, it was not without its surprises.


  • 1 The plan
    • 1.1 Divide and conquer
  • 2 Learning points
    • 2.1 Have…


Read Full Post
JacekThePie2 JacekThePie2 3 October 2022
0

Working @Fandom.com: Data Engineering

Have you ever looked for information about new TV series like House of the Dragon or The Rings of Power? A new Marvel movie has been released and you want to catch up with what has happened so far in the universum? Maybe you are a game addict and Minecraft is your middle name? Or wonder who this famous Geralt is? Or are you passionate about any other things, like music, books, or public transportation in San Francisco? The chances are very high that you have visited one of the 382 535 174 Fandom articles on one of the 282 503 Fandoms!


That sounds like a lot! Yes, a lot of data: pages, traffic, ads!


  • 1 How big is Big Data at Fandom?
  • 2 What do we do?
    • 2.1 Data processing
    • 2.2 DataOps
    • 2.3 MLOps
  • 3 Cross-team cooperation
  • 4 What do we exactly do?
    • 4.1 Near real-time…



Read Full Post
DonaldChronosKing DonaldChronosKing 21 July 2022
0

Tech Talk: Go For Java Developers


Hi, Fandom fans!

Last month I did a Fandom-internal tech talk, on the topic of learning how to develop software in Go, from the perspective of an experienced Java software developer with a deep understanding of the JVM. The same talk was a hit with my own team when I first gave it earlier in the year, but that instance of the talk wasn’t recorded.

Luckily, we had the foresight to record it this time, and now it’s available on YouTube. Fully captioned by me, the presenter. Here it is:

Talk: Go For Java Developers (Fandom internal, 2022-06-15)

And here are the slides in Google Docs:

Go For Java Developers

And here is the transcript:

[Me:]

First I’m going to paste my slides into the chat so folks can follow along if they would rather do it themselves…


Read Full Post
Arturekm Arturekm 15 April 2022
0

How do we onboard new hires in mobile team

At Fandom, we believe that a well-structured onboarding is a crucial part of successfully hiring new talent no matter if you have a distributed team, work from the office or in a hybrid solution. The onboarding process should be well known and understood by new hires, the team and other people from your organizations.

The Mobile Apps team breaks down the process into ten days. Each day is devoted to a theme and during the day a new hire will meet with one person on the team for approximately 60 minutes to go through the content together.

Day 1 - Intro

On the first day, the new member is likely to be already busy with other onboarding meetings, e.g. with the IT, finance and HR teams. To not overwhelm them, we start with an intro day. During th…

Read Full Post
Bmanczak Bmanczak 16 December 2021
0

Playwright on Jenkins at scale with HTML reports

A comprehensive guide to running Playwright tests in Parallel on Jenkins

If you’re anything like me, sometimes you need to integrate old battle-tested solutions with new and shiny technologies. In today episode, my perennial butler called Jenkins was asked to run end-to-end tests using Playwright, the new hot and steamy test automation framework.



To be able to run any tests on Jenkins, we need tests. If you have your own repo, skip this part.

For those of you who want to try stuff from scratch, run npm init playwright demo in your terminal. Project creator will ask you a few questions. I chose my project to be TypeScript and disabled GitHub Actions. This resulted in creation of 1 test tests/example.spec.ts`: in place, our job will still be faile…



Read Full Post
Ritaly-dev Ritaly-dev 10 December 2021
0

Keep programmers’ decisions documentation up-to-date and simple with Architecture Decision Records

Developing a system over years is hard, as is writing clean, maintainable, up-to-date code. But believe me, writing easy-to-maintain AND up-to-date documentation is not trivial either.

Once upon a time, I heard at a conference that software engineers are not good at making a long-term commitment to their code. We all love greenfields more than diving into decades-long bloated projects, sounds familiar?

But actually, that's not true. At least not for every mature project. The difference is in the documentation.


  • 1 Spooky story time
  • 2 What is ADR?
  • 3 Benefits
  • 4 Qualities of an effective ADR
  • 5 When should I write ADR?
  • 6 Where should ADR be stored?
  • 7 ADR template


Of course, code grows over time. Dependencies get complicated, and decisions made in the past that w…



Read Full Post
Mnowicka Mnowicka 10 November 2021
3

Platform Team’s Technical Decision Process - Actor Migration Case Study

As the Platform team at Fandom we are responsible for providing core functionalities and shared services used by all the teams that work on the Unified Community Platform. As a part of our job, we are responsible for keeping our platform secure, performant, and technically up-to-date.

That includes keeping MediaWiki (the wiki engine we are using) upgraded to the newest version, providing users with a secure platform with the newest features. This article describes our decision-making process for a project which was blocking us from upgrading MediaWiki to the new version.


  • 1 What is MediaWiki and why do we need to keep it up-to-date?
    • 1.1 What was blocking us from migrating to the newest version of MediaWiki?
  • 2 Challenges (aka problems) we have face…


Read Full Post
Gaperlinski Gaperlinski 17 August 2021
0

Five Ways in which D&D Beyond Benefits from Dogfooding

Eating your own dog food, or “dogfooding”, is the practice of using one's own products or services. The rationale is that real-world usage of a product by the developers leads to improved quality and confidence in the product. First pioneered by Microsoft and adopted by a number of “big players”, including Facebook, dogfooding has perhaps been overused as a cover-up of extended QA on company employees. While it is true that there are some risks associated with the practice, when executed well and paired with reliable quality control, it can prove to be an effective technique.

Most of the engineering teams working on the D&D Beyond toolkit incorporate dogfooding into their process. In the context of my team, this practice means that each spr…

Read Full Post
Ritaly-dev Ritaly-dev 23 April 2021
0

Auto linking Jira ticket in PR description via Github Action

I was looking for an easy way to generate a URL to a Jira ticket in the pull request description or as the first comment. The ticket number/code should be automatically extracted from the branch name.

Neat, automated solutions, but unfortunately, all the top articles from Google led me to the opposite problem - how to see PRs or commits from GitHub in Jira.

The final solution is really simple, so maybe someone will use it.


(or any other project management tool)


First things first - What is Github Action?

GitHub Action is a tool added to Github that allows you to automate your entire software development workflow such as CI/CD, custom steps, E2E testing, etc.

There is a lot of things you can just drop into GitHub Action and not have to worry about…



Read Full Post

Advertisement