Moving from being a DBA on a proprietary RDBMS for 20 years to working on an open source RDBMS is teaching me a lot of lessons. Many of them are lessons I expected – remembering how and when to ask for help, getting used to not being expected to be the expert on everything, and technical details around how MySQL does things differently. I’d like to talk about some of the lessons I didn’t really expect.
Is X a MySQL Thing or a “My Employer” Thing
When I come across a new term, a practice, or even a technical detail, I often have to ask (myself or others) a question like: “Is this a MySQL thing or a MySQL at My Employer thing?”
As an expert on IBM Db2 with experience consulting in a number of companies, it was easy for me to see the lines between the RDBMS and how a particular employer used the RDBMS. I have seen and discussed a lot of conventional and not so conventional uses of Db2. The distinction is less obvious when I’m not as familiar with the RDBMS.
A simple example of this is foreign keys. At work, we don’t use them, and the main reason is that we use a schema migration tool and also use Ruby on Rails. The checks that foreign keys provide are handled at the application layer. Having foreign keys would break the processes for schema migration that we use for true CI/CD. Coming in from a company whose idea of CI/CD was deploys once every six weeks, it took a bit to understand that this wasn’t due to any lacking in MySQL, and may not be a common practice in the MySQL community, but just how my employer does things.
Every piece of software, including proprietary and very established RDBMSes, encounter errors. At some point they’re going to have some problem to work through or around. Last week was my first experience with a MySQL failure that took a database down, and it was a doozy.
When encountering an error with Db2 that causes a crash or data issue, you call IBM support. You call IBM Support even before you’ve verified it wasn’t something related to known causes like hardware failures or obvious problems with OS or applications. They’ll ask a ton of questions and likely ask for logs of various sorts from the issue. If the issue stopped, they may ask you to try to recreate the issue with additional logging enabled. With the information that you provide, they’ll help with both immediate and long-term root cause of the issue and suggest a course of action – be it changing configuration in some way, or even writing a patch just for you. Sometimes the answer is you just made a mistake, and they’ll make that clear, but still help you identify the mistake and correct it. Now this is a bit of a simplification because IBM support is not particularly easy to deal with. There are many pitfalls along the way including them telling you to upgrade, recommending patently wrong courses of action, or the dreaded “Working as Designed” designator.
The approach for a similar issue on MySQL (without any paid support) is strikingly different. With a fairly large MySQL implementation, my employer decides not to pay for support from MySQL or Percona, but to instead employ people who can handle deep problems. This means that the course of action falls entirely on our teams. First, fixing the problem in production without data loss and low down time because we are prepared for even unexpected failures, and secondly digging into the MySQL code ourselves to see if we can find some clue about where the problem stems from so we can either suggest a fix to the code ourselves or at least point to where exactly things went wrong. Generally opening an issue with MySQL with “I had this problem” won’t help anyone.
General RDBMS Concepts
Another separation that is fascinating for me is discovering which concepts are general RDBMS concepts and which are specific to a platform or vary between platforms. While concepts like the buffer pool hit ratio and page cleaning are very similar at a high level, the details of how many buffer pools there are, how work is mapped to them, and the algorithm for deciding what to retain in the buffer pool are either slightly or very different. I was recently talking with another Db2 expert who has moved to MySQL, and his comment was that it felt like MySQL stole a lot of the terminology from Db2. I think that isn’t the case, but more that the areas where that feels true are the more general RDBMS concepts that apply to many platforms.
None of this should direct whether a company chooses to use a proprietary or open-source RDBMS, but I do feel like the RDBMS category choice should at least not clash too badly with an organization and a corporate culture. I’ve seen some astounding cultures of blame in my career, and have hated them with every fiber of my being. I wonder if a proprietary RDBMS better fits a culture of blame, because ultimately the finger can be pointed at IBM or whoever the vendor is. I’ve also seen some corporate cultures that took a wonderful, collaborative, blameless approach to problem solving, and still used a proprietary RDBMS. The expertise of your database team looks different for each, but both require deep expertise, particularly at larger scales.
Overall, I feel like I’ve been particularly lucky in this transition. I have at least three people at work I can ask the dumb questions and know they won’t make fun of me for it, and will take the time to explain in detail down to whatever level I need. I still have moments where I learn some technical detail and my immediate reaction is “What kind of podunk RDBMS is this?!?!”, but they’re now followed by marveling at the people I work with and the fascinating large scale we’re working at.