How Oracle Has Done Nothing to Change MySQL

Last night at the Oracle OpenWorld MySQL Community Reception, there were lots of old and new friends milling about.  It occurred to me that there is one very important thing Oracle has NOT changed about the MySQL world - the rock stars and higher-ups are still readily accessible.

One of the things I love about being in the open source community is that you can have an in-depth conversation with someone, and only later on find out that this person is famous.  For the most part, rock stars and important people are readily accessible.  They stay in the same hotels that attendees do, they take the same elevators, they are not whisked away by bodyguards, and they do not play the "don't you know who I am?" card.

Now, it's not surprising that the community rock stars like Mark Callaghan, Baron Schwartz, Giuseppe Maxia and Sarah Novotny are still as accessible as ever.  However, Ed Screven and Thomas Ulin were also around for the party, and I can confirm that Thomas was one of the last dozen or so to leave (Ronald Bradford and I closed out the party and were the last to leave).

So, kudos to Oracle for not keeping your VIPs locked up in a bunker.  I am very glad to see this aspect of open source culture still thriving.

Advocating For Our Clients - Part 1 Cultural


What do companies need from their database professional, MySQL or otherwise? How can we exceed those expectations as a remote team?  This is my first in a series of blog posts discussing exactly how we do so at PalominoDB - regardless if the technology is MySQL, MongoDB, Cassandra, Oracle or ottherwise.


The majority of our clients are start-ups.  Some are small teams experiencing their first three year growth pains, while others are in the three to seven year period, have proven the effectiveness of their business model, yet retain a strong sense of start-up culture and personality.  When looking for staff, their focus is rightly on people who have passion, drive and personalities that fit with their unique corporate cultures.  How can a remote resource, much less one that is not dedicated to one company full-time, understand not just clients’ product and technology, but the people, the schedules and the drive that support clients' success?

Quite often clients want PalominoDB to have a single point of contact who gives us individual tasks and who functions as a filter between their organization and ours.  While we will work with whatever model is requested, this method builds a certain level of isolation that can limit our effectiveness in the bigger picture.  Being in operations requires a certain push and pull with engineering and product groups to meet business demands and ensure availability and performance. We also require knowledge of a company's business goals and project portfolio.  Otherwise, how can we react with urgency at the appropriate times?  How can we know which issues require escalation and which require push back? 

Once we understand company strategy and priorities, we can start to tailor the decisions we make to our clients’ needs. For example, if we know a particular system is crucial to the success of a client's key project, we are much more inclined to work until 2 am to complete the project or to meet a release deadline. If we know that two months of late nights and weekend work are crucial to helping a client with a customer launch, beat the competition and grow successful, my staff and I willingly devise a plan to support that customer. However, if we perceive that a customer’s demands for last minute changes or large amounts of off- hours work come from poor planning, poor communication or a poorly prioritized product plan, we are much more inclined to put our efforts into improving the underlying processes around change management and project planning. 

As operational professionals, we understand the importance of urgency and the product delivery speed that the modern start-up must work with.  Because of the breadth of our experience, we also know that if production teams had their way, all tasks would be P1s, all reports would be real time and there would never be any downtime (and all work would happen for free) and we act in accordance with this desire to the best of our ability.  When we know that the task we are working on is crucial to our clients’ ability to maintain their competitive edge, we are motivated to work the 12 hour days needed to get it done.  Alternatively, when we know a date is flexible, we can choose not to tax our operations team and evoke the risks associated with working too many hours and making crucial decisions under fatigue. As CEO and principal at PalominoDB, it is my job to work within my clients availability and take care of my staff  Work-life balance is not simply a concept to which I pay lip-service; I believe that a happy, rested and alert operations staff is essential to customer up-time and to keeping human mistakes to a minimum. 

Another question we often get asked is how do we as remote team members correctly align with business so that we can support them at their pace and intensity?  We’ve had the most success with regular knowledge- shares and participation in operational team meetings.  While taking part in our clients’ company-wide sessions has been unnecessary, we have found that attending operations and architecture team meetings where information can be shared down and around is an excellent start.  Getting onto operational team distribution lists is another method we use to learn about what is going on.  Obviously, every DBA on our team cannot do these things for every one of PalominoDB's clients, but the primary DBA assigned to a particular client can, and, as they filter out relevant details, they can share pertinent information with the rest of the team. 

Having that primary DBA serve as a client’s advocate is crucial, and a point I will continually discuss in my writings.  It is the primary DBA who asks questions when information is not forthcoming, who reviews the org charts and introduce themselves to engineers, project managers and QA/release folks.  The primary DBA gets contact info from all of these folk, documents it in CRM, and plugs it into GTalk, Skype or whatever medium is appropriate.  The primary DBA will hang out in a clients’ IRC and campfire rooms and soak up everything they see.  The primary DBA even reads powerpoints (yes really)! Finally, and most importantly, the primary DBA makes on-site visits.  PalominoDB’s operations team makes it a point to try and come on-site at least once every two months.  Some of that time is spent in meetings and some of that time is spent simply dining or hanging out.  Regardless, these on-site visits allow us all to connect, to put names to faces, and to get to know each other. It helps to ensure that our clients understand that PalominoDB is not a faceless company full of replaceable DBAs.  We are a company made up of individuals with skills, quirks, personalities (usually BIG ONES), and we know our clients are the same. 

Does this take time? Yes. However, I ask our clients to think about the savings in cost they accrue by using us instead of maintaining a full-time staff.  The extra time spent on meetings, emails and IRC conversations does not add much in overall cost, yet it is invaluable when building relationships.  Constant contact replaces the water-cooler meetings and impromptu conversations at lunch. That small investment of time in camaraderie and in team-building pays-off in more ways than you can imagine.

PalominoDB's Proactive Support

I'm rather passionate about what we do here at PalominoDB.  When I tell people about what makes us different from the competition, I often discuss the proactive work we do.  We are not a reactive company.  We know that proactive reviews are what keep a database up and running smoothly, and we definitely want to prevent those late night pages that everybody loves.  So what do we do to make this happen?  

We start with daily health checks.  These are aided by scripts, but include the Primary DBA reviewing the last 4 days of core cacti graphs (or a similar trending tool, as we require clients to have one) for anomalous behavior.  Core indicates key metrics that point out workload shifts - CPU, IOWait, Load Average, Swapping, SQL Query Type Counters and InnoDB Row Activity.  We verify backup logs are error free, and we make sure nothing has shown up in the MySQL error logs that proves unusual.  Finally, we review the Nagios alerts of the last day, making sure criticals are followed up on, that nothing has been acknowledged and forgotten and that alerts are enabled.

Once a week, the primary DBA then reviews all cacti graphs, not just core ones.  They review the dailies and ensure all items that have come up are being acted on and they verify tickets are not stalling.

Once a month, we do SQL Reviews of systems, unless we have been requested to do them more frequently based on release schedules.  Using mk-query-digest, general logs with microsecond patches or tcpdumps, we put together a list of the top executed and the slowest queries and provide our clients with a list of recommended changes to indexes, datatypes, data model changes, caching suggestions and query rewrites.  Once provided, we follow up regularly to ensure that crucial changes are being implemented in a timely manner.

Then we have activities that we do quarterly.  Some clients prefer these be done more frequently and that is fine by us!  These include:

  • Recovery tests if regular refreshes of other environments don't test this.
  • Ensuring tools are at up to date versions.
  • Ensuring runbooks and documentation are up to date.
  • Capacity reviews of existing workloads and hardware.
  • Security audits to ensure that changes since the last audit do not violate security policy.
  • Monitoring and trending audits to ensure that appropriate checks are in place and that all needed graphs are graphing.

This is where we start in our proactive service.  The end is unlimited, only depending on our clients' needs.  Any opportunity we can have to solve an issue before it is noticed is an opportunity that we relish and that our clients appreciate.  Of course, the real test of our mettle is the number of pages waking us up in the middle of the night.  If we have to wear our asbestos suits to work, then we are not doing our job!  

Why use PalominoDB?

 [edited June 8th to correct typo; the figure given is per YEAR, not per month]

Why PalominoDB:

Efficiency in Time and Cost - 

The average salary for a full-time DBA is $80-120K per year, not counting payroll and benefit costs.  You can retain PalominoDB for your senior needs at only $37,200 a year based on a 20 hour retainer.  This leaves the balance for you to invest in a staff of flexible, competent, systems administrators who can handle day to day tasks. A retainer with PalominoDB includes access to our senior and principal personnel, engineers, and systems administrators.  Whatever your need, PalominoDB can fulfill it.  As you have access to an entire team, you are never left short-handed on holidays or during flu season...and we will never leave you for another company.
Team integration -

PalominoDB isn’t just a remote database services company.  From day one, we jump in and work as hard as we can to become part of your team.  We don’t sit in the wings, waiting for you to give us work or for your systems to page us.  We plug into your team’s IRC room and get into your monitoring systems.  We provide constant communication through the use of instant messaging and email.  We attend your team meetings or we set up meetings with you - and we ensure that they are efficient meetings.  Our company integrates into your systems rather than forcing you to integrate with ours.  We are confident you will come to see us as a part of your full-time workforce rather than a group of consultants.

Proactive service -

We review systems daily.  We look for anomalies in workloads.  We check error logs, and we ensure successful backups. We research the benefits of upgraded binaries, new parameters and technologies, and how they can be applied for our clients.  If you aren’t actively generating work for us, we use those hours for proactive work - SQL Reviews, Capacity Reviews and Backup/Recovery tests.  We understand how busy small businesses can be, and we make sure to keep tickets moving.  We do not require babysitting; it is our job to free up your time so you can focus on the rest of your infrastructure.  We can even provide help and recommendations on the rest of your environments.

Extensive Experience -

The professionals at PalominoDB have been managing production systems since before the dot com era.  We’ve seen explosive growth and scaled companies through years, not months, of development.  Our experience is not just with particular technologies, but with operational process, change management, incident and problem management, documentation, and configuration management.  We see the big picture, and work in it using our breadth of experience to do so.  And bottom line, we follow our mantra of three key principles:  Keep production safe, keep production simple, and document and share everything.

Oracle is not removing InnoDB!


I was asked "What's the deal with Oracle removing InnoDB?"  I had not heard this, so I did some research.  It took less than 5 minutes to figure out what happened, and it was easy to see where the confusion is.

On the MySQL products page at the matrix of MySQL editions includes "MySQL Classic" which is free, "MySQL Standard" which costs $2k per year, "MySQL Enterprise" which costs $5k per year and "MySQL Cluster Carrier Grade" which costs $10k per year.

Indeed, the "MySQL Classic" does not include InnoDB.  What happened was that folks assumed that, because it was free, it was the MySQL Community edition we all know and love.

This is not true.  How do I know?  Because just above the matrix is a set of links to each edition, and if you click the "MySQL Classic" link you get to which explains "MySQL Classic Edition is the ideal embedded database for ISVs, OEMs and VARs developing read-intensive applications using the MyISAM storage engine."


So calm down, folks. 

Managing mistakes, apologies and other Grown Up Business.

Here at PalominoDB, we recently had the opportunity to work through a few mistakes and their aftermath.  Why is this pertinent to a blog by a database management team?  Regardless of your use of MySQL, Oracle, Cassandra or whatever technology you've deemed appropriate, it is the management of operations that becomes the true differentiator in your service providing.  Dealing with these recent mistakes got me thinking on what a crucial component of growth these moments are.  In the world of operations, mistakes can be quite visible, and are not uncommon to any of us.  One might argue that one of the biggest parts of our jobs is managing risk and mitigating impact of mistakes that will inevitably occur.

In the moment, it is easy to get very upset with someone who makes a mistake, or two or even three in a week.  I've learned from the best clients on how to deal with mistakes, by how they deal with ours. Mistakes are typically insights into a broken process, a bad behavior pattern or a lack of information and tools.  If you do manage risk and have planned appropriately, you should be able to survive and handle the immediate effects of the mistake.  Then, you have an opportunity to identify the issues, and improve them.  Anyone who sees a mistake (and I mean a true mistake of ignorance, versus a mistake of gross negligence) as a chance only to yell, blame and generally act discourteously is not interested in growth.  

I've found in life that growth is inevitably fostered by mistakes, occasional hardship, and even pain.  With an attitude of growth, these become quite manageable, if not welcome.  Recognizing this should help you reduce your stress around your own mistakes, and focus on the growth opportunities present.

So how do you deal with that mistake?  I learned a long time ago that the ability to truly, genuinely apologize is another one of those crucial life skills, and here is the perfect time to use that skill.  An immediate acknowledgment of your mistake, recognition of the impact your mistake has made, and a sincere request to offer amends are the parts of a good apology.  Don't focus on avoiding responsibility, trying to pass the buck, share blame, etc...  You are responsible for your part in this, and that is what you are apologizing for.

Next, you dig into what happened, and why it happened.  Perform a post-mortem, talk in ways that do not make anyone feel attacked, and find out where the root issue is.  Rather than focusing on blame and punishment, focus on the core issue, and put together a concise plan of attack to address the issue once it is exposed.  With this approach, continued improvement, and an environment that fosters openness and transparency becomes the norm.

Sheeri has joined PalominoDB

About six weeks ago I posted about leaving Pythian. I had a month off in which I spent quality time with my husband, including packing up our apartment to move two blocks away. I also spent some time doing some planning and organizing for OpenSQLCamp Boston, happening in 6 weeks. Many people have been wondering what my next move is.

Where we've been...

You might notice that I haven’t blogged in oh, 2 years.  How remiss of me.  The only defense I have is that we’ve been non-stop busy!  Since then we’ve built our client portfolio, brought in two new team members, presented at MySQL Conference and still managed to get into any number of shenanigans.   Still, that is no excuse.  Let me give a summary of some of the interesting things we’ve been up to:

The Prototype

The first start-up stage I’ve worked within is the prototype phase. Within this phase traffic is not an issue for performance or scale, it’s about functionality. Low traffic and small datasets can hide atrocious code quite easily. The nice thing about this stage is that you should not have to invest a lot of time or money into your database and instead can focus on functionality and business development. Over-engineering at this point can be a devastating waste of very precious resources.

The Start-up and the Database

My first senior production database role was at an established start-up, Preview Travel, that had just been purchased by a similarly established, but better positioned start-up, Travelocity. Since then, I’ve worked with start-ups in all stages of growth and I’ve seen definite patterns in how database infrastructures are designed, implemented and maintained (or the lack thereof). I’ve seen these phases of growth presented elsewhere, often at a level of granularity that didn’t work for me. I am a believer of simplicity wherever possible.

Syndicate content
Website by Digital Loom