Business

Last Day at PalominoDB

I have been the Community Liaison and a Senior DBA at PalominoDB for 15 months, and doing remote DBA work for 4 years.  In that time I have learned that "consultant" need not be a dirty word, and that in the DBA world it is actually extremely valuable to have a remote DBA with lots of outside experience, and a team of remote DBAs for when your primary contact is sick or goes on holiday.

As with everything, there are downsides to remote database management.  Even though there is a lot of architecture experience among the remote DBAs I know, we are not often invited to architecture meetings.  This is because time is the unit of currency, and while sitting in an hour-long meeting to give 5 minutes of feedback can save hours down the road, it's hard to see that.  Many clients have gotten around this by having all DDL checked and performed by remote DBAs, and that helps a lot.

There is also no ownership - we can recommend solutions and technologies, but the client makes the actual decision about whether something needs to be done or not.  I look forward to actually owning architecture after 4 years of making "strong recommendations".

Since folks will ask, I have taken a job as a Senior DBA/Architect with Mozilla, starting Monday.  A former co-worker told me about the job; I was not particularly looking for anything, but I was intrigued.

I have said before that it is hard to find a good in-house DBA if you are not a huge company like Facebook or Google or Yahoo, and that is still true.  At Mozilla, they are 100% open and public about their ideas, and they do a lot of behind-the-scenes work.  Sound familiar?

They also allow their developers to develop on whatever platforms work best.  Their biggest database is their crash reporting database (and they do read it, so do submit your crashes).  They have MySQL, PostgreSQL, and MongoDB, and are starting to move some applications around, as developers are not always aware of what platforms will work best.  There is another DBA, so I will not be alone, but I expect to be just as challenged at Mozilla as I have been at PalominoDB and Pythian.

So, to keep up with PalominoDB, you can:

- like the PalominoDB page on Facebook

- follow @palominodb on Twitter

- connect with Laine on LinkedIn

- follow PalominoDB on LinkedIn

- continue to read PalominoDB's blog and Planet MySQL

 

To keep up with me, you can:

- follow @sheeri on Twitter

- read my blog and Planet MySQL

- connect with me on LinkedIn

- subscribe to the OurSQL podcast (details at http://www.oursql.com/?page_id=2)

Announcing PalominoDB's Non Profit Program

I've always had a dream of being able to use what we are doing at PalominoDB not only for our for profit clients, but for those who go out day to day helping those who need it.  We always try to work with clients who make people's lives better, but there are those whose entire purpose is to provide aid, education and empowerment to those who are disadvantaged or whose freedoms are at risk.  I am constantly inspired by companies such as Worldreader (http://blog.worldreader.org/) are a perfect example - who work to provide e-Readers to those who have no access to books or libraries in places such as Kenya.  The challenge has always been how to provide support from our team when they are constantly busy.  In a growing organization, resources are tight, and we are blessed with non-stop work from world class clients.

 

As we've grown to a sustainable size, I've had more time to think about issues such as this, and I would like to announce our newest program at PalominoDB - donation of hours.  A good portion of our clients are on retainer agreements with a monthly minimum.  Sometimes work is light, sometimes it is heavy and some clients just keep us around as insurance and rarely use our hours.  Regardless, we have to staff for a certain workload and thus must enforce the minimum.  I've always found myself frustrated at having to charge for hours not worked, and constantly brainstorm ways to provide maximum benefit to our regular clients.  

 

PalominoDB would now like to announce our donation of hours program - whereby we are setting up relationships with non-profits who can use our resources - and the hours for this work will be donated by our clients who have unused hours and wish to see them go to a good cause, rather than paying the 50% unused hours rate.  Clients can also donate a fixed amount of hours per month to the program.  We will start our program with one non-profit, Worldreader, and will solicit for other companies who can make effective use of our resources.   Should you know of any deserving companies, please don't hesitate to let us know!

 

You can see a video of Worldreader's work here.  Please take a look at their donations page here, as there are great opportunities to donate for e-readers or books or to sponsor classes and schools.

PalominoDB at PerconaLive

PalominoDB is very excited about our participation in the upcoming PerconaLive conference in London.  We'll have two of our European staff presenting.  On Monday, Jonathan Levin will be doing a tutorial on Advanced MySQL Scaling Strategies for Developers, and on Tuesday Rene Cannao will be presenting on MySQL Backup and Recovery Tools and Techniques.  Rene and Jonathan are two of our newer team members, and represent an exciting growth in staff outside of the US; Jonathan being in the UK and Rene being in Malta.  I know I'm thrilled to get out to London to meet a lot of new folks in our community.

We did get a chance to present at PerconaLive in NYC this year as well, which was quite a lot of fun, and the positive reception Sheeri's session got was gratifying.  Percona has become a huge part of our community and has provided great value in their information share, tools, services and now conferences - including the MySQL Conference & Expo 2012.  Having been involved in professional MySQL consulting and remote support for four years has been quite the adventure, and the fact that so many companies find space to create, share and prosper only shows the viability of open source software and the communities behind it.  

I know we at PalominoDB are quite proud to share space with companies such as Pythian, Blue Gecko, SkySQL and, of course, Percona. We are proud to support open source solutions across the board, and are even more excited to have grown to a place where we have the resources to contribute back to them, to non-profits and to the growth of our clients and every person on our team.  Here's to an exciting and brilliant future, and a great conference!

How Oracle Has Done Nothing to Change MySQL

Last night at the Oracle OpenWorld MySQL Community Reception, there were lots of old and new friends milling about.  It occurred to me that there is one very important thing Oracle has NOT changed about the MySQL world - the rock stars and higher-ups are still readily accessible.

One of the things I love about being in the open source community is that you can have an in-depth conversation with someone, and only later on find out that this person is famous.  For the most part, rock stars and important people are readily accessible.  They stay in the same hotels that attendees do, they take the same elevators, they are not whisked away by bodyguards, and they do not play the "don't you know who I am?" card.

Now, it's not surprising that the community rock stars like Mark Callaghan, Baron Schwartz, Giuseppe Maxia and Sarah Novotny are still as accessible as ever.  However, Ed Screven and Thomas Ulin were also around for the party, and I can confirm that Thomas was one of the last dozen or so to leave (Ronald Bradford and I closed out the party and were the last to leave).

So, kudos to Oracle for not keeping your VIPs locked up in a bunker.  I am very glad to see this aspect of open source culture still thriving.

Advocating For Our Clients - Part 1 Cultural

 

What do companies need from their database professional, MySQL or otherwise? How can we exceed those expectations as a remote team?  This is my first in a series of blog posts discussing exactly how we do so at PalominoDB - regardless if the technology is MySQL, MongoDB, Cassandra, Oracle or ottherwise.

Cultural:  

The majority of our clients are start-ups.  Some are small teams experiencing their first three year growth pains, while others are in the three to seven year period, have proven the effectiveness of their business model, yet retain a strong sense of start-up culture and personality.  When looking for staff, their focus is rightly on people who have passion, drive and personalities that fit with their unique corporate cultures.  How can a remote resource, much less one that is not dedicated to one company full-time, understand not just clients’ product and technology, but the people, the schedules and the drive that support clients' success?

Quite often clients want PalominoDB to have a single point of contact who gives us individual tasks and who functions as a filter between their organization and ours.  While we will work with whatever model is requested, this method builds a certain level of isolation that can limit our effectiveness in the bigger picture.  Being in operations requires a certain push and pull with engineering and product groups to meet business demands and ensure availability and performance. We also require knowledge of a company's business goals and project portfolio.  Otherwise, how can we react with urgency at the appropriate times?  How can we know which issues require escalation and which require push back? 

Once we understand company strategy and priorities, we can start to tailor the decisions we make to our clients’ needs. For example, if we know a particular system is crucial to the success of a client's key project, we are much more inclined to work until 2 am to complete the project or to meet a release deadline. If we know that two months of late nights and weekend work are crucial to helping a client with a customer launch, beat the competition and grow successful, my staff and I willingly devise a plan to support that customer. However, if we perceive that a customer’s demands for last minute changes or large amounts of off- hours work come from poor planning, poor communication or a poorly prioritized product plan, we are much more inclined to put our efforts into improving the underlying processes around change management and project planning. 

As operational professionals, we understand the importance of urgency and the product delivery speed that the modern start-up must work with.  Because of the breadth of our experience, we also know that if production teams had their way, all tasks would be P1s, all reports would be real time and there would never be any downtime (and all work would happen for free) and we act in accordance with this desire to the best of our ability.  When we know that the task we are working on is crucial to our clients’ ability to maintain their competitive edge, we are motivated to work the 12 hour days needed to get it done.  Alternatively, when we know a date is flexible, we can choose not to tax our operations team and evoke the risks associated with working too many hours and making crucial decisions under fatigue. As CEO and principal at PalominoDB, it is my job to work within my clients availability and take care of my staff  Work-life balance is not simply a concept to which I pay lip-service; I believe that a happy, rested and alert operations staff is essential to customer up-time and to keeping human mistakes to a minimum. 

Another question we often get asked is how do we as remote team members correctly align with business so that we can support them at their pace and intensity?  We’ve had the most success with regular knowledge- shares and participation in operational team meetings.  While taking part in our clients’ company-wide sessions has been unnecessary, we have found that attending operations and architecture team meetings where information can be shared down and around is an excellent start.  Getting onto operational team distribution lists is another method we use to learn about what is going on.  Obviously, every DBA on our team cannot do these things for every one of PalominoDB's clients, but the primary DBA assigned to a particular client can, and, as they filter out relevant details, they can share pertinent information with the rest of the team. 

Having that primary DBA serve as a client’s advocate is crucial, and a point I will continually discuss in my writings.  It is the primary DBA who asks questions when information is not forthcoming, who reviews the org charts and introduce themselves to engineers, project managers and QA/release folks.  The primary DBA gets contact info from all of these folk, documents it in CRM, and plugs it into GTalk, Skype or whatever medium is appropriate.  The primary DBA will hang out in a clients’ IRC and campfire rooms and soak up everything they see.  The primary DBA even reads powerpoints (yes really)! Finally, and most importantly, the primary DBA makes on-site visits.  PalominoDB’s operations team makes it a point to try and come on-site at least once every two months.  Some of that time is spent in meetings and some of that time is spent simply dining or hanging out.  Regardless, these on-site visits allow us all to connect, to put names to faces, and to get to know each other. It helps to ensure that our clients understand that PalominoDB is not a faceless company full of replaceable DBAs.  We are a company made up of individuals with skills, quirks, personalities (usually BIG ONES), and we know our clients are the same. 

Does this take time? Yes. However, I ask our clients to think about the savings in cost they accrue by using us instead of maintaining a full-time staff.  The extra time spent on meetings, emails and IRC conversations does not add much in overall cost, yet it is invaluable when building relationships.  Constant contact replaces the water-cooler meetings and impromptu conversations at lunch. That small investment of time in camaraderie and in team-building pays-off in more ways than you can imagine.

PalominoDB's Proactive Support

I'm rather passionate about what we do here at PalominoDB.  When I tell people about what makes us different from the competition, I often discuss the proactive work we do.  We are not a reactive company.  We know that proactive reviews are what keep a database up and running smoothly, and we definitely want to prevent those late night pages that everybody loves.  So what do we do to make this happen?  

We start with daily health checks.  These are aided by scripts, but include the Primary DBA reviewing the last 4 days of core cacti graphs (or a similar trending tool, as we require clients to have one) for anomalous behavior.  Core indicates key metrics that point out workload shifts - CPU, IOWait, Load Average, Swapping, SQL Query Type Counters and InnoDB Row Activity.  We verify backup logs are error free, and we make sure nothing has shown up in the MySQL error logs that proves unusual.  Finally, we review the Nagios alerts of the last day, making sure criticals are followed up on, that nothing has been acknowledged and forgotten and that alerts are enabled.

Once a week, the primary DBA then reviews all cacti graphs, not just core ones.  They review the dailies and ensure all items that have come up are being acted on and they verify tickets are not stalling.

Once a month, we do SQL Reviews of systems, unless we have been requested to do them more frequently based on release schedules.  Using mk-query-digest, general logs with microsecond patches or tcpdumps, we put together a list of the top executed and the slowest queries and provide our clients with a list of recommended changes to indexes, datatypes, data model changes, caching suggestions and query rewrites.  Once provided, we follow up regularly to ensure that crucial changes are being implemented in a timely manner.

Then we have activities that we do quarterly.  Some clients prefer these be done more frequently and that is fine by us!  These include:

  • Recovery tests if regular refreshes of other environments don't test this.
  • Ensuring tools are at up to date versions.
  • Ensuring runbooks and documentation are up to date.
  • Capacity reviews of existing workloads and hardware.
  • Security audits to ensure that changes since the last audit do not violate security policy.
  • Monitoring and trending audits to ensure that appropriate checks are in place and that all needed graphs are graphing.

This is where we start in our proactive service.  The end is unlimited, only depending on our clients' needs.  Any opportunity we can have to solve an issue before it is noticed is an opportunity that we relish and that our clients appreciate.  Of course, the real test of our mettle is the number of pages waking us up in the middle of the night.  If we have to wear our asbestos suits to work, then we are not doing our job!  

Why use PalominoDB?

 [edited June 8th to correct typo; the figure given is per YEAR, not per month]

Why PalominoDB:

Efficiency in Time and Cost - 

The average salary for a full-time DBA is $80-120K per year, not counting payroll and benefit costs.  You can retain PalominoDB for your senior needs at only $37,200 a year based on a 20 hour retainer.  This leaves the balance for you to invest in a staff of flexible, competent, systems administrators who can handle day to day tasks. A retainer with PalominoDB includes access to our senior and principal personnel, engineers, and systems administrators.  Whatever your need, PalominoDB can fulfill it.  As you have access to an entire team, you are never left short-handed on holidays or during flu season...and we will never leave you for another company.
 
Team integration -

PalominoDB isn’t just a remote database services company.  From day one, we jump in and work as hard as we can to become part of your team.  We don’t sit in the wings, waiting for you to give us work or for your systems to page us.  We plug into your team’s IRC room and get into your monitoring systems.  We provide constant communication through the use of instant messaging and email.  We attend your team meetings or we set up meetings with you - and we ensure that they are efficient meetings.  Our company integrates into your systems rather than forcing you to integrate with ours.  We are confident you will come to see us as a part of your full-time workforce rather than a group of consultants.

Proactive service -

We review systems daily.  We look for anomalies in workloads.  We check error logs, and we ensure successful backups. We research the benefits of upgraded binaries, new parameters and technologies, and how they can be applied for our clients.  If you aren’t actively generating work for us, we use those hours for proactive work - SQL Reviews, Capacity Reviews and Backup/Recovery tests.  We understand how busy small businesses can be, and we make sure to keep tickets moving.  We do not require babysitting; it is our job to free up your time so you can focus on the rest of your infrastructure.  We can even provide help and recommendations on the rest of your environments.

Extensive Experience -

The professionals at PalominoDB have been managing production systems since before the dot com era.  We’ve seen explosive growth and scaled companies through years, not months, of development.  Our experience is not just with particular technologies, but with operational process, change management, incident and problem management, documentation, and configuration management.  We see the big picture, and work in it using our breadth of experience to do so.  And bottom line, we follow our mantra of three key principles:  Keep production safe, keep production simple, and document and share everything.

Oracle is not removing InnoDB!

 

I was asked "What's the deal with Oracle removing InnoDB?"  I had not heard this, so I did some research.  It took less than 5 minutes to figure out what happened, and it was easy to see where the confusion is.

On the MySQL products page at http://mysql.com/products/ the matrix of MySQL editions includes "MySQL Classic" which is free, "MySQL Standard" which costs $2k per year, "MySQL Enterprise" which costs $5k per year and "MySQL Cluster Carrier Grade" which costs $10k per year.

Indeed, the "MySQL Classic" does not include InnoDB.  What happened was that folks assumed that, because it was free, it was the MySQL Community edition we all know and love.

This is not true.  How do I know?  Because just above the matrix is a set of links to each edition, and if you click the "MySQL Classic" link you get to http://mysql.com/products/classic/ which explains "MySQL Classic Edition is the ideal embedded database for ISVs, OEMs and VARs developing read-intensive applications using the MyISAM storage engine."

 

So calm down, folks. 

Managing mistakes, apologies and other Grown Up Business.

Here at PalominoDB, we recently had the opportunity to work through a few mistakes and their aftermath.  Why is this pertinent to a blog by a database management team?  Regardless of your use of MySQL, Oracle, Cassandra or whatever technology you've deemed appropriate, it is the management of operations that becomes the true differentiator in your service providing.  Dealing with these recent mistakes got me thinking on what a crucial component of growth these moments are.  In the world of operations, mistakes can be quite visible, and are not uncommon to any of us.  One might argue that one of the biggest parts of our jobs is managing risk and mitigating impact of mistakes that will inevitably occur.

In the moment, it is easy to get very upset with someone who makes a mistake, or two or even three in a week.  I've learned from the best clients on how to deal with mistakes, by how they deal with ours. Mistakes are typically insights into a broken process, a bad behavior pattern or a lack of information and tools.  If you do manage risk and have planned appropriately, you should be able to survive and handle the immediate effects of the mistake.  Then, you have an opportunity to identify the issues, and improve them.  Anyone who sees a mistake (and I mean a true mistake of ignorance, versus a mistake of gross negligence) as a chance only to yell, blame and generally act discourteously is not interested in growth.  

I've found in life that growth is inevitably fostered by mistakes, occasional hardship, and even pain.  With an attitude of growth, these become quite manageable, if not welcome.  Recognizing this should help you reduce your stress around your own mistakes, and focus on the growth opportunities present.

So how do you deal with that mistake?  I learned a long time ago that the ability to truly, genuinely apologize is another one of those crucial life skills, and here is the perfect time to use that skill.  An immediate acknowledgment of your mistake, recognition of the impact your mistake has made, and a sincere request to offer amends are the parts of a good apology.  Don't focus on avoiding responsibility, trying to pass the buck, share blame, etc...  You are responsible for your part in this, and that is what you are apologizing for.

Next, you dig into what happened, and why it happened.  Perform a post-mortem, talk in ways that do not make anyone feel attacked, and find out where the root issue is.  Rather than focusing on blame and punishment, focus on the core issue, and put together a concise plan of attack to address the issue once it is exposed.  With this approach, continued improvement, and an environment that fosters openness and transparency becomes the norm.

Sheeri has joined PalominoDB

About six weeks ago I posted about leaving Pythian. I had a month off in which I spent quality time with my husband, including packing up our apartment to move two blocks away. I also spent some time doing some planning and organizing for OpenSQLCamp Boston, happening in 6 weeks. Many people have been wondering what my next move is.

Syndicate content
Website by Digital Loom