Blog

Using Amazon Web Services for MySQL at Scale Presentation

PalominoDB CEO Laine Campbell presented this class at the O'Reilly Velocity Conference, NYC Oct 2013. The class is broken into two videos and is a total of 85 minutes.

Click here for Part 1 - Click here for Part 2

Laine will explain the options for running MySQL at high volumes at Amazon Web Services, exploring options around database as a service, hosted instances/storages and all appropriate availability, performance and provisioning considerations using real-world examples from Call of Duty, Obama for America and many more.

Laine will show how to build highly available, manageable and performant MySQL environments that scale in AWS—how to maintain then, grow them and deal with failure. Some of the specific topics covered are:

  • Overview of RDS and EC2 -- pros, cons and usage patterns/antipatterns.
  • Implementation choices in both offerings: instance sizing, ephemeral SSDs, EBS, provisioned IOPS and advanced techniques (RAID, mixed storage environments, etc...)
  • Leveraging regions and availability zones for availability, business continuity and disaster recovery.
  • Scaling patterns including read/write splitting, read distribution, functional dataset -partitioning and horizontal dataset partitioning (aka sharding)
  • Palomino cluster tool (OSS) and Jetpants for building and managing large sharded MySQL in EC2
  • Configuration management, backup/recovery, monitoring and trending considerations.
  • Common failure modes -- AZ and Region failures, EBS corruption, EBS performance inconsistencies and more.
  • Cost structures, estimating and tracking cost.
  • Managing and mitigating cost with various instance and storage options.

Click here for Part 1
Click here for Part 2

PalominoDB in Argentina & India in Fall of 2013

It will be a busy Fall season for the staff at PalominoDB as we will be sponsoring and speaking at two major international conferences.

First up is the MYSQL NOSQL & Cloud Expo in Buenos Aires, Argentina, OCT 15-16 2013. PalominoDB CTO Jay Edwards & DBA Emanuel Calvo will be featured speakers at this 2nd annual conference dedicated to building communities in Latin America.

Next is the AIOUG-SANGAM 13 in Hyderabad India, Nov 8-9, 2013.
PalominoDB CTO Jay Edwards and VP of International Operations Shiv Iyer, will be representing PalominoDB as a Silver Sponosor at one of the largest events in India.

PalominoDB announces new Vice President of International Operations, Shiv Iyer.

Shiv IylerShiv is the Vice President of International Operations at PalominoDB. He has been a database operations strategist for companies such as; LinkedIn, Microsoft, SUN, National Iran Oil Co. and lastminute.com.

He has taught and been a keynote speaker for database technology conferences globally, speaking on; Database Architecture, High Performance, Database Health Check, 24*7*365 DBA Operations, High Availability and Managed Database Service Operations.

He is a Database Architect specialist in PostgreSQL, MySQL, Oracle and SQL Server and passionate about DBA process automation and maximum return on investment from database infrastructure. He is the founder and CEO for Ask DB Experts, a boutique private label Database Architect Consulting Practice & Remote DBA services company based out of Bangalore, India.

He is a marathon runner and very active in supporting NGOs in fund raising marathon events.

Welcome Shiv!

PalominoDB CEO Laine Campbell at Velocity NYC Conference Oct 14th

Laine Campbell CEO PalominoDBPalominoDB CEO Laine Campbell will be presenting "Using Amazon Web Services for MySQL at Scale" at Velocity NYC Conference Oct 14th 2013 at 11am.

Laine will explain the options for running MySQL at high volumes at Amazon Web Services, exploring options around database as a service, hosted instances/storages and all appropriate availability, performance and provisioning considerations using real-world examples from Call of Duty, Obama for America and many more. Laine will show how to build highly available, manageable and performant MySQL environments that scale in AWS—how to maintain then, grow them and deal with failure. Some of the specific topics covered are:

  1. Overview of RDS and EC2 – pros, cons and usage patterns/antipatterns.
  2. Implementation choices in both offerings: instance sizing, ephemeral SSDs, EBS, provisioned IOPS and advanced techniques (RAID, mixed storage environments, etc…)
  3. Leveraging regions and availability zones for availability, business continuity and disaster recovery.
  4. Scaling patterns including read/write splitting, read distribution, functional dataset partitioning and horizontal dataset partitioning (aka sharding)
  5. Palomino cluster tool (OSS) and Jetpants for building and managing large sharded MySQL in EC2
  6. Configuration management, backup/recovery, monitoring and trending considerations.
  7. Common failure modes – AZ and Region failures, EBS corruption, EBS performance inconsistencies and more.
  8. Cost structures, estimating and tracking cost.
  9. Managing and mitigating cost with various instance and storage options.

For more info visit the Velocity NYC Website

PalominoDB CEO Laine Campbell to speak at LGBTQ MBA Conference

PalominoDB CEO Laine Campbell will be a special guest speaker at this year's, Reaching Out LGBTQ MBA Conference.

Calling her "...an experienced tech entrepreneur, CEO, world traveler and self-starter," event organizers hope Laine's involvement "will help ignite and encourage more members of the LGBT community to successfully pursue careers in tech or entrepreneurship."

Now in its 16th year, the conference promotes the visibility, education, and networking capabilities of LGBTQ business leaders around the globe and annually draws over 1,200 students and executives.

Their mission is to empower LGBT MBA students to become professionals who will lead the way to equality in business education, in the workplace and throughout society.

 This year the conference will be held at the Hilton Riverside in New Orleans, LA on Oct 17-19, 2013. The theme this year is "Building Visible Leaders: Connecting a Community" as the organization believes that together, they will continue to change history and advance full LGBT equality.

Key Information
2013 Reaching Out LGBT MBA Conference
Date: Oct 17-19, 2013
Location: Hilton Riverside in New Orleans, LA

PGBR 2013 Porto Velho insights

It's been 4 years since the last time I spoke at the Postgres Brazilia community event (last time I spoke was at PGCon 2009 Sao Paolo - PyReplica project) and seems that the community is still growing and vibrant.   The talks given at the meeting were amazing.  PalominoDB was there with two talks: "Postgres and FTS" and "Postgres on AWS",  which premiered PalominoDB as  the new "Amazon Advanced Technology Partner".

 

Something that caught my attention during the event was a large number of talks about PostGIS. It seemed to me that most of the projects that are being developed are related to geolocalizing and mapping, specifically in the Amazon region.  

 

Other interesting talks were from Alvaro Hernández and Joseph Conway.  Alvaro Hernández spoke about the "billion table project ". He discovered the real limit on the table creation in a cluster: 2.147.483.483. I don't know anybody yet who wants to reach that level on a production environment, but the road to find that number was entertaining and enriching.   Joseph Conway delighted us with an Amazing PL/R talk, showing the power of analytics in Postgres combined with R language.

 

The event was at the same time that Postgres 9.3 was delivered in the beta 2 version and surprisingly there wasn't any talk about the new features incoming.

 

I just can't write this post withuot a special mention to the organization of the event, specially to Luis Bueno and Alex Lopes.

PGBR 2014 is planned to be located in Florianópolis and PGBR 2015 is planned to be located in Bahia. We’re looking forward to our future participation with more talks!

PalominoDB will match up to $500 of donations to the Ada Initiative

PalominoDB is proud to be the first company participating in a matching donation challenge for the Ada Initiative. Palomino will match up to $500 of donations in the next 24 hours, until 3pm PDT August 20th (20:00 UTC August 20).

The Ada Initiative helps women get and stay involved in open source, open data, open education, and other areas of free and open technology and culture. These communities are changing the future of global society. If we want that society to be socially just and to serve the interests of all people, women must be involved in its creation and organization. Their Impostor Syndrome training program helps women more accurately judge their abilities, and their AdaCamp unconference introduces women to other open tech/culture entrepreneurs and potential co-founders. You can visit them here and donate here.

PalominoDB CEO Laine Campbell says, “Palomino supports the Ada Initiative because they are a woman-owned non-profit that is always striving to bring more women into their technology, support and leadership teams. As CEO, I’ve experienced the challenges of breaking into tech as a woman, as well as encouraged other women to grow, contribute and participate in the technical ecosystem. We’re excited to support and contribute to the Ada Initiative with its focus on open source community and technology.”

Join PalominoDB and Ada Initiative in supporting women in open source, open data, and other areas of open tech/culture! Donate now!

PalominoDB named as one of the first "Advanced Technology Partners" by AWS

Amazon Web Services PartnerPalominoDB is proud to announce that we have become one of the first to gain the public designation of  AWS Advanced Consulting Partner!

Signaling our company's deepening investment in AWS, we've been recognized as one of the top global APN partners. It recognizes our ability to provide significant value to our large and diverse base of customers using AWS as a business platform. Building a highly available modern cloud infrastructure is difficult. With this partnership, we will gain access to resources that will help us make those implementations easier. We're excited to be working more closely with AWS and look forward to the benefits it will provide!

Critical MySQL 5.6 bug: GRANTs and replication

Critical MySQL 5.6 bug: any user with GRANT privileges can unwillingly cause all replicas to break

 

The latest major release of MySQL brought us a lot of new and exciting features. As always, new features come with brand new bugs waiting to bite you in the least expected way.

I was implementing a monitoring system for a client and needed to create a new dedicated user. I went ahead and granted the usual basic privileges but later, I discovered that this user also needed the REPLICATION CLIENT privilege because the monitoring needed it to check for replication status.

No big deal, let's grant this as well. Unfortunately I did not get it right at first:

grant replication client on database.* to user@'10.%';
ERROR 1221 (HY000): Incorrect usage of DB GRANT and GLOBAL PRIVILEGES

Ouch! My bad! This is a global privilege and I should grant it on "*.*":

grant replication client on *.* to user@'10.%';
Query OK, 0 rows affected (0.10 sec)

I flushed privilege table and moved forward to other things.

 

After a short while, a page came in: replication had stopped on a replica of the database I was working on.  I quickly connected to the paging replica and issued a SHOW SLAVE STATUS:

        Seconds_Behind_Master: NULL

         Last_SQL_Error: The incident LOST_EVENTS occured on the master. Message: error writing to the binary log

LOST EVENTS?  I heard about those in MySQL Cluster before, but we are using stock 5.6 replication here.  A bit of digging on the MySQL bugzilla site lead me to the following bug:

Bug #68892 Invalid use of GRANT command breaks replication

 

Basically, any GRANT command that errors out will break replication, and you will have to skip the offending event on ALL replicas (or inject an empty transaction if GTID in use).

 

The reason behind this is the fact that MySQL 5.6 writes the following in the binlog when your grant fails:

#130410 18:06:40 server id 1 end_log_pos 874 CRC32 0x4d815dbc
# Incident: LOST_EVENTS
RELOAD DATABASE; # Shall generate syntax error
# at 874

The purpose is to avoid that a grant that fails on the master due to insufficient privileges, succeeds on the replicas instead (due to the fact that replication thread runs with SUPER privileges). To avoid that, replication is stopped with an error.

Bottom line is, any user on 5.6 which has GRANT privileges should be careful when GRANTing - as you have seen, it is very easy to break replication completely.

Verified on MySQL 5.6.12.

Insights from the PgCon 2013

PgCon 2013 was attended by 256 people across the globe.  Attendees had the opportunity to enjoy tutorials, talks and an excellent unconference (this last deserves a special mention).

I lectured a talk related with Full text search using Sphinx and Postgres (you can find the slides at http://t.co/lgFoLq37EC, and all of the talks have been recorded).  The quality of the talks in general was quite good, but I don't want to repeat what you will find in other posts.

The unconference was attended quite late into the evening. You can find a schedule of it, as well as the minutes of some of the talks that happened (and others that didn't also) here.

There was a special emphasis on the pluggable storage feature, albeit most agree that it will be a very difficult feature to implement in the near versions. A topic related to this, was the Foreign Data Wrapper enhancements.

Pluggable Storage engine was extended after. The main reason of why everybody agrees with this feature, is because an API for the storage will allow companies to collaborate with code and avoid forks to other projects.

There was a long discussion also about migrations on the hall, using pg_upgrade.

The features about replication were bi-directional and logical replication.

Full text search unconference discussion was pretty interesting. Oleg Bartunov and Alexander showed a really interesting work coming up for optimizing GIN indexes. According to their benchmarks, Postgres could improve the performance significantly.

There were a lot of discussion I missed, due the wide number of tracks and "hall spots". But th emajority of attendees I heard agreed that the unconference was quite exciting and granted the possibility to bring many new ideas.

 

 

 

 

Syndicate content
Website by Digital Loom