Thanks to your generosity, we have reached our target of £56,000 and the funds in place to make the planned server upgrades a reality. A big Thank You to the huge number of people and companies who contributed, and especially to our launch partners Mapbox and Mapzen!
It's upgrade time! OpenStreetMap is growing, servers are projected to hit capacity by mid 2015. OpenStreetMap is a global community of dedicated and resourceful people, giving their time to make the best map of the world. Those mappers’ contributions are the largest part of the project. However, contributors need infrastructure such as a repository to share their contributions and services that help them make the map and talk to each other. We are raising funds to keep these services running and improve them, so that everyone else can continue making an amazing map.
The OpenStreetMap project is a global community of dedicated and resourceful people, giving their time to make the best map of the world. Those mappers' contributions are the largest part of the project. However, OpenStreetMap.org is inching towards capacity. Our last funding drive was exactly designed to bring us to this point based on growth projections. We are now raising funds for an infrastructure upgrade that will sustain OpenStreetMap.org into 2016.
Secondary database server
The secondary database server, until recently the primary, has
filled up almost all of its available disk space — which is great,
because it’s filling up with awesome mapping data generated by the
many thousands of dedicated contributors world-wide. We want to be
able to continue to store that growing set of data, and run the
website and API on it.
At the moment, the disks on that machine are configured in a
high performance setup and we will try to make some space
available by reconfiguring in a slightly lower perfomance
setup. This may mean slower queries to the secondary machine, and
worse API performance. This would be even worse, should the primary
server fail and all the website and API database queries move to the
secondary. We don’t want the site and API to be slower — we want a
great experience for those dedicated contributors.
To avoid this, it would be better to have a machine closer in
specification and performance to the primary database server in
case it is necessary to fail over to it.
Tile rendering servers
The primary tile rendering machine is also quickly filling up the disk
space available to it — which is also great, as it’s filling with
the detailed geometries produced all the time by the global
We want to be able to continue to serve up minutely-updated tiles to
everyone who is helping make OpenStreetMap more complete and more
detailed — or even just looking around. Unfortunately, this machine
is already stuffed full of high performance, high capacity disks,
which means we need to look at replacing it with something more modern
The secondary machine is going flat-out and the bottleneck is the
transfer of data to and from the disks. The machine could keep the
tiles more up-to-date if we removed that bottleneck, which means
upgrading the RAID controller and adding a mirror disk for the main
We know that out-of-copyright (OOC) maps and aerial imagery are both
very useful sources for mapping activities, and we’re not happy that
the machine which used to serve this data is currently down. The
reason is that the large external disk arrays used to store the
imagery are very old and consume too much power for the home they are
in while our friends at UCL re-build the building which used to be
To fix this, we can get a smaller, more modern and more power
efficient server to serve that data, and get this service back online
where it’s useful to everyone.
Services machine replica
The “services” machine currently runs a mix of stuff, including the
planet download site, backups and the internal configuration
server. All of these are also backed-up remotely, for safety. However,
it would take some time to recover those files from a remote backup,
simply due to the greater bandwidth within the server room. We would
like to have a “live” backup of this machine to minimise any downtime
when the primary machine, or the site it’s hosted at, aren’t
Sadly, nothing lasts forever. And hard disks, SSDs, RAID controller
batteries, PSUs, fans, cables and many other computer components don’t
even come close! Every year, we replace many of these items which have
failed or come to the end of their lives. Many are replaced under
warranty, but we still have to spend money on replacing or fixing
those which are out of warranty or not covered.
The Operations Working Group has scaled OpenStreetMap since 2009. We're here to maintain servers day-to-day from applying security patches to fine tuning the stack. From time to time we apply larger upgrades, here are some of the more notable ones.
For a recent, detailed history of OWG activity, please see the OWG log.
2013 Funding Drive
- Added a new database server for greater reliability through redundancy and speedier editing.
- Added a new services server for greater planet capacity and reliability.
- Upgraded disks for the primary tile rendering server.
2013 budget: £40,000
2011 Funding Drive
2011 budget: £15,000
Details & budget
Secondary database server
The defining characteristic of a database server, for a database which
doesn’t fit entirely within RAM, is I/O performance. For that reason,
we choose a chassis which is able to accomodate a very large number of
disks — in this case, a whopping 72 2.5” 10K disks. Although we
don’t need all 72 (yet) we do need 25 900GB disks and it’s good to
know that the capacity is there for future expansion.
To keep as much of the data accessible as quickly as possible, we also
want lots of RAM to use as system cache. 256GB seems to be the largest
that it’s reasonable to get.
Finally, we add on a couple of good quality, enterprise grade
SSDs. For reliability and especially under hard power-off conditions,
you still want enterprise grade despite improvements in the pro-sumer
Tile rendering servers
Tile serving is about really good disk read performance, lots of
memory and a good CPU. It’s possible to fit almost all of a rendering
database in memory, if you buy enough. The optimal spot at the moment
seems to be getting about 128GB RAM and a couple of pro-sumer SSDs for
the most often accessed disk data. The amount of data being read
requires a good RAID controller, too. In this case, we’re looking at 8
600GB 15k SAS disks with a couple of 512GB Samsung 840 Pro SSDs.
It’s all about the I/O — the secondary tile server can’t handle it
at the moment, but the addition of a good RAID controller and a pair
of 512GB Samsung 840 Pro SSDs will soon sort that out.
Serving aerial imagery is also about good disk performance, same as
serving tiles but the aerial imagery tiles tend to be quite large and
not rendered as often. Reprojecting and slicing up aerial imagery can
be CPU intensive though, so the specification for an aerial
imagery server looks a lot like that of a tile server but with more,
slower disk — 128GB RAM, a couple of low-voltage E5-2650Ls and 6 4TB
Services machine replica
The services machine does a lot of odd jobs; it serves GPS traces to
the website machines, it serves the
planet, stores backups, aggregates
log files, runs the Chef configuration server and serves internal DNS
requests. All of this means it needs to be a general-purpose
machine, with quite a lot of RAM, decent CPUs and a lot of space for
storing all those GPS traces, dumps and planets. A suitable spec is a
couple of decent CPUs (E5-2650LV2s) with 256GB RAM and 9 4TB SAS2
Parts on hardware fail with alarming regularity, especially the
mechanical parts. Usually these are covered by the standard warranty
we get on all new machines, but some machines aren’t covered —
either because they’re donated or we buy them out of warranty to get a
good deal. The failures themselves are random, and therefore pretty
hard to predict, but a good rule of thumb is about 10% of the overall
budget as a contingency, so that’s about what we’ve estimated here.
All numbers approximates based on market prices.
Who's doing the upgrade?
The OpenStreetMap Foundation's Operations Working Group. Read more about the Operations Working Group over here.
What's the OpenStreetMap Foundation?
The OpenStreetMap Foundation is the non-profit organisation which supports the OpenStreetMap project by maintaining its servers, providing a legal structure around the project's database, by organizing an annual conference and more. Learn more about the OpenStreetMap Foundation on osmfoundation.org.
We plan our hardware funding drives based on 6 to 12 month projections of scaling the current platform. We've determined that at current growth, we'd see first significant downgrades in performance by mid-2015 and now's the time for building ahead.
Who is receiving funds?
The funding drive is an iniative by the OpenStreetMap Foundation's Operations Working Group (OWG) - all funds are collected by the OpenStreetMap Foundation and made available to the Operations Working Group.
Are you a 501(c)(3)?
No. The OpenStreetMap Foundation is a UK-registered not-for-profit organisation that supports the OpenStreetMap Project. It is not a 501(c)(3) in the sense of the Internal Revenue Code of the US Internal Revenue Service (IRS).
How long will these funds last?
We have designed this funding drive to allow for investments that will let us scale OpenStreetMap's infrastructure into 2016.
Why only scale into 2016?
Everything else would be overbuilding. We also don't usually scale the same components. For example, while we're upgrading the secondary database server this time, we'll probably not need to touch it the next time around.
What's going to happen once you've used these funds?
There will be another funding drive for hardware :) Seriously, the model of focused funding drives is working very well. It allows for a very clear value for money and it's what made it possible to scale OpenStreetMap from a server under a desk to supporting over 90 million edited objects each month.
What else can we do to support OpenStreetMap?
Join us! Aside from funds to upgrade our hardware what we need most are people. Whatever your background - technical or not - you can help OpenStreetMap. Here are just a few of the many possible ways to get more involved:
If you have contributed data to OpenStreetMap, or time to helping organise mapping activities: thank you for everything you’ve already done!
OpenStreetMap only survives because of the work put in by thousands of hard-working and selfless volunteer contributors. If you are one of these people, then you are already doing a great thing for the project.
If you would like to do more, then you can:
- Join the Foundation and support the body which supports (but doesn’t control) the project. By being a part of the Foundation, your voice can be heard in discussions about how the Foundation is run.
- Join a Working Group and give your time to solving those issues which require greater commitment. Being on a Working Group, as the name suggests, is hard work and requires diligence, but can make a huge difference to the project.
- Donate money to the hardware funding drive. Whatever you can spare will be gratefully received and put towards hardware which is necessary for the continued growth and success of the project.
Whether you use OpenStreetMap data in your products and services, or are just interested in helping, corporate sponsorship of OpenStreetMap events and hardware helps the project to keep going and be inclusive.
If your company would like to help the project grow and succeed, then you can:
- Join the Foundation as a corporate member. Your support of the Foundation will be greatly appreciated and publicly lauded.
- Donate time. Some of your employees might jump at the chance to be more involved with OSM, so why not give them a “20% time” to join a Working Group or contribute time to a project which helps OSM?
- Donate money to the hardware funding drive. Any contribution will help and is gratefully received.
How can I donate more?
Get in touch