"We need to ensure that if memcached is down that the site can still continue" sounds a lot like
"We need to ensure that if postgresql is down that the site can still continue" to me.

Currently memcached stores important stuff, the only way I can see us fixing this ticket is by moving the sessions elsewhere (which is MBS-4419).

Perhaps this should be read as "the site should still work for all tasks that don't require memcached, i.e., anything read-only, when memcached is out of commission". I agree that the main fix is to move away from memcached, but it's still definitely true that our webservice shouldn't go down because memcached crashed.

+1 to Ian.

The web site and web service should be able to continue running when the release editor cannot create sessions. The current failure of the entire site has nothing to do with how we store release editor sessions – in fact the same will be true of a more resilient data store.

And MBS-4419 isn't quite actionable at this point in time. It seems that there are two tasks conflated in that bug report – I'll respond on that bug.

Please fix this and get it deployed ASAP. Then we can spend time on 4419.

The webservice seems unaffected, I haven't been able to reproduce the problem where the webservice doesn't work if memcached is unavailable. In theory this could be a difference between production and me dev setup, but that seems unlikely.

For the website itself as far as I can see that only affected logged in users, as for them an attempt is made to obtain an existing session and store it again – a work-around for that is on code review. http://codereview.musicbrainz.org/r/2171/

Shipped to production today, but it is spamming oddlog:

2012-09-03 10:34:22.462181500 value for memkey:catalyst_session#0.01#MusicBrainz::Server is not defined at local/lib/perl5/Cache/Memcached.pm line 500.
2012-09-03 10:34:22.462183500 Use of uninitialized value $val in length at local/lib/perl5/Cache/Memcached.pm line 502.
2012-09-03 10:34:22.462185500 Use of uninitialized value $val in concatenation (.) or string at local/lib/perl5/Cache/Memcached.pm line 521.
2012-09-03 10:34:22.627633500 Use of uninitialized value $key in pattern match (m//) at local/lib/perl5/Catalyst/Plugin/Session/Store/Memcached.pm line 78.
2012-09-03 10:34:22.627749500 value for memkey:catalyst_session#0.01#MusicBrainz::Server is not defined at local/lib/perl5/Cache/Memcached.pm line 500.
2012-09-03 10:34:22.627770500 Use of uninitialized value $val in length at local/lib/perl5/Cache/Memcached.pm line 502.
2012-09-03 10:34:22.627772500 Use of uninitialized value $val in concatenation (.) or string at local/lib/perl5/Cache/Memcached.pm line 521.
2012-09-03 10:34:22.638502500 Use of uninitialized value $key in pattern match (m//) at local/lib/perl5/Catalyst/Plugin/Session/Store/Memcached.pm line 78.