Other What's ruining your life right now?

Much obliged for the detailed reply - far more detailed than I expected, so thank you for that!

I've found that the algorithms used by Spotify for its "Discover weekly" and "Daily mix" playlists are actually pretty good - although I've been using them so often that the same bands keep coming up! I did see services to transfer playlists, and that'll definitely come in handy.

Oddly enough, I've found that the list of integrations for Spotify to be quite limited. Plex is a perfect example - there is no official Spotify support in Plex (and I use Plex a lot), yet there's an official Tidal integration. When I still had the Roku the Spotify app on there was utter garbage: by the time I ditched the Roku you couldn't even see your own Spotify playlists, you had to manually search for everything (which is a total ballache on a remote that doesn't have a keyboard). I have a Shield now, so this is less of an issue...

Thanks for the info; I'm going to give the trial a go and see how I get on. A Tidal family account is double the price of a Spotify family account, but if it's worth it we can easily absorb that extra cost.

 

Can't drive in today, roads around where I live are too treacherous for a motorbike. This means I have to get a lift in with the other half; meaning that I can no longer drive myself home in the afternoon to work from home; meaning that I have to reschedule a telephone interview, because there are no meeting rooms available and I don't really want to do that at work anyway.

 

I half expected you to be posting a picture of a nail in your tyre or something.

 

Oh please no, not on fresh rubber!

 

Do what they do here, toilet cubicles, fire escapes, other peoples meeting rooms or so called "quiet areas" are all fair game for private conversations

 

https://uk.camelcamelcamel.com/

I'm in a sort of consumer stasis out of having no idea how much things should cost, or what's a "good deal" any more.

But aside from that... what sort of operation are these clowns running here?
- Three disks fail and bring down your entire operation for at least a week?
- Replacing said three disks somehow costs $15000?
- When was your last backup? I feel like losing a few days of users and price history, possibly even referral clicks isn't the end of the world... so are we talking months? Hiring a recovery company reeks of this sort of desperation.
- Asking for donations to fix things? Maybe I've misjudged the scale of the operation here, but I was thinking that the service wasn't entirely altruistic and boatloads of referral clicks should do a pretty decent job of keeping the lights on.
- Mostly, how are you not running something with the seasonality of retail in AWS to start with?

When your revenue stream is entirely dependant on being immediately available, I feel like they should have been taking availability more seriously. I'm sure this ordeal is quite the learning experience.

 

I'm struggling to get my head around this one, unless they were using uber expensive multi-TB SSDs

 

I find it plausible that there would be given that latency is key to user experience here. But if it's the case, I find it even more ridiculous that there's not a greater availability architected in... like matching their 8-16TB SSDs with an async replica on HDDs for a comparatively tiny incremental cost. And if they're in the financial position that they need to ask for donations to get up and running, I feel rather than spanking their budget on top-end enterprise kit "because enterprise", they could have built something a helluvalot more resilient with consumer grade kit.

Of course, to even begin to ponder that, you have to get past the fact that they're running on what appears to be home-rolled kit racked up in a colo, which I find utterly bepuzzling.

 

'we had backups, but they're old so we didn't/couldn't use them and spunked away $30k on data recovery instead...'

...so what you're saying is you *didn't* have backups?

 

It's not really an ideal situation for a telephone interview. There were no meeting rooms available and there are rarely quiet parts of our office. It's fine, I've rescheduled.

Their backups either aren't taken regularly or their backup protocol failed. Either way it boils down to what @RedFlames said - they don't have backups. DB backup/restore is something that we've been doing for freakin' decades now, so not having a tested and reliable backup protocol in place for your production system is beyond contempt.

Little tip for that site in the future: Amazon Aurora. Do it. Do it now and get the MySQL variant. Full backups at least once a day and incremental diffs, meaning you can restore your DB back to any point in time that you choose (up to about 3 weeks ago IIRC). The MySQL variant has a feature called "Backtrack" which performs the rollback in-place: no need to take your cluster or nodes offline, or provision a new master node from the backups, or any of that awkward stuff (PostgreSQL variant doesn't support this yet). You don't even have to write the backup scripts/jobs or anything - just click buttons in a web control panel.

This is on top of the existing Aurora benefits like read-only replicas, nodes spread across AZs, zero replication delay, etc. It's pricier than standard RDS but it is so worth it - it's saved my arse on more than one occasion.

 

IMO you'd only spend that much on recovery if you absolutely had to.... and you'd only absolutely have to if you had no backups.

...and if your most recent backup can, under any circumstances, be 'too old to be usable', you're not backing up frequently enough [and don't actually have a backup],

 

It's a co-operative run jointly by tribes in South American jungles and polar bears in Svalbard, hence the name.

 

Actually they run almost entirely on potato-powered servers.

 

i think i pulled a muscle in my ribs...

i was assisting my mother and sister pack for their holiday, and while i was lifting the case suddenly it felt like i had been stabbed in the chest... i played it down alot to them as i didnt want to jeopardise their holiday

but all day ive been in bed in agony, and its at the point where ive taken painkillers ( for the first time in my life ) to try to dull the pain, every time i move or breath it sends shooting sharp pains through my left hand side

 

Eh. Working in an MSP has turned the 'shock knob' all the way down to 2 on what people will try and get away with instead of paying for things to be done properly.

I'm assuming the data recovery costs were so high because three failed disks with a two failure capacity implies RAID (...6?), which also suggest mucho-longo and arduous task of getting at least two of those disks to readable so the entire RAID set can be frankensteined to life for a data copy. Hell, last time I dealt with data recovery it was for a payroll PC (Why the data wasn't on the server? That version of the software - Strangely, not Sage - cost more and ewspending money) and it cost them £1200 for a 500gb drive on a 36 hour turn around.

I dread to think how many sites and services we all use daily that have a poorly backed up, or not at all, single point of failure.

 

In my experience, whether you pay $30k or $300k, you're not getting owt useful off a failed RAID set.

 

Have to get up in 6 hours and I'm not tired at all. Will be a fun day ahead.

 

Make sure you lie awake for the next 5h30 thinking about how soon you have to get up, and then fall into the deepest sleep you've ever had.