undefined | Better HN

0 pointsruben81adelaide3y ago0 comments

> Does Cloudwatch confirm that RDS "needed" to scale?

Actually you pointed out to a clue that I missed. I should have checked cloud watch!

The free space graphs shows that at some point something got the 200 GB of space that we originally assigned.

This is a good clue that I am going to dig in.

Thanks for your feedback!

EDIT: Cloudwatch metrics in RDS has been the key to find the source of the issue.

0 comments

viraptor3y ago

So what was it? My money would be on (in order): Wal, logs, unexpected indices growth.

ruben81adelaideOP3y ago

The things started to go from bad to worse when work_mem parameter has not been set up correctly.

Some queries that requires a large amount of memory to process started to use disk.

Once the autoscaling kicked in, even if we would have realized about it, wouldn't have helped as you are locked out of the system.

The auto-scaling event was triggered earlier than our alerts for low disk available.

What I don't know yet, is that we have been moving services for multiple days, and the service that required the work_mem parameter was in production in that AWS for 48 hours before started to use disk rather than memory to process the SQL Queries.

synicalx3y ago

Interesting problem to come across, sounds like this is a scenario where RDS and the lack of host access/visibility was a bit of a handicap. Glad you found the issue though, hopefully things go smooth next time!

j / k navigate · click thread line to collapse