Panic! cdbUsers table empty!

[Bizbuzz]

Hi,

Been developing a system for a while now with mobile apps and PC/Mac software.

All my users seems to be gone in the cdbUsers table! All other tables (each user have 2 tables) are still there. What is this? 

Do you have logs or something to see when it was emptied? Could be somewhere in the latest 7 days this happened. Do you keep backups ?

Besides of that the LiveCloud Manager is working very slowly. UK region.

Br /John

Added. Maybe cdbUsers is not empty just unresponsive ? attached a screenshot from LiveCloud Manager trying to create user.

[efrain.c]

Hi John,

Thank you for brining this to our attention.

Let me start by saying that the issue has been resolved. Your guess was correct. The Users server was unresponsive due to corrupted data. This prevented the server from processing any requests. The server was able to process all of the backed up requests after we removed the corrupted data.

To answer your questions, yes we do have transactions logs and backups of your data.

Please let us know if you run into any issues or any of your data is missing.

[Bizbuzz]

Hi John,

Thank you for brining this to our attention.

Let me start by saying that the issue has been resolved. Your guess was correct. The Users server was unresponsive due to corrupted data. This prevented the server from processing any requests. The server was able to process all of the backed up requests after we removed the corrupted data.

To answer your questions, yes we do have transactions logs and backups of your data.

Please let us know if you run into any issues or any of your data is missing.

Hi! Thanks! 
Can you somehow do something to avoid this in the future? Or perhaps monitor things 24/7 and get notifications if it does happens? 

Is the US droplet more stable or better monitored as you yourselves are using it (I guess anyway)?

No for now, but further in future, do you have any plans for some kind of failover solution if the server becomes unavailable. 

Anyway, thanks for solving this issue now.

[mark_talluto]

Hi John,

We have identified an issue with non-ascii characters being used in the naming of columns. We had a meeting this morning where we discussed some potential solutions to the root of the problem. We have had this issue happen to us twice now. We moved the matter up to the highest priority.

We usually have monitoring software that does regular health checks on each instance. Our current problem is that we closed out our offices and need to relocate the visual monitor to each of our homes. This goal has been moved up in priority as well.

We knew that LiveCloud has been performing exceptionally well. I think its high reliability has lulled us into thinking we could delay our monitoring system migration. Complex systems like this benefit from automated system health checks. We need to bring it online as soon as possible. We should have that up and running in a week or so.

We appreciate the support all of you have provided as we work to improve LiveCloud.