Bug/Site Problem: Submission Notifications Not Appearing

[Vrghr]

This is definitely still going on. Was just browsing new art by category, and noticed multiple instances of new art by folks on my watch list. None of which has shown up on my "New" notifications. Some submissions were only hours old, others from a day or more ago.

While it is possible to search out new art/story/etc. submissions using the Browse and Search tools, it's far harder to find new journal submissions.

Looks like this has been going on at least 5 days now. Any clues about estimated fix times?

Vrghr

 

[quoting_mungo]

At this point it seems the servers are hell-bent on throwing off any estimate we make; anything I can say would be a wild guess at this point. The rollback that's hogging the table and causing the issues is about halfway done by number of rows, but no idea if it's going to decide to blaze or slog through those remaining ones.

(Disclaimer: The above is obviously not a very technical explanation. The "about halfway" figure is accurate, though.)

 

[STrRedWolf]

Probably another five days. They updated the notice and said "it's half way done" so I'm going to employ SWAG research and say another five days.

I'm not even going to touch FA in terms of any notifications.

 

[Vrghr]

At this point it seems the servers are hell-bent on throwing off any estimate we make; anything I can say would be a wild guess at this point. The rollback that's hogging the table and causing the issues is about halfway done by number of rows, but no idea if it's going to decide to blaze or slog through those remaining ones.

(Disclaimer: The above is obviously not a very technical explanation. The "about halfway" figure is accurate, though.)

Thanks muchly for the news, "technical" or not. *grins*

Wuffy's curious now - - When the system finally gets done hashing its way thru the database, are we going to get deluged with a sudden influx of "new" notifications for everything that happened since the last? Or are those missing notices lost for good?

Vrghr's betting on the latter, but was just wondering...

 

[PheagleAdler]

Probably another five days. They updated the notice and said "it's half way done" so I'm going to employ SWAG research and say another five days.

I'm not even going to touch FA in terms of any notifications.

SWAG!

That's a hopeful estimation, I guess? In the meantime I'll keep my artists a click away in my bookmarks, just to stay updated.

 

[STrRedWolf]

Full update from Chase w/Dragoneer's approval: http://www.furaffinity.net/journal/6311970

In short for this problem: They're making code changes to not only stagger out notifications for less lag, but also to not send notices to those banned or dead. They're also cleaning up notices for the banned/dead users, which is taking so dang long. Millions of notifications have to be removed. ETA: Thursday.

No word on recovering lost notifications!

 

[PheagleAdler]

Full update from Chase w/Dragoneer's approval: http://www.furaffinity.net/journal/6311970

In short for this problem: They're making code changes to not only stagger out notifications for less lag, but also to not send notices to those banned or dead. They're also cleaning up notices for the banned/dead users, which is taking so dang long. Millions of notifications have to be removed. ETA: Thursday.

No word on recovering lost notifications!

Thank God...wish they'd done that ages ago. Those users don't need to be sent notifications.

 

[DrakeDragon]

Pretty sure the notification group staggering was done a year ago http://www.furaffinity.net/journal/5311872/ unless that "beta" really has been sitting unfinished for a year?

 

[quoting_mungo]

The staggering was done quite some time back, yes.
The current issues are caused by the cleanup query for banned/deceased/permasuspended accounts' notifications locking up the database table, being cancelled partway through due to the issues that caused, and the rollback deciding to take forever (continuing to hog database time better spent elsewhere).

 

[DrakeDragon]

I still don't understand why they didn't make the site read-only before performing such enormous database maintenance. :T

 

[GamerFox]

Cuz then people would whine.

FA can't really win in any case.

 

[DrakeDragon]

"People whining" vs. "a site that acts like it's working but actually isn't (plus people whining)"

I know which option I'D choose

Yeah I know, wah wah furries can't see their porn. What has me mad about this isn't that, so much. It's that (1) they did this massive DB maintenance without planning it or letting any of the users know they were doing it until AFTER it had already begun and was causing problems; (2) they decided to do this during the holiday season when commission artists do a lot of business; (3) they haven't made any attempt to FIX the issue or at least put the site in read-only mode to keep it from happening, instead letting it continue to eat all our notifications, which is somehow better than having the site be completely inaccessible; and (4) they haven't responded to any of the concerns about this quasi-outage even though it's been going on for the better part of a week now.

It's nonexistent communication all the time with FA.

 

[Harbinger]

So once thus us all over will we get the notifications we missed or just the ones after the fix?

 

[GamerFox]

From what I told, the lost notifications are gone forever.

 

[Kalmor]

From what I told, the lost notifications are gone forever.

Nope. If required we can run a script to re-send them.

 

[Felixpath]

Nope. If required we can run a script to re-send them.

Define "if required." Isn't it required for every user who has missed submissions and journals? On the other hand, what's the consequence of suddenly flooding every user's inbox with a week's worth of notifications?

 

[Kalmor]

Define "if required." Isn't it required for every user who has missed submissions and journals?

It was just an offer from one of the tech staff (in our communication channels). It isn't set in stone yet.

On the other hand, what's the consequence of suddenly flooding every user's inbox with a week's worth of notifications?

I do not know.

 

[GamerFox]

I do not know.

Maybe hold off on that; that might just overtax the server a bit more and get us back into a mess.

Also, are they tying the notice's percentage meter directly to the progress percentage that the query's doing?

 

[Thanatos2k]

The missing notifications definitely should be sent, and the site should be read only until that purging finishes. Missing notifications are worse than the site being down for both artists and viewers, because if the site was down the artists would wait to upload until things were working. Countless pieces of artwork might as well be lost to the void since people depend on notifications to know whether they exist or not.

 

[Kalmor]

Also, are they tying the notice's percentage meter directly to the progress percentage that the query's doing?

Nope.

 

[GamerFox]

Then how is the number updating? You aren't having someone go and change the number every time the process moves another percentage point are you?

 

[rwpikul]

More likely it gets updated when said someone happens to check the progress.

As for a re-send: I can survive without the submission notices being resent, (I have a couple of fairly easy ways to deal with that), journals and comments are more important. I would _like_ everything resent, but its not worth overloading the system to get them out.

 

[Amber]

More likely it gets updated when said someone happens to check the progress.

As for a re-send: I can survive without the submission notices being resent, (I have a couple of fairly easy ways to deal with that), journals and comments are more important. I would _like_ everything resent, but its not worth overloading the system to get them out.

You are still getting your comment notifications currently. What's not being sent out are notifications that get sent to multiple users. Since comment notifications only need to be sent to one user they are being sent fine.

 

[Socks the Fox]

It looks to me that the batching system that's currently being affected the most by this. Therefore, it seems notifications that get sent out to 1000+ watchers are broken (namely, submissions and journals). It wouldn't surprise me if the first 1000 watchers of a popular artist are getting notifications and the rest aren't, too. It seems to me that people with less than 1000 watchers are sending out notifications fine (as I do get a few from various artists).

My vote is to send out backlogged notifications.

 

[GamerFox]

Oh yeah, how much time will sending the backlogged notifications take? Another 5 days? Dump 'em.