Bug/Site Problem: Submission Notifications Not Appearing

[Danza]

Many users on my twitter were telling me that art I had uploaded wasn't showing at all in their inboxes, despite the art being up for over an hour. Other artists I've spoken to are having the same problem. Any idea what is causing it and if its going to get fixed?

For now. I'm gonna hold of posting anything until its fixed. Artwork posted where no one in your watch list gets notified isn't very good.

 

[Thanatos2k]

I can confirm this is happening, but infrequently. Here's some visual proof. Here's a snapshot of some of my submission notifications.

http://s28.postimg.org/8abwtzb7f/missing.jpg (NSFW)


As you can see, there's the 2/8 picture missing in Banana-of-Doom's upload sequence. But if you check their gallery....


http://s30.postimg.org/5okgyl1wf/notmissing.jpg (NSFW)

There it is. So yes, submission notifications are being missed, and this should be investigated immediately.

 

[PheagleAdler]

I can confirm this is happening, but infrequently. Here's some visual proof. Here's a snapshot of some of my submission notifications.

http://s28.postimg.org/8abwtzb7f/missing.jpg (NSFW)


As you can see, there's the 2/8 picture missing in Banana-of-Doom's upload sequence. But if you check their gallery....


http://s30.postimg.org/5okgyl1wf/notmissing.jpg (NSFW)

There it is. So yes, submission notifications are being missed, and this should be investigated immediately.

I missed that exact same notification. So this bug is possibly affecting specific notifications?

 

[sethtriggs]

This is happening to me; the latest three submissions I've made apparently did not generate notifications.

Now in the submission process I noticed that it took a very, very long time after I filled out the description, tags and all the other details. I then ended up with an error which is obscured for security reasons.

In another window, I checked my gallery and the submissions actually did post. So I think some sort of bug has happened for me right at the point where the notification is generated, hence the error after the long delay.

Also, this worked just fine yesterday, so I don't know if this is a load issue or what. The images I'm uploading are PNG, but those are usually fine anyway. Editing the submission file also works fine.

 

[Kalmor]

Can confirm that there is currently something up with the way notifications are delivered, we do not know what the nature of the issue (be it heavy load or something else) is yet but are working on finding and solving it.

 

[PheagleAdler]

Can confirm that there is currently something up with the way notifications are delivered, we do not know what the nature of the issue (be it heavy load or something else) is yet but are working on finding and solving it.

Thanks, hopefully it is fixed soon, I just realized I missed all of the submissions an artist posted 8 hours ago, none of them showed up.

 

[Kalmor]

We found the issue. We have a system that splits notifications into 1k notification batches if the uploader has more than 1k watchers, we stagger the sending of these batches a little to reduce load on the servers. However, they shouldn't be taking this long. Apparently some of these batches fail to send (get timed out).

The issue may be an unintended consequence of us rolling back 13230568 "cleanup query rows" (no that number is not a typo <.< >.>), which will take about half a week to complete. So things will be like this until then at least, unless we find a fix in the meantime.

This is my understanding of the situation. Yak or Neer may post a more informed explanation soon.

CORRECTION: Half a week was incorrect, more like 6-8 hours. Woo!

 

[sethtriggs]

We found the issue. We have a system that splits notifications into 1k notification batches if the uploader has more than 1k watchers, we stagger the sending of these batches a little to reduce load on the servers. However, they shouldn't be taking this long. Apparently some of these batches fail to send (get timed out).

The issue may be an unintended consequence of us rolling back 13230568 "cleanup query rows" (no that number is not a typo <.< >.>), which will take about half a week to complete. So things will be like this until then at least, unless we find a fix in the meantime.

This is my understanding of the situation. Yak or Neer may post a more informed explanation soon.

Ahhhh that makes a lot of sense! The people I'm still getting submissions from often don't have a huge amount of watchers. I happen to have 1,450 watchers so that would explain why some people aren't getting mine. (One fan actually reported to me that he received a notification from me).

I'm hoping this can be fixed too! But at least the other stuff is working. The database error I receive when the submit operation completes would be because of the notification system then, right?

Thanks in advance...

 

[Danza]

We found the issue. We have a system that splits notifications into 1k notification batches if the uploader has more than 1k watchers, we stagger the sending of these batches a little to reduce load on the servers. However, they shouldn't be taking this long. Apparently some of these batches fail to send (get timed out).

The issue may be an unintended consequence of us rolling back 13230568 "cleanup query rows" (no that number is not a typo <.< >.>), which will take about half a week to complete. So things will be like this until then at least, unless we find a fix in the meantime.

This is my understanding of the situation. Yak or Neer may post a more informed explanation soon.

CORRECTION: Half a week was incorrect, more like 6-8 hours. Woo!

Thanks for the confirmation! It was happening all day as well on the 22nd [for me at least]. Nearly all my followers on twitter were reporting my uploaded works where NEVER showing in their inboxes. For me today I've only received one submission notification so I think it's happening to a lot of artists and they don't even know about it :O

 

[Kritterfox]

Same issue here, I had a buncha pals tell me that it never showed up, and on top of it now I'm getting the phantom submission in my watchlist thinger.

 

[Kalmor]

Same issue here, I had a buncha pals tell me that it never showed up, and on top of it now I'm getting the phantom submission in my watchlist thinger.

Phantom submissions can be removed by ~not~ nuking the notifications, but rather deleting at least one notification (to reset the counter) and then nuking them all.

 

[Kritterfox]

Phantom submissions can be removed by ~not~ nuking the notifications, but rather deleting at least one notification (to reset the counter) and then nuking them all.

I have never nuked but just go bit by bit so I'm not sure why it popped up! Oh well

 

[HoshiKitsunuki]

I am also finding that there are a lot of artists I have been watching who have been posting things and I NEVER got notifications for them. For a lot longer ago than just today. Weeks into their submissions there are things I haven't seen till I look at their profile pages. I have a few users in the #Furaffinity channel on IRC who are encountering errors when they try to submit art to the site as well.

 

[Azuriae]

Yeah, there's definitely something broken somewhere.

I got home and only had 16 notifications total. Saw something from an artist I'm watching on the front page; I never got a notification for it.

 

[Kolbo]

Having this problem now. Submission notes aren't showing up.

 

[kopareigns]

Also going to throw my voice in here. Based on how many notifications I usually get, only about 20% actually showed. Moreover, this bug also affected journal notifications.

 

[PheagleAdler]

We found the issue. We have a system that splits notifications into 1k notification batches if the uploader has more than 1k watchers, we stagger the sending of these batches a little to reduce load on the servers. However, they shouldn't be taking this long. Apparently some of these batches fail to send (get timed out).

The issue may be an unintended consequence of us rolling back 13230568 "cleanup query rows" (no that number is not a typo <.< >.>), which will take about half a week to complete. So things will be like this until then at least, unless we find a fix in the meantime.

This is my understanding of the situation. Yak or Neer may post a more informed explanation soon.

CORRECTION: Half a week was incorrect, more like 6-8 hours. Woo!

should it be working now?

 

[thoron]

Thats what I want to know.

 

[thoron]

Doesn't seem to be working. Submissions are still failing to show up in the inbox.

 

[HoshiKitsunuki]

Also going to throw my voice in here. Based on how many notifications I usually get, only about 20% actually showed. Moreover, this bug also affected journal notifications.

Based on this happening I have art I am not even bothering to post if my watchers may not be notified. This is a much bigger problem than it seems when the artists can't alert their watchers of the art they are uploading. Clients are expecting uploads and communication here.

 

[PheagleAdler]

yeah, can we get this fixed before I need to shave?

 

[Danza]

Based on this happening I have art I am not even bothering to post if my watchers may not be notified. This is a much bigger problem than it seems when the artists can't alert their watchers of the art they are uploading. Clients are expecting uploads and communication here.

Exactly. For me FA is as good as down when I can't post any commission journals, auctions, updates etc because no one is getting notified. It's already hurting my income :/

 

[idejtauren]

...this probably explains why yesterday seemed so quiet on FA. Barely any notifications.

 

[Zenia]

Yeah, I keep getting that mysterious error when I try to upload something, but when I check my gallery it is in there. ... And the two things I have submitted have received far less attention than my stuff usually gets, so I assume it isn't showing up in peoples inboxes.

 

[quoting_mungo]

The rollback is taking longer than expected; it's currently about 1/3 of the way done and notifications are likely to remain glitchy until it's finished.