Azure servers down 40.121.152.108 / 52.224.109.74

[hutleytj]

I have a 100% completed work unit that is not being uploaded to 40.114.52.201 (work server) or 52.224.109.74 (collection server) - not sure which is the upload one...

is there a way to upload manually? or force the upload?

[Neil-B]

No ... The server is down ... You simply have to be patient and let the client do what it is designed to do - which is keep retrying to send at intervals until such time as either the server is up and can receive it or it passes the expiration deadline and gets dumped by the client ... The client should be continuing to get WUs and process them from other servers without interruption.

[bruce]

Ongoing discussions between FAH Server admins and azure server admins. Apparently a critical boot device is off-line.

[IgnoredAmbience]

The two work units I have awaiting upload both expire tomorrow, hopefully the servers will be restored by then.

[Foxbat]

I heard reports of a CloudFlare outage July 17th. https://blog.cloudflare.com/cloudflare- ... y-17-2020/
Outage was 20:25 UTC through 22:10. The last contact with four of the Azure cloud servers was about two hours before this. Depending on how often the sanity check is performed, is this a possible cause of the Azure cloud outage?

[Cruddson001]

Hi not sure where to post this but I have a complete work unit that was never uploaded?

I had to switch my computer off for a few weeks.

See log pasted below:

Code: Select all

04:48:07:WU01:FS00:Upload 34.80%
04:48:13:WU01:FS00:Upload 46.70%
04:48:19:WU01:FS00:Upload 56.77%
04:48:25:WU01:FS00:Upload 65.93%
04:48:31:WU01:FS00:Upload 76.01%
04:48:37:WU01:FS00:Upload 86.08%
04:48:43:WU01:FS00:Upload 95.24%
04:48:48:ERROR:WU01:FS00:Exception: 10001: Server responded: HTTP_BAD_GATEWAY
04:54:57:WU02:FS00:0xa7:Completed 92500 out of 125000 steps (74%)
05:03:11:WU02:FS00:0xa7:Completed 93750 out of 125000 steps (75%)
05:11:21:WU02:FS00:0xa7:Completed 95000 out of 125000 steps (76%)
05:19:31:WU02:FS00:0xa7:Completed 96250 out of 125000 steps (77%)
05:27:43:WU02:FS00:0xa7:Completed 97500 out of 125000 steps (78%)
05:35:55:WU02:FS00:0xa7:Completed 98750 out of 125000 steps (79%)
05:44:07:WU02:FS00:0xa7:Completed 100000 out of 125000 steps (80%)
05:46:37:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14818 run:641 clone:0 gen:241 core:0xa7 unit:0x000001192879986c5ea8de43810a1dc4
05:46:37:WU01:FS00:Uploading 6.83MiB to 40.121.152.108
05:46:37:WU01:FS00:Connecting to 40.121.152.108:8080
05:46:41:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
05:46:41:WU01:FS00:Connecting to 40.121.152.108:80
05:46:43:WU01:FS00:Upload 4.58%
05:46:49:WU01:FS00:Upload 15.57%
05:46:55:WU01:FS00:Upload 24.72%
05:47:01:WU01:FS00:Upload 33.88%
05:47:07:WU01:FS00:Upload 45.79%
05:47:13:WU01:FS00:Upload 55.86%
05:47:19:WU01:FS00:Upload 65.02%
05:47:25:WU01:FS00:Upload 75.09%
05:47:31:WU01:FS00:Upload 84.25%
05:47:37:WU01:FS00:Upload 94.32%
05:47:43:WARNING:WU01:FS00:Exception: Failed to send results to work server: 10001: Server responded: HTTP_BAD_GATEWAY
05:47:43:WU01:FS00:Trying to send results to collection server
05:47:43:WU01:FS00:Uploading 6.83MiB to 52.224.109.74
05:47:43:WU01:FS00:Connecting to 52.224.109.74:8080
05:47:45:WARNING:WU01:FS00:WorkServer connection failed on port 8080 trying 80
05:47:45:WU01:FS00:Connecting to 52.224.109.74:80
05:47:49:WU01:FS00:Upload 6.41%
05:47:55:WU01:FS00:Upload 16.48%
05:48:01:WU01:FS00:Upload 26.56%
05:48:07:WU01:FS00:Upload 37.54%
05:48:13:WU01:FS00:Upload 47.62%
05:48:19:WU01:FS00:Upload 56.77%
05:48:25:WU01:FS00:Upload 66.85%
05:48:31:WU01:FS00:Upload 76.01%
05:48:37:WU01:FS00:Upload 86.08%
05:48:43:WU01:FS00:Upload 96.15%
05:48:48:ERROR:WU01:FS00:Exception: 10001: Server responded: HTTP_BAD_GATEWAY
05:52:23:WU02:FS00:0xa7:Completed 101250 out of 125000 steps (81%)
06:00:36:WU02:FS00:0xa7:Completed 102500 out of 125000 steps (82%)
06:08:46:WU02:FS00:0xa7:Completed 103750 out of 125000 steps (83%)

Mod Edit: Added Code Tags - PantherX

[bruce]

HTTP_BAD_GATEWAY indicates that the server is overloaded or cannot accept the upload for some other reason.

In this case, both servers have been down. They may be struggling to handle the surge load.

[Erock]

I have the same Problem. I guess the WU was folded for nothing because it deletes itself soon.

[hutleytj]

Still (23 July 7pm) unable to upload a completed work unit from Work Queue 00 to 52.224.109.74...

[Neil-B]

... and still server down as shown on the server status link at top of this page.

When the issue is resolve (if indeed it is) the server will be restarted - until such time the client will not be able to connect to it and upload because it is down ... if the expiration deadline is reached before the server is up then the WU will be dumped by the client.

Looks like one of the four servers that went done on the 17th was up at least for a bit yesterday (22nd) but is now done again so my guess is they are still troubleshooting the issue(s)

[RichieDoubleU]

The 52.... was started about half an hour ago and has meanwhile collected my "orphan" WU. So that problem is solved.
The 40... is still down at the moment.

[comixgoddess]

My work unit finally uploaded successfully to 52.224.109.74, less than 24 hours before the expiration.

[gbowman]

Sorry for the troubles, and thanks for your patience! We're working with the Azure folks to get these machines up to accept work units.

[hutleytj]

Still (23 July 7pm) unable to upload a completed work unit from Work Queue 00 to 52.224.109.74...

Friday 24 July

My work unit has now uploaded to the collection server (at least, it is not in a queue to upload anymore)