Possible Server Issue: VERY slow downloads of WU's.

[RamonetB]

I've been seeing an issue over the weekend where the majority of my CPU / SMP WU's using core A7 are downloading extremely slowly. As in, taking almost 2 hours at times for a 3MB - 15MB file.

All the ones that are being affected seem to be getting their WU's from 171.67.108.101.

Here is a log entry:

Code: Select all

5:50:32:WU01:FS00:Connecting to 171.67.108.45:8080
15:50:32:WU01:FS00:Assigned to work server 171.67.108.101
15:50:32:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:48 from 171.67.108.101
15:50:32:WU01:FS00:Connecting to 171.67.108.101:8080
15:50:33:WU01:FS00:Downloading 2.94MiB
15:50:52:WU00:FS00:0xa7:Completed 200000 out of 200000 steps (100%)
15:50:54:WU00:FS00:0xa7:Saving result file ../logfile_01.txt
15:50:54:WU00:FS00:0xa7:Saving result file frame8.trr
15:50:57:WU00:FS00:0xa7:Saving result file md.log
15:50:57:WU00:FS00:0xa7:Saving result file science.log
15:50:57:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
15:50:57:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
15:50:58:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:13126 run:76 clone:7 gen:8 core:0xa7 unit:0x0000000bab436c655898c999bd5830c3
15:50:58:WU00:FS00:Uploading 15.53MiB to 171.67.108.101
15:50:58:WU00:FS00:Connecting to 171.67.108.101:8080
15:51:01:WU00:FS00:Upload complete
15:51:01:WU00:FS00:Server responded WORK_ACK (400)
15:51:01:WU00:FS00:Final credit estimate, 7090.00 points
15:51:01:WU00:FS00:Cleaning up
15:51:50:WU01:FS00:Download 2.12%
15:52:00:WU01:FS00:Download 6.37%
15:52:08:WU01:FS00:Download 10.62%
15:52:46:WU01:FS00:Download 12.75%
16:03:13:WU01:FS00:Download 14.87%
16:03:20:WU01:FS00:Download 19.12%
16:03:39:WU01:FS00:Download 21.25%
16:03:46:WU01:FS00:Download 23.37%
16:05:03:WU01:FS00:Download 25.50%
16:07:33:WU01:FS00:Download 27.62%
16:07:49:WU01:FS00:Download 31.87%
16:08:13:WU01:FS00:Download 34.00%
16:08:20:WU01:FS00:Download 38.25%
16:08:59:WU01:FS00:Download 40.37%
16:09:09:WU01:FS00:Download 44.62%
16:09:26:WU01:FS00:Download 48.87%
16:09:32:WU01:FS00:Download 51.00%
16:09:42:WU01:FS00:Download 55.25%
16:14:09:WU01:FS00:Download 57.37%
16:14:23:WU01:FS00:Download 61.62%
16:16:53:WU01:FS00:Download 63.75%
16:18:55:WU01:FS00:Download 65.87%
16:19:05:WU01:FS00:Download 68.00%
16:21:22:WU01:FS00:Download 70.12%
16:21:32:WU01:FS00:Download 72.25%
16:21:47:WU01:FS00:Download 74.37%
16:24:10:WU01:FS00:Download 76.50%
16:24:24:WU01:FS00:Download 80.75%
16:24:30:WU01:FS00:Download 82.87%
16:24:43:WU01:FS00:Download 87.12%
16:24:50:WU01:FS00:Download 89.24%
16:25:03:WU01:FS00:Download 93.49%
16:35:26:WU01:FS00:Download 95.62%
16:35:33:WU01:FS00:Download 97.74%

Other servers (so far, I haven't done a thorough log audit) don't seem to be exhibiting this issue. I've also verified network connections and speeds on my end. Everything else appears in order.

Is something odd going on with this server?

Thanks,

-Kirk

[RamonetB]

Just to add some more details to this:

It is certainly something to do with this server or a connection to it: 171.67.108.101

After clearing out the work folder and restarting the client, the same systems will quickly download a WU from a different one, such as: 155.247.166.219.
But after one or two WU's, it will return to 171.67.108.101 and the slow downloads are back.

Bummer.

-Kirk

[Joe_H]

No one else is reporting download speed issues with this server, possibly the problem is between your system and there. Can you provide the results of a traceroute to that server?

[bruce]

For comparison, here's what I'm seeing.

Code: Select all

  3    14 ms    14 ms    34 ms  lax1_cr1_3_0_306.dslextreme.com [66.51.202.17]
  4    23 ms    23 ms    25 ms  hurricane-electric.as6939.any2ix.coresite.com [206.72.210.122]
  5    34 ms    32 ms    40 ms  100ge8-2.core1.pao1.he.net [184.105.81.237]
  6     *       31 ms    33 ms  stanford-university.10gigabitethernet1-4.core1.pao1.he.net [216.218.209.118] 
  7    22 ms    23 ms    24 ms  west-rtr-vlan8.SUNet [171.64.255.193]
  8    22 ms    23 ms    24 ms  vspe1.stanford.edu [171.67.108.101]

Trace complete.

I'm on the west coast close to 171.67.108.101 but a bit farther from 155.247.166.219

Code: Select all

  3    15 ms    14 ms    13 ms  lax1_cr1_3_0_306.dslextreme.com [66.51.202.17]
  4    16 ms    13 ms    14 ms  ge-101-0-0-16.r00.lsanca07.us.bb.gin.ntt.net [129.250.203.113]
  5    15 ms    15 ms    13 ms  be3025.ccr41.lax04.atlas.cogentco.com [154.54.9.29]
  6    13 ms    15 ms    13 ms  be2965.ccr22.lax01.atlas.cogentco.com [154.54.45.1]
  7    69 ms    25 ms    24 ms  be2932.ccr22.phx02.atlas.cogentco.com [154.54.45.161]
  8    33 ms    34 ms    34 ms  be2930.ccr21.elp01.atlas.cogentco.com [154.54.42.78]
  9    47 ms    49 ms    48 ms  be2927.ccr41.iah01.atlas.cogentco.com [154.54.29.221]
 10    64 ms    64 ms    63 ms  be2687.ccr41.atl01.atlas.cogentco.com [154.54.28.69]
 11    73 ms    73 ms    73 ms  be2112.ccr41.dca01.atlas.cogentco.com [154.54.7.157]
 12    76 ms    79 ms    78 ms  be3161.rcr21.phl01.atlas.cogentco.com [154.54.5.114]
 13    79 ms    78 ms    79 ms  be3163.rcr21.phl03.atlas.cogentco.com [154.54.7.2]
 14    78 ms    81 ms    76 ms  38.140.148.74
 15    79 ms   317 ms    80 ms  155.247.119.90
 16    80 ms    79 ms    79 ms  conwellcore-tu-internet-vs.is.temple.edu [155.247.119.227]
 17    80 ms   316 ms    81 ms  tu-internalhosts-vs.is.temple.edu [155.247.119.233]
 18    77 ms    79 ms    78 ms  vav3.ocis.temple.edu [155.247.166.219]

Trace complete.

Both look normal to me.

[RamonetB]

Bruce, Joe,

I'm starting to see the same behavior on other servers as well. So it definitely must be something happening on my side or something in between.
Traceroute suggests there is no trouble, though, at least on 171.67.108.101.

This particular computer is currently experiencing slow downloads but the traceroute response is peppy:

Code: Select all

8  et-8-1-0.2364.rtsw.chic.net.internet2.edu (198.71.45.156)  16.342 ms 172.30.8.10 (172.30.8.10)  1.465 ms et-8-1-0.2364.rtsw.chic.net.internet2.edu (198.71.45.156)  16.273 ms
 9  et-8-1-0.2364.rtsw.chic.net.internet2.edu (198.71.45.156)  16.363 ms  16.285 ms  16.310 ms
10  et-3-1-0.4070.rtsw.kans.net.internet2.edu (198.71.47.207)  27.364 ms et-1-0-0.109.rtr.hous.net.internet2.edu (198.71.45.16)  41.695 ms  41.677 ms
11  et-7-1-0.4070.rtsw.losa.net.internet2.edu (198.71.45.21)  73.862 ms  73.825 ms et-1-0-0.109.rtr.hous.net.internet2.edu (198.71.45.16)  41.595 ms
12  137.164.26.200 (137.164.26.200)  73.792 ms et-7-1-0.4070.rtsw.losa.net.internet2.edu (198.71.45.21)  74.443 ms 137.164.26.200 (137.164.26.200)  73.721 ms
13  hpr-svl-hpr3--lax-hpr3-100ge.cenic.net (137.164.25.74)  80.867 ms  80.792 ms  81.076 ms
14  hpr-stan-ge--svl-hpr2.cenic.net (137.164.27.162)  81.430 ms  81.320 ms hpr-svl-hpr3--lax-hpr3-100ge.cenic.net (137.164.25.74)  81.140 ms
15  west-rtr-vlan9.sunet (171.66.255.193)  81.829 ms hpr-stan-ge--svl-hpr2.cenic.net (137.164.27.162)  81.242 ms  81.209 ms
16  west-rtr-vlan9.sunet (171.66.255.193)  82.152 ms vspe1.stanford.edu (171.67.108.101)  81.110 ms !X  81.217 ms !X

Although I am experience lack of responses now to 155.247.166.219 (though this could just be a protocol issue, I CAN ping the server)

Code: Select all

 7  et-8-1-0.2364.rtsw.chic.net.internet2.edu (198.71.45.156)  16.341 ms  16.476 ms  16.217 ms
 8  et-8-1-0.4070.rtsw.wash.net.internet2.edu (198.71.45.57)  33.302 ms  33.315 ms et-8-1-0.2364.rtsw.chic.net.internet2.edu (198.71.45.156)  16.340 ms
 9  et-8-1-0.4070.rtsw.wash.net.internet2.edu (198.71.45.57)  33.342 ms surveyor-gw.magpi.net (216.27.100.17)  37.581 ms  37.551 ms
10  remote.temple.magpi.net (216.27.98.234)  36.865 ms surveyor-gw.magpi.net (216.27.100.17)  37.459 ms  37.454 ms
11  * * remote.temple.magpi.net (216.27.98.234)  37.225 ms
12  * * *
13  * * *
14  * * *
15  * * *

I could suspect something odd happening on the new equipment I'm testing, but I'm also seeing this on tried and true hardware.
I guess I better start looking further down my network line. I'm not sure what is going on.

[bruce]

In addition to "ping" you should be able to open http://(Ip address) or http://(Ip address:port) though I'm not sure that just looking at the homepage is going to give you any indication fo upload/download speeds.