Bziped summary files corrupted

[Kakao]

Can you detail the specific problem you have with the downloaded files and maybe someone here can host them for you as a relay to avoid the issue?

I can download the files from my web server in San Diego, California, where I keep those files backups. That is a ssh download. Although very slow, they arrive without errors. I think I have a problem with everything from California as I can watch youtube videos and download other content at the connection full speed.

The speed would be just an inconvenience if it weren't by the corrupted files. It looks like something changed about the Stanford's http server configuration.

I will have to change the KakaoStats code to download from the web server but that will obviously require a coding session. That is because the files are not http exposed as the current download code requires. I will need to either expose then or change the current code to make a ssh download. I can only start it on the weekend.

[HaloJones]

OK. Happy to hear you may have a solution.

[noorman]

.

Kakao,

I downloaded the file from your link (in my WinXP) and unpacked it with Winrar; to me the file seems uncorrupted ...

my traceroute gave me this:

Code: Select all

Tracing route to vspm27.stanford.edu [171.65.103.94]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  [router]
  2    <1 ms    <1 ms    <1 ms  [router]
  3     7 ms     8 ms     7 ms  d54C3F001.access.telenet.be [84.195.240.1]
  4     9 ms     9 ms     9 ms  dD5E0C521.access.telenet.be [213.224.197.33]
  5     8 ms    10 ms     8 ms  dD5E0FD59.access.telenet.be [213.224.253.89]
  6     8 ms     7 ms     8 ms  dD5E0FDB9.access.telenet.be [213.224.253.185]
  7    10 ms    10 ms     9 ms  ae0.anr11.ip4.tinet.net [77.67.65.177]
  8    22 ms    42 ms    19 ms  xe-1-0-0.lon11.ip4.tinet.net [89.149.185.170]
  9    21 ms    25 ms    35 ms  te7-6.mpd02.lon01.atlas.cogentco.com [130.117.15.49]
 10    98 ms    96 ms    96 ms  te0-0-0-1.mpd21.jfk02.atlas.cogentco.com [154.54.30.129]
 11   141 ms   218 ms   211 ms  te1-8.mpd01.ord01.atlas.cogentco.com [154.54.29.157]
 12   168 ms   168 ms   163 ms  te0-1-0-2.mpd21.mci01.atlas.cogentco.com [66.28.4.185]
 13   292 ms   215 ms   206 ms  te2-2.mpd01.sfo01.atlas.cogentco.com [154.54.6.38]
 14   185 ms   215 ms   203 ms  te7-4.mpd01.sjc04.atlas.cogentco.com [154.54.28.82]
 15   170 ms   172 ms   170 ms  Stanford_University2.demarc.cogentco.com [66.250.7.138]
 16   172 ms   175 ms   176 ms  boundarya-rtr.Stanford.EDU [68.65.168.33]
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20   171 ms   171 ms   180 ms  vspm27.Stanford.EDU [171.65.103.94]

Trace complete.

.

part of the unpacked file looks like this:

Code: Select all

Wed Apr 14 10:50:00 PDT 2010
name	newcredit	sum(total)	team
PS3	1908894914	8037921	0
anonymous	1326616315	6495933	0
PDC	527070688	9644	1
mutsu	210771427	130924	11108
AtlasFolder	172276138	315802	36167
ChasR	153427598	231306	32
Leganfuh	143451489	140142	75255
eastms.edu	134667803	1111084	1714
_	132702717	730830	0
anonymous	121951048	43475	1
Scott_H	118546443	353954	2654
wayne	116650005	133649	46429
[Zebulon.fr]_Cobra	103389154	111407	51
Christian_Bargmann	99501616	172194	1971
van_arnam	97621713	187973	50625
Michael_McCord,_M.D.	95168086	124027	11108
HayesK	87130337	102801	32
Tigerbiten	85996460	136098	33
kennish	77972716	41581	39340
wingrider	77145738	40012	53562
OverClocking-Masters.com	76076097	98262	51
Computekinc.us	74443526	102033	32
clamatowas	74410014	43090	111065
Spider_Monkey	71640893	119493	11108

.

seems normal enough to me

NOTE: must add that I downloaded that via my high-speed cable network

.

[noorman]

.

a while ago, someone thought that the extra delays after "boundarya-rtr.Stanford.EDU [68.65.168.33]" are from the Stanford firewall ...

my delay seems to be equal to that of toTOW (3 lines of time outs)

.

[artoar_11]

This is my traceroute, if it will help. PPPOE - 16Mbit/s.

Code: Select all

Tracing route to vspm27.stanford.edu [171.65.103.94]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  [router]
  2     2 ms     1 ms     1 ms  in-int.PPPoE2.escom.bg [195.24.89.9]
  3     2 ms     1 ms     1 ms  195.24.88.2
  4     5 ms     5 ms     5 ms  94.156.248.121
  5     9 ms     5 ms     5 ms  ae0-431.sof10.ip4.tinet.net [77.67.67.113]
  6    42 ms    41 ms    41 ms  xe-10-0-0.fra23.ip4.tinet.net [89.149.186.250]
  7    56 ms    59 ms    41 ms  te1-7.mpd02.fra03.atlas.cogentco.com [130.117.14.85]
  8   137 ms   138 ms   137 ms  te2-4.mpd01.ymq02.atlas.cogentco.com [154.54.28.93]
  9   152 ms   152 ms   152 ms  te8-7.mpd02.ord01.atlas.cogentco.com [66.28.4.57]
 10   173 ms   174 ms   174 ms  te0-4-0-0.mpd22.mci01.atlas.cogentco.com [154.54.30.178]
 11   204 ms   204 ms   204 ms  te4-7.mpd01.sfo01.atlas.cogentco.com [154.54.24.105]
 12   216 ms   216 ms   216 ms  te4-2.mpd01.sjc04.atlas.cogentco.com [154.54.2.166]
 13   216 ms   232 ms   216 ms  Stanford_University2.demarc.cogentco.com [66.250.7.138]
 14   216 ms   214 ms   211 ms  bnda-rtr-1.Stanford.EDU [68.65.168.33]
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18   221 ms   210 ms   211 ms  vspm27.Stanford.EDU [171.65.103.94]

Trace complete.

[Kakao]

.

a while ago, someone thought that the extra delays after "boundarya-rtr.Stanford.EDU [68.65.168.33]" are from the Stanford firewall ...

my delay seems to be equal to that of toTOW (3 lines of time outs)

.

So everybody gets 3 timeouts before reaching the target and I never get there even with max hops set to 60

Code: Select all

$ traceroute -m 60 vspm27.stanford.edu
traceroute to vspm27.stanford.edu (171.65.103.94), 60 hops max, 60 byte packets
 1  10.1.1.1 (10.1.1.1)  1.403 ms  2.042 ms  2.713 ms
 2  BrT-L10-bsace704-vrdef.dsl.brasiltelecom.net.br (201.10.168.254)  25.361 ms  34.849 ms  41.281 ms
 3  BrT-G7-1-2-740-bsace-core02.g.brasiltelecom.net.br (201.10.248.93)  33.660 ms  39.602 ms  41.954 ms
 4  po2-0.core02.mia03.atlas.cogentco.com (154.54.10.13)  180.361 ms  180.655 ms  181.211 ms
 5  te4-4.ccr01.mia03.atlas.cogentco.com (154.54.2.42)  185.834 ms  187.429 ms  191.833 ms
 6  te3-3.ccr01.mia01.atlas.cogentco.com (154.54.2.153)  198.812 ms te2-3.ccr01.mia01.atlas.cogentco.com (154.54.24.233)  155.750 ms te9-3.ccr01.mia01.atlas.cogentco.com (154.54.28.241)  180.727 ms
 7  te8-2.ccr01.iah01.atlas.cogentco.com (154.54.24.193)  216.769 ms  217.069 ms te7-3.mpd01.iah01.atlas.cogentco.com (154.54.28.77)  195.035 ms
 8  te4-2.mpd01.lax01.atlas.cogentco.com (154.54.0.141)  234.378 ms te3-2.mpd02.lax01.atlas.cogentco.com (154.54.0.245)  236.895 ms te9-2.mpd02.lax01.atlas.cogentco.com (154.54.0.253)  235.875 ms
 9  te7-4.mpd01.sfo01.atlas.cogentco.com (154.54.3.50)  278.402 ms te4-5.mpd01.sfo01.atlas.cogentco.com (154.54.3.94)  270.813 ms  271.234 ms
10  te7-4.mpd01.sjc04.atlas.cogentco.com (154.54.28.82)  367.097 ms * *
11  Stanford_University2.demarc.cogentco.com (66.250.7.138)  267.943 ms  272.032 ms  340.796 ms
12  bnda-rtr-1.Stanford.EDU (68.65.168.33)  323.310 ms  324.436 ms  333.791 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *
37  * * *
38  * * *
39  * * *
40  * * *
41  * * *
42  * * *
43  * * *
44  * * *
45  * * *
46  * * *
47  * * *
48  * * *
49  * * *
50  * * *
51  * * *
52  * * *
53  * * *
54  * * *
55  * * *
56  * * *
57  * * *
58  * * *
59  * * *
60  * * *

[HaloJones]

I get the three timeouts after boundarya-rtr.stanford.edu but then I get in.

[JPinTO]

Kakao: I just wanted to express my gratitude for your stats page. It's a very simple but effective design for conveying a lot of information. Thanks in advance for your work in making it operational again!!!

- JP

[Kakao]

I managed to implement a half solution. First I exposed the backup files at the web server through http. Then I pointed the download script to them. But the problems persisted showing that the problem is not with the Stanford's http server. Then I tried the hardened download script rewritten for the development version of KakaoStats and it proved to be more resilient to http connection problems successfully downloading the last few updates.

So the problem is not solved. I just have a more resilient script working and the download is very slow. My issue is very likely a routing problem which I will investigate during the next days.

There are many updates still missing in the database but I will manually feed then until the weekend.

[Kijad]

Glad to see it's at least partially up-and-running though, and isn't something like "oh the server HDDs all failed."

Keep on keepin' on, let us know if we can be of help in any way.

[noorman]

.

So, you are having trouble getting from the San Diego area to the San Jose area

Should be a small-hop trip though ...
( I had to get that file to the western seaboard of Europe )


.

[Kakao]

No. There is no problem from San Diego to San Jose. I can get the files extremely fast when downloading from San Jose to San Diego. The problem is downloading to Brasília, Brazil from both San Diego and San Jose. That is where the routing problem is. There is no problem getting content from anywhere else to Brasília.

[noorman]

No. There is no problem from San Diego to San Jose. I can get the files extremely fast when downloading from San Jose to San Diego. The problem is downloading to Brasília, Brazil from both San Diego and San Jose. That is where the routing problem is. There is no problem getting content from anywhere else to Brasília.

.

Why, then, does a traceroute to kakaostats.com guide me to a server in the San Diego area ?

.

[Kakao]

The web side of KakaoStats lives on the web server in San Diego. That is where the pages are built. The data to build the pages live in the database server in Brasília. So whenever a page is built by the web server it sends a sql query to the database. Actually there is lots of caching so the queries are much less frequent.

A database server is a much more demanding software than a web server. It would be costly to rent a machine in a data center to run the database. So it sits right in my house.

The machine in my house has 8GB memory, two HDs in Raid 1. Nothing fancy. But in the data center they charge every bit you add to the base machine.

[noorman]

.

OK, I get it.
Now I see why you need this data packet to be sent from California to Brasil

That 's also the reason why my traceroute didn't detect any obvious problems

Hope you can trace the culprit (soon) / good luck

.