Page 3 of 3
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Tue Aug 11, 2009 2:56 pm
by tear
A-ha! Doesn't the error message look familiar?
I'm not sure if the bug has been there for a long-er time or if it was introduced just recently
(due to attempted "optimization" of some kind perhaps) but... I can bet any money it's the
Squid.
The way Stanford servers send response is unusual (on TCP layer) but still valid.
Response bodies as sent by testA and testD (these are of interest) look like this (yes, they are identical):
HTTP/1.1 200 OK
Content-Type: text/html
Cache-Control: no-cache
Transfer-Encoding: chunked
Connection: close
17
<html><b>OK</b></html>
0
testA sends complete response in one TCP packet:
Code: Select all
08:46:35.274722 IP 85.11.66.61.http > 192.168.2.110.58969: P 1:151(150) ack 170 win 54 <nop,nop,timestamp 2156808957 127686672>
0x0000: 4520 00ca 8772 4000 2c06 6c3d 550b 423d E....r@.,.l=U.B=
0x0010: c0a8 026e 0050 e659 c7c2 cd93 6eb7 aa6a ...n.P.Y....n..j
0x0020: 8018 0036 db45 0000 0101 080a 808e 4afd ...6.E........J.
0x0030: 079c 5810 4854 5450 2f31 2e31 2032 3030 ..X.HTTP/1.1.200
0x0040: 204f 4b0d 0a43 6f6e 7465 6e74 2d54 7970 .OK..Content-Typ
0x0050: 653a 2074 6578 742f 6874 6d6c 0d0a 4361 e:.text/html..Ca
0x0060: 6368 652d 436f 6e74 726f 6c3a 206e 6f2d che-Control:.no-
0x0070: 6361 6368 650d 0a54 7261 6e73 6665 722d cache..Transfer-
0x0080: 456e 636f 6469 6e67 3a20 6368 756e 6b65 Encoding:.chunke
0x0090: 640d 0a43 6f6e 6e65 6374 696f 6e3a 2063 d..Connection:.c
0x00a0: 6c6f 7365 0d0a 0d0a 3137 0d0a 3c68 746d lose....17..<htm
0x00b0: 6c3e 3c62 3e4f 4b3c 2f62 3e3c 2f68 746d l><b>OK</b></htm
0x00c0: 6c3e 0a0d 0a30 0d0a 0d0a l>...0....
...whereas testD sends bold part in one packet (08:48:08.878319) and blue part in the other packet (08:48:08.929282):
Code: Select all
08:48:08.878319 IP 85.11.66.61.http > 192.168.2.110.58972: P 1:10(9) ack 170 win 54 <nop,nop,timestamp 2156902559 127780277>
0x0000: 4520 003d 6eab 4000 2c06 8591 550b 423d E..=n.@.,...U.B=
0x0010: c0a8 026e 0050 e65c cd83 d629 c551 7ec2 ...n.P.\...).Q~.
0x0020: 8018 0036 2d20 0000 0101 080a 808f b89f ...6-...........
0x0030: 079d c5b5 4854 5450 2f31 2e31 20 ....HTTP/1.1.
08:48:08.878367 IP 192.168.2.110.58972 > 85.11.66.61.http: . ack 10 win 92 <nop,nop,timestamp 127780514 2156902559>
0x0000: 4500 0034 baca 4000 4006 259b c0a8 026e E..4..@.@.%....n
0x0010: 550b 423d e65c 0050 c551 7ec2 cd83 d632 U.B=.\.P.Q~....2
0x0020: 8010 005c 461c 0000 0101 080a 079d c6a2 ...\F...........
0x0030: 808f b89f ....
08:48:08.929282 IP 85.11.66.61.http > 192.168.2.110.58972: FP 10:151(141) ack 170 win 54 <nop,nop,timestamp 2156902610 127780277>
0x0000: 4520 00c1 6eac 4000 2c06 850c 550b 423d E...n.@.,...U.B=
0x0010: c0a8 026e 0050 e65c cd83 d632 c551 7ec2 ...n.P.\...2.Q~.
0x0020: 8019 0036 6300 0000 0101 080a 808f b8d2 ...6c...........
0x0030: 079d c5b5 3230 3020 4f4b 0d0a 436f 6e74 ....200.OK..Cont
0x0040: 656e 742d 5479 7065 3a20 7465 7874 2f68 ent-Type:.text/h
0x0050: 746d 6c0d 0a43 6163 6865 2d43 6f6e 7472 tml..Cache-Contr
0x0060: 6f6c 3a20 6e6f 2d63 6163 6865 0d0a 5472 ol:.no-cache..Tr
0x0070: 616e 7366 6572 2d45 6e63 6f64 696e 673a ansfer-Encoding:
0x0080: 2063 6875 6e6b 6564 0d0a 436f 6e6e 6563 .chunked..Connec
0x0090: 7469 6f6e 3a20 636c 6f73 650d 0a0d 0a31 tion:.close....1
0x00a0: 370d 0a3c 6874 6d6c 3e3c 623e 4f4b 3c2f 7..<html><b>OK</
0x00b0: 623e 3c2f 6874 6d6c 3e0a 0d0a 300d 0a0d b></html>...0...
0x00c0: 0a .
Now, TCP is a stream protocol so either peer is at liberty to send data in
arbitrary chunks.
Nothing prevents server from sending one byte per packet. It's inefficient but still, no matter
how fragmented the data are, client MUST NOT expect that a single read()/recv() will return
a desired chunk. This is Unix and stream sockets 101.
Yes, it qualifies as a bug report to Squid folks.
tear
P.S.
Who's yer daddy!?
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Tue Aug 11, 2009 4:54 pm
by SanHolo
Nice work tear! I've sent an email to our IT wizards with a link to your last post, maybe they can use your work and tell the Squid people to take care of the bug.
I'll keep you posted.
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Tue Aug 11, 2009 5:02 pm
by tear
Having given it more thoughts -- use of tools that do some kind of inspection of HTTP responses (or anything that attempts
to look *into* HTTP responses/conversations for that matter) could also result in observed behavior.
In other words -- if such tool (firewall component/configuration perhaps) that is not able to handle chunked
responses (read: is poorly designed) and drops connection instead....
Next logical step would be to test just the squid without any network "interference".
Your IT folks on the other hand know their setup so they should be able to draw conclusions without testing I suppose...
tear
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Wed Aug 12, 2009 7:44 am
by SanHolo
Very nice, got a response from our IT wizards last night and just tested - they have one of the proxies running with another configuration and that one works for me. I'll switch to that one after the currently running WUs are done. Thanks for the temporary fix, tear!
The IT guy pointed me to a related blogpost:
http://squidproxy.wordpress.com/2008/04 ... -decoding/
Maybe they're going to try this config setting on the main proxy, but since I have another one at my disposal I'm now fine.
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Wed Aug 12, 2009 1:12 pm
by tear
You are welcome
Hmm, I should have used "fragmented" term rather than "chunked" as (as I see) I introduced confusion.
It's not "chunked" Transfer-Encoding (on HTTP level) that I had in mind but TCP packet fragments -- my bad.
Anyway, you're saying they have another configuration that works for you?
Out of curiosity -- do you know *how* this other configuration is different than the other one?
Thanks,
tear
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Wed Aug 12, 2009 3:20 pm
by 314159
Thank you for your excellent support on this one, tear.
Let's not lose this thread.
It might be useful in the future.
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Wed Aug 12, 2009 3:46 pm
by Beberg
Nice work tear
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Wed Aug 12, 2009 5:55 pm
by tear
Pascal, John, Adam,
Your appreciation is appreciated (did it sound cliché?).
<bows>
tear
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Wed Aug 12, 2009 10:33 pm
by SanHolo
tear wrote:Out of curiosity -- do you know *how* this other configuration is different than the other one?
I'd have to find out tomorrow, but I assume they did not update that one proxy to the same squid version as the main one which caused all this fuzz. Maybe something like a backup proxy that nobody knows about but the inner circle.
Keep on bowing, still standing ovations going on here.
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Thu Aug 13, 2009 8:21 am
by Beberg
Yea, let us know so we can tell people that have this problem they need to use Squid version X or better, since I'm sure they will fix the bug quickly now that Tear sent them a bug report, and it should be a fast fix.
On this end I made some code changes, but they wont be rolled out for at least a while, not that we really have absolute control over where the packets split up...
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Thu Aug 13, 2009 5:28 pm
by tear
Beberg wrote:Yea, let us know so we can tell people that have this problem they need to use Squid version X or better, since I'm sure they will fix the bug quickly now that Tear sent them a bug report, and it should be a fast fix.
Actually I haven't sent bug report to squid folks as I have no way of verifying (beyond any doubt) it's squid
(see
doubts post) and I'm sure you know attitude towards vague bug reports.
I think IT guys from SanHolo's Uni have better environment to isolate the bug -- I would need to install squid...
Beberg wrote:On this end I made some code changes, but they wont be rolled out for at least a while, not that we really have absolute control over where the packets split up...
True, though aggregating payloads of sendto()/write() to (at least) full-lines (as far as response line + headers are concerned) should improve our chances...
tear
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Fri Aug 14, 2009 10:46 pm
by SanHolo
I guess the squid folks know about an issue there, see the link to the blog post. And since our IT dept was working with (some of) the squid team, I assume they've told them about the problem. Anything you want me to do? The proxy error page of the working proxy does not tell me the version of squid (or whether it's even squid), is there another way? The unworking proxy is 3.0.STABLE18.
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Sat Aug 15, 2009 3:27 pm
by tear
SanHolo wrote:I guess the squid folks know about an issue there, see the link to the blog post.
I am pretty sure it is not the same problem -- I had attempted to explain
here
but forgot about the bottom line
SanHolo wrote:Anything you want me to do? The proxy error page of the working proxy does not tell me the version of squid (or whether it's even squid), is there another way? The unworking proxy is 3.0.STABLE18.
I'll venture a guess squid adds extra header in responses -- you'd need to do a manual retrieval of a(ny) site
through the proxy (using telnet).
Just call
telnet yourproxy yourproxyport. Then it's only the matter of sending valid... (invalid will do too) request.
For instance, you can type
GET / HTTP/1.1 and press Enter *two* times. That should return an error message along
with squid version (either in HTTP headers or HTML body below).
tear
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Tue Aug 18, 2009 11:22 am
by SanHolo
Sorry, am abroad again, but just tried it with a VPN-Session: Nope, the working proxy does not return a signature, only the success/error message. The non-working proxy does, however, return it's signature, which I posted before: squid/3.0.STABLE18
thx!
Re: SQUID Proxy problems with assign.stanford.edu
Posted: Tue Aug 18, 2009 3:45 pm
by tear
Well then... I don't know of any other way to check the version (though given that I don't know squid
it doesn't mean there aren't any).
tear