SQUID Proxy problems with assign.stanford.edu

Moderators: Site Moderators, FAHC Science Team

Post Reply
tear
Posts: 254
Joined: Sun Dec 02, 2007 4:08 am
Hardware configuration: None
Location: Rocky Mountains

Re: SQUID Proxy problems with assign.stanford.edu

Post by tear »

A-ha! Doesn't the error message look familiar?

I'm not sure if the bug has been there for a long-er time or if it was introduced just recently
(due to attempted "optimization" of some kind perhaps) but... I can bet any money it's the
Squid.

The way Stanford servers send response is unusual (on TCP layer) but still valid.

Response bodies as sent by testA and testD (these are of interest) look like this (yes, they are identical):
HTTP/1.1 200 OK
Content-Type: text/html
Cache-Control: no-cache
Transfer-Encoding: chunked
Connection: close

17
<html><b>OK</b></html>

0

testA sends complete response in one TCP packet:

Code: Select all

08:46:35.274722 IP 85.11.66.61.http > 192.168.2.110.58969: P 1:151(150) ack 170 win 54 <nop,nop,timestamp 2156808957 127686672>
	0x0000:  4520 00ca 8772 4000 2c06 6c3d 550b 423d  E....r@.,.l=U.B=
	0x0010:  c0a8 026e 0050 e659 c7c2 cd93 6eb7 aa6a  ...n.P.Y....n..j
	0x0020:  8018 0036 db45 0000 0101 080a 808e 4afd  ...6.E........J.
	0x0030:  079c 5810 4854 5450 2f31 2e31 2032 3030  ..X.HTTP/1.1.200
	0x0040:  204f 4b0d 0a43 6f6e 7465 6e74 2d54 7970  .OK..Content-Typ
	0x0050:  653a 2074 6578 742f 6874 6d6c 0d0a 4361  e:.text/html..Ca
	0x0060:  6368 652d 436f 6e74 726f 6c3a 206e 6f2d  che-Control:.no-
	0x0070:  6361 6368 650d 0a54 7261 6e73 6665 722d  cache..Transfer-
	0x0080:  456e 636f 6469 6e67 3a20 6368 756e 6b65  Encoding:.chunke
	0x0090:  640d 0a43 6f6e 6e65 6374 696f 6e3a 2063  d..Connection:.c
	0x00a0:  6c6f 7365 0d0a 0d0a 3137 0d0a 3c68 746d  lose....17..<htm
	0x00b0:  6c3e 3c62 3e4f 4b3c 2f62 3e3c 2f68 746d  l><b>OK</b></htm
	0x00c0:  6c3e 0a0d 0a30 0d0a 0d0a                 l>...0....
...whereas testD sends bold part in one packet (08:48:08.878319) and blue part in the other packet (08:48:08.929282):

Code: Select all

08:48:08.878319 IP 85.11.66.61.http > 192.168.2.110.58972: P 1:10(9) ack 170 win 54 <nop,nop,timestamp 2156902559 127780277>
	0x0000:  4520 003d 6eab 4000 2c06 8591 550b 423d  E..=n.@.,...U.B=
	0x0010:  c0a8 026e 0050 e65c cd83 d629 c551 7ec2  ...n.P.\...).Q~.
	0x0020:  8018 0036 2d20 0000 0101 080a 808f b89f  ...6-...........
	0x0030:  079d c5b5 4854 5450 2f31 2e31 20         ....HTTP/1.1.
08:48:08.878367 IP 192.168.2.110.58972 > 85.11.66.61.http: . ack 10 win 92 <nop,nop,timestamp 127780514 2156902559>
	0x0000:  4500 0034 baca 4000 4006 259b c0a8 026e  E..4..@.@.%....n
	0x0010:  550b 423d e65c 0050 c551 7ec2 cd83 d632  U.B=.\.P.Q~....2
	0x0020:  8010 005c 461c 0000 0101 080a 079d c6a2  ...\F...........
	0x0030:  808f b89f                                ....
08:48:08.929282 IP 85.11.66.61.http > 192.168.2.110.58972: FP 10:151(141) ack 170 win 54 <nop,nop,timestamp 2156902610 127780277>
	0x0000:  4520 00c1 6eac 4000 2c06 850c 550b 423d  E...n.@.,...U.B=
	0x0010:  c0a8 026e 0050 e65c cd83 d632 c551 7ec2  ...n.P.\...2.Q~.
	0x0020:  8019 0036 6300 0000 0101 080a 808f b8d2  ...6c...........
	0x0030:  079d c5b5 3230 3020 4f4b 0d0a 436f 6e74  ....200.OK..Cont
	0x0040:  656e 742d 5479 7065 3a20 7465 7874 2f68  ent-Type:.text/h
	0x0050:  746d 6c0d 0a43 6163 6865 2d43 6f6e 7472  tml..Cache-Contr
	0x0060:  6f6c 3a20 6e6f 2d63 6163 6865 0d0a 5472  ol:.no-cache..Tr
	0x0070:  616e 7366 6572 2d45 6e63 6f64 696e 673a  ansfer-Encoding:
	0x0080:  2063 6875 6e6b 6564 0d0a 436f 6e6e 6563  .chunked..Connec
	0x0090:  7469 6f6e 3a20 636c 6f73 650d 0a0d 0a31  tion:.close....1
	0x00a0:  370d 0a3c 6874 6d6c 3e3c 623e 4f4b 3c2f  7..<html><b>OK</
	0x00b0:  623e 3c2f 6874 6d6c 3e0a 0d0a 300d 0a0d  b></html>...0...
	0x00c0:  0a                                       .
Now, TCP is a stream protocol so either peer is at liberty to send data in arbitrary chunks.
Nothing prevents server from sending one byte per packet. It's inefficient but still, no matter
how fragmented the data are, client MUST NOT expect that a single read()/recv() will return
a desired chunk. This is Unix and stream sockets 101.

Yes, it qualifies as a bug report to Squid folks.


tear


P.S.
Who's yer daddy!? :mrgreen:
One man's ceiling is another man's floor.
Image
Top
SanHolo
Posts: 13
Joined: Tue Jun 02, 2009 8:54 am

Re: SQUID Proxy problems with assign.stanford.edu

Post by SanHolo »

Nice work tear! I've sent an email to our IT wizards with a link to your last post, maybe they can use your work and tell the Squid people to take care of the bug. :D :D

I'll keep you posted.
Top
tear
Posts: 254
Joined: Sun Dec 02, 2007 4:08 am
Hardware configuration: None
Location: Rocky Mountains

Re: SQUID Proxy problems with assign.stanford.edu

Post by tear »

Having given it more thoughts -- use of tools that do some kind of inspection of HTTP responses (or anything that attempts
to look *into* HTTP responses/conversations for that matter) could also result in observed behavior.

In other words -- if such tool (firewall component/configuration perhaps) that is not able to handle chunked
responses (read: is poorly designed) and drops connection instead....

Next logical step would be to test just the squid without any network "interference".
Your IT folks on the other hand know their setup so they should be able to draw conclusions without testing I suppose...


tear
One man's ceiling is another man's floor.
Image
Top
SanHolo
Posts: 13
Joined: Tue Jun 02, 2009 8:54 am

Re: SQUID Proxy problems with assign.stanford.edu

Post by SanHolo »

Very nice, got a response from our IT wizards last night and just tested - they have one of the proxies running with another configuration and that one works for me. I'll switch to that one after the currently running WUs are done. Thanks for the temporary fix, tear!

The IT guy pointed me to a related blogpost: http://squidproxy.wordpress.com/2008/04 ... -decoding/
Maybe they're going to try this config setting on the main proxy, but since I have another one at my disposal I'm now fine.
Top
tear
Posts: 254
Joined: Sun Dec 02, 2007 4:08 am
Hardware configuration: None
Location: Rocky Mountains

Re: SQUID Proxy problems with assign.stanford.edu

Post by tear »

You are welcome :-)

Hmm, I should have used "fragmented" term rather than "chunked" as (as I see) I introduced confusion.
It's not "chunked" Transfer-Encoding (on HTTP level) that I had in mind but TCP packet fragments -- my bad.

Anyway, you're saying they have another configuration that works for you?

Out of curiosity -- do you know *how* this other configuration is different than the other one?

Thanks,
tear
One man's ceiling is another man's floor.
Image
Top
314159
Posts: 232
Joined: Sun Dec 02, 2007 2:46 am
Location: http://www.teammacosx.org/

Re: SQUID Proxy problems with assign.stanford.edu

Post by 314159 »

Thank you for your excellent support on this one, tear. :!:

Let's not lose this thread.
It might be useful in the future. :ewink:
John (from the central part of the Commonwealth of Virginia, U.S.A.)

A friendly visitor to what hopefully will remain a friendly Forum.
With thanks to all of the dedicated volunteers on the staff here!!
Top
Beberg
Pande Group Member
Posts: 307
Joined: Sat Dec 01, 2007 9:05 pm

Re: SQUID Proxy problems with assign.stanford.edu

Post by Beberg »

Nice work tear :)
Top
tear
Posts: 254
Joined: Sun Dec 02, 2007 4:08 am
Hardware configuration: None
Location: Rocky Mountains

Re: SQUID Proxy problems with assign.stanford.edu

Post by tear »

Pascal, John, Adam,


Your appreciation is appreciated (did it sound cliché?).

<bows>


tear
One man's ceiling is another man's floor.
Image
Top
SanHolo
Posts: 13
Joined: Tue Jun 02, 2009 8:54 am

Re: SQUID Proxy problems with assign.stanford.edu

Post by SanHolo »

tear wrote:Out of curiosity -- do you know *how* this other configuration is different than the other one?
I'd have to find out tomorrow, but I assume they did not update that one proxy to the same squid version as the main one which caused all this fuzz. Maybe something like a backup proxy that nobody knows about but the inner circle. :D :D

Keep on bowing, still standing ovations going on here. :wink:
Top
Beberg
Pande Group Member
Posts: 307
Joined: Sat Dec 01, 2007 9:05 pm

Re: SQUID Proxy problems with assign.stanford.edu

Post by Beberg »

Yea, let us know so we can tell people that have this problem they need to use Squid version X or better, since I'm sure they will fix the bug quickly now that Tear sent them a bug report, and it should be a fast fix.

On this end I made some code changes, but they wont be rolled out for at least a while, not that we really have absolute control over where the packets split up...
Top
tear
Posts: 254
Joined: Sun Dec 02, 2007 4:08 am
Hardware configuration: None
Location: Rocky Mountains

Re: SQUID Proxy problems with assign.stanford.edu

Post by tear »

Beberg wrote:Yea, let us know so we can tell people that have this problem they need to use Squid version X or better, since I'm sure they will fix the bug quickly now that Tear sent them a bug report, and it should be a fast fix.
Actually I haven't sent bug report to squid folks as I have no way of verifying (beyond any doubt) it's squid
(see doubts post) and I'm sure you know attitude towards vague bug reports.

I think IT guys from SanHolo's Uni have better environment to isolate the bug -- I would need to install squid...
Beberg wrote:On this end I made some code changes, but they wont be rolled out for at least a while, not that we really have absolute control over where the packets split up...
True, though aggregating payloads of sendto()/write() to (at least) full-lines (as far as response line + headers are concerned) should improve our chances...


tear
One man's ceiling is another man's floor.
Image
Top
SanHolo
Posts: 13
Joined: Tue Jun 02, 2009 8:54 am

Re: SQUID Proxy problems with assign.stanford.edu

Post by SanHolo »

I guess the squid folks know about an issue there, see the link to the blog post. And since our IT dept was working with (some of) the squid team, I assume they've told them about the problem. Anything you want me to do? The proxy error page of the working proxy does not tell me the version of squid (or whether it's even squid), is there another way? The unworking proxy is 3.0.STABLE18.
Top
tear
Posts: 254
Joined: Sun Dec 02, 2007 4:08 am
Hardware configuration: None
Location: Rocky Mountains

Re: SQUID Proxy problems with assign.stanford.edu

Post by tear »

SanHolo wrote:I guess the squid folks know about an issue there, see the link to the blog post.
I am pretty sure it is not the same problem -- I had attempted to explain here
but forgot about the bottom line :roll:
SanHolo wrote:Anything you want me to do? The proxy error page of the working proxy does not tell me the version of squid (or whether it's even squid), is there another way? The unworking proxy is 3.0.STABLE18.
I'll venture a guess squid adds extra header in responses -- you'd need to do a manual retrieval of a(ny) site
through the proxy (using telnet).

Just call telnet yourproxy yourproxyport. Then it's only the matter of sending valid... (invalid will do too) request.

For instance, you can type GET / HTTP/1.1 and press Enter *two* times. That should return an error message along
with squid version (either in HTTP headers or HTML body below).


tear
One man's ceiling is another man's floor.
Image
Top
SanHolo
Posts: 13
Joined: Tue Jun 02, 2009 8:54 am

Re: SQUID Proxy problems with assign.stanford.edu

Post by SanHolo »

Sorry, am abroad again, but just tried it with a VPN-Session: Nope, the working proxy does not return a signature, only the success/error message. The non-working proxy does, however, return it's signature, which I posted before: squid/3.0.STABLE18

thx!
Top
tear
Posts: 254
Joined: Sun Dec 02, 2007 4:08 am
Hardware configuration: None
Location: Rocky Mountains

Re: SQUID Proxy problems with assign.stanford.edu

Post by tear »

Well then... I don't know of any other way to check the version (though given that I don't know squid
it doesn't mean there aren't any).


tear
One man's ceiling is another man's floor.
Image
Top
Post Reply

Return to “Issues with a specific server”