Below is a trace of the AFS network activity resulting from
<B>ls -l /afs/ncsa/.u8/bertini/</B>,<BR>
from the machine brick7 (Linux 2.4.18 + OpenAFS 1.2.8)
which took 1 minute 12 seconds (!) to finish.  
<P>
I don't have read access to .../bertini/, so none of the
information gets cached, and the behavior is fairly repeatable.
<P>
Apparent timing behavior is similar when I do this from an SGI AFS client,
though I haven't tried to trace the network there.
<P>
<HR>
<P>
A full brick7<->crystal tcpdump trace from two such successive
"ls -l ..." commands is at <A HREF="afstrace.tcpdump">afstrace.tcpdump</A>,
readable with e.g. <B>tcpdump -r afstrace.tcpdump -x</B>.
<P>
<HR>
<P>
Below are some packet headers for a few typical seconds
out of this sequence.
<P>
Several of the gaps follow the pattern:
<UL>
 <LI> brick7 sends a message to crystal
 <LI> after about 2 seconds with no reply, brick7 re-sends same packet
 <LI> after another 1 second or so, crystal replies,
 <LI> and brick7 promptly sends another packet.
</UL>
Here are the packet headers.  I've inserted blank lines to mark the
time-gaps, where no traffic was flowing for more than a
few hundred milliseconds.
<PRE>
13:38:02.609117 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:02.609245 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
13:38:02.609673 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:02.609980 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 300 (DF)
13:38:02.610739 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:02.610844 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
13:38:02.611271 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:02.611791 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 288 (DF)
13:38:02.845238 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 65 (DF)

13:38:05.093841 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 288 (DF)
13:38:05.615073 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:05.615248 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)

13:38:07.644127 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)

13:38:08.625294 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:08.625674 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 252 (DF)

13:38:10.704506 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 252 (DF)

13:38:11.635122 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:11.635381 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)

13:38:13.764866 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)

13:38:14.645156 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:14.645555 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 240 (DF)

13:38:16.825232 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 240 (DF)

13:38:17.655129 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
13:38:17.655284 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)

13:38:19.885594 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
</PRE>

Here's the same extract, with packet contents (including IP headers) listed.

<PRE>
13:38:02.609117 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97ae 4000 fd11 878a 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 fb49 be9b 4f25
			 fa7b 7ff4 0000 0038 0000 0000 0000 0066
			 0400 0000 e24d 0001 0000 000d
13:38:02.609245 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 6435 be9b 4f25
			 fa7b 7ff4 0000 0039 0000 0001 0000 006d
			 0105 0000 0000 0001 0000 0084 736a 0748
			 0000 008e 0000 0085
13:38:02.609673 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97af 4000 fd11 8789 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 b3df be9b 4f25
			 fa7b 7ff4 0000 0039 0000 0000 0000 0067
			 0400 0000 29b6 0001 0000 000d
13:38:02.609980 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 300 (DF)
			 4500 0148 0000 4000 4011 db2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0134 c6f2 be9b 4f25
			 fa7b 7ff4 0000 003a 0000 0001 0000 006e
			 0105 0000 0000 0001 0000 009b 0000 0016
			 736a 0748 0000 0076 0000 0089 736a 0748
			 0000 008a 0000 0267 736a 0748 0000 0074
			 0000 01e0 736a 0748 0000 0036 0000 06ca
			 736a 0748 0000 002a 0000 026a 736a 0748
			 0000 0084 0000 0105 736a 0748 0000 0075
			 0000 03ce 736a 0748 0000 006f 0000 027d
			 736a 0748 0000 0192 0000 0295 736a 0748
			 0000 019c 0000 0296 736a 0748 0000 019e
			 0000 0297 736a 0748 0000 00f8 0000 08dc
			 736a 0748 0000 0294 0000 08da 736a 0748
			 0000 0041 0000 06ea 736a 0748 0000 0055
			 0000 070a 736a 0748 0000 0114 0000 0715
			 736a 0748 0000 005d 0000 071a 736a 0748
			 0000 01b6 0000 0776 736a 0748 0000 00a3
			 0000 08a9 736a 0748 0000 00a9 0000 0b55
			 736a 0748 0000 0298 0000 08c4 736a 0748
			 0000 002d 0000 08f9
13:38:02.610739 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97b0 4000 fd11 8788 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 dd93 be9b 4f25
			 fa7b 7ff4 0000 003a 0000 0000 0000 0068
			 0400 0000 0000 0001 0000 000d
13:38:02.610844 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 6445 be9b 4f25
			 fa7b 7ff4 0000 003b 0000 0001 0000 006f
			 0105 0000 0000 0001 0000 0084 736a 0748
			 0000 0076 0000 0089
13:38:02.611271 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97b1 4000 fd11 8787 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 1cc3 be9b 4f25
			 fa7b 7ff4 0000 003b 0000 0000 0000 0069
			 0400 0000 c0ce 0001 0000 000d
13:38:02.611791 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 288 (DF)
			 4500 013c 0000 4000 4011 db39 8d8e 404e
			 8d8e 030d 1b59 1b58 0128 42b9 be9b 4f25
			 fa7b 7ff4 0000 003c 0000 0001 0000 0070
			 0105 0000 0000 0001 0000 009b 0000 0015
			 736a 0748 0000 008a 0000 0267 736a 0748
			 0000 0074 0000 01e0 736a 0748 0000 0036
			 0000 06ca 736a 0748 0000 002a 0000 026a
			 736a 0748 0000 0084 0000 0105 736a 0748
			 0000 0075 0000 03ce 736a 0748 0000 006f
			 0000 027d 736a 0748 0000 0192 0000 0295
			 736a 0748 0000 019c 0000 0296 736a 0748
			 0000 019e 0000 0297 736a 0748 0000 00f8
			 0000 08dc 736a 0748 0000 0294 0000 08da
			 736a 0748 0000 0041 0000 06ea 736a 0748
			 0000 0055 0000 070a 736a 0748 0000 0114
			 0000 0715 736a 0748 0000 005d 0000 071a
			 736a 0748 0000 01b6 0000 0776 736a 0748
			 0000 00a3 0000 08a9 736a 0748 0000 00a9
			 0000 0b55 736a 0748 0000 0298 0000 08c4
			 736a 0748 0000 002d 0000 08f9
13:38:02.845238 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 65 (DF)
			 4500 005d 97b2 4000 fd11 8765 8d8e 030d
			 8d8e 404e 1b58 1b59 0049 ccea a13a cfea
			 fa59 80a0 0000 0002 0000 0000 0000 0003
			 0221 0000 f1d5 0001 0000 0000 0000 0002
			 0000 0001 0000 0002 0800 5d0a 5400 0016
			 3c00 0005 a400 0000 2000 0000 04


13:38:05.093841 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 288 (DF)
			 4500 013c 0000 4000 4011 db39 8d8e 404e
			 8d8e 030d 1b59 1b58 0128 42b6 be9b 4f25
			 fa7b 7ff4 0000 003c 0000 0001 0000 0071
			 0107 0000 0000 0001 0000 009b 0000 0015
			 736a 0748 0000 008a 0000 0267 736a 0748
			 0000 0074 0000 01e0 736a 0748 0000 0036
			 0000 06ca 736a 0748 0000 002a 0000 026a
			 736a 0748 0000 0084 0000 0105 736a 0748
			 0000 0075 0000 03ce 736a 0748 0000 006f
			 0000 027d 736a 0748 0000 0192 0000 0295
			 736a 0748 0000 019c 0000 0296 736a 0748
			 0000 019e 0000 0297 736a 0748 0000 00f8
			 0000 08dc 736a 0748 0000 0294 0000 08da
			 736a 0748 0000 0041 0000 06ea 736a 0748
			 0000 0055 0000 070a 736a 0748 0000 0114
			 0000 0715 736a 0748 0000 005d 0000 071a
			 736a 0748 0000 01b6 0000 0776 736a 0748
			 0000 00a3 0000 08a9 736a 0748 0000 00a9
			 0000 0b55 736a 0748 0000 0298 0000 08c4
			 736a 0748 0000 002d 0000 08f9
13:38:05.615073 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97b3 4000 fd11 8785 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 dd8f be9b 4f25
			 fa7b 7ff4 0000 003c 0000 0000 0000 006a
			 0400 0000 0000 0001 0000 000d
13:38:05.615248 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 624e be9b 4f25
			 fa7b 7ff4 0000 003d 0000 0001 0000 0072
			 0105 0000 0000 0001 0000 0084 736a 0748
			 0000 008a 0000 0267



13:38:07.644127 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 624b be9b 4f25
			 fa7b 7ff4 0000 003d 0000 0001 0000 0073
			 0107 0000 0000 0001 0000 0084 736a 0748
			 0000 008a 0000 0267



13:38:08.625294 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97b4 4000 fd11 8784 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 dd8d be9b 4f25
			 fa7b 7ff4 0000 003d 0000 0000 0000 006b
			 0400 0000 0000 0001 0000 000d
13:38:08.625674 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 252 (DF)
			 4500 0118 0000 4000 4011 db5d 8d8e 404e
			 8d8e 030d 1b59 1b58 0104 bf5a be9b 4f25
			 fa7b 7ff4 0000 003e 0000 0001 0000 0074
			 0105 0000 0000 0001 0000 009b 0000 0012
			 736a 0748 0000 002a 0000 026a 736a 0748
			 0000 0084 0000 0105 736a 0748 0000 0075
			 0000 03ce 736a 0748 0000 006f 0000 027d
			 736a 0748 0000 0192 0000 0295 736a 0748
			 0000 019c 0000 0296 736a 0748 0000 019e
			 0000 0297 736a 0748 0000 00f8 0000 08dc
			 736a 0748 0000 0294 0000 08da 736a 0748
			 0000 0041 0000 06ea 736a 0748 0000 0055
			 0000 070a 736a 0748 0000 0114 0000 0715
			 736a 0748 0000 005d 0000 071a 736a 0748
			 0000 01b6 0000 0776 736a 0748 0000 00a3
			 0000 08a9 736a 0748 0000 00a9 0000 0b55
			 736a 0748 0000 0298 0000 08c4 736a 0748
			 0000 002d 0000 08f9



13:38:10.704506 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 252 (DF)
			 4500 0118 0000 4000 4011 db5d 8d8e 404e
			 8d8e 030d 1b59 1b58 0104 bf57 be9b 4f25
			 fa7b 7ff4 0000 003e 0000 0001 0000 0075
			 0107 0000 0000 0001 0000 009b 0000 0012
			 736a 0748 0000 002a 0000 026a 736a 0748
			 0000 0084 0000 0105 736a 0748 0000 0075
			 0000 03ce 736a 0748 0000 006f 0000 027d
			 736a 0748 0000 0192 0000 0295 736a 0748
			 0000 019c 0000 0296 736a 0748 0000 019e
			 0000 0297 736a 0748 0000 00f8 0000 08dc
			 736a 0748 0000 0294 0000 08da 736a 0748
			 0000 0041 0000 06ea 736a 0748 0000 0055
			 0000 070a 736a 0748 0000 0114 0000 0715
			 736a 0748 0000 005d 0000 071a 736a 0748
			 0000 01b6 0000 0776 736a 0748 0000 00a3
			 0000 08a9 736a 0748 0000 00a9 0000 0b55
			 736a 0748 0000 0298 0000 08c4 736a 0748
			 0000 002d 0000 08f9



13:38:11.635122 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97b5 4000 fd11 8783 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 dd8b be9b 4f25
			 fa7b 7ff4 0000 003e 0000 0000 0000 006c
			 0400 0000 0000 0001 0000 000d
13:38:11.635381 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 62a5 be9b 4f25
			 fa7b 7ff4 0000 003f 0000 0001 0000 0076
			 0105 0000 0000 0001 0000 0084 736a 0748
			 0000 002a 0000 026a



13:38:13.764866 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 62a2 be9b 4f25
			 fa7b 7ff4 0000 003f 0000 0001 0000 0077
			 0107 0000 0000 0001 0000 0084 736a 0748
			 0000 002a 0000 026a



13:38:14.645156 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97b6 4000 fd11 8782 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 dd89 be9b 4f25
			 fa7b 7ff4 0000 003f 0000 0000 0000 006d
			 0400 0000 0000 0001 0000 000d
13:38:14.645555 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 240 (DF)
			 4500 010c 0000 4000 4011 db69 8d8e 404e
			 8d8e 030d 1b59 1b58 00f8 3cb4 be9b 4f25
			 fa7b 7ff4 0000 0040 0000 0001 0000 0078
			 0105 0000 0000 0001 0000 009b 0000 0011
			 736a 0748 0000 0084 0000 0105 736a 0748
			 0000 0075 0000 03ce 736a 0748 0000 006f
			 0000 027d 736a 0748 0000 0192 0000 0295
			 736a 0748 0000 019c 0000 0296 736a 0748
			 0000 019e 0000 0297 736a 0748 0000 00f8
			 0000 08dc 736a 0748 0000 0294 0000 08da
			 736a 0748 0000 0041 0000 06ea 736a 0748
			 0000 0055 0000 070a 736a 0748 0000 0114
			 0000 0715 736a 0748 0000 005d 0000 071a
			 736a 0748 0000 01b6 0000 0776 736a 0748
			 0000 00a3 0000 08a9 736a 0748 0000 00a9
			 0000 0b55 736a 0748 0000 0298 0000 08c4
			 736a 0748 0000 002d 0000 08f9



13:38:16.825232 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 240 (DF)
			 4500 010c 0000 4000 4011 db69 8d8e 404e
			 8d8e 030d 1b59 1b58 00f8 3cb1 be9b 4f25
			 fa7b 7ff4 0000 0040 0000 0001 0000 0079
			 0107 0000 0000 0001 0000 009b 0000 0011
			 736a 0748 0000 0084 0000 0105 736a 0748
			 0000 0075 0000 03ce 736a 0748 0000 006f
			 0000 027d 736a 0748 0000 0192 0000 0295
			 736a 0748 0000 019c 0000 0296 736a 0748
			 0000 019e 0000 0297 736a 0748 0000 00f8
			 0000 08dc 736a 0748 0000 0294 0000 08da
			 736a 0748 0000 0041 0000 06ea 736a 0748
			 0000 0055 0000 070a 736a 0748 0000 0114
			 0000 0715 736a 0748 0000 005d 0000 071a
			 736a 0748 0000 01b6 0000 0776 736a 0748
			 0000 00a3 0000 08a9 736a 0748 0000 00a9
			 0000 0b55 736a 0748 0000 0298 0000 08c4
			 736a 0748 0000 002d 0000 08f9



13:38:17.655129 < crystal.ncsa.uiuc.edu.afs3-fileserver > brick7.ncsa.uiuc.edu.afs3-callback: udp 32 (DF)
			 4500 003c 97b7 4000 fd11 8781 8d8e 030d
			 8d8e 404e 1b58 1b59 0028 dd87 be9b 4f25
			 fa7b 7ff4 0000 0040 0000 0000 0000 006e
			 0400 0000 0000 0001 0000 000d
13:38:17.655284 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 63aa be9b 4f25
			 fa7b 7ff4 0000 0041 0000 0001 0000 007a
			 0105 0000 0000 0001 0000 0084 736a 0748
			 0000 0084 0000 0105



13:38:19.885594 > brick7.ncsa.uiuc.edu.afs3-callback > crystal.ncsa.uiuc.edu.afs3-fileserver: udp 44 (DF)
			 4500 0048 0000 4000 4011 dc2d 8d8e 404e
			 8d8e 030d 1b59 1b58 0034 63a7 be9b 4f25
			 fa7b 7ff4 0000 0041 0000 0001 0000 007b
			 0107 0000 0000 0001 0000 0084 736a 0748
			 0000 0084 0000 0105

</PRE>
  [...]