 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
To the network gurus
We are currently experiencing some network slowdowns that we have not
experienced in the past and I am trying to narrow down what is going on.
In the past one could expect 1/2 a second delay when loading images or
other data.
More recently we have seen 5+ minutes of delay.
Our applications make heavy use of images - one application loads (3)
5MB images at once. This application is used by 3-5 people in the
office at the same time.
We run a Windows 2003 SBS domain controller with a Linux based file
server. The file server is only that - a file server with Linux From
Scratch so there is nothing else running on it.
So far I have looked at file locks and 'top' to try to see if anything
stands out, but I am far from being a network administrator and am
searching to find where the problem may lie.
What should I be looking at to help determine where the problem is
originating from?
Any help is greatly appreciated.
Thanks
Tom
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Le 20/12/2011 13:35, Tom Austin a écrit :
> To the network gurus
>
> We are currently experiencing some network slowdowns that we have not
> experienced in the past and I am trying to narrow down what is going on.
>
> In the past one could expect 1/2 a second delay when loading images or
> other data.
> More recently we have seen 5+ minutes of delay.
>
> Our applications make heavy use of images - one application loads (3)
> 5MB images at once. This application is used by 3-5 people in the
> office at the same time.
>
> We run a Windows 2003 SBS domain controller with a Linux based file
> server. The file server is only that - a file server with Linux From
> Scratch so there is nothing else running on it.
>
> So far I have looked at file locks and 'top' to try to see if anything
> stands out, but I am far from being a network administrator and am
> searching to find where the problem may lie.
>
> What should I be looking at to help determine where the problem is
> originating from?
>
>
> Any help is greatly appreciated.
First consider each system which might be part of the problem (client &
server) and then the network.
Is the load on client higher than average ? (does it use swap ?)
Is the load on server higher than usual ? (does it use swap ?)
And now for the real problem, of the network:
* Is there any system which has been added or removed recently ?
(for instance, one with the same Windows name as your server... maybe
someone copied the samba setting for another system... oops! the name
must be changed)
* Does any system reboot recently ?
* Is the DHCP server still running ?
(the last two are connected: if your client restarts without getting
DHCP info, it will be in a default network... which might be a bit
difficult to reach the normal network)
Check on the server the "ifconfig" statistic ("ifconfig -a")
...
RX packets:121678837 errors:0 dropped:0 overruns:0 frame:0
TX packets:230675697 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9614830711 (9.6 GB) TX bytes:333118590778 (333.1 GB)
Interrupt:17
Are you with numbers instead of 0 ?
If you have manageable routers, did you check if someone is not eating
all the bandwidth with some "new" application (like "let's use skype !",
a torrent client is such fun, "I'm bored, so let's listen to
internet-radio", there is so many lovely video on youtube with dancing
cats... ). Hopefully the router can make a bit of graphs per port to
display the average traffic.
And with a bit of installation, put "Wireshark" on both client and
server, and make a capture in promiscious mode on both systems while
loading your image. Compare the captures, check the usage of bandwidth
for your application (are you 100% of packets, with lot of losts, or a
mere 0.01% in a storm of broadcast or other packets ? What is the
time-spacing of the packets for your application flow ?(need to isolate
the connection based on port number) ?
(Notice: you will have to install pcap driver on Windows for Wireshark
to capture... also, you need administrator right on both systems)
Did you check the cables ?
Any upgrade/update of systems ?
May be your office is now part of botnet, attacking some target in a
DDOS attack.
--
A good Manager will take you
through the forest, no mater what.
A Leader will take time to climb on a
Tree and say 'This is the wrong forest'.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Tue, 20 Dec 2011 07:35:35 -0500, Tom Austin wrote:
> What should I be looking at to help determine where the problem is
> originating from?
With network problems, I tend to start with a sniffer to see if there's
something weird going on on the network. If you can determine the source
of the delays (like a name lookup failure/retry cycle, lots of CRC errors
or other issues that are causing problems), that can often point to a
hardware issue or some other cause that you hadn't considered.
Jim
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 12/20/2011 8:16 AM, Le_Forgeron wrote:
>
> First consider each system which might be part of the problem (client&
> server) and then the network.
>
> Is the load on client higher than average ? (does it use swap ?)
>
> Is the load on server higher than usual ? (does it use swap ?)
>
We have changed some of the applications that we are using.
One new application loads (3) 5 MB images at the same time and we have
3-4 people using it at the same time.
But we have had other applications process all of the images in a
directory in rapid succession without issue.
We have also recently changed from using access .mdb files to sqLite
database files.
> And now for the real problem, of the network:
> * Is there any system which has been added or removed recently ?
> (for instance, one with the same Windows name as your server... maybe
> someone copied the samba setting for another system... oops! the name
> must be changed)
> * Does any system reboot recently ?
> * Is the DHCP server still running ?
> (the last two are connected: if your client restarts without getting
> DHCP info, it will be in a default network... which might be a bit
> difficult to reach the normal network)
>
No network changes per se
no server software or hardware was changed
no network components were changed (firewall / switch)
We run a small shop
1 windows 2003 SB server in use for 7 years
1 Linux (Linux From Scratch) file server in use for 4 years
10 Windows XP computers
both have been in use for many years with few changes
> Check on the server the "ifconfig" statistic ("ifconfig -a")
> ...
> RX packets:121678837 errors:0 dropped:0 overruns:0 frame:0
> TX packets:230675697 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:9614830711 (9.6 GB) TX bytes:333118590778 (333.1 GB)
> Interrupt:17
>
> Are you with numbers instead of 0 ?
>
I ran "ip -s link" and got no errors:
1: eth0: <BROADCAST,MULTICAST,SLAVE,UP,10000> mtu 1500 qdisc pfifo_fast
master bond0 qlen 1000
link/ether 00:30:48:8e:a9:da brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
3451334983 1139981492 0 0 0 6235
TX: bytes packets errors dropped carrier collsns
4014767917 1149015701 0 0 0 0
2: eth1: <BROADCAST,MULTICAST,SLAVE,UP,10000> mtu 1500 qdisc pfifo_fast
master bond0 qlen 1000
link/ether 00:30:48:8e:a9:da brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
2758140294 1137482626 0 0 0 6235
TX: bytes packets errors dropped carrier collsns
4012883940 1148897644 0 0 0 0
3: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
RX: bytes packets errors dropped overrun mcast
2668631 23838 0 0 0 0
TX: bytes packets errors dropped carrier collsns
2668631 23838 0 0 0 0
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,10000> mtu 1500 qdisc noqueue
link/ether 00:30:48:8e:a9:da brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
1914507981 2277464118 0 0 0 12470
TX: bytes packets errors dropped carrier collsns
3732684561 2297913345 0 0 0 0
> If you have manageable routers, did you check if someone is not eating
> all the bandwidth with some "new" application (like "let's use skype !",
> a torrent client is such fun, "I'm bored, so let's listen to
> internet-radio", there is so many lovely video on youtube with dancing
> cats... ). Hopefully the router can make a bit of graphs per port to
> display the average traffic.
>
I have a cheap DELL gigabit switch that has some management capabilities.
I will see what I can do to take a look at some of this.
> And with a bit of installation, put "Wireshark" on both client and
> server, and make a capture in promiscious mode on both systems while
> loading your image. Compare the captures, check the usage of bandwidth
> for your application (are you 100% of packets, with lot of losts, or a
> mere 0.01% in a storm of broadcast or other packets ? What is the
> time-spacing of the packets for your application flow ?(need to isolate
> the connection based on port number) ?
>
> (Notice: you will have to install pcap driver on Windows for Wireshark
> to capture... also, you need administrator right on both systems)
>
I know exactly what you are talking about - but without experience of
looking at this data it will likely take me too long to use this method.
If I need to go this route, then we should be calling someone in with
more experience.
>
> Did you check the cables ?
>
Visually everything looks good - we have talked about switching out
cables to isolate some parts of the network and will likely do so in the
next week or so.
> Any upgrade/update of systems ?
>
> May be your office is now part of botnet, attacking some target in a
> DDOS attack.
True - scary but true.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 12/20/2011 11:14 AM, Jim Henderson wrote:
> On Tue, 20 Dec 2011 07:35:35 -0500, Tom Austin wrote:
>
>> What should I be looking at to help determine where the problem is
>> originating from?
>
> With network problems, I tend to start with a sniffer to see if there's
> something weird going on on the network. If you can determine the source
> of the delays (like a name lookup failure/retry cycle, lots of CRC errors
> or other issues that are causing problems), that can often point to a
> hardware issue or some other cause that you hadn't considered.
>
> Jim
Wireshark has been suggested - I may look into trying it if I get the time.
Thanks
Tom
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Wed, 21 Dec 2011 09:15:01 -0500, Tom Austin wrote:
> On 12/20/2011 11:14 AM, Jim Henderson wrote:
>> On Tue, 20 Dec 2011 07:35:35 -0500, Tom Austin wrote:
>>
>>> What should I be looking at to help determine where the problem is
>>> originating from?
>>
>> With network problems, I tend to start with a sniffer to see if there's
>> something weird going on on the network. If you can determine the
>> source of the delays (like a name lookup failure/retry cycle, lots of
>> CRC errors or other issues that are causing problems), that can often
>> point to a hardware issue or some other cause that you hadn't
>> considered.
>>
>> Jim
>
>
> Wireshark has been suggested - I may look into trying it if I get the
> time.
That's a good one, and the one I use. I find it's better to do this
rather than try to guess - it can save time spent looking at blinkenlights
on switches and whatnot, because it tells you exactly what's happening on
the wire.
Jim
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Le 2011-12-21 18:35, Jim Henderson a écrit :
> On Wed, 21 Dec 2011 09:15:01 -0500, Tom Austin wrote:
>
>> On 12/20/2011 11:14 AM, Jim Henderson wrote:
>>> On Tue, 20 Dec 2011 07:35:35 -0500, Tom Austin wrote:
>>>
>>>> What should I be looking at to help determine where the problem is
>>>> originating from?
>>>
>>> With network problems, I tend to start with a sniffer to see if there's
>>> something weird going on on the network. If you can determine the
>>> source of the delays (like a name lookup failure/retry cycle, lots of
>>> CRC errors or other issues that are causing problems), that can often
>>> point to a hardware issue or some other cause that you hadn't
>>> considered.
>>>
>>> Jim
>>
>>
>> Wireshark has been suggested - I may look into trying it if I get the
>> time.
>
> That's a good one, and the one I use. I find it's better to do this
> rather than try to guess - it can save time spent looking at blinkenlights
> on switches and whatnot, because it tells you exactly what's happening on
> the wire.
>
> Jim
Keep in mind that in a switched environment, the switch will only send
the packets that your workstation should see (i.e.: packets intended for
you and broadcasts/multicasts). Without configuring the switch for
port-replication, you will not be able to see everything that goes on in
the network.
In other words, take automatic diagnostics of "high level of broadcasts
from XYZ" with a grain of salt since - by design - it's the only frames
coming from that station that you will see.
--
/*Francois Labreque*/#local a=x+y;#local b=x+a;#local c=a+b;#macro P(F//
/* flabreque */L)polygon{5,F,F+z,L+z,L,F pigment{rgb 9}}#end union
/* @ */{P(0,a)P(a,b)P(b,c)P(2*a,2*b)P(2*b,b+c)P(b+c,<2,3>)
/* gmail.com */}camera{orthographic location<6,1.25,-6>look_at a }
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Thu, 22 Dec 2011 10:17:36 -0500, Francois Labreque wrote:
> Keep in mind that in a switched environment, the switch will only send
> the packets that your workstation should see (i.e.: packets intended for
> you and broadcasts/multicasts). Without configuring the switch for
> port-replication, you will not be able to see everything that goes on in
> the network.
>
> In other words, take automatic diagnostics of "high level of broadcasts
> from XYZ" with a grain of salt since - by design - it's the only frames
> coming from that station that you will see.
Very true - but if you take a trace from the station that's having
difficulty, most of the time that will give you enough information to go
on.
But if it doesn't, then yes, you need to enable port replication (if the
switch supports it) or do something to otherwise look at all the traffic
going through it.
Jim
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On 12/22/2011 10:17 AM, Francois Labreque wrote:
> Le 2011-12-21 18:35, Jim Henderson a écrit :
>> On Wed, 21 Dec 2011 09:15:01 -0500, Tom Austin wrote:
>>
>>> On 12/20/2011 11:14 AM, Jim Henderson wrote:
>>>> On Tue, 20 Dec 2011 07:35:35 -0500, Tom Austin wrote:
>>>>
>>>>> What should I be looking at to help determine where the problem is
>>>>> originating from?
>>>>
>>>> With network problems, I tend to start with a sniffer to see if there's
>>>> something weird going on on the network. If you can determine the
>>>> source of the delays (like a name lookup failure/retry cycle, lots of
>>>> CRC errors or other issues that are causing problems), that can often
>>>> point to a hardware issue or some other cause that you hadn't
>>>> considered.
>>>>
>>>> Jim
>>>
>>>
>>> Wireshark has been suggested - I may look into trying it if I get the
>>> time.
>>
>> That's a good one, and the one I use. I find it's better to do this
>> rather than try to guess - it can save time spent looking at
>> blinkenlights
>> on switches and whatnot, because it tells you exactly what's happening on
>> the wire.
>>
>> Jim
>
> Keep in mind that in a switched environment, the switch will only send
> the packets that your workstation should see (i.e.: packets intended for
> you and broadcasts/multicasts). Without configuring the switch for
> port-replication, you will not be able to see everything that goes on in
> the network.
>
> In other words, take automatic diagnostics of "high level of broadcasts
> from XYZ" with a grain of salt since - by design - it's the only frames
> coming from that station that you will see.
>
Yep, I am with you on this one - we have a switch - and likely in the
mix I would have forgotten about it's 'benefits'
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
