On 12/20/2011 8:16 AM, Le_Forgeron wrote:
>
> First consider each system which might be part of the problem (client&
> server) and then the network.
>
> Is the load on client higher than average ? (does it use swap ?)
>
> Is the load on server higher than usual ? (does it use swap ?)
>

We have changed some of the applications that we are using.
One new application loads (3) 5 MB images at the same time and we have 
3-4 people using it at the same time.

But we have had other applications process all of the images in a 
directory in rapid succession without issue.

We have also recently changed from using access .mdb files to sqLite 
database files.


> And now for the real problem, of the network:
> * Is there any system which has been added or removed recently ?
>   (for instance, one with the same Windows name as your server... maybe
> someone copied the samba setting for another system... oops! the name
> must be changed)
> * Does any system reboot recently ?
> * Is the DHCP server still running ?
> (the last two are connected: if your client restarts without getting
> DHCP info, it will be in a default network... which might be a bit
> difficult to reach the normal network)
>

No network changes per se
	no server software or hardware was changed
	no network components were changed (firewall / switch)

We run a small shop
	1 windows 2003 SB server    in use for 7 years
	1 Linux (Linux From Scratch) file server  in use for 4 years
	10 Windows XP computers
both have been in use for many years with few changes

> Check on the server the "ifconfig" statistic ("ifconfig -a")
>            ...
>            RX packets:121678837 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:230675697 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:9614830711 (9.6 GB)  TX bytes:333118590778 (333.1 GB)
>            Interrupt:17
>
> Are you with numbers instead of 0 ?
>

I ran "ip -s link" and got no errors:
1: eth0: <BROADCAST,MULTICAST,SLAVE,UP,10000> mtu 1500 qdisc pfifo_fast 
master bond0 qlen 1000
     link/ether 00:30:48:8e:a9:da brd ff:ff:ff:ff:ff:ff
     RX: bytes  packets  errors  dropped overrun mcast
     3451334983 1139981492 0       0       0       6235
     TX: bytes  packets  errors  dropped carrier collsns
     4014767917 1149015701 0       0       0       0
2: eth1: <BROADCAST,MULTICAST,SLAVE,UP,10000> mtu 1500 qdisc pfifo_fast 
master bond0 qlen 1000
     link/ether 00:30:48:8e:a9:da brd ff:ff:ff:ff:ff:ff
     RX: bytes  packets  errors  dropped overrun mcast
     2758140294 1137482626 0       0       0       6235
     TX: bytes  packets  errors  dropped carrier collsns
     4012883940 1148897644 0       0       0       0
3: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     RX: bytes  packets  errors  dropped overrun mcast
     2668631    23838    0       0       0       0
     TX: bytes  packets  errors  dropped carrier collsns
     2668631    23838    0       0       0       0
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,10000> mtu 1500 qdisc noqueue
     link/ether 00:30:48:8e:a9:da brd ff:ff:ff:ff:ff:ff
     RX: bytes  packets  errors  dropped overrun mcast
     1914507981 2277464118 0       0       0       12470
     TX: bytes  packets  errors  dropped carrier collsns
     3732684561 2297913345 0       0       0       0



> If you have manageable routers, did you check if someone is not eating
> all the bandwidth with some "new" application (like "let's use skype !",
> a torrent client is such fun, "I'm bored, so let's listen to
> internet-radio", there is so many lovely video on youtube with dancing
> cats... ). Hopefully the router can make a bit of graphs per port to
> display the average traffic.
>

I have a cheap DELL gigabit switch that has some management capabilities.
I will see what I can do to take a look at some of this.


> And with a bit of installation, put "Wireshark" on both client and
> server, and make a capture in promiscious mode on both systems while
> loading your image. Compare the captures, check the usage of bandwidth
> for your application (are you 100% of packets, with lot of losts, or a
> mere 0.01% in a storm of broadcast or other packets ? What is the
> time-spacing of the packets for your application flow ?(need to isolate
> the connection based on port number) ?
>
> (Notice: you will have to install pcap driver on Windows for Wireshark
> to capture... also, you need administrator right on both systems)
>

I know exactly what you are talking about - but without experience of 
looking at this data it will likely take me too long to use this method. 
  If I need to go this route, then we should be calling someone in with 
more experience.

>
> Did you check the cables ?
>

Visually everything looks good - we have talked about switching out 
cables to isolate some parts of the network and will likely do so in the 
next week or so.

> Any upgrade/update of systems ?
>

> May be your office is now part of botnet, attacking some target in a
> DDOS attack.

True - scary but true.

Post a reply to this message