==============================
General
==============================
Sometimes there is a need to determine where from connections are coming to Oracle DB.
==============================
On Oracle host (199.188.111.86)
==============================
A. Find the suspected remote host
netstat -ano | grep 1521 | wc -l
490
netstat -ano | grep 1521 | grep 192.168.110.4 | wc -l
98
B. Find the open ports on the suspected remote host.
netstat -ano | grep 1521 | grep 192.168.110.4 | grep keep
tcp 0 0 199.188.111.86:1521 199.188.111.4:52605 ESTABLISHED keepalive (5306.17/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:11265 ESTABLISHED keepalive (6096.62/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:55128 ESTABLISHED keepalive (4930.29/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:9951 ESTABLISHED keepalive (2355.54/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:51404 ESTABLISHED keepalive (4717.04/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:51391 ESTABLISHED keepalive (6858.24/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:51403 ESTABLISHED keepalive (4935.00/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:55433 ESTABLISHED keepalive (5940.11/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:11205 ESTABLISHED keepalive (3176.02/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:57786 ESTABLISHED keepalive (1346.87/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:9792 ESTABLISHED keepalive (3342.94/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:57787 ESTABLISHED keepalive (1291.57/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:9776 ESTABLISHED keepalive (2204.64/0/0)
tcp 0 0 199.188.111.86:1521 199.188.111.4:9791 ESTABLISHED keepalive (3343.07/0/0)
==============================
On remote host (199.188.111.4)
==============================
A. Get the process details, by protocol@host:port
lsof -iTCP@199.188.111.4:9776
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
notificat 16316 iu 14u IPv4 2078111001 0t0 TCP esp-tel-2-aps-3:9776->esp-tel-2-ora-1:ncube-lm (ESTABLISHED)
lsof -iTCP@199.188.111.4:9791
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
notificat 12764 iu 13u IPv4 2078297343 0t0 TCP esp-tel-2-aps-3:9791->esp-tel-2-ora-1:ncube-lm (ESTABLISHED)
B. Get the process details, by PID
ps -ef | grep 12764
iu 12764 12471 1 Nov26 ? 01:28:21 /starhome/igate/slv-tel-sp2/ig1//notification --PROCESSNAME=SLV-TELSP-01_SMS-02_A --EVENTNAME=SLV-TELSP-01_SMS-02_A_XEVT --CONFIG=VIP.INI
Showing posts with label Network. Show all posts
Showing posts with label Network. Show all posts
Tuesday, December 1, 2015
Thursday, November 5, 2015
Sunday, November 1, 2015
Network Connectivity Testing commands
====================
Index
====================
ping
====================
The ping command verifies connectivity by sending Internet Control Message Protocol (ICMP) echo packets to a host and listening for an echo reply.
Times on the Internet can range from 80ms (milliseconds) to 250ms, which is considered normal during peak hours.
You can use the ping utility to test both the host name and IP address of the host for DNS resolution.
A successful IP ping and failed host name ping could indicate name resolution issues.
Example:
ping www.domain.com
ping 123.123.123.123
ping options
–t: continuous echo requests to a host.
ping 123.123.123.123 –t
nslookup examples
A. nslookup with no parameters
would return:
Your router's server and address.
D:\Users\akaplan>nslookup
Default Server: isr-01.starhome.local
Address: 100.111.100.444
B. nslookup with server name as parameter
would return:
First section: Server and address of the server that provided you with the information.
Second section: domain name and IP address of the input server name.
D:\Users\akaplan>nslookup cms
Server: isr-01.starhome.local
Address: 100.111.100.444
Name: cms.starhome.local
Address: 100.223.333.127
Index
====================
ping
telnet
tracert
nslookup
====================ping
====================
The ping command verifies connectivity by sending Internet Control Message Protocol (ICMP) echo packets to a host and listening for an echo reply.
Times on the Internet can range from 80ms (milliseconds) to 250ms, which is considered normal during peak hours.
You can use the ping utility to test both the host name and IP address of the host for DNS resolution.
A successful IP ping and failed host name ping could indicate name resolution issues.
Example:
ping www.domain.com
ping 123.123.123.123
ping options
–t: continuous echo requests to a host.
ping 123.123.123.123 –t
====================
telnet
====================
The Telnet command is used to test a variety of services for connectivity.
You can use it to test for SMTP, SQL or Remote Desktop connectivity.
This is a good test to use for service or port blocks resulting from a firewall configuration.
Usage
telnet <server> <port>
Example:
telnet mail.domain.com 25
telnet 123.123.123.123 3389
The response of a successful connection will be different for each service, but a failed connection will always respond with a variation of the following message:
"Could not open connection to the host, on port n: Connect failed"
====================
Trace Route
====================
A Trace Route command displays the path a packet takes to reach a destination from the machine that it is executed on.
It does this by sending Internet Control Message Protocol (ICMP) echo request messages to the destination. The path is displayed as a list in the order of which it heard back from each node that it passed through on its way to the destination.
The numbers on the trace show the time it's taking to travel to each of those locations.
Times on the Internet can range from 80ms (milliseconds) to 250ms.
A Trace Route will include at least 4 pieces of information for each hop; the number of the hop, the Round Trip Time (RTT is displayed in milliseconds or ms) it takes to get from the interface of the current hop and then back again to your machine, the IP address of the interface for that hop and the hostname corresponding to the IP address of the hop.
The default is to send out 3 packets to each hop. This is done in case a packet is lost and allows you to get an idea of whether or not there is a variance in the time for a specific hop.
A high number on the first external hop from you machine is a good indication of possible Local Area Network (LAN) issues.
An asterisk (*) indicates an echo request that was lost.
These can be the result of security implementations of firewalls or Access Control List (ACL’s).
Additionally, routers may be configured not to respond to this type of traffic. You may see a row of three asterisks with no IP address or hostname. The trace may then continue responding normally again and display the destination results.
Windows Usage
tracert domain.com
tracert 123.345.567.789
You can see the output results in the following example:
Tracing route to domain.com over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 111.111.111.111
2 <1 ms <1 ms <1 ms 222.222.222.222
3 1 ms 1 ms 1 ms 111.222.111.222
4 1 ms 1 ms 12 ms 222.111.222.111
5 * * * Request timed out.
6 14 ms 13 ms 13 ms domain.com
Trace complete.
====================
nslookup
====================
The Nslookup command allows you to displays information that you can use to diagnose Domain Name System (DNS) infrastructure.
It is helpful when you want to query a specific DNS record.
Command Usage:
1. Enter nslookup
2.Type set type=MX and then press ENTER.
3. Enter domain.
Or use the single line command:
nslookup -querytype=mx domain.com
nslookup examples
A. nslookup with no parameters
would return:
Your router's server and address.
D:\Users\akaplan>nslookup
Default Server: isr-01.starhome.local
Address: 100.111.100.444
B. nslookup with server name as parameter
would return:
First section: Server and address of the server that provided you with the information.
Second section: domain name and IP address of the input server name.
D:\Users\akaplan>nslookup cms
Server: isr-01.starhome.local
Address: 100.111.100.444
Name: cms.starhome.local
Address: 100.223.333.127
Thursday, March 27, 2014
Common Network Issues. netstat, lsof, tracert, tcpdump
===========================
Reference
===========================
Very good reference: Network Commands Reference
Site for running commands. Just provide the IP, and run command: network-tools.com
===========================
ping
telnet
netcat
tracert
netstat
lsof
ifconfig
tcpdump
mii-show
MTU and Packet size
ORA-03113 and MTU ping <IP>
D:\Users\akaplan>ping 11.222.333.444
Pinging 11.222.333.444 with 32 bytes of data:
Reply from 11.222.333.444: bytes=32 time=2ms TTL=127
Reply from 11.222.333.444: bytes=32 time=1ms TTL=127
Reply from 11.222.333.444: bytes=32 time=1ms TTL=127
Ping statistics for 11.222.333.444:
Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 2ms, Average = 1ms
Telnet
Reference: Troubleshooting with Telnet
Useful options with telnet:
telnet <some_server> - 23 is the default port for telnet
telnet <some_server> 3000 - Try to connect on port 3000.
netcat
Reference: netcat reference
Netcat is a computer networking service for reading from and writing to network connections using TCP or UDP.
tracert <IP>
D:\Users\akaplan>tracert 111.222.333.444
Tracing route to AAA-BBB-1-CCC-2 [111.222.333.444]
over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 44.555.666.1
2 1 ms 1 ms 1 ms AAA-BBB-1-CCC-2 [111.222.333.444]
Trace complete.
netstat command
netstat reference
The netstat command is used to display the TCP/IP network protocol statistics and information.
netstat flags
t - TCP active sockets
u - UDP active sockets
w - raw active sockets
x - Unix active sockets
l - Listening sockets
a - all (Active and Listening) sockets
o - timer information
p - PID/Process of the process that is using a socket
n - numeric. Do not resolve host, port, user from their numeric value.
(-n stands for Good for knowing current open connections and on which port.
netstat output fields on Linux
sudo netstat -plnt
List tcp ports that are being listened on, along with the name of each listener's daemon and its PID.
sample output:
sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 3686/mysqld
tcp 0 0 :::443 :::* LISTEN 2218/httpd
tcp 0 0 :::80 :::* LISTEN 2218/httpd
tcp 0 0 :::22 :::* LISTEN 1051/sshd
netstat -ano
List all ports with timer info.
sample output on Linux
Proto Recv-Q Send-Q Local Address Foreign Address State Timer
tcp 0 0 0.0.0.0:199 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:2381 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:2301 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 10.666.3.152:2301 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 111.666.222.1:2301 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:9494 127.0.0.1:32802 ESTABLISHED off (0.00/0/0)
tcp 4 0 55.227.3.555:55118 55.227.3.555:199 CLOSE_WAIT off (0.00/0/0)
tcp 0 0 127.0.0.1:5302 127.0.0.1:32775 ESTABLISHED keepalive (3334.13/0/0)
These paranmeters are configured in the file /etc/sysctl.conf.
Three parameters related to keepalive:
net.ipv4.tcp_keepalive_time
net.ipv4.tcp_keepalive_intvl
net.ipv4.tcp_keepalive_probes
The net.ipv4.tcp_keepalive_time parameter is the time before the first keepalive packet is sent out.
As long as there is TCP/IP socket communications going on, no keepalive packets are needed, but if the amount of time in seconds specified in net.ipv4.tcp_keepalive_time passes without any communication on a TCP/IP socket connection, then the Linux OS will begin sending keepalive packets.
Once keepalive packets begin being sent out, they will be sent every net.ipv4.tcp_keepalive_intvl time (in seconds).
Keepalive packets are a two-way exchange.
When one device sends a keepalive packet to another, the receiving device sends a quick acknowledgement packet back.
This way, both devices know the communication link between them is OK.
If the device sending the keepalive packet does not get a response back, it sends another keepalive packet after the net.ipv4.tcp_keepalive_intvl passes.
After enough keepalive packets are sent and no response is received, the sending device will assume the link is down, close the socket, and try to re-establish communications.
The number of keepalive packets sent before the device will reset if it does not get a response is configured in the net.ipv4.tcp_keepalive_probes parameter.
lsof
lsof Reference
lsof Reference II
The -i option without further qualification lists all open Internet socket files.
To refine the search, add network name, address, protocol name, service name or port number.
For example:
To know what process has a connection open to or from host my.server.com:
lsof -i@my.server.com
sudo netstat -lptu
sudo netstat -tulpn
=============================
ifconfig
=============================
tcpdump -i eth0
tcpdump -c 10 -i eth0
tcpdump -w some_file.pcap -i eth0
Write output to a file some_file.pcap.
tcpdump -n -tttt -i eth0
Capture packets with readable timestamp
tcpdump -w xpackets.pcap -i eth0 dst 10.181.140.216 and port 22
tcpdump -w comm.pcap -i eth0 src xxx.xxx.xxx.001 and port 22 and dst xxx.xxx.xxx.002 and port 22
=============================
=============================
MTU and Packet size
=============================
MTU: Maximum Transmission Unit
The default size for Packet size is 1400 bytes.
How to set the correct MTU size (Linksys Reference)
ping [url / local server or IP address] –f –l xxxx
Where xxxx is the packet size (value) which you will test.
For example:
>ping 10.16.39.122 -f -l 1400
Pinging 10.16.39.122 with 1400 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Ping statistics for 10.16.39.122:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
Pinging 10.16.39.122 with 1248 bytes of data:
Reply from 10.16.39.122: bytes=1248 time=312ms TTL=58
Ping statistics for 10.16.39.122:
Reference
===========================
Very good reference: Network Commands Reference
Site for running commands. Just provide the IP, and run command: network-tools.com
===========================
Most Useful Network commands:
===========================ping
telnet
netcat
tracert
netstat
lsof
ifconfig
tcpdump
mii-show
MTU and Packet size
ORA-03113 and MTU ping <IP>
D:\Users\akaplan>ping 11.222.333.444
Pinging 11.222.333.444 with 32 bytes of data:
Reply from 11.222.333.444: bytes=32 time=2ms TTL=127
Reply from 11.222.333.444: bytes=32 time=1ms TTL=127
Reply from 11.222.333.444: bytes=32 time=1ms TTL=127
Ping statistics for 11.222.333.444:
Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 2ms, Average = 1ms
Telnet
Reference: Troubleshooting with Telnet
Useful options with telnet:
telnet <some_server> - 23 is the default port for telnet
telnet <some_server> 3000 - Try to connect on port 3000.
netcat
Reference: netcat reference
Netcat is a computer networking service for reading from and writing to network connections using TCP or UDP.
tracert <IP>
D:\Users\akaplan>tracert 111.222.333.444
Tracing route to AAA-BBB-1-CCC-2 [111.222.333.444]
over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 44.555.666.1
2 1 ms 1 ms 1 ms AAA-BBB-1-CCC-2 [111.222.333.444]
Trace complete.
netstat command
netstat reference
The netstat command is used to display the TCP/IP network protocol statistics and information.
netstat flags
t - TCP active sockets
u - UDP active sockets
w - raw active sockets
x - Unix active sockets
l - Listening sockets
a - all (Active and Listening) sockets
o - timer information
p - PID/Process of the process that is using a socket
n - numeric. Do not resolve host, port, user from their numeric value.
(-n stands for Good for knowing current open connections and on which port.
netstat output fields on Linux
Proto The protocol (tcp, udp, raw) used by the socket.
Recv-Q The count of bytes not copied by the user program connected to this socket.
Send-Q The count of bytes not acknowledged by the remote host.
Local Address Address and port number of the local end of the socket.
Unless the --numeric (-n) option is specified, the socket address is resolved to
its canonical host name (FQDN), and the port number is translated into
the corresponding service name.
Foreign Address Address and port number of the remote end of the socket; analogous to "Local Address."
State The state of the socket.
Since there are no states in raw mode and usually no states used in UDP, this column may be left blank.
Normally this can be one of several values:
ESTABLISHED The socket has an established connection.
SYN_SENT The socket is actively attempting to establish a connection.
SYN_RECV A connection request has been received from the network.
FIN_WAIT1 The socket is closed, and the connection is shutting down.
FIN_WAIT2 Connection is closed, and the socket is waiting for a shutdown from the remote end.
TIME_WAIT The socket is waiting after close to handle packets still in the network.
CLOSE The socket is not being used.
CLOSE_WAIT The remote end has shut down, waiting for the socket to close.
LAST_ACK The remote end has shut down, and the socket is closed. Waiting for acknowledgement.
LISTEN The socket is listening for incoming connections. Such sockets are not included in the output unless you specify the --listening (-l) or --all (-a) option.
CLOSING Both sockets are shut down but we still don't have all our data sent.
UNKNOWN The state of the socket is unknown.
User The username or the user id (UID) of the owner of the socket.
PID/Program name Slash-separated pair of the process id (PID) and process name of the process that owns the socket. --program causes this column to be included. You will also need superuser privileges to see this information on sockets you don't own.
sudo netstat -plnt
List tcp ports that are being listened on, along with the name of each listener's daemon and its PID.
sample output:
sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 3686/mysqld
tcp 0 0 :::443 :::* LISTEN 2218/httpd
tcp 0 0 :::80 :::* LISTEN 2218/httpd
tcp 0 0 :::22 :::* LISTEN 1051/sshd
netstat -ano
List all ports with timer info.
sample output on Linux
Proto Recv-Q Send-Q Local Address Foreign Address State Timer
tcp 0 0 0.0.0.0:199 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:2381 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:2301 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 10.666.3.152:2301 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 111.666.222.1:2301 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:9494 127.0.0.1:32802 ESTABLISHED off (0.00/0/0)
tcp 4 0 55.227.3.555:55118 55.227.3.555:199 CLOSE_WAIT off (0.00/0/0)
tcp 0 0 127.0.0.1:5302 127.0.0.1:32775 ESTABLISHED keepalive (3334.13/0/0)
netstat -a sample output on Linux
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost.localdomain:2301 *:* LISTEN
tcp 0 0 kuk-lol-1-cdc-1:2301 *:* LISTEN
tcp 0 0 111.666.222.1:2301 *:* LISTEN
tcp 0 0 localhost.localdomain:9494 localhost.localdomain:32802 ESTABLISHED
tcp 4 0 kuk-lol-1-cdc-1:55118 kuk-lol-1-cdc-1:smux CLOSE_WAIT
tcp 0 0 localhost.localdom:hacl-cfg localhost.localdomain:32775 ESTABLISHED
Oracle on Linux
netstat -plnt | grep 1521
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:1521 0.0.0.0:* LISTEN -
netstat -ano | grep 1521
tcp 0 0 172.19.242.20:60414 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60419 ESTABLISHED keepalive (2272.24/0/0)
tcp 0 0 172.19.242.20:26383 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:26360 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:26383 ESTABLISHED keepalive (7183.74/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:63903 ESTABLISHED keepalive (3879.65/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60413 ESTABLISHED keepalive (2272.42/0/0)
tcp 0 0 172.19.242.20:65271 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:18177 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:26274 172.19.242.20:1521 TIME_WAIT timewait (14.60/0/0)
tcp 0 0 172.19.242.20:19698 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60421 ESTABLISHED keepalive (2272.27/0/0)
tcp 0 0 172.19.242.20:60259 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
netstat -ano | grep 1521 | grep keep
tcp 0 0 172.19.242.20:1521 172.19.242.20:21059 ESTABLISHED keepalive (3971.96/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:65271 ESTABLISHED keepalive (1977.41/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:17904 ESTABLISHED keepalive (5046.05/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:19687 ESTABLISHED keepalive (3513.32/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:21065 ESTABLISHED keepalive (3972.04/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:25412 ESTABLISHED keepalive (5916.66/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:27742 ESTABLISHED keepalive (7185.41/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:13877 ESTABLISHED keepalive (4141.11/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60429 ESTABLISHED keepalive (1524.39/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:27745 ESTABLISHED keepalive (7185.46/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:63721 ESTABLISHED keepalive (3106.11/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:27743 ESTABLISHED keepalive (7185.43/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60258 ESTABLISHED keepalive (1524.05/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:18177 ESTABLISHED keepalive (5075.28/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:63888 ESTABLISHED keepalive (3130.35/0/0)
iu@lka-mob-1-aps-1:~/workarea>% netstat -ano | grep 1521 | grep keep | wc -l
140
kepalive - The timer column
Reference
netstat -o output
tcp keepalive
Example: keepalive (6176.47/0/0)
The timer column has two fields
<1st field> <2nd field>
The 1st field can have three values:
keepalive - when the keepalive timer is ON for the socket
on - when the retranmission timer is ON for the socket
off - none of the above is ON
The 2nd field has three sub-fields, which are keepalive parameters.
(6176.47/0/0) -> (a/b/c)
a = Timer value (keepalive/retransmission timer, depending on first field)
b = Number of retransmissions that have occurred
c = Number of keepalive probes that have been sent
tcp 0 0 *:hacl-cfg *:* LISTEN
tcp 0 0 localhost.localdomain:2381 *:* LISTENtcp 0 0 localhost.localdomain:2301 *:* LISTEN
tcp 0 0 kuk-lol-1-cdc-1:2301 *:* LISTEN
tcp 0 0 111.666.222.1:2301 *:* LISTEN
tcp 0 0 localhost.localdomain:9494 localhost.localdomain:32802 ESTABLISHED
tcp 4 0 kuk-lol-1-cdc-1:55118 kuk-lol-1-cdc-1:smux CLOSE_WAIT
tcp 0 0 localhost.localdom:hacl-cfg localhost.localdomain:32775 ESTABLISHED
Oracle on Linux
netstat -plnt | grep 1521
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:1521 0.0.0.0:* LISTEN -
netstat -ano | grep 1521
tcp 0 0 172.19.242.20:60414 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60419 ESTABLISHED keepalive (2272.24/0/0)
tcp 0 0 172.19.242.20:26383 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:26360 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:26383 ESTABLISHED keepalive (7183.74/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:63903 ESTABLISHED keepalive (3879.65/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60413 ESTABLISHED keepalive (2272.42/0/0)
tcp 0 0 172.19.242.20:65271 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:18177 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:26274 172.19.242.20:1521 TIME_WAIT timewait (14.60/0/0)
tcp 0 0 172.19.242.20:19698 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60421 ESTABLISHED keepalive (2272.27/0/0)
tcp 0 0 172.19.242.20:60259 172.19.242.20:1521 ESTABLISHED off (0.00/0/0)
netstat -ano | grep 1521 | grep keep
tcp 0 0 172.19.242.20:1521 172.19.242.20:21059 ESTABLISHED keepalive (3971.96/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:65271 ESTABLISHED keepalive (1977.41/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:17904 ESTABLISHED keepalive (5046.05/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:19687 ESTABLISHED keepalive (3513.32/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:21065 ESTABLISHED keepalive (3972.04/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:25412 ESTABLISHED keepalive (5916.66/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:27742 ESTABLISHED keepalive (7185.41/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:13877 ESTABLISHED keepalive (4141.11/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60429 ESTABLISHED keepalive (1524.39/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:27745 ESTABLISHED keepalive (7185.46/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:63721 ESTABLISHED keepalive (3106.11/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:27743 ESTABLISHED keepalive (7185.43/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:60258 ESTABLISHED keepalive (1524.05/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:18177 ESTABLISHED keepalive (5075.28/0/0)
tcp 0 0 172.19.242.20:1521 172.19.242.20:63888 ESTABLISHED keepalive (3130.35/0/0)
iu@lka-mob-1-aps-1:~/workarea>% lsof -i@lka-mob-1-aps-1 | grep 21059
java 30629 iu 165u IPv4 781945850 0t0 TCP lka-mob-1-aps-1:19694->lka-mob-1-aps-1:ncube-lm (ESTABLISHED)
iu@lka-mob-1-aps-1:~/workarea>% lsof -i@lka-mob-1-aps-1 | grep 65271
java 30629 iu 165u IPv4 781945850 0t0 TCP lka-mob-1-aps-1:19694->lka-mob-1-aps-1:ncube-lm (ESTABLISHED)
iu@lka-mob-1-aps-1:~/workarea>% lsof -i@lka-mob-1-aps-1 | grep 17904
java 17501 iu 145u IPv4 781647249 0t0 TCP lka-mob-1-aps-1:60247->lka-mob-1-aps-1:ncube-lm (ESTABLISHED)
iu@lka-mob-1-aps-1:~/workarea>% lsof -i@lka-mob-1-aps-1 | grep 19687
java 47682 iu 137u IPv4 745088409 0t0 TCP lka-mob-1-aps-1:65271->lka-mob-1-aps-1:ncube-lm (ESTABLISHED)
iu@lka-mob-1-aps-1:~/workarea>% netstat -ano | grep 1521 | grep keep | wc -l
140
In this case - there is an java executable, running many processes, that are constantly connected to Oracle.
kepalive - The timer column
Reference
netstat -o output
tcp keepalive
Example: keepalive (6176.47/0/0)
The timer column has two fields
<1st field> <2nd field>
The 1st field can have three values:
keepalive - when the keepalive timer is ON for the socket
on - when the retranmission timer is ON for the socket
off - none of the above is ON
The 2nd field has three sub-fields, which are keepalive parameters.
(6176.47/0/0) -> (a/b/c)
a = Timer value (keepalive/retransmission timer, depending on first field)
b = Number of retransmissions that have occurred
c = Number of keepalive probes that have been sent
These paranmeters are configured in the file /etc/sysctl.conf.
Three parameters related to keepalive:
net.ipv4.tcp_keepalive_time
net.ipv4.tcp_keepalive_intvl
net.ipv4.tcp_keepalive_probes
The net.ipv4.tcp_keepalive_time parameter is the time before the first keepalive packet is sent out.
As long as there is TCP/IP socket communications going on, no keepalive packets are needed, but if the amount of time in seconds specified in net.ipv4.tcp_keepalive_time passes without any communication on a TCP/IP socket connection, then the Linux OS will begin sending keepalive packets.
Once keepalive packets begin being sent out, they will be sent every net.ipv4.tcp_keepalive_intvl time (in seconds).
Keepalive packets are a two-way exchange.
When one device sends a keepalive packet to another, the receiving device sends a quick acknowledgement packet back.
This way, both devices know the communication link between them is OK.
If the device sending the keepalive packet does not get a response back, it sends another keepalive packet after the net.ipv4.tcp_keepalive_intvl passes.
After enough keepalive packets are sent and no response is received, the sending device will assume the link is down, close the socket, and try to re-establish communications.
The number of keepalive packets sent before the device will reset if it does not get a response is configured in the net.ipv4.tcp_keepalive_probes parameter.
lsof
lsof Reference
lsof Reference II
The -i option without further qualification lists all open Internet socket files.
To refine the search, add network name, address, protocol name, service name or port number.
For example:
To know what process has a connection open to or from host my.server.com:
lsof -i@my.server.com
To limit results only to TCP (or UDP)/ specific port number / service name, you can add
those discriminators to the -i information:
lsof -iTCP@my.server.com:ftp-data
Connecting lsof with netstat
For example, this is the netstat output
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp 0 0 vic.1023 ipscgate.login ESTABLISHED
What process is connected to service name login on ipscgate?
Option A. - Using service name.
Use lsof's -i option:
>lsof -iTCP@ipscgate:login
COMMAND PID USER FD TYPE DEVICE SIZE/OFF INODE NAME
rlogin 25023 abe 3u inet 0x10144168 0t184 TCP vic.cc:1023-> ipscgate.cc:login
Option B. - using PCB.
Use netstat -A option:
Notice the 0x10144168 in the DEVICE column of the lsof output.
That's the protocol control block (PCB) address.
Many netstat applications will display it with -A option:
netstat -A
PCB Proto Recv-Q Send-Q Local Address Foreign Address (state)
10144168 tcp 0 0 vic.1023 ipscgate.login ESTABLISHED
Now find this process using lsof:
lsof -i | grep 10144168
rlogin 25023 abe 3u inet 0x10144168 0t184 TCP vic.cc:1023-> ipscgate.cc:login
More lsof examples
List all network connections
lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
avahi-dae 515 avahi 13u IPv4 6848 0t0 UDP *:mdns
avahi-dae 515 avahi 16u IPv6 6851 0t0 UDP *:52060
cupsd 1075 root 5u IPv6 22512 0t0 TCP ip6-localhost:ipp (LISTEN)
List all network files in use by a specific process
lsof -i -a -p 234
You can also use the following
lsof -i -a -c ssh
The above command will list the network files opened by the processes starting with ssh.
List processes which are listening on a particular port
lsof -i :25
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
exim4 2541 Debian-exim 3u IPv4 8677 TCP localhost:smtp (LISTEN)
List all TCP or UDP connections
lsof -i tcp;
lsof -i udp;
How to identify what is the process that is holding the port?
sudo lsof -ithose discriminators to the -i information:
lsof -iTCP@my.server.com:ftp-data
For example, this is the netstat output
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp 0 0 vic.1023 ipscgate.login ESTABLISHED
What process is connected to service name login on ipscgate?
Option A. - Using service name.
Use lsof's -i option:
>lsof -iTCP@ipscgate:login
COMMAND PID USER FD TYPE DEVICE SIZE/OFF INODE NAME
rlogin 25023 abe 3u inet 0x10144168 0t184 TCP vic.cc:1023-> ipscgate.cc:login
Option B. - using PCB.
Use netstat -A option:
Notice the 0x10144168 in the DEVICE column of the lsof output.
That's the protocol control block (PCB) address.
Many netstat applications will display it with -A option:
netstat -A
PCB Proto Recv-Q Send-Q Local Address Foreign Address (state)
10144168 tcp 0 0 vic.1023 ipscgate.login ESTABLISHED
Now find this process using lsof:
lsof -i | grep 10144168
rlogin 25023 abe 3u inet 0x10144168 0t184 TCP vic.cc:1023-> ipscgate.cc:login
More lsof examples
List all network connections
lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
avahi-dae 515 avahi 13u IPv4 6848 0t0 UDP *:mdns
avahi-dae 515 avahi 16u IPv6 6851 0t0 UDP *:52060
cupsd 1075 root 5u IPv6 22512 0t0 TCP ip6-localhost:ipp (LISTEN)
List all network files in use by a specific process
lsof -i -a -p 234
You can also use the following
lsof -i -a -c ssh
The above command will list the network files opened by the processes starting with ssh.
List processes which are listening on a particular port
lsof -i :25
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
exim4 2541 Debian-exim 3u IPv4 8677 TCP localhost:smtp (LISTEN)
List all TCP or UDP connections
lsof -i tcp;
lsof -i udp;
How to identify what is the process that is holding the port?
sudo netstat -lptu
sudo netstat -tulpn
for example:
:~>% lsof -i | grep 43230
java
13927 user 171u
IPv4 4074139617 0t0 TCP
server-aps-1:43230->server-ora-1:proc (ESTABLISHED)
=============================
ifconfig
=============================
ifconfig is used to view the current network configuration
>% ifconfig
eth0 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:90
inet addr:192.168.69.216 Bcast:192.168.69.255 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8387572303 errors:0 dropped:0 overruns:0 frame:0
TX packets:8615008840 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7334475993483 (6.6 TiB) TX bytes:5923454851965 (5.3 TiB)
eth0:0 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:90
inet addr:192.168.69.231 Bcast:192.168.69.255 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:1 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:90
inet addr:192.168.69.223 Bcast:192.168.69.255 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:3 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:90
inet addr:192.168.69.229 Bcast:192.168.69.255 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:5 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:90
inet addr:192.168.69.222 Bcast:192.168.69.255 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:6 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:90
inet addr:192.168.69.206 Bcast:192.168.69.255 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:94
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:34480079 errors:0 dropped:0 overruns:0 frame:0
TX packets:34936034 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2594272693 (2.4 GiB) TX bytes:3063799522 (2.8 GiB)
eth3 Link encap:Ethernet HWaddr 28:92:4A:2F:6F:96
inet addr:192.168.20.20 Bcast:192.168.20.127 Mask:255.255.255.128
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:251475120 errors:0 dropped:0 overruns:0 frame:0
TX packets:34910610 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:249823838895 (232.6 GiB) TX bytes:3060739778 (2.8 GiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1861721437 errors:0 dropped:0 overruns:0 frame:0
TX packets:1861721437 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1339967244643 (1.2 TiB) TX bytes:1339967244643 (1.2 TiB)
=============================
tcpdump
=============================
Useful Options
-A Print each packet (minus its link level header) in ASCII
-C count Exit after receiving count packets.
-d Dump the compiled packet-matching code in a human readable form to standard output and stop.
-i By default, all the packets flowing through all the interfaces would be captured. With -i option only traffic from particular ethernet interface.would be captured.
tcpdump -i eth1
-l Make stdout line buffered.
Useful if you want to see the data while capturing it.
For Example: tcpdump -l | tee dat or tcpdump -l > dat & tail -f dat
-n Leave the IP address.
Don't convert addresses (i.e., host addresses, port numbers, etc.) to names.
Don't convert addresses (i.e., host addresses, port numbers, etc.) to names.
-O --no-optimize Do not run the packet-matching code optimizer.
This is useful only if you suspect a bug in the optimizer.
-Q Quick (quiet?) output. Print less protocol information so output lines are shorter.
-v, -vv, -vvv When parsing and printing, produce more (and more) verbose output.
-w - Write output to a file
-w - Write output to a file
Examples:
tcpdump host some_host
To print all packets arriving at or departing from some_host
tcpdump host some_host and \( hostA or hostB \)
To print traffic between some_host and either hostA or hostB
tcpdump ip host my_host and not hostA
To print all IP packets between my_host and any host except hostA
tcpdump net ucb-ether
To print all traffic between local hosts and hosts at ucb-ether network
Useful tcpdump commands
tcpdump -D
Display all Available Interfaces.
Capture traffic only to eth0
With -c option you can specify the number of packets to capture. -
In this example, capture only 10 packets.
In this example, capture only 10 packets.
Without -c tcpdump would captute packets until you cancel the tcpdump command.
tcpdump -w some_file.pcap -i eth0
Write output to a file some_file.pcap.
tcpdump -r some_file.pcap
Read generated file some_file.pcap.
Read generated file some_file.pcap.
tcpdump -n -i eth0
In the trace, would be displayed IP address.
In the trace, would be displayed IP address.
Without conversion to DNS.
tcpdump -i eth0 tcp
Capture only TCP packets.
Capture only TCP packets.
tcpdump -i eth0 port 22
Capture packets from a particular port.
tcpdump -i eth0 src 10.20.30.40
Capture packets from a source IP.
tcpdump -i eth0 dst 10.10.50.50
Capture packets from a destination IP.
Other tcpdump commands
tcpdump -n -tttt -i eth0
Capture packets with readable timestamp
tcpdump -w g_1024.pcap greater 1024
Read packets longer than 1024 bytes and log data to file g_1024.pcap
tcpdump -w g_1024.pcap less 1024
Read packets less than 1024 bytes and log data to file g_1024.pcap
tcpdump -i eth0 <protocol>
for example tcpdump -i eth0 arp
Receive only the packets of a specific protocol type
Protocol options: fddi, tr, wlan, ip, ip6, arp, rarp, decnet, tcp and udp.
tcpdump -v or tcpdump -vv or tcpdump -vvv
Receive verbose output
tcpdump -w xpackets.pcap -i eth0 dst 10.181.140.216 and port 22
Receive packets with destination IP 10.181.140.216 and port 22
tcpdump -w comm.pcap -i eth0 src xxx.xxx.xxx.001 and port 22 and dst xxx.xxx.xxx.002 and port 22
Captures all ssh packets flowing between the source and destination addresses.
File comm.pcap can be analyzed using any network protocol analyzer tool.
=============================
mii-show
=============================
mii-show is used to see current open networks status.
>% mii-show
Detected 4 Interfaces
eth0 is { elink=yes speed=100Mb autoneg=on name=eth0 duplex=Full }
eth1 is { elink=no speed=Unknown autoneg=on name=eth1 duplex=Half }
eth2 is { elink=yes speed=100Mb autoneg=on name=eth2 duplex=Full }
eth3 is { elink=yes speed=100Mb autoneg=on name=eth3 duplex=Full }
MTU and Packet size
=============================
MTU: Maximum Transmission Unit
The default size for Packet size is 1400 bytes.
How to set the correct MTU size (Linksys Reference)
ping [url / local server or IP address] –f –l xxxx
Where xxxx is the packet size (value) which you will test.
For example:
>ping 10.16.39.122 -f -l 1400
Pinging 10.16.39.122 with 1400 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Ping statistics for 10.16.39.122:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
Here are the results that you may get after doing the ping test:
• Four (4) replies received: This means that the packet size entered is either within or the actual MTU size used within your network.
• Destination net unreachable: This means that there was no path or route to the destination or the address.
• Request Timed Out: This means that within the default wait time period (1 second), there was no response.
• Packet needs to be fragmented but DF set: This means that the packet size you entered is too high for your MTU value.
• Bad parameter –f: This means that you have typed the command incorrectly.
Handle case for Packet needs to be fragmented but DF set
Try with packet size of 1248:
>ping 10.16.39.122 -f -l 1248
Pinging 10.16.39.122 with 1248 bytes of data:
Reply from 10.16.39.122: bytes=1248 time=312ms TTL=58
Reply from 10.16.39.122: bytes=1248 time=312ms TTL=58
Reply from 10.16.39.122: bytes=1248 time=338ms TTL=58
Reply from 10.16.39.122: bytes=1248 time=339ms TTL=58
Ping statistics for 10.16.39.122:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 312ms, Maximum = 339ms, Average = 325ms
The -l flag would set the MTU size in current cmd window.
To change the settings in general, need to access the Router, and per server enter MTU value..
=============================
ORA-03113 and MTU
=============================
To see the current MTU settings:
netsh interface ip show interface
The pre-requirement is to have netsh service running.
=============================
ORA-03113 and MTU
=============================
ORA-03113: end-of-file on communication channel.
Consider this common scenario: Oracle Client and Oracle Server communicate via Network.
It may happen the MTU on the Client router is set to too high value.
In this case, the client requests from the remote database are not received back, and error ORA-03113 is thrown:
The solution would be to lower the MTU to the default value of 1400, or even lower.
Subscribe to:
Posts (Atom)