SAP Training 2.doc
SAP Training Canton Feb/Mar 2001
Three Tier Client/Server Model 3
Overview 3
Flow 4
Application Server 4
Database Server 5
Performance Troubleshooting 6
Performance Bottlenecks 6
Performance problems and troubleshooting techniques 6
Dialog response 6
Bottom-Up Troubleshooting 7
Unix 7
Database 10
Basis Layer 14
Dialog (per process) 17
Batch (per process) 17
Top-Down Troubleshooting 19
Acute Problems 19
Qualitative Data 19
Quantitative Data 19
Transaction overview 23
Pro: Distribution of workload
Con: Lots of moving parts
Ø Check with Central Instance which server is least loaded
Ø Servers check with each other, one responds to CI
Ø Make a straight connection to that server
Ø Dialog = Interactive communication (client/server communication)
Ø Update = Communication with database
Ø Spool = Formatting spool requests and passing it on to the host.
Ø Reebok Environment » 330 work processes » 330 Informix logons (threads)
Ø Chronic/Widespread – Bottom-up
Ø Acute – Top-down
Ø Measured from Dispatcher to Dispatcher
Ø CPU (ABAP time on the Apps server)
Ø DB (CPU and IO on USUNX07)
Ø Roll (context swapping of user data)
Ø Wait
Ø Load/Gen (compiling of ABAP code)
Ø Processing
Ø OSS Collector
v Basic, generalized data of the Operating System
v ST06 runs OSS Collector for the instance you’re logged on to
v Collector runs hourly
v Has an option to compare all servers
v On the USUNX07
ü Idle time not less then 20%
ü Not to much swap action
ü Paging must be < 20% of main memory
ü Disk utilization should be < 50%
ü Disk queue should be low
ü Disk wait time should be < 50/100 ms
v On an Apps server
ü Dialog instances have low CPU usage
ü Batch instances use more CPU (max should be < 20%)
Ø Perview (pv)
Ø Glance plus (gpm)
v Disk 80%
v CPU 80%
v MEM 90%
v Swap 20%/main mem (no reactivations or deactivations)
Ø SAR (sar –d 5 5) – Will show hot disks
HP-UX usunx07 B.11.00 U 9000/800 03/05/01
08:01:43 device %busy avque r+w/s blks/s avwait avserv
Average c0t6d0 10.32 0.56 9 145 5.03 27.44
Average c3t10d0 6.40 0.56 7 137 4.88 19.55
Average c13t0d0 3.12 0.50 9 38 4.69 4.18
Average c9t0d0 2.56 0.50 9 40 5.31 3.15
Average c15t0d0 3.16 0.50 8 34 5.46 3.74
Average c19t0d0 3.04 0.50 9 38 5.00 3.25
Average c17t0d5 23.99 0.50 76 704 5.08 4.23
Average c7t0d5 27.15 0.50 76 714 5.12 4.79
Average c11t0d5 25.63 0.50 76 706 5.14 4.73
Average c5t0d5 25.75 0.50 75 704 5.05 4.81
Average c15t0d1 0.24 0.50 0 2 4.31 8.36
Average c19t0d1 0.16 0.50 0 1 6.34 4.84
Average c17t0d4 0.88 0.50 2 7 5.61 4.75
Average c7t0d4 1.12 0.50 2 9 5.06 4.93
Average c11t0d4 0.96 0.50 2 7 4.96 4.88
Average c5t0d4 0.56 0.50 1 4 5.87 5.15
Average c9t0d2 0.04 0.50 0 0 3.01 8.41
Average c15t0d2 0.04 0.50 0 0 4.22 5.97
Average c19t0d2 0.04 0.50 0 0 3.59 8.37
Average c17t0d3 0.12 0.50 0 2 4.57 3.71
Average c7t0d3 0.36 0.50 1 2 5.24 6.74
Average c11t0d3 0.20 0.50 0 1 4.34 8.88
Average c5t0d3 0.16 0.50 0 2 5.49 4.61
Average c13t0d3 1.08 0.50 4 24 5.81 3.25
Average c9t0d3 1.52 0.50 12 70 5.09 1.57
Average c15t0d3 4.52 0.50 9 38 5.20 4.88
Average c19t0d3 0.44 0.50 1 4 4.69 5.39
Average c17t0d2 0.08 0.50 0 0 6.13 7.12
Average c7t0d2 0.52 0.50 2 6 5.15 2.50
Average c13t0d4 4.68 0.55 25 746 5.54 4.69
Average c17t0d1 6.00 0.50 35 141 5.00 1.68
Average c5t0d1 3.64 0.50 15 60 4.89 2.26
Average c13t0d5 0.52 0.50 1 4 6.11 5.00
Average c9t0d5 0.84 0.63 2 11 6.39 8.48
Average c15t0d5 0.04 0.50 0 0 3.72 7.95
Average c19t0d5 7.76 0.50 37 149 4.96 1.92
Ø Dmesg – Shows system messages
btlan6: Initializing 10/100BASE-TX card at 0/0/0....
HP PCI Mux Device Driver, FILESET PCI-MUX-KRN Version B.11.00.02
Copyright (c) 1997 Hewlett-Packard Company
pci_mux0: Driver Enabled for board 1 at HW path (3/1/0).
fddi4: INITIALIZING: 0150 PCI FDDI card in slot 4/1/0
fddi4: INITIALIZING: 0150 PCI FDDI card in slot 5/1/0
btlan6: Initializing 10/100BASE-TX card at 7/2/0....
System Console is on SPP DUART0 Interface
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 4194304
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0 - major is 31, minor is 0x6000; start = 207712, size = 2097152
Starting the STREAMS daemons-phase 1
Create STCP device files
Starting the STREAMS daemons-phase 2
B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 8384512 Kbytes, lockable: 7267600 Kbytes, available: 7412892 Kbytes
Ø /var/adm/syslog/syslog.log – System log
Mar 5 05:35:18 usunx07 ftpd[26592]: FTP session closed
Mar 5 05:35:20 usunx07 ftpd[26476]: FTP session closed
Mar 5 05:45:11 usunx07 ftpd[27540]: FTP LOGIN FROM rbkusibmmf.reebok.com [10.101.226.250], prdftp
Mar 5 05:45:12 usunx07 ftpd[27540]: FTP session closed
Mar 5 06:10:00 usunx07 : su : + tty?? root-informix
Mar 5 06:10:06 usunx07 ftpd[29417]: FTP LOGIN FROM usunx24.reebok.com [10.101.202.124], mrcadm
Mar 5 06:10:06 usunx07 ftpd[29417]: FTP session closed
Mar 5 06:11:56 usunx07 ftpd[26470]: FTP session closed
Mar 5 06:20:09 usunx07 ftpd[397]: FTP LOGIN FROM usunx24.reebok.com [10.101.202.124], mrcadm
Mar 5 06:20:09 usunx07 ftpd[397]: FTP session closed
Mar 5 06:30:10 usunx07 ftpd[974]: FTP LOGIN FROM usunx24.reebok.com [10.101.202.124], mrcadm
Mar 5 06:30:10 usunx07 ftpd[974]: FTP session closed
Mar 5 07:00:15 usunx07 ftpd[2858]: FTP LOGIN FROM rbkusibmmf.reebok.com [10.101.226.250], prdftp
Mar 5 07:00:22 usunx07 ftpd[2858]: FTP session closed
Mar 5 07:10:00 usunx07 : su : + tty?? root-informix
Mar 5 07:10:05 usunx07 ftpd[3461]: FTP LOGIN FROM usunx24.reebok.com [10.101.202.124], mrcadm
Mar 5 07:10:06 usunx07 ftpd[3461]: FTP session closed
Mar 5 07:16:57 usunx07 : su : + td bderzhav-informix
Mar 5 07:20:07 usunx07 ftpd[4909]: FTP LOGIN FROM usunx24.reebok.com [10.101.202.124], mrcadm
Mar 5 07:20:08 usunx07 ftpd[4909]: FTP session closed
Mar 5 08:01:36 usunx07 : su : + tc tvlot-informix
Ø ST04
v Unfortunatly this transaction is pretty useless because the numbers are squed due to 64/32 bit limitations
Ø Onstat –p
v Read quality > 95% (% cached reads)
v Write quality > 82% (% cached writes)
v Buffers shouldn’t be to big because the flush point will take to long (see checkpoint duration in the logs). During a flush point the system halts.
Informix Dynamic Server Version 7.31.FC7X8 -- On-Line -- Up 1 days 15:47:41 -- 3878280 Kbytes
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
503518339 555131625 7922394078 93.64 13790707 12327788 63066879 78.13
isamtot open start read write rewrite delete commit rollbk
6517994423 31894184 148483570 5618257153 4541512 1474594 4032022 533747 52
gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0
ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 10391 230004.69 173810.34 403 806
bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
49301821 3608 3967053865 0 0 3039 1409042 81640
ixda-RA idx-RA da-RA RA-pgsused lchwaits
178228743 4854740 1054788 183924904 2789177
Ø Onstat –m
v Last 20 lines of the informix message log
Informix Dynamic Server Version 7.31.FC7X8 -- On-Line -- Up 1 days 15:48:12 -- 3878280 Kbytes
Message Log File: /informix/PRD/online.usunx07.prd.log
05:56:16 Checkpoint Completed: duration was 2 seconds.
06:03:19 Logical Log 101488 Complete.
06:03:20 Logical Log 101488 - Backup Started
06:05:33 Logical Log 101488 - Backup Completed
06:06:27 Checkpoint Completed: duration was 2 seconds.
06:16:37 Checkpoint Completed: duration was 2 seconds.
06:26:47 Checkpoint Completed: duration was 1 seconds.
06:30:56 Logical Log 101489 Complete.
06:30:58 Logical Log 101489 - Backup Started
06:33:33 Logical Log 101489 - Backup Completed
06:36:55 Checkpoint Completed: duration was 2 seconds.
06:47:05 Checkpoint Completed: duration was 1 seconds.
06:57:12 Checkpoint Completed: duration was 1 seconds.
07:07:22 Checkpoint Completed: duration was 1 seconds.
07:17:29 Checkpoint Completed: duration was 1 seconds.
07:27:36 Checkpoint Completed: duration was 1 seconds.
07:37:45 Checkpoint Completed: duration was 1 seconds.
07:47:53 Checkpoint Completed: duration was 1 seconds.
07:58:01 Checkpoint Completed: duration was 0 seconds.
08:08:09 Checkpoint Completed: duration was 1 seconds.
Ø ...
wydra66