IntelliMagic Vision for z/OS Data Collection Quick Start Guide

Please watch this short video first to understand the data collection process, Data Collection Video.

This webpage serves as an overview of the recommended procedures for gaining the most out of your IntelliMagic Vision Performance Assessment. IntelliMagic will perform this analysis based on data created by IBM’s Resource Measurement Facility (RMF) and collected through the System Management Facility (SMF) data sets. In addition to the SMF/RMF data, the analysis will rely on background and configuration information provided to IntelliMagic, through various system commands or other data sources detailed in this document. We process your data by sysplex and by day, so all related data needs to be separated to this level of granularity (or lower). For more information about SMF data and interval lengths, please refer to Appendix K: SMF Intervals. For Near Real Time processing considerations, please refer to Appendix H: Near Real Time (NRT) considerations.

The following steps summarize the process detailed in this document:

• Frequently Asked Questions
• Identify Required Data
• Extract
• Sort
• RMFPack and SMF2ZRF
• FTP

The following tables provide guidance on the required data for each module. For a description of each module please refer to Appendix A.

Following extraction, the data must be sorted into date/time stamp order (just as with a standard SMF/RMF report). Our sort process will split the raw data into SMF/RMF (SMF7x), SMF type 30, 42, 100, 101, 102, 110, 115, 116, 119, 120 and tape files. If you have additional optional SMF data (i.e. 105, 124, 206) we recommend you direct that to the SMFOTH file. All data from a single Sysplex should be sorted into a single set of files. If there are multiple Sysplexes, each should have their own set of files.

We provide sample JCL members for the sort in the RMFPACK.JCL.XMIT. See the following member:

• A2SORT provides the JCL for sorting the records for Systems, Db2, CICS, TCP/IP, WebSphere, MQ, Disk and Tape modules

At this point in the process you have extracted, sorted and used the RMFPACK utility to create several compressed files.

Now you need to use FTP to transfer these files to IntelliMagic using the FTP credentials you were provided in the IntelliMagic data collection meeting.

In order to provide the highest possible security when transferring data to IntelliMagic, we recommend the usage of a secure file transfer mechanism (FTPS or SFTP).

In order to ease the processing of the data it is important to make sure you have unique filenames. When transferring data to IntelliMagic please use filenames in the format of:

Date.HHMM.Company.Sysplex.LPAR.SMFRecord.ZRF
(where: Date = YYMMDD; HHMM = 2300 if not NRT or ENDing hour if NRT; Company (8 chars or less); Sysplex (8 chars or less); LPAR = optional LPAR name if sending by LPAR; Record (7X, 30, 42, tape, 115, 116, 119, 120, 100, 101, 102, 1101, 1102, or OTHer))

Where the symbolics (#DATE etc) are replaced by current values. A few examples of properly formatted file names are in the table below:

We provide sample JCL members for the FTP in the RMFPACK.JCL.XMIT. See the following members:

• A4FTP invokes a REXX EXEC to issue the FTP commands for sending the records for Systems, Db2, CICS, TCP/IP, WebSphere, MQ, Disk and Tape modules.  This REXX EXEC will retry the FTP if it fails for any reason.  Will also use FTPS by uncommenting SYSFTPD and STDENV DD statements.
• A4FTPFTP provides the JCL for FTPing the records for Systems, Db2, CICS, TCP/IP, WebSphere, MQ, Disk and Tape modules.  Use this FTP job if first sending to a local FTP server.
• A4FTPS uses FTPS to transfer files.  It is similar to A4FTP but is a secure method of transferring data.
• A4SFTP uses SFTP to transfer files.

Transferring the data to IntelliMagic requires the ability to send the data from your z/OS environment to our public FTPS/SFTP server. There are two mechanisms for transferring the data directly from your z/OS environment to our FTPS/SFTP server. The preferred method is the secure method. This will be referred to as the z/OS Secure Option and uses strong encryption facilitated by Public key certificate installation on z/OS. Details on how to configure this are available in Appendix F: z/OS Secure Option.

If security requirements within your environment preclude you from directly using FTP from your z/OS environment to our server, then you will need to investigate the Proxy FTP option below or contact your network security team to identify the appropriate way to transfer large files from your environment to our FTP/SFTP server. For an example of the Proxy approach please see Appendix G: Proxy FTP.

Firewall Considerations

It may be necessary to open a port or a range of ports to transfer data to IntelliMagic’s FTP Server(s). This is also dependant on the method of transfer. If you are unable to initiate a successful transfer, you will want to verify that the following IP addresses and ports are whitelisted for the method of transfer you are using:

Appendices: Links to each Appendix

Appendix A: Module Definitions
Appendix B: Gathering BVIR data – IBM TS7700
Appendix C: Gathering DCOLLECT data – DISK Component
Appendix D: GEOXPARM
Appendix E: Gathering Additional Tape Management Configuration Data
Appendix F: z/OS Secure Option
Appendix G: Proxy FTP
Appendix H: Near Real Time (NRT) considerations
Appendix I: Help getting dates added to the FTP files
Appendix J: Gathering the CICS Data Dictionary and Statistics
Appendix K: SMF Intervals
Appendix L: Gathering Cache and ESS Statistics
Appendix M: Configuring TCP/IP for SMF 119.12 zERT Records
Appendix N: Configuring FTPS client for AT-TLS
Appendix O: Cloud Database Retention Settings
Appendix P: Configuring CICS and TCP/IP zERT Groups

Appendix A: Module Definitions

The purpose of this appendix is to define the modules that are offered. IntelliMagic recommends that SMF record types for all modules be collected and transmitted so that they can be analyzed. The data that is required for each module can be found in Identify Required Data.

z/OS Systems Module: The purpose of this module is to review the z/OS systems ecosystem including WLM, Coupling Facility, XCF, Physical and Virtual Storage to determine if there are any availability concerns.

Db2 Module: The purpose of this module is to review the Db2 subsystems and enables performance analysts to more effectively and efficiently manage and optimize their z/OS Db2 environment.

CICS Module: The purpose of this module is to review the CICS regions and enables performance analysts to more effectively and efficiently manage and optimize their z/OS CICS transactions.

MQ Module: The purpose of this module is to review MQ enterprise messaging and provide the kind of automated intelligence needed to effectively manage today’s far-reaching MQ environments.

WebSphere Module: The purpose of this module is to review the WebSphere Application Server for z/OS transaction logging records (SMF 120 subtype 9), enabling you to track transaction response time and resource consumption.

MLC Assessment: The MLC assessment portion of the z/OS Systems module attempts to determine if IBM Monthly License Charge (MLC) software costs can be reduced through various optimizations.

TCP/IP & zERT Module: The zERT portion of the z/OS Systems module aids with the difficult task of z/OS security policy compliance.

z/OS Disk & Replication Module: The  DISK module reviews the I/O ecosystem including the I/O subsystem, storage system(s), replication and FICON director components and determine if there are any performance, capacity or availability concerns.

z/OS TAPE Module: The z/OS Tape module reviews the Tape ecosystem including the logical devices, virtual tape system, physical tape system and the virtual tape inventory through the Tape Management Catalog in order to identify performance, capacity and availability concerns.

z/OS Connect Module: The  z/OS Connect module enables performance analysts to manage and optimize their mainframe API environment, and to profile the requests coming into your system for essential management reporting and resource planning.

Appendix J: Gathering the CICS Data Dictionary and Statistics

The CICS data dictionary is used to customize the CICS 110.1 records (usually done to reduce the volume of data). If that customization is implemented at your site, the data dictionary record is required to process the 110 records. There are two ways to include the CICS data dictionary with your CICS 110.1 records.

1. This can be created using the IBM DFHMNDUP job. Sample JCL provided below:

//SMFMNDUP JOB (accounting information),CLASS=A,
// MSGCLASS=A
//****************************************************************
//* Step 1 – Create new dictionary record and output to SYSUT4
//****************************************************************
//MNDUP EXEC PGM=DFHMNDUP
//STEPLIB DD DSN=CICSTS54.CICS.SDFHLOAD,DISP=SHR
// DD DSN=mct.table.loadlib,DISP=SHR
//SYSUT4 DD DSN=dict.record.dsn,DISP=(NEW,CATLG),
// UNIT=SYSDA,SPACE=(TRK,(1,1))
//SYSPRINT DD SYSOUT=A
//SYSUDUMP DD SYSOUT=A
//SYSIN DD *
MCT=NO
SYSID=PRD1
GAPPLID=AOR98M12
DATE=2022158
TIME=090000
/*
//****************************************************************
//* Step 2 – Run RMFPACK on data set with dictionary record from
//* step 1 (same process as used for previous SMF data)
//****************************************************************

//****************************************************************
//* Step 3 – FTP RMFPACK output from step 2 should be named
//* DYYYYMMDD.$SYSPLEXNAME.CICS.DATA.DICTIONARY.ZRF
//* If you have multiple data dictionaries per SYSPLEX please
//* let us know.
//****************************************************************

• First line of STEPLIB concatenation: specify CICS library containing the DFHMNDUP program.
• Second line of STEPLIB concatenation: specify library from the DFHRPL concatenation that contains the modified monitoring control table (MCT) used for CICS region(s) that are related to the 110 data
• SYSUT4 = specifies a data set to hold the dictionary record. Specify the DISP parameter according to whether the data set already exists, or a new one is to be created and cataloged.
• MCT= specifies the suffix of the monitoring control table (MCT) used in the CICS run for which you are analyzing performance data. If your CICS region ran with the system initialization parameter MCT=NO (which results in a default MCT dynamically created by CICS monitoring domain) you should specify MCT=NO otherwise specify the suffix of the DFHMCT member.
• SYSID=xxxx specifies the system identifier of the MVS system that owns the SMF data sets.
• GAPPLID=name specifies the APPLID specified on either the APPLID system initialization parameter, or the generic APPLID in an XRF environment for which you are analyzing performance data.
• DATE=yyyyddd specifies the Julian date to be included in the dictionary record.
• TIME=hhmmss specifies the time in hours minutes seconds in the dictionary record.

2. Gather the CICS data from a region that was restarted

CICS region startup processing automatically writes the CICS dictionary record in the SMF recording process so if 110 records are being cut by SMF, the CICS dictionary will be written out when a CICS region is started.

A day of data collected when CICS region maintenance normally occurs is not typically of interest for performance at peak processing times; however, this collection period would include a CICS restart. It is likely the same dictionary is used for many or all regions within a sysplex, but this should be confirmed to ensure a unique dictionary is applied for the regions in scope for the SMF data collected.

Gathering the CICS Statistics

The CICS Statistics are recorded in the CICS 110.2 records.  These are required statistic types that must be included in your CICS 110.2 records.

CICS STATISTICS SUMMARY

Appendix K: SMF Intervals

Internal processing in IntelliMagic Vision is performed on a Sysplex Boundary. We want the SMF data from all LPARS in a Sysplex, and if multiple Sysplexes attach to the same hardware, then we want these Sysplexes together in the same interest group. By processing the data in this manner, an interest group will provide an accurate representation of the hardware’s perspective of activity and allow an evaluation of whether this activity is below, equal to, or above the hardware’s capability. It’s also true that the shorter the interval, the more accurate the data will be in showing peaks and lulls. The shortest interval you can define is 1 minute. This would typically be the average of 60 samples (1 cycle per second). It’s always a balancing act between the accuracy of the data and the size/cost of storing and processing the data.

In IntelliMagic Vision we also bridge across SMF record types. Accomplishing this feat requires that the SMF/RMF or SMF/CMF data sources are coordinated. There are two variables to manage this.

1) Interval Length (generally 15-30 minutes)
2) Start of the interval (often the top of the hour)

IntelliMagic Vision will throw out data from an LPAR when the data is not recorded in the same interval as all the other LPARs. To handle such variation would impact the reporting accuracy of the Sysplex or consolidated Hardware views. There are overrides available in Reduce to set the interval length, so your options are override or correct. It is possible to override/normalize to a higher interval length, but not a shorter one.

You may see message
RMF0109E 12 System D001 has an interval length of 1800 s, cannot be normalized to standard interval length of 900 seconds. Records will be dropped.

If you want to know how SMF is recorded, issue the operator command ‘D SMF’ or examine the active SMFPRMxx parmlib member. Pay attention to the settings of INTVAL and SYNCVAL parameters (specified in minutes).

RMF

If you want to know how RMF is recorded, issue the operator command ‘F RMF,D ZZ’ or examine the active ERBRMFxx Parmlib member. The Settings of SYNC and INTERVAL are of interest.

If you specify SYNC(SMF) then the INTERVAL is ignored. SMF Values for INTVAL and SYNCVAL are used.
If you specify SYNC(RMF, mm) then INTERVAL is used and recording starts at mm.
NOSYNC says INTERVAL is used (and I think the default interval start value is 00).

To have RMF use the same settings as SMF,  specify ERBRMFxx value SYNC(SMF). That way, if you change the SMF settings, then RMF will follow.

CMF

Some customers use CMF from BMC to provide their RMF type support. In this situation, issue the operator command ‘F MNVZPAS,DC=STATUS’.
This will produce a series of console messages…
BBDDA101I ———- Regular Interval Information ———-
BBDDA101I Base Cycle Time (in 1/100s) ……: 100
BBDDA101I Record Interval Time (in 1/100s) .: 90000
BBDDA101I Number of Cycles/Interval ……..: 900 (100%)
BBDDA101I #of Cycles in current Interval …: 89 ( 9%)
BBDDA101I Total #of Cycles ……………..: 5620394
BBDDA101I Total #of Regular Intervals ……: 6277

Pay particular attention to the number of cycles per interval. In this case it’s 900 seconds which equates to 15 minutes. CMF parameters that control Interval length and the start of the interval are:

REPORT CPM

,INTERVAL={30|nn|HOUR|QTR|HALF|SMF}

,SMF=YES

,SYNCH={00|nn|HOUR|QTR|HALF|SMF}

To have CMF use the same settings as SMF, in CMF specify SMF=YES,SYNC=SMF.  That way, if you change the SMF settings, then CMF will follow.

SMF Interval Data

The default SMF interval is 30 minutes and some customers will balk at using 15 minutes because they think it will double their SMF recording. This is likely not the case. Most SMF record types are written because an event occurred, not because a timer expired. Interval recording impacts type 30 Subtype 2 and type 42 and 7x data only. So yes, there’s an increase, but your daily SMF is not going to double. Depending on the number of records, this is a sample of the size of the SMF Interval Records.

MQ Interval Data

By default MQ writes its accounting data once every 30 minutes (STATIME=30), and the intervals are not synchronized. We strongly recommend that you configure MQ to use SMF synchronized recording by setting STATIME=0 in the CSQ6SYSP system parameter module. The benefits are that the SMF data is synchronized between MQ subsystems, and with other SMF information about MQ, such as provided with job (30), data set (42) and RMF records. It also makes it easier for IntelliMagic Vision to process the intervals, as the native interval support in MQ tends to produce somewhat variable interval lengths.

Appendix L: Gathering Cache and ESS Statistics

The 74.5 record contains cache statistics and must be enabled.  The 74.7 record contains Director Control Unit Port (CUP) statistics and must be enabled.  The 74.8 record contains Array and Link statistics and must be enabled.  Enablement of this data collection is different between RMF and CMF.  If replication is present, port type of SYNC or ASYNC must be supplied by the customer for each DSS and distance between sites(miles/km or local)

Ideally 74.5 and 74.8 data collection need to be from only one LPAR in the SYSPLEX, but it should be from the LPAR that has connectivity to all DSSs and reporting volumes online. There’s no problem in specifying CACHE and ESS in more than one (or all) LPARs, IntelliMagic Vision throws out the duplicate records when they are encountered. Some customers prefer to specify CACHE and ESS in more than one LPAR, because not all LPARs are active in all intervals, and they want continuous recording of this information.

FICON Director/Switch statistics are stored on the switch platform itself and it does not need to be accessed by more than one LPAR out of all the LPARs that run data through the switch. Every CEC and LPAR in every Sysplex that might be attached to a switching device can retrieve the switch information using a single LPAR which can then cut the 74-7 RMF/CMF records for all CECs, LPARs and Sysplexes.

Brocade tells their students that they should enable FICON switch data collection in only one LPAR of their Sysplex. They should also make sure that FICON STATS=NO has been set in the IECIOSxx member of all LPARs except in the one LPAR where data collection is enabled in the Sysplex.

In 2007, it was warned that obtaining Director Statistics (74.7) data from multiple LPARs could result in Missing Interrupts and boxed devices. Following this guideline will mean only the selected LPAR will report switch activity/errors, even with multiple Sysplexes involved.

In cases where Remote Copy exists, it’s also useful to have a volume from the secondary DSS online to the primary host. That way we gather 74.8 information from both ends of the remote copy and can see both send and receive activity in the remote copy session(s).

Not all hardware vendors fully support the 74.8 records, but in no case is the request from RMF given an error.

To enable in either RMF or CMF follow the instructions below accordingly.

RMF

The ERBRMFxx Parmlib member needs to be updated accordingly:

74.5 cache statistics collection – specify CACHE (NOCACHE is the default)

74.7 FICON director data collection – specify FCD (the default is NOFCD – disabled).

74.8 ESS Array and Link statistics – specify ESS (NOESS is the default).

CMF

To turn on the 74.5 and 74.8 cache statistics use the CMF statement CACHE RECORDS=ALL.  Specifying CACHE will collect 74.5 records and specifying ESS will collect 74.8 records.

To turn on 74.7 FICON switch data collection use the CMF statement FICONSW.

Appendix M: Configuring TCP/IP for SMF 119.12 zERT Records

The 119.12 record contains zERT statistics and must be enabled.  To enable in zERT logging follow the instructions below to use the SMFCONFIG statement to provide SMF logging for TCP/IP and the GLOBALCONFIG statement to activate zERT in-memory monitoring.

Using SMFCONFIG to turn on SMF logging allows you to request that standard subtypes are assigned to the TCP/IP SMF records. The SMFPARMS statement provides a similar capability but requires the installation to select the subtype numbers to be used. Use the SMFCONFIG statement instead of SMFPARMS.   See more information about accounting for SMF records in z/OS Communications Server: IP Configuration Guide.  You turn the zERT discovery function on in the TCP/IP stack by specifying the GLOBALCONFIG ZERT parameter in the TCP/IP profile data set. Enabling zERT causes the TCP/IP stack to record and collect cryptographic protection information for new TCP and Enterprise Extender connections that terminate at the stack.  For more information on enabling zERT, see GLOBALCONFIG statement in z/OS Communications Server: IP Configuration Reference.

The SMFCONFIG profile statement controls only whether SMF records are written to the MVS™ SMF data sets.  More information about the SMFCONFIG statement is contained in this link https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.halz001/smfconfigstatement.htm

Syntax

Tip: Specify the parameters for these statement in any order.

Appendix N: Configuring FTPS client for AT-TLS

Appendix O: Cloud Database Retention settings

These are the default retention settings for SaaS customers.

Appendix P: Configuring CICS and TCP/IP zERT Groups

IntelliMagic Vision CICSGROUP names are used for reporting about the transaction data, providing reports per CICS group, or drilldowns to reports per CICS group. If a CICS group cannot be found or no CICS groups are defined, *Unassigned* will be used as CICS group name.  CICSGROUP combines CICS region data by APPLID.  Combine like CICS regions into a single CICSGROUP.

The TCP/IP zERT records (119.12) provide a log of all connections made by the mainframe TCP/IP system, listing the security properties for each session. The records are intended to help identify what protection is used, and what is not yet protected at the right level. The records provide very detailed information about the security settings for each protocol (TLS, SSH or IPSEC). While the raw records are different for each format, IntelliMagic Vision uses just one table, mapping the common fields between the protocols to common fields in the database.

IntelliMagic Vision adds the ability to specify masks for IP addresses, such that Intranet, Partner and 3rd party addresses can be easily identified in the reporting. You can define these IP-ranges with masks to suit your environment. IntelliMagic Vision applies these masks while processing the raw records, so the mapping is frozen in the database.