How can I solve "WER046A SORT CAPACITY EXCEEDED" in SYNCSORT JCL

Question

I am trying to sort a data set and writing into a new data set, Every time i am executing the job, its abending (ABEND=U0016).

message form SYSOUT.

    WER276B  SYSDIAG= 24646562, 29667262, 29667262, 27500165            
    WER164B  307,288K BYTES OF VIRTUAL STORAGE AVAILABLE, MAX REQUESTED,
    WER164B     0 BYTES RESERVE REQUESTED, 307,272K BYTES USED          
    WER036B G=5174,B=1,BIAS=99 
    WER162B 75 PREALLOCATED SORTWORK TRACKS, 3,750,000 DYNAMICALLY 
             ALLOCATED, 
    WER162B 26,721,480 ACQUIRED IN 2,230 SECONDARY EXTENTS, 0 RELEASED, 
             TOTAL 30,471,555 TRACKS USED
    WER046A  SORT CAPACITY EXCEEDED              
    WER493I  ZIIP PROCESSOR USED                 
    WER211B  SYNCSMF  CALLED BY SYNCSORT; RC=0000
    WER449I  SYNCSORT GLOBAL DSM SUBSYSTEM ACTIVE
    WER066A  APROX RCD CNT        30430982 

Here is my code:

    //STEP50  EXEC SORTD                                             
    //SORTIN   DD DSN=FILEXYZ(0),                          
    //            DISP=SHR,DCB=BUFNO=192                             
    //SORTOUT  DD DSN=FILE2XXY,                            
    //            DISP=(NEW,CATLG,DELETE),RETPD=365,VOL=(,,,99),     
    //            DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),    
    //            UNIT=(TAPE,2)                                      
    //SYSOUT   DD SYSOUT=*                                           
    //SYSPRINT DD SYSOUT=*                                           
    //SYSIN    DD *                                                  
           SORT FIELDS=(19,5,PD,A,8,6,PD,A,50,2,ZD,A)                  
           OUTREC IFTHEN=(WHEN=(70,18,CH,EQ,C' encoding="IBM037"'),    
                         OVERLAY=(70:C'  encoding="UTF-8"'))             
           OPTION DYNALLOC=(SYSDA,255)                                 
    /*  

Here is code from SORTD from the jcl:

       4 //STEP40  EXEC SORTD                                                  
       5 XXSORTD   PROC CYLS=1,SYSFICH='*',                                    
         XX     DMPCLAS='1,DEST=ABNORMAL'                                      
         XX*                                                                   
         XX*    REMOVED SEP PARAMETER 89/07/20                                 
         XX*                                                                   
       6 XXSORTD    EXEC PGM=SORT                                              
       7 //SYSOUT   DD SYSOUT=*                                                
         X/SYSOUT   DD SYSOUT=&SYSFICH                                         
         IEFC653I SUBSTITUTION JCL - SYSOUT=*                                  
       8 XXSORTWK01 DD DSN=&WORK1,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK1,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
       9 XXSORTWK02 DD DSN=&WORK2,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK2,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      10 XXSORTWK03 DD DSN=&WORK3,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK3,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      11 XXSORTWK04 DD DSN=&WORK4,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK4,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      12 XXSORTWK05 DD DSN=&WORK5,SPACE=(CYL,(&CYLS)),UNIT=SORTWORK            
         IEFC653I SUBSTITUTION JCL - DSN=&WORK5,SPACE=(CYL, 
         (1)),UNIT=SORTWORK  
      13 XXSYSABEND DD SYSOUT=&DMPCLAS                                         
         IEFC653I SUBSTITUTION JCL - SYSOUT=1,DEST=ABNORMAL                    
      14 XXSYSUDUMP DD SYSOUT=&DMPCLAS                                         
         IEFC653I SUBSTITUTION JCL - SYSOUT=1,DEST=ABNORMAL                    
      15 //SORTIN   DD DSN=FILEXYZ(0),                               
         //            DISP=SHR,DCB=BUFNO=192                                  
      16 //SORTOUT  DD DSN=FILE2XXY,                                 
         //            DISP=(NEW,PASS,DELETE),RETPD=365,VOL=(,,,99),           
         //            DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),         
         //            UNIT=TAPE                                               
      17 //SYSPRINT DD SYSOUT=*                                                
      18 //SYSIN    DD *     

Please suggest me some hints to over come this issue. Thank you.

Answer 1

I'll go out-on-a-limb here and assume that the files in question here are the same as previous question with same file attributes by same OP at: How can I reduce CPU in SORT operation

With 80,000,000 records at an LRECL of 30050, your input file is approximately 2.2TB.

A generally accepted rule-of-thumb for sort work space is 1.3x the input file size. So, you’ll need to target ~2.85TB (I.e ~3.7 million CYLS or ~55 million tracks) of sort work space.

With SyncSort’s maximum of 255 SORTWK DDs, you’ll need to acquire ~14,500 CYLs across each of the supported 255 DDs to accommodate your space requirements. This is a bit extreme, and will likely present a number of challenges, but there are a few options that could allow you to achieve this.

Disclaimer - I have not personally used SyncSort myself, but after perusing the docs, these concepts look to be consistent with other sort products that I do have experience with.

Rely on DYNALLOC

You have specified DYNALLOC=(SYSDA,255) and this is generally the recommended approach. However, there are a couple of issues with your current setup:

• By using the SORTD PROC, you are robbing yourself of 5 DDs that could have been more appropriately utilized. The hardcoded SORTWK DDs in the PROC with SPACE=(CYL,(1)) limit the total work space you can acquire because DYNALLOC now only has 250 DDs to work with. As @SaggingRufus suggests, calling SYNCSORT directly would avoid the 1-CYL allocations of the PROC DDs and allow DYNALLOC to work with all 255 DDs.

• SyncSort is not projecting the necessary work space accurately. Based on the output that you provided, SyncSort initially acquired 75 tracks (1 CYL for each of 5 hardcoded DDs in the SORTD proc… in other words, 5 DDs at 1 CYL * 15 Tracks-per-CYL = 75) explicitly, then dynamically acquired an additional 3,750,000 tracks (1,000 CYLs (or 15,000 tracks) * 250 DDs). It then acquired an additional 26.7 million tracks (1.78MM CYLs) across 2,230 secondary extents, which indicates an average of 799 CYLs (or 11,983 tracks) per secondary extent and an average of 8.75 secondary extents per volume/DD. All of this results in a total of ~1.6TB (~2MM Cyls) being allocated for sort work space, which is way below the estimated work space requirements for a 2.2TB input file. Even if we apply the DYNALLOC attributes to the hardcoded SORTWK DDs from the PROC, that only adds about 32GB to the total work space and is still woefully inadequate. So, as @phunsoft suggests, adding the SIZE=Ennnnnnnn parameter may be necessary for SyncSort to project work space appropriately.

• Additional note - The fact that you're getting an average of <10 secondary extents per DD/volume indicates that you're exhausting the available free space on volumes in the work space pool for your environment. You should coordinate with your storage admins to ensure that enough disk space is available for your work space.

So, bypassing the PROC to provide access to all 255 DDs and providing an estimated record count could allow SyncSort to more reliably calculate the necessary work space, but this amount of data in a single sort is likely an outlier and default calculations may not be sufficient. Also, the available free space in your work space pool may be a limiting factor.

Hardcode SORTWK DDs

If you hardcode the SORTWK DDs in your JCL, you bypass the DDs in the PROC and eliminate some of the uncertainty of the DYNALLOC calculations. This is actually the approach that I, personally, would take first in this scenario… hardcode each SORTWK DD with the best possible options to acquire ~14,500 CYLS on each DD. So:

• Hardcode each of 255 DDs (SORTWK01 - SORTWKFF) in job

• Include SPACE=(CYL,(4000,1000)) on each DD (even though SyncSort will supposedly calculate secondary allocations itself)

• Include DSNTYPE=LARGE on each DD to allow greater than 4G per volume (each SORTWK is restricted to 1 volume) The aggregate of these DDs should provide sufficient work space for your sort requirements.

Each of the above options assume that sufficient vols/extents/overall-space are available in the work space pool for your shop. I’d recommend coordinating with your storage management team to ensure availability of resources

Use MAXSORT

SyncSort provides the MAXSORT capability to accommodate extremely large data sets. It sorts sections of an input file into intermediate files, and re-uses SORTWK space for subsequent sections, until the intermediate files for each section can be combined into a single output file. There are particular requirements for using MAXSORT. As referenced earlier, I don’t have access to SyncSort, so I can’t test this capability myself. You'll have to research the SyncSort doc yourself for the details of this capability.

Answer 2

Since your sort input is read from tape, sort cannot reliably calculate the sort work space needed. Try adding the SIZE=Ennnnnnn parameter to the SORT statement. nnnnnnn is the number of records to be read. E tells sort this is an estimate.

Answer 3

try calling SYNCSORT directly instead of calling this proc. This PROC only has 5 workfiles set up. It is usually best to let SYNCSORT allocate as may workfiles as needed rather than telling it how many it can use.

I believe it is the workfiles that have run out of space. Try this:

//SORT10 EXEC PGM=SYNCSORT,                                   
//             PARM=('INCORE=OFF,DYNALLOC=(SYSDA,255)',EQUALS)
//SORTIN   DD DSN=FILEXYZ(0),                          
//            DISP=SHR,DCB=BUFNO=192                             
//SORTOUT  DD DSN=FILE2XXY,                            
//            DISP=(NEW,CATLG,DELETE),RETPD=365,VOL=(,,,99),     
//            DCB=(RECFM=FB,LRECL=30050,BLKSIZE=0,BUFNO=192),    
//            UNIT=(TAPE,2)
//SYSOUT   DD  SYSOUT=*
//SYSPRINT DD  SYSOUT=*
//SORTMSG  DD  SYSOUT=*
//SYSIN    DD *                                                  
       SORT FIELDS=(19,5,PD,A,8,6,PD,A,50,2,ZD,A)                  
       OUTREC IFTHEN=(WHEN=(70,18,CH,EQ,C' encoding="IBM037"'),    
                     OVERLAY=(70:C'  encoding="UTF-8"'))             
       OPTION DYNALLOC=(SYSDA,255)                                 
/*