This suite of programs will describe and demonstrate how to merge or append (concatenate) multiple files of various record lengths into a single file. For example, the first file is a sequential file with fixed-length records of 80 bytes. The second file is a sequential file with fixed-length records of 60 bytes. The third file is a sequential file with fixed-length records of 40 bytes.
This package includes a job (JCL Member) that will create the test files. The techniques for merging or appending the files will use IEBGENER, IDCAMS, ICETOOL, ICEMAN, DFSORT or SORT.
Note: The jobs in this suite of programs would be much simpler if all the input files had the same characteristics (i.e. the same file type with the same record length).
Within the scope of this document we will use the term 'merge' when the task will actually merge the files and intersperse the individual records of the files based on the content of a user-defined field within each record. This approach requires the input files to be sorted into sequence based on the content of the user-defined field.
Within the scope of this document we will use the term 'append' when the task simply copies the first file to a new file and then appends the second and third file to the end of the new file. The sequence of the records will be the same as the sequence within the original files. The technique may be referred to as file concatenation.
The process of merging or appending files is fairly simple when all the input files have the same characteristics or properties such as file type and record length. It does require extra consideration if the file properties or record lengths are different. This document uses files with different record lengths.
We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.
Copyright © 1987-2019
This section describes the JCL Members included with this suite of programs.
Append Files using IEBGENER
This batch job (MRGCPYJ1) uses a single JCL member (MRGCPYJ1,jcl) with multiple job steps to merge the records from three (3) input file with different record lengths into a single file with a record length that is equal-to the largest record size of the input files.
The following is a description for each job step following by a listing of the JCL Member.
The following (MRGCPYJ1.jcl) is the JCL Member.
Append Files using IDCAMS
This batch job (MRGCPYJ2) uses a single JCL member (MRGCPYJ2.jcl) with multiple job steps to merge the records from three (3) input file with different record lengths into a single file with a record length that is equal-to the largest record size of the input files.
The following is a description for each job step following by a listing of the JCL Member.
The following (MRGCPYJ2.jcl) is the JCL Member.
Append Files using SORT
This example uses a single JCL member (MRGCPYJ3.jcl) to append (or concatenate) three (3) input files with different record lengths into a single file with a record length that is equal-to or greater-than the largest record size of the input files.
This batch job (MRGCPYJ3) uses a single JCL member (MRGCPYJ3.jcl) with multiple job steps to append (or concatenate) three (3) input files with different record lengths into a single file with a record length that is equal-to the largest record size of the input files.
The following is a description for each job step following by a listing of the JCL Member.
The following (MRGCPYJ3.jcl) is the JCL Member.
Merge Files using ICETOOL
This batch job (MRGCPYJ4) uses a single JCL member (MRGCPYJ4.jcl) with multiple job steps to merge the records from three (3) input file with different record lengths into a single file with a record length that is equal-to the largest record size of the input files.
The following is a description for each job step following by a listing of the JCL Member.
The following (MRGCPYJ4.jcl) is the JCL Member.
The jobs included in this suite of programs will execute on an IBM Mainframe System with ZOS or a Linux, UNIX or Windows System with Micro Focus Enterprise Developer. Therefore, it will be necessary to have a properly configured system that is capable of executing JCL that use the standard data file utility programs. programs. The techniques for merging the files will use IEBGENER, IDCAMS, SORT, ICETOOL, ICEMAN and IEFBR14.
Next, a set of three test files will be necessary. The following sub-section describes a job for creating the test files.
Create Three Test Files
This job uses a single JCL member (MRGCPYJ8.jcl) to create the three (3) test files with different record lengths to be used as input for the jobs that do the file merging.
This section provides additional detail about the jobs and programs that are used to support this set of programs.
Hex-Dump for IEBGENER File
A utility program (HX80DUMP.cbl) that was generated using SimoTime technology is used to create a HEX-dump file for the content of the merged file created by IEBGENER.
Hex-Dump, JCL Member
The following is the JCL member (MRGDMPJ1.jcl) that is used to execute the program that creates a HEX-Dump file for the merged file that was created using IEBGENER.
Hex-Dump, Sample Output
The following is the output produced by the Hex-Dump program. The input file is the merged file that was created using IEBGENER.
Hex-Dump for IDCAMS File
A utility program (HX80DUMP.cbl) that was generated using SimoTime technology is used to create a HEX-dump file for the content of the merged file created by IEBGENER.
Hex-Dump, JCL Member
The following is the JCL member (MRGDMPJ2.jcl) that is used to execute the program that creates a HEX-Dump file for the merged file that was created using IDCAMS.
Hex-Dump, Sample Output
The following is the output produced by the Hex-Dump program. The input file is the merged file that was created using IDCAMS.
This suite of programs will describe and demonstrate how to merge or append (concatenate) multiple files of various record lengths into a single file. This document may be used to assist as a tutorial for new programmers or as a quick reference for experienced programmers.
In the world of programming there are many ways to solve a problem. This document and the links to other documents are intended to provide a greater awareness of the Data Management and Application Processing alternatives.
The documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.
SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please send an e-mail to: [email protected] or call 415 883-6565. We appreciate hearing from you.
Software Agreement and Disclaimer
Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.
SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided 'AS IS' without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.
Downloads and Links
This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.
Note: A SimoTime License is required for the items to be made available on a local system or server.
Current Server or Internet Access
The following links may be to the current server or to the Internet.
Note:The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the icon.
Explore How to Create an Empty Key-Sequenced-Data-Set (or VSAM, KSDS) without using IDCAMS.
Explore a detailed example of using data set concatenation that includes sample COBOL, JCL and Test Data.
Explore the non-Relational Data Connection for more examples of accessing methodologies and coding techniques for Data Files and VSAM Data Sets.
Explore the JCL Connection for more examples of JCL functionality with programming techniques and sample code.
Explore An Enterprise System Model that describes and demonstrates how Applications that were running on a Mainframe System and non-relational data that was located on the Mainframe System were copied and deployed in a Microsoft Windows environment with Micro Focus Enterprise Server.
Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.
Explore The File Status Return Codes to interpret the results of accessing VSAM data sets and/or QSAM files.
Internet Access Required
The following links will require an internet connect.
This suite of programs and documentation is available to download for review and evaluation purposes. Other uses will require a SimoTime Software License. Link toan Evaluation zPAK Optionthat includes the program members, documentation and control files.
A good place to start isThe SimoTime Home Pagefor access to white papers, program examples and product information. This link requires an Internet Connection
ExploreThe Micro Focus Web Sitefor more information about products and services available from Micro Focus. This link requires an Internet Connection.
Glossary of Terms
Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.
Comments or Feedback
This document was created and is maintained by SimoTime Technologies. If you have any questions, suggestions, comments or feedback please use the following contact information.
We appreciate hearing from you.
Company Overview
SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.
Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.
Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.
Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com
I have a 6 row input file which consists of a field(Position 1 to 6) that contains a different value on every line. Based on the different values contained in this field the other fields (From position 7 -80) will be moved to to a single row in output.
E.G.
Input:
Output:
I need to find out how to read these all in as different rows and then output to a single row. I've tried using something similar to the code to this:
But this will move the data onto the correct position on seperate rows like this:
So now I think I need to do a sort in one step and a merge in another step. I would prefer to do it in one if it's possible however. I'd appreciate any help on this. Thanks.
cschneid
7,60711 gold badge2020 silver badges2828 bronze badges
Jack DonovanJack Donovan
1 Answer
You add a sequence-number to each record.
The use WHEN=GROUP to copy data from one record to one or more subsequent records.
You use OUTFIL INLUDE= to just pick up the final record.
You need to do a bit of planning. The sixth record will contain all the data, but perhaps not yet in the order that you want. Either with an IFTHEN=(WHEN=(logical expression) (to identify the sixth record) on the INREC or with the BUILD on the OUTREC, you can do your final formatting.
You need to change the
something each time, it will be receivingposition:sourceposition,length
The DFSORT manuals are very good, there is a Getting Started for those new to the product, and everything you'll ever need is in the Application Programming Guide,
Bill WoodgerBill Woodger
12.5k44 gold badges3333 silver badges3939 bronze badges
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Not the answer you're looking for? Browse other questions tagged mergemainframejcldfsort or ask your own question.
In Microsoft Excel, concatenation generally refers to combining the contents of two or more cells in a worksheet into a third, separate cell using either; this process can be accomplished by either using the CONCATENATE function or the concatenation operator. Take a look at the different ways you can combine cells in Excel, potentially improving your current workflow.
Understanding the Excel CONCATENATE FunctionCONCATENATE Syntax and Arguments
A function's syntax refers to the layout of the function and includes the function's name, brackets, comma separators, and arguments.
Text1 (required) can be actual text such as words or numbers, blank spaces surrounded by quotation marks, or cell references to the location of data in a worksheet.
Text2, Text3, ... Text255 (optional up to 255 text entries) can be added to the CONCATENATE function to a maximum of 8,192 characters – including spaces. Each entry must be separated by a comma.
Neither method of concatenation automatically leaves a blank space between words, which is fine when joining two parts of a compound word like Baseball into one or combining two series of numbers like 123456.
Joining first and last names or an address, however, need the space so it must be included in the concatenation formula, as shown in rows four, five, and six of the accompanying image.
Concatenating Number Data
Even though numbers can be concatenated, as seen in row six of the example image, the result 123456 is no longer considered a number by the program but is now seen as text data.
The resulting data in cell C7 cannot be used as arguments for certain math functions such as SUM and AVERAGE. If such an entry is included with a function's arguments, it is treated like other text data and ignored.
One indication is that the concatenated data in cell C7 is aligned to the left, which is the default alignment for text data. The same result occurs if the CONCATENATE function is used instead of the concatenate operator.
How to Use the Excel CONCATENATE Function
Although it is possible to type the complete function in manually, many people find it easier to use the dialog box to enter a function's arguments, since the dialog box takes care of entering brackets, commas and, in this example, the quotation marks surrounding the blank space.
The steps below cover entering the function using the dialog box into cell C2.
The concatenated name Mary Jones appears in cell C4.
When you click on cell C4, the complete function appears in the formula bar above the worksheet.
Displaying the Ampersand in Concatenated Text
There are times where the ampersand character is used in place of the word and, such as in company names as shown in row six of the example image.
To display the ampersand as a text character rather than have it act as the concatenation operator, you surround it in double quotation marks like other text characters, as shown in the formula in cell D6.
It should be noted that in this example, spaces are present on either side of the ampersand to separate that character from the words on either side. To achieve this result, you enter space characters on either side of the ampersand inside the double quotation marks in this fashion: ' & '.
Similarly, if a concatenation formula that uses the ampersand as the concatenation operator is used, the space characters and the ampersand surrounded by double quotes must also be included to have it appear as text in the formula results.
For example, the formula in cell D6 could be replaced with the above formula to achieve the same results.
Input file Layout:01 to 10 - 10 Digit Acct#53 to 01 - An indicator with values 'Y' or 'N'71 to 10 - Time stamp(Rest of the fields are insignificant for this sort)
While sorting the input file by splitting and eliminating duplicates in two ways result in different results. I wanna know why?
Casei: Splitting and Eliminating duplicates in the same step.
Caseii: Splitting and eliminating duplicates in two different steps:
These two steps are resulting different output. Do u see any difference between both cases? Please clarify.
Raja ReddyRaja Reddy
47288 gold badges1818 silver badges3737 bronze badges
2 Answers
You are asking to sort on an Account Number (10 characters ascending) then on an Indicator (1 character ascending).These two fields alone determine the key of the record - Timestamp is not part of the sort key. Consequently if thereare two or more records with the same key they could be placed in any (random) order by the sort. No tellingwhat order the Timestamp values will appear.
Keeping the above in mind, consider what happens when you have two records with the same key but differentTimestamp values. One of these Timestamp values meets the given INCLUDE criteria and the other one doesn't.The SUM FIELDS=NONE parameter is asking to remove duplicates based on the key. It does this by groupingall of the records with the same key together and then selecting the last one in the group. Since keydoes not include the Timestamp the choosen record is essentially a random event. Consequently it is unpredictableas to whether you get the record that meets the subsequent INCLUDE condition.
There are a couple of ways to fix this:
A stable sort causes records having the same sort key to maintain their same relative positions after the sort.This will preserve the original order of the Timestamp values in your file given the same key. When the removal of duplicates occurs DFSORT will choose the last record from the set of duplicates. This should bring the predicability to the duplicate elimination process you are looking for. Specifya stable sort by adding an OPTIONS EQUALS control card before the SORT card.
EDIT Comment:...picks the VERY FIRST record
The book I based my original answer on clearly stated the last record in a group of records with the samekey would be selected when SUM=NONE is specified. However, it is alwaysbest to consult the vendors own manuals. IBM's DFSORT Application Programming Guide only statesthat one record with each key will be selected. However,it also has the following note:
The FIRST operand of ICETOOL's SELECT operator can be used to perform the same function as SUM FIELDS=NONE with OPTION EQUALS. Additionally, SELECT's FIRSTDUP, ALLDUPS, NODUPS, HIGHER(x), LOWER(y), EQUAL(v), LASTDUP, and LAST operands can be used to select records based on other criteria related to duplicate and non-duplicate keys. SELECT's DISCARD(savedd) operand can be used to save the records discarded by FIRST, FIRSTDUP, ALLDUPS, NODUPS, HIGHER(x), LOWER(y), EQUAL(v), LASTDUP, or LAST. See SELECT Operator for complete details on the SELECT operator.
Based on this information I would suggest using ICETOOL's SELECT operator to select the correct record.
Sorry for the misinformation.
NealBNealB
14.9k22 gold badges3131 silver badges5959 bronze badges
The problem is as NealB identified.
The easiest thing to do is to 'get rid of' the records you don't want by date before the SORT. The SORT will take less time. This assumes that SORTOUT is not required. If it is, you have to keep your INCLUDE= on the OUTFILs.
SELECT is a good option. SELECT uses OPTION EQUALS by default. The below Control Cards can be included in an xxxxCNTL dataset, and action from the SELECT with USING(xxxx). SELECT gives you greater flexibility than SUM (you can get the last, amongst other things).
The whole task sounds flawed. If there are records per account with different dates, I'd expect either the first date or the last date, or something else specific, to be required, not just whatever record happens to be hanging around at the end of the SUM.
Or, if the Y/N cover all records:
Concatenate Records In DfsortBill WoodgerBill Woodger![]()
12.5k44 gold badges3333 silver badges3939 bronze badges
Not the answer you're looking for? Browse other questions tagged sortingduplicate-removaljcl or ask your own question.
This document will describe and demonstrate how to do data file concatenation using the function provided with JCL, Utility Programs and COBOL Programs. It is possible to process two (2) or more files as a single data set by concatenating the DD statements.
We have made a significant effort to ensure the documents and software technologies are correct and accurate. We reserve the right to make changes without notice at any time. The function delivered in this version is based upon the enhancement requests from a specific group of users. The intent is to provide changes as the need arises and in a timeframe that is dependent upon the availability of resources.
Copyright © 1987-2019 Programming Objectives
The following is a list of the functions provided in this example.
Programming Input and Output
The following shows the contents of the three test files that are used to demonstrate the data set concatenation capabilities.The first file contains three records.
The second file contains the following three records.
The third file contains the following three records.
The following files show the results that are produced using a COBOL program and the SORT Utility program. The first file is the results produced by the job step (DUPE80S2) that executes the COBOL program
The second file is the results produced by the job steps (SORT80S3 and EXTENDS4)) that executes the Sort program to read and sort the concatenated files into a single, sorted sequentail file based on the content of the first five positions in the records. The final step uses IEBGENER to add a record to the file.
Programming Requirements
This suite of samples programs will run on the following platforms.
Programming Overview
The following shows the processing logic by job step for the JCL Member that demonstrates the usage of data set concatenation.
Color Associations: The
This sample suite of programs has two JCL members. The first JCL member will execute the demonstration program. The second JCL member will create the test files used by the job that does data set concatenation.
Run a Job, Data Set Concatenation
The following is the JCL member (CONCATJ1.jcl) required to run the job that does data set concatenation. This job will run on an IBM Mainframe System with ZOS or a Linux, UNIX or Windows System with Micro Focus Enterprise Server or Studio.
Create the Test Files
The following is the JCL member (CONCATJ8.jcl) that is used to create the test files. This job will run on an IBM Mainframe System with ZOS or a Linux, UNIX or Windows System with Micro Focus Enterprise Server or Studio.
The following is the COBOL program (DUPE8080.cbl) that is used to read the three concatenated data sets and write a new, single data set containing all the records. This job will run on an IBM Mainframe System with ZOS or a Linux, UNIX or Windows System with Micro Focus Enterprise Server or Studio.
Note: The DD statements in the JCL will present the three concatenated data sets to the COBOL program as if they were one sequential file. The COBOL program will see the first record in the first file as the starting point. The COBOL program will not get an End-of-File result until after the last record in the third data set has been read.
This document will describe and demonstrate how to do data file concatenation using the functions provided with JCL, Utility Programs and COBOL Programs. This document may be used to assist as a tutorial for new programmers or as a quick reference for experienced programmers.
In the world of programming there are many ways to solve a problem. This document and the links to other documents are intended to provide a greater awareness of the Data Management and Application Processing alternatives.
The documentation and software were developed and tested on systems that are configured for a SIMOTIME environment based on the hardware, operating systems, user requirements and security requirements. Therefore, adjustments may be needed to execute the jobs and programs when transferred to a system of a different architecture or configuration.
SIMOTIME Services has experience in moving or sharing data or application processing across a variety of systems. For additional information about SIMOTIME Services or Technologies please send an e-mail to: [email protected] or call 415 883-6565. We appreciate hearing from you.
Software Agreement and Disclaimer
Permission to use, copy, modify and distribute this software, documentation or training material for any purpose requires a fee to be paid to SimoTime Technologies. Once the fee is received by SimoTime the latest version of the software, documentation or training material will be delivered and a license will be granted for use within an enterprise, provided the SimoTime copyright notice appear on all copies of the software. The SimoTime name or Logo may not be used in any advertising or publicity pertaining to the use of the software without the written permission of SimoTime Technologies.
SimoTime Technologies makes no warranty or representations about the suitability of the software, documentation or learning material for any purpose. It is provided 'AS IS' without any expressed or implied warranty, including the implied warranties of merchantability, fitness for a particular purpose and non-infringement. SimoTime Technologies shall not be liable for any direct, indirect, special or consequential damages resulting from the loss of use, data or projects, whether in an action of contract or tort, arising out of or in connection with the use or performance of this software, documentation or training material.
Downloads and Links
This section includes links to documents with additional information that are beyond the scope and purpose of this document. The first group of documents may be available from a local system or via an internet connection, the second group of documents will require an internet connection.
Note: A SimoTime License is required for the items to be made available on a local system or server.
Current Server or Internet Access
The following links may be to the current server or to the Internet.
Note:The latest versions of the SimoTime Documents and Program Suites are available on the Internet and may be accessed using the icon. If a user has a SimoTime Enterprise License the Documents and Program Suites may be available on a local server and accessed using the icon.
Explore how to merge or append multiple files into a single file. This suite of programs will describe and demonstrate the process using multiple input files of various record lengths.
Explore the JCL Connection for more examples of JCL functionality with programming techniques and sample code.
Explore a brief Overview of JCL Syntax that includes a description of data set concatenation using DD statements. Job Control Language (or JCL) specifies how programs are executed on the mainframe. JCL functions are the interface between the programs and the operating system.
Explore the non-Relational Data Connection for more examples of accessing methodologies and coding techniques for Data Files and VSAM Data Sets.
Explore An Enterprise System Model that describes and demonstrates how Applications that were running on a Mainframe System and non-relational data that was located on the Mainframe System were copied and deployed in a Microsoft Windows environment with Micro Focus Enterprise Server.
Explore The ASCII and EBCDIC Translation Tables. These tables are provided for individuals that need to better understand the bit structures and differences of the encoding formats.
Explore The File Status Return Codes to interpret the results of accessing VSAM data sets and/or QSAM files.
Internet Access Required
The following links will require an internet connect.
This suite of programs and documentation is available to download for review and evaluation purposes. Other uses will require a SimoTime Software License. Link toan Evaluation zPAK Optionthat includes the program members, documentation and control files.
A good place to start isThe SimoTime Home Pagefor access to white papers, program examples and product information. This link requires an Internet Connection
ExploreThe Micro Focus Web Sitefor more information about products and services available from Micro Focus. This link requires an Internet Connection.
Glossary of Terms
Explore the Glossary of Terms for a list of terms and definitions used in this suite of documents and white papers.
Comments or Feedback
This document was created and is maintained by SimoTime Technologies. If you have any questions, suggestions, comments or feedback please use the following contact information.
We appreciate hearing from you.
Company Overview
SimoTime Technologies was founded in 1987 and is a privately owned company. We specialize in the creation and deployment of business applications using new or existing technologies and services. We have a team of individuals that understand the broad range of technologies being used in today's environments. Our customers include small businesses using Internet technologies to corporations using very large mainframe systems.
Quite often, to reach larger markets or provide a higher level of service to existing customers it requires the newer Internet technologies to work in a complementary manner with existing corporate mainframe systems. We specialize in preparing applications and the associated data that are currently residing on a single platform to be distributed across a variety of platforms.
Preparing the application programs will require the transfer of source members that will be compiled and deployed on the target platform. The data will need to be transferred between the systems and may need to be converted and validated at various stages within the process. SimoTime has the technology, services and experience to assist in the application and data management tasks involved with doing business in a multi-system environment.
Whether you want to use the Internet to expand into new market segments or as a delivery vehicle for existing business functions simply give us a call or check the web site at http://www.simotime.com
Comments are closed.
|
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |