Datastage Interview Questions and Answers

Q. What is DataStage?

• Design jobs for Extraction, Transformation and Loading(ETL).
• Ideal tool data integration projects such as data warehouses, data marts and system migrations.
• Import,export,create and managed metadata for use within jobs.
• Schedule, run and monitor jobs all within DataStage
• Administers your Datastage development and execution environments.

Q. What are the components of Ascential Data Stage?

• Client Components - Administrator, Director, Manager, and Designer.
• Server Components - Repository, Server and Plug-ins.

Q. Which Command is required to check the Datastage Job Status?

• dsjob -status

Q. What is version Control?

• Version Control stores different versions of DS jobs runs different versions of same job reverts to previous version of a job view version histories.

Q. What is the difference between DataStage and Informatica?

• DataStage support parallel processing which informatica doesn't.
• Links are object in the DataStage ,in Informatica it’s a port to port connectivity.
• In Informatica it’s easy to implement Slowly Changing Dimensions which is little bit complex in DataStage.
• DataStage doesn't support complete error handling.

Q. What is a merge?

• Merge is a stage that is available in both parallel and server jobs.
• The merge stage is used to join two tables(server/parallel) or two tables/datasets(parallel).
• Merge requires that the master table/dataset and the update table/dataset to be sorted.
• Merge is performed on a key field, and the key field is mandatory in the master and update dataset/table.

Q. Tell us part/components of configuration file:

• Node
• ServerName
• Pools
• FastName
• ResourceDisk

Q. In which two situations would you use the Web Services Client stage?

• You need the Web service to act as either a data source or a data target during an operation.
• You do not need both input and output links in a single web service operation.

Q. What are Sequencers?

• A sequencer allows you to synchronize the control flow of multiple activities in a job sequence. It can have multiple input triggers as well as multiple output triggers.

Q. How to unlock a Datastage job?

• Clean-up Resources in Director
• Clear Status File in Director
• DS.Tools in Administrator
• DS.Tools in UNIX

Q. What is Active and Passive Stage?

• Passive Stages are used for data extraction and loading ,while Active Stage are used to implements and process the business rules.

Q. How do you generate Sequence number in Datastage?

• Using the Routine
• KeyMgtGetNextVal
• KeyMgtGetNextValConn
• They can also be done by Oracle Sequence.

Q. What are the Job parameters?

• These Parameters are used to provide Administrative access and change run time values of the job.EDIT > JOBPARAMETERS. In that Parameters Tab we can define the name,prompt,type,value.

Q. What is the difference between Hashfile and Sequential File?

• Hash file stores the data based on hash algorithm and on a key value. A sequential file is just a file with no key column. Hash file used as a reference for look up. Sequential file cannot.

Q. What is the difference between Account and Directory options?

• Account option create the hashfile in the current project itself using condition and for Directory option we have to specify the path of the directory.

Q. What is the difference between Server Job and Parallel Jobs?

• Server Jobs works in sequential way while parallel jobs work in parallel fashion (Parallel Extender work on the principal of pipeline and partition) for Inpur/Output processing.

Q. What are the types of jobs we have in Datastage?

• Server Jobs
• Parallel Jobs
• Mainframe Jobs

Q. What is Staging Variable?

• These are the temporary variables created in transformer for calculation.

Q. What are Routines?

• Routines are the functions which we develop in BASIC Code for required tasks, which we Datastage is not fully supported (Complex).

Q. Unix Command to stop Datastage Engine?

• /bin/uv –admin –stop

Q. Unix Command to start Datastage Engine? 

• /bin/uv –admin –start

Q. Unix command to check datastage jobs running at server?

• ps -ef | grep phantom

Q. Unix Command to check Datastage sessions running at backend?

•  netstat –na | grep dsr 
•  netstat –a | grep dsr 
•  netstat –a | grep dsrpc

Q. Where datastage temprory dataset files stored while running a Datastage parallel Job?

• ResourceScratchDisk

Q. You need to invoke a job from the command line that is a multi-instance enabled. What is the correct syntax to start a multi-instance job?

• dsjob -run
• -mode NORMAL

Q. A client must support multiple languages in selected text columns when reading from DB2 database. Which two actions will allow selected columns to support such data?

• Choose Unicode setting in the extended column attribute.
• Choose NVar/NVarchar as data types.

Q. What are the types of Containers?
        
       There are Two types of containers namely

• Local Container
• Shared Container

Q. Define data aggregation?

• An operational data source usually contains records of individual transactions such as product sales. If the user of a data warehouse only needs a summed total, you can reduce records to a more manageable number by aggregating the data.

Q. What are stage variables?

• An intermediate processing variable that retains value during read and doesn’t pass the value into target column.

Q. Which two system variables/techniques must be used in a parallel Transformer derivation to generate a unique sequence of integers across partitions?

• @PARTITIONNUM
• @NUMPARTITIONS

Q. What is the default Array Size in OCI stage?

• 32767

Q. What is the default cache size of Datastage?

• 256MB

Q. Which is a part of Managing active sessions in Datastage?

• Viewing all active sessions
• Setting session limits
• Opening user details
• Disconnecting a session

Q. What can you do from the Administrator client?

• Set up user permissions for projects
• Purge job log file
• Set Environment variable default value
• Add, delete, and move InfoSphere® DataStage® projects

Q. What all are the different way to run a job?

• Using DataStage Director and DataStage Administration Client.

Q. How can we improve the performance in DataStage?

In server canvas we can improve performance in two ways:

• Firstly we can increase the memory by enabling interprocess row buffering in job properties
• Secondly by inserting an IPC stage we break a process into two processes.We can use this stage to connect two passive stages or two active stages.

Q. What is orabulk Stage?

• This Stage is used to Bulk Load the Oracle Target Database.

Q. How do you schedule or monitoring the job?

• Using the DataStage Director we can schedule or monitor the job.

Q. What is APT_CONFIG in datastage?

• Datastage understands the architecture of the system through this file(APT_CONFIG_FILE). For example this file consists information of node names, disk storage information etc.
• APT_CONFIG is just an environment variable used to idetify the *.apt file.
• Dont confuse that with *.apt file that has the node's information and Configuration of SMP/MMP server.

Q. What are orabulk and bcp stages?

• These are called as plug-in stages orabulk is used when v have bulk data in oracle then v go for orabulk for other than oracle database we go for bcp stages.

Q. Why fact table is in normal form?

• A fact table consists of measurements of business requirements and foreign keys of dimensions tables as per business rules.

Q. What is the difference between the local container and shared container?

• Local Container is local to the particular job in which we developed the container.
• Shared Container is can be used in any other jobs also.

Q. What can we do with DataStage Director?

• Validating
• Scheduling
• Executing
• Monitoring Jobs (server Jobs).

Q. Which algorithm you used for your hashfile?

• It uses GENERAL or SEQ.NUM. algorithm

Q. What all the types of Jobs you developed?

• Server Job
• Parallel Job
• Sequencer Job
• Container Job.

Q. How do you register plug-ins?

• Using DataStage Manager.

Q. What are iConv and oConv?

These are the Date functions, which we use to convert the Dates from internal format to External format

• iConv - External to Internal
• oConv- Internal to External

Q. How do you convert the columns to rows in DataStage?

• Using Pivot Stage.

Q. What are the types of input links in datastage?

DataStage supports two types of input link:

• Stream : A link representing the flow of data. This is the principal type of link, and is used by both active and passive stages.
• Reference : A link representing a table lookup. Reference links are only used by active stages. They are used to provide information that might affect the way data is changed, but do not supply the data to be changed.

Q. Define dynamic arrays?

• Dynamic arrays map the structure of DataStage file records to character string data. Any character string can be a dynamic array. A dynamic array is a character string containing elements that are substrings separated by delimiters.

Q. Where the DataStage stored his repository?

• DataStage stored his repository in IBM Universe Database.

Q. What are the command line functions that import and export the DS jobs?

• dsimport.exe : imports the DataStage components.
• dsexport.exe : exports the DataStage components.

Q. What is transformer stage?

• Transformer stages do not extract data or write data to a target database. They are used to handle extracted data, perform any conversions required, and pass data to another Transformer stage or a stage that writes data to a target data table.

Q. How do you schedule or monitoring the job?

• Using the DataStage Director we can schedule or monitor the job.

Q. Define universe stage?

• A stage that extracts data from or loads data into a Universe database using SQL. Used to represent a data source, an aggregation step, or a target data table.

Q. How we can reuse the components?

• Using the Shared and Local Containers.

Q. What are the Types of Jobs?

• Server Jobs : Designed, developed and compiled are done in Designer. Validated, Scheduled, Executed and Monitoring are done in Director. Win NT/ Unix system Server.
• Parallel Jobs : Designed, developed and compiled are done in Designer Validated, Scheduled, Executed and Monitoring are done in Director. Server should be on Unix System Only.
• Mainframe Jobs : Designed and developed are done in Designer (win) Complied, Validated, Scheduled, Executed and Monitored in Mainframe Systems Only.

Q. In which two situations would not use the Web Services Client stage?

• You want to deploy a service.
• You need to create a WSDL.

Q. Which two actions can improve sort performance in a DataStage job?

• Specify only the key columns which are necessary.
• Minimize the number of sorts used within a job flow.
• Adjusting the "Restrict Memory Usage" option in the Sort stage.

Q. You are experiencing performance issues for a given job. You are assigned the task of understanding what is happening at run time for the given job. What are the first two steps you should take to understand the job performance issues?

• Run job with $APT_TRACE_RUN set to true.
• Review the objectives of the job.

Q. Your customer asks you to identify which stages in a job are consuming the largest amount of CPU time. Which product feature would help identify these stages?

• $APT_PM_PLAYER_TIMING

Q. Which three statements describe a DataStage installation in a clustered environment?

• The conductor node will create the job score and consolidate messages to the DataStage log.
• For clustered implementations, appropriate node pools are useful to reduce data shipping.
• Compiled job and all shared components must be available across all servers.

Q. Which three defaults are set in DataStage Administrator?

• Project level defaults for environment variables.
• Project level default for compile options
• Project level default for Runtime Column Propagation

Q. Which two environment variables should be set to "True" to allow you to see operator process statistics at run-time in the job log?

• $APT_PM_PLAYER_MEMORY
• $APT_PM_PLAYER_TIMING

Q. Which three statements are true about National Language Support (NLS)

• NLS must be selected during installation to use it.
• Within an NLS enabled DataStage environment, maps are used to convert external data into UTF-#6.
• Reading or writing 7-bit ASCII data from a database does not require NLS support.

Q. Which three statements describe a DataStage installation in a clustered environment?

• The conductor node will create the job score and consolidate messages to the DataStage log.
• For clustered implementations, appropriate node pools are useful to reduce data shipping.
• Compiled job and all shared components must be available across all servers.

Q. Upon which two conditions does the number of data files created by a File Set depend?

• The number of processing nodes in the default node pool
• The number of disks in the export or default disk pool connected to each processing node in the default node pool

Q. Which command line switch can be used to return the most recent start time for a given job?

• dsjob -jobinfo

Q. You are working on a project that contains a large number of jobs contained in many folders. You would like to review the jobs created by the former developer of the project. How can you find these jobs?

• Use the Advanced Find feature contained in the Designer interface.

Q. Which two statements are true for named node pools?

• Using appropriately named node pools can allow separation of buffering from sorting disks.
• Using appropriately named node pools constraints will limit stages to be executed only on the nodes defined in the node pools.

Q. Which three methods can be used to import metadata from a Web Services Description Language (WSDL document)?

• Web Service Function Definitions
• XML Table Definitions
• Web Services WSDL Definitions

Q. What are two tasks that can create DataStage projects?

• Install the DataStage engine.
• Add new projects from DataStage Administrator.

Q. Which two conditions does the No of data files created by a Dataset Set depend?

• The numbers of processing nodes in the default node pool.
• The number of disks in the export or default disk pool connected to each processing node in the default node pool.

Q. Which requirement must be met to read from a database in parallel using the ODBC Enterprise stage?

• Specify the partition col property.

Q. For APT_DISABLE_COMBINATION which statements are true?

• Disabling generates more processes requiring more system resources and memory.
• Globally disables operator combining.

Q. Techniques you will use to abort a job in Transformer stage?

• Create a dummy output link with a constraint that tests for the condition to abort on set the "Abort After Rows" property to #.

Q. The dsrpcd daemon is the means by which processes that represent DataStage jobs are started? 

• The environment that DataStage processes inherit when they are started is the same environment as for dsrpcd. ODBC drivers and some plug-ins require that certain directories are included in the shared library environment variable setting for dsrpcd.

Q. A job design consists of an input sequential file, a Modify stage, followed by a Filter stage and an output Sequential File stage. The job is running on an SMP machine with a configuration file defined with three nodes. No environment variables were set for the job. How many osh processes will this job create?

• 9

Q. Using a DB2 for z/OS source database, a 200 million row source table with 30 million distinct values must be aggregated to calculate the average value of two column attributes. What would provide optimal performance while satisfying the business requirements?

• Select all source rows using a DB2 API stage and then Aggregate using a Sort Aggregator.

Q. In your DB2 database you have column names that use characters # and $. Which two steps should be completed to allow DataStage to convert these characters into internal format and back as necessary?

• Set environment variable $DS_ENABLE_RESERVED_CHAR_CONVERT to true.
• Avoid using the strings __035__ and __036__ in your IBM DB2 column names.

Q. When invoking a job from a third-party scheduler, it is often desirable to invoke a job and wait for its completion in order to return the job's completion status. Which three commands would invoke a job named "BuildWarehouse" in project DevProject and wait for the job's completion?

• dsjob -run -jobstatus DevProject BuildWarehouse
• dsjob -run -userstatus DevProject BuildWarehouse
• dsjob -run -wait DevProject BuildWarehouse

Parameters Using Parameter/Value Set/Value File in Datastage

When a file based parameter set is used, the parameter set picks up the contents of the file at the time the job is started. This provides excellent flexibility as long job could manipulate a parameter set file, and leave new parameters for a subsequent job. The key is that the parameters are not compiled into the job or the parameter set. The following steps detail the creation and use of file based parameter sets.


The following steps detail the creation and use of file based parameter sets. 
File > New > Other > Parameter Set

In the general tab, name the parameter set. It should be alphanumeric and underscore characters only and no spaces
For example: my_file_parm_set

   1. In the Parameters tab, create the parameters and defaults

    2. In the Values tab, put the name of the file in the "Value File Name" column. Specify a file name only. No path component

Use alphanumeric characters etc. and no spaces. In the same row, list values for each parameter defined in the Parameters tab. The table should look like this:

         3. Click OK and save the parameter set.

      4. At this point the parameter set file should have been created using the following system:

Projects<project_name>/ParameterSets/<parm_nat_name>/<parm_file_name>

In our example the file name would be: InformationServer\Server\Projects\foo2\ParameterSets\my_file_parm_set\parm_file.txt

The contents of the file looks like this:

foo=Parm file value for foo

bar=Parm file value for bar 

In the job, navigate to Edit > Job Properties and select the Parameters tab. 

Click Add Parameter Set and select the newly created parameter set:

              ******Create the Sequence Job*****

      5. Create a new sequence and add a Job Activity stage to it. Double click on the stage to edit its properties then click on the button to select a Job name. 

    6. Once the job is loaded, its parameters become visible in the Parameters section of the Job Activity screen. The new parameter set should be in the list. Click on the drop down selector in the Value Expression column and select the parameter file:

Click OK and save the sequence.

      7. Now every time the sequence is run, parameters are read from the file at run time and passes to the job. Any preceding jobs in the sequence can change the file to dynamically modify the parameters.

Note: When the number of parameters in a parameter set is changed, all jobs using the parameter set must be recompiled.

*******************************************************************************

New Features In IBM Datastage 9.1

New features and Enhancement on IBM InfoSphere Information Server, Version 9.1

The new and changed features and documentation updates are described in details in the following different sections.

Index 

1.    InfoSphere Information Server for Data Integration

·         InfoSphere Data Click

·         InfoSphere DataStage

2.    InfoSphere Information Server for Data Quality

·         InfoSphere Data Quality Console

·         InfoSphere Information Analyzer

·         InfoSphere QualityStage

3.    InfoSphere Metadata Asset Manager

4.    Common capabilities across the InfoSphere Information Server suite

·         Administering

·         Connecting to external sources

·         InfoSphere Blueprint Director

·         InfoSphere Metadata Asset Manager

·         InfoSphere Metadata Workbench

·         Migrating

5.    InfoSphere Business Information Exchange

·         InfoSphere Business Glossary

·         InfoSphere Business Glossary Client for Eclipse

6.    Documentation changes included in the Version 9.1 release

·         Documentation introduced or enhanced with Version 9.1

Below is the detailed description about newly added features in Datastage v9.1:

1. InfoSphere Information Server for Data Integration

•    InfoSphere Data Click

InfoSphere Data Click helps users retrieve data and provision systems with agility. Users can offload individual tables or entire schemas to generate sandbox environments for personal or group development work. The simple interface enables users of any skill level to complete the data integration task. InfoSphere Data Click inherits the built-in data governance features of the InfoSphere Information Server platform.

InfoSphere Data Click generates both design and operational metadata to support data lineage and impact analysis. InfoSphere Data Click assets also support linkages to the business glossary so that users can establish trust in the sources of information that are used. Also, administrators can define policies that control the data integration activity so that users cannot exceed limits that are based on enterprise requirements.

InfoSphere Data Click is installed when you install InfoSphere Information Server for Data Integration. InfoSphere Data Click activities are governed from InfoSphere Blueprint Director. You install InfoSphere Data Click as a plug-in into InfoSphere Blueprint Director.

•   InfoSphere DataStage

Workload management

You can now use the workload management service in InfoSphere Information Server to allow the administrator to set system resource policies and prioritization of workload classes. The policies and workload classes control the execution of parallel and server jobs.

Web-based job runtime management

Administration and management of the operational environment is simplified by extending the Operations Console. Authorized users can now define the workload management policies, and can run, stop, and reset integration jobs within the projects that they administer.

Balanced optimization for Hadoop

Extending the HDFS features in Version 8.7, you can now use the Balanced Optimization features of InfoSphere DataStage to push sets of data integration processing and related data I/O into a Hadoop cluster. InfoSphere DataStage adds integration with Oozie workflows, as well as real-time integration with InfoSphere Streams.

Support for IBM Rational Team Concert™ as a source control system

You can now use Rational Team Concert as a source control system in IBM InfoSphere Information Server Manager.

XML design and performance optimization enhancements

InfoSphere DataStage 9.1 includes new features to help you work with the type of large XML schemas that are often seen in industry standards. You can use one new feature, the schema view, to narrow the scope of a large XSD to only the subset of the schema tree that you want to work with.

When you narrow the scope, you can focus on a particular business challenge and parse and compose XML documents more easily. Other new features include user-specified parallelization for greater performance, extended support for XSD typing, and usability and productivity improvements in XML job editing through schema search and mapping intelligence.

2.   InfoSphere Information Server for Data Quality

• InfoSphere Data Quality Console

InfoSphere Data Quality Console is a new unified, browser-based interface that you can use to monitor and track data quality exceptions that are generated by InfoSphere Information Server products and components. Exceptions are entities that are generated by a condition or event and that might require additional information or investigation. For example, records that do not meet the conditions of data rules in InfoSphere Information Analyzer might be considered exceptions. The following screen capture shows how you can view a subset of exception descriptors by specifying search criteria, which include search terms and attributes.

• InfoSphere Information Analyzer

Predefined rule definitions

A key challenge in assessing and monitoring information quality is starting the process to validate key business requirements. Instead of starting that process without assistance, you can start by using predefined data quality rule definitions.

New installations of this release include more than 100 predefined rule definitions for basic and common domains. Also included are more than 60 predefined rule definitions that are designed to validate standardized address data. Although the rule definitions are optimized for US data, they can be modified for any country or region.

The data domains that are represented include the following domains:

·         Personal identity, such as age, date of birth, and national identifier

·         Asset identity, such as IP address information

·         Financial

·         Orders and sales

·         Data classification, such as identifier, indicator, code, date, and quantity

·         Completeness, which checks whether a field exists

·         Data format, such as alphabetic and numeric

·         Address data

User-named output tables for data rules

When you create data rules, you can specify that you want a user-named rule output table to be created in addition to the system rule output tables. User-named output tables can be simple or advanced. Use a simple table if you plan to use the rule output from one rule to create subsequent rules. Use an advanced table if you want to collect rule output from multiple data rules into one table. Also, you might want to create an advanced user-named table if you plan to use the rule output from multiple rules to create subsequent rules. An advanced user-named table is an additional physical table with copied records, which means that it requires additional storage space.

Distinct output records

You can now specify whether you want only distinct output records or all output records in the rule output table.

Task sequencing

You can now use task sequences to group multiple InfoSphere Information Analyzer jobs that are to be executed sequentially. In this release, task sequencing is available only by using the HTTP API and CLI, and only rules, rule sets, and metrics are supported for task sequencing.

 

• InfoSphere QualityStage

Standardization Rules Designer

The new Standardization Rules Designer provides an intuitive and efficient framework that you can use to enhance standardization rule sets. You can use the browser-based interface to add or modify classifications, lookup tables, and rules.

You can also import sample data to validate that the enhancements to the rule set work with your data. The following screen capture shows a part of the Standardization Rules Designer in which you can add or modify a rule by mapping input values from an example record to output columns. This rule splits concatenated values in an input address record by mapping each part of the input value to a different output column.

New rule sets

The following rule sets are now available:

·   The PHPROD rule set is a rule set for pharmaceutical data. The rule set demonstrates how you can use rules to standardize description data from the health industry.

·     The RUNAMEL rule set can be used to standardize Russian names.

·    The RUADDRL rule set can be used to standardize Russian addresses and area information.

Rule set enhancements

The predefined rule sets are enhanced in the following ways:

·         The domain-specific rule sets can be used with the Standardization Rules Designer.

·       The CNNAME, HKCNAME, and HKNAME rule sets now have special options for name processing.

·      The CNADDR, CNAREA, CNPHONE, HKADDR, HKCADDR, and HKPHONE rule sets now have user modification subroutines.

·       The CNPHONE and HKPHONE rule sets are enhanced in several ways. For example, input data can be converted to half-width characters.

Sample data available for predefined jobs and tutorial

Sample data is now provided for the predefined standardization jobs that you can use to generate standardized data and the frequency information for that data.The installation media also now contains sample data and other files that are required for the InfoSphere QualityStagetutorial.

3.   InfoSphere Metadata Asset Manager

Enhanced documentation of import and export bridges

Individual reference topics for each bridge contain prerequisites, frequently asked questions, troubleshooting information, and detailed help for each parameter. Individual PDF guides to using BI bridges contain customized information for imports from IBM Cognos, SAP BusinessObjects, Microsoft, and Oracle BIEE.

Mapping documents for each import bridge show how each metadata class in the source tool is displayed inInfoSphere Information Server.

Asset interchange and istool command line

The following functions are documented:

·         Exporting and importing InfoSphere Streams assets.

·         Exporting and importing InfoSphere Data Quality Console assets

·      Generating business glossary content from InfoSphere Data Architect glossary        models

·         Generating business glossary content from logical data models

InfoSphere QualityStage

To help you learn about the new Standardization Rules Designer, tutorials are provided that use data from the product and address domains.

New and updated topics provide information about the standardization process and standardization rule sets:

·         Standardization workflow

·         Developing rule sets

·         Enhancing standardization rule sets by using the Standardization Rules Designer

4. Common capabilities across the InfoSphere Information Server suite

•    Administering

New repository administration tool

The InfoSphere DataStage and QualityStage operations database and the InfoSphere QualityStage Standardization Rules Designer database are typically installed by the installation program unless you are using a database other than DB2 or unless you want to create them yourself.

To assist in the management of repositories that are not installed by the installation program, the RepositoryAdmin command line tool is provided. You can also use the RepositoryAdmin tool for other purposes, such as to assist you in relocating a repository to another server or to update a connection to a repository.

New database for InfoSphere QualityStage

The InfoSphere QualityStage Standardization Rules Designer is supported by an additional database for your Version 9.1 installation.

• Connecting to external sources

Stage for IBM Operational Decision Management

IBM Operational Decision Management allows customers to externalize complex business rules from applications. With the new ILOG JRules stage, you can invoke complex business rules within the context of a job.

InfoSphere Streams connector

The new InfoSphere Streams connector enables integration between InfoSphere Streams and InfoSphere DataStage. You can use the InfoSphere Streams connector to send data from an InfoSphere DataStage job to an InfoSphere Streams job, and also to send data from an InfoSphere Streams job to an InfoSphere DataStage job.

Unstructured Data stage

Use the new Unstructured Data stage to extract information, such as formulas or document authors, from Microsoft Excel files. The stage supports style sheets for .xls and .xlsx file types.

Java™ Integration stage

You can use the new Java Integration stage to integrate your code into your job design by writing your Java code using the Java Integration stage API. The Java Integration stage API defines interfaces and classes for writing Java code that can be invoked from within InfoSphere DataStage and QualityStage parallel jobs.

Support for new data sources

The following connectors and stages are now available:

·         DB2 connector for IBM DB2 for Linux, UNIX, and Microsoft Windows, Version 10.1.x

·         DB2 connector for IBM DB2 for z/OS  , Version 10

·         MQ connector for IBM WebSphere MQ, Version 7.1.x and 7.5.x

·         Informix stage for IBM Informix, Version 11.7

·         Streams connector for IBM InfoSphere Streams 3.0

·         Teradata connector for Teradata Database 13.10 and 14.0

·         Oracle connector for Oracle Database 11g Release2

·         Sybase stage for Sybase ASE, Version 15.7 and Sybase IQ, Version 15.4

·         Netezza connector for Netezza 4.6x, 6.0.x, and 7.0.x

·         ODBC connector for DataDirect ODBC, Version 7.0.x

·         ILOG JRules stage for ILOG-JRules 7.1.x and WODM 8.0.x

·         Big Data File stage for IBM BigInsight 1.4 and Cloudera CH4.0

• InfoSphere Blueprint Director

Publication of blueprints

Blueprints can now be published to the metadata repository of InfoSphere Information Server so that other users can view or use them.

InfoSphere Metadata Asset Manager

Import metadata by bridge from additional tools

Import support was added for the following tools and types of metadata:

·         CA ERwin Data Modeler 8. Logical and physical data models.

·      IBM Cognos, Version 10. Business intelligence (BI) models, BI reports, and related implemented data resources.

·         IBM InfoSphere Streams MetaBroker, Endpoints and tuples.

·       Oracle BI Enterprise Edition. Business intelligence (BI) models, BI reports, and related data resources.

Export metadata

You can now use the OMG CWM 1 XMI 1 bridge to export the contents of databases and database schemas to XML files that are compliant with the OMG CWM XMI file format.

Create and edit data connections

When you import by using a connector, you can now create a data connection, use an existing data connection, or edit an existing data connection. Data connections are saved to the metadata repository.

Automatic creation of metadata interchange servers

Metadata interchange servers that enable import from bridges and connectors are now created automatically during installation.

• InfoSphere Metadata Workbench

Enhancements in Manage Lineage utility

You can now select or clear InfoSphere DataStage projects to be included in lineage. Previously, the Manage Lineage utility included all jobs in a selected project. In addition, you can run the Manage Lineage utility on database views without selecting a InfoSphere DataStage project to link the database view to its source database table.

Integration with IBM InfoSphere Blueprint Director

You can browse, query, and display published blueprints. You can display the blueprint diagram.

Integration with IBM InfoSphere Information Analyzer

You can browse, query, and display published rule definitions and published rule set definitions.

You can browse, query, display, and include for lineage the InfoSphere DataStage Data Rules stage and its relationship to the published data rule.

Integration with Big Data platform

You can browse, query, display, and include for lineage the InfoSphere DataStage Unstructured Data, Big Data File, and Streams Connector stages.

Integration with IBM InfoSphere Business Glossary

You can browse, query, display, and assign assets to information governance rules and information governance policies.

You can query and display the new Is A and Has A term relationships. In previous versions, only the parent category of the term was displayed.

Integration with IBM InfoSphere DataStage

You can browse, query, display, and include for lineage the InfoSphere DataStage Java Client and Java Transformer stages. You can display additional database stage properties: the server, database, schema, and table properties of the stage.

You can display additional data file stage properties: the file and location properties of the stage.

Integration with IBM InfoSphere Data Click

You can browse, query, display, and include for lineage published Change Data Capture (CDC) subscriptions from InfoSphere Data Click. In addition, you can invoke the CDC subscription process from a blueprint diagram.

Importing assets into the metadata repository

You can generate database, data file, and business intelligence (BI) report assets from a CSV file for later import into the metadata repository.

• Migrating

New migration functions

To help you to migrate automatically, you can now use two new migration wizards. The wizards automate the process of exporting and importing databases, profiles, and directories that are associated with InfoSphere Information Server. The wizards collect information about your computer and InfoSphere Information Server configuration. The information is then used to export and import your system.

The migration wizards support all three server tiers: the services tier, the engine tier, and the metadata repository tier. When you export or imports by using the wizards, all tiers that are installed on the computer are backed up simultaneously.

5.   InfoSphere Business Information Exchange

• InfoSphere Business Glossary

Expanded enterprise information governance policies and rules

Now, in addition to creating and managing terms and categories, you can create and manage information governance policies and information governance rules. Information governance policies and rules describe the way that information should be used and managed to comply with business objectives. You can define relationships among the policies and rules and between the policies and rules and other metadata information assets.

Advanced term relationships

You can use new relationships between terms to express hierarchies of type and containment. The relationships enable consumers of the information to understand the meaning of terminology more fully, in the context of other terms.

Single sign-on for Windows users

Integration with Windows desktop authentication enables users who are logged in to Windows to work with InfoSphere Business Glossary immediately, without requiring a separate login process.

Web-based access to blueprints

You can now define information about blueprints and view published blueprints directly from the business glossary.

Dynamic display of external content from OSLC providers

OSLC (Open Services for Lifecycle Collaboration) is a method of communicating among different systems. InfoSphere Business Glossary can now be a consumer of OSLC services from Rational Asset Manager and Rational Software Architect Data Manager.

The metadata content that is stored in these OSLC providers is displayed dynamically in the business glossary. The dynamic display ensures that data is synchronized and eliminates the need for separate data transfer procedures.

Enhanced integration with InfoSphere Information Analyzer

In previous releases, you were able to view the results of table and column analysis, including valid values for columns. You can now browse, search, view details of, and assign published data rule definitions and data rule set definitions to business glossary assets.

• InfoSphere Business Glossary Client for Eclipse

Information governance policy and information governance rule assets

You can now browse, search, and display the properties of two new InfoSphere Business Glossary assets: information governance policies and information governance rules. You can assign an information governance rule to an asset, such as a database table, so that the information governance rule governs the asset.

Import and export of glossary assignments

Earlier versions supported import and export term assignments. In version 9.1, you can import and export glossary assignments, which include both, term assignments and information governance rule assignments.

Advanced term relationships from InfoSphere Business Glossary

Two new term relationships, Is A and Has A, are included in the Properties view of a term. You can view the supertype and subtype relationship between terms in the Term Type Hierarchy view.

Business Process Modeling Notation (BPMN) model elements

You can now view and remove term assignments in BPMN model elements that are displayed in IBM Rational Software Architect. With the Business Process Model Integration API, you can build functions to add, remove, and get term assignments to BPMN model elements.

Local indexing

Local term assignments and local information governance rule assignments are now indexed to improve search and display performance.

6. Documentation introduced or enhanced with Version 9.1

Introduction to InfoSphere Information Server

This information is more complete and streamlined to help you understand how the suite and its components interact. Diagrams show where each component fits in the suite architecture, and scenarios explain how each component might be used to solve real business problems.

InfoSphere Business Glossary

New topics provide information about populating your business glossary by using the command line:

·        Generating business glossary content from InfoSphere Data Architect glossary model (*.ndm) files.

·         Generating business glossary content from logical data models

InfoSphere DataStage

The quality of information is improved and task steps are clarified in the InfoSphere DataStage tutorial.

More troubleshooting information, with focus on client login and job runtime issues, is provided. The enhanced troubleshooting information includes information about specific operating systems and information about how to prevent errors.