Mainframe Datatype Scrubbing

Jul 20, 2005

I have a million record mainframe flat file that I BULK INSERT into a
SQL table with CHAR(fieldlength) deined for every column to prevent
import errors.

Once imported I "INSERT INTO ... SELECT
dbo.MyScrubFunction(columnN),..."
My scrub functions will take for example a char(8) YYYYMMDD date field
and return either a valid datetime variable or a NULL for 8-spaces or
8-zeros....or return a MONEY datatype by dividing by 100.

PROBLEM: This is extremely SLOW!
QUESTION: Should I do multiple "UPDATE ...SET" statements back into
CHAR() columns, then let SQL Server do the CHAR() to DATETIME and
MONEY conversions itself? What is the most efficient or recommended
method to transform/scrub imported data?

P.S. I also have to convert low-values and reformat dates which I
also use my own UDFs for.

THANKS

View 1 Replies


ADVERTISEMENT

SSIS - Migrating Mainframe Data To SQL 2005 Only Mainframe Group Level Exposed

Apr 25, 2007

In using SSIS to migrate data from mainframe to SQL 2005, I had a situation where only group level data was exposed through the ODBC to SSIS, so I pulled this information as varchar on the SQL destination side. Now I would like to break that group into the individual numeric columns I need on SQL Server. However, the positive and negative sign did not convert because it came of character. I can write something to convert the positive signs to positive numbers; however I cannot do the negative because I would need get rid of the leading zeros in order to place the negative sign before the number. Is there anything I could have done to get SSIS to do the conversion like it did for every one-to-one mapping?

View 9 Replies View Related

Scrubbing Data

Dec 20, 2007

I dont know if anyone else uses this, but I'm trying to scrub a database on my dev server. The procedure is called uspScrubDatabase. When I execute this I get an error message:

Server: Msg 201, Level 16, State 4, Procedure uspScrubDatabase, Line 0
Procedure 'uspScrubDatabase' expects parameter '@ErrorCode', which was not supplied.

The guy that usually does this is on vacation so I have no one to ask.

Can anyone assist?

Thanks.

View 3 Replies View Related

Data Scrubbing Problem

Apr 24, 2008

Hello,
Consider the following string:

EHILLIS /CGROGAN

How would you clean that up to produce the following result:

EHILLIS /CGROGAN


I have tried a couple of things with REPLACE, but nothing has worked. The name of the column is 'Rep'.

Any ideas?


Thank you for your help!

cdun2

View 5 Replies View Related

FTP From A Mainframe (z/os) Box

Sep 30, 2005

Has anyone been able to use an ftp task to pull a flat file from a z/os mainframe? The ftp task appears to want the remote file to begin with a /, which pulls you into unix system services on z/os.

View 7 Replies View Related

FTP From A Mainframe (z/os)

Mar 29, 2006

Being that this issue first surfaced September 2005 and it's now six months later (March 29, 2006), has the problem with recognizing VMS datasets (not requiring a '/') been addressed?

View 2 Replies View Related

DTS - How To Get DB2 Data From Mainframe?

Jun 15, 2000

We would like to use DTS to pull data from DB2 on the mainframe into SQL Server 7.0 databases. Does anyone know what 3rd party products are required for us to do this (and which ones are the best to use)? What has to be installed on the mainframe and on the SQL Server?

View 2 Replies View Related

Replication To Mainframe Db2 OSZ

Apr 22, 2004

What would I need in order to setup replication between SQL2000 SP3 and MVS Subsystem running Db2 database . I mean, does it need any 3rd party tool or regular replication setup between SQL and Db2 is enough

View 1 Replies View Related

How To FTP To Host(Mainframe)?

Jan 24, 2008

Hi,

I am trying to send files over to an ftp server on Mainframe using scripts that is provided in this link, but I can't get it to work.
The problem is the sendfile method concatinates the source file name to the MVS dataset name which causes the issue.

Here's my code (FTPConn is a connecation mgr for FTP):


Imports System

Imports System.Data

Imports System.Math

Imports Microsoft.SqlServer.Dts.Runtime



Public Class ScriptMain


Public Sub Main()



Dim mgr As ConnectionManager

mgr = Dts.Connections("FTPConn")



Dim conn As FtpClientConnection

conn = New FtpClientConnection(mgr.AcquireConnection(DBNull.Value))



Dim toFiles As String

toFiles = "ftpp.a.b.xxxxx.yyyyyy.int"

Dim fromFiles(0) As String

fromFiles(0) = "c: est.txt"

Try


conn.Connect()

conn.SendFiles(fromFiles, toFiles, True, True)

Catch ex As Exception





Finally

conn.Close()

End Try





Dts.TaskResult = Dts.Results.Success

End Sub



End Class



Output:


SSIS package "Package.dtsx" starting.

Error: 0xC001602A at Package, Connection manager "FTPConn": An error occurred in the requested FTP operation. Detailed error description: 200 Representation type is Ascii NonPrint

200 Port request OK.

501 Invalid data set name "ftpp.a.b.xxxxx.yyyyyy.int/test.txt". Use MVS Dsname conventions.

View 13 Replies View Related

DTS LOADING FROM MainFrame To SQL Server

Sep 18, 2000

hi
I am loading from Mainframe to Sqlserver.
Right now i am truncating the tables in sqlserver connecting to mainframe and downloading the tables to Sql server. The problem is sometimes i fail to connect to Mainframe. So is there any way of checking whether i can connect to mainframe and if succesful truncate the tables.
Thanks

View 1 Replies View Related

DTS FTP Task From Mainframe Problem

Aug 30, 2002

I am trying to use the FTP Task within a DTS Package to copy a file from the Mainframe (OS390). We have SQL2000 with SP2 running on NT. This process works when I do it manually with ftp. At first, when I would execute the package it would just hang like it is copying but it would never end.

Then, I added a Dynamic Properties Task to the package. I used the Dynamic Properties Task to set up a Constant with the actual name of the Source File located on the Mainframe. When I ran this I received "Invalid string format for source file name".

Next, I removed the Dynamic Properties Task from the DTS Package, saved the package, and executed it. The package executed successfully, but when I looked in the folder on my hardrive the file that was supposed to be copied was not there. In order to get the package to execute successfully it appears that I have to go through the following process: add the Dynamic Properties Task, set up a constant to change the Source File name, save the package, delete the Dynamic Properties Task, save the package and then execute it.

Also, on one execution of this package, it brought across the file from the Mainframe to my hard drive but the file was in Binary format.

What do I have to do to get the FTP Task within a DTS Package to work? Would you specify in detail exactly how the Dynamic Properties Task needs to be set up? How can I specify how to change the file format from Binary to ASCII in the DTS Package?

Thanks, in advance.

View 1 Replies View Related

Any Way To Build A Bridge To DB2 Mainframe?

Jan 18, 2008

Hi,

is there a solution outside for getting of DB2 for OS/390 into SQL server?
I mean not replication or copying of data with flat files or an ETL tool, but a kind of integration of DB2 tables as they would be "normal" mssql tables.

Oracle has Transparent Gateways for os/z DB2 and many other non-Oracle databases, is there some similar for SQL Server?

Could it be realisable with an ODBC client for DB2?

Thanks

View 1 Replies View Related

Replication To Mainframe Db2 OS390

Feb 26, 2004

Can I get drivers to setup Replication to Mainframe . I really don't want to install HIS

Thanks

View 4 Replies View Related

Cannot Import File From Mainframe

Mar 8, 2006

Hello Everybody!
Im trying to migrate my SQL 2000 packages that“s currently working in the company production enviroment to SSIS packages. So, in the 2000 version i got the flat file from Mainframe and i had to do a trick to transform all the columns to match the same size as the example above:

1432 1138734217 1144256628<CRLF>
1432 1138734217<CRLF>
1433 1136657788 1122441177<CRLF>
1433 1125554545 1122441177<CRLF>
1433 1192925544 1122441177<CRLF>

So, when i import the file, first of all i have to transform the Text file to another texfile file fixing the size to 32..... but you can see that in the second row im receiving a CRLF and if i try to import without the trick, the preview of the ragged file show me the columns desorganized...

Someone knows how to import it without transform to another text file with fixed length?
Thanks
Cleber

View 2 Replies View Related

Extract From Table - Send To Mainframe

Jan 24, 2007

Hi,

I want to extract all fields from a table and ftp the output file to the mainframe. My problem is the columns in the extracted file do not line up, which is required by the mainframe program. Example column would be this column of last names:
Smith
Jones
Sullivan

How could I format the extracted data to look like this:
Smith
Jones
Sullivan

Thanks,
Jeff

View 8 Replies View Related

Script/FTP Task A Mainframe Destination

Sep 21, 2007

Hello,

Has anyone had any success sending or receiving file(s) from either Script or FTP task? I've Google and found examples and no luck for me. The main idea is to send a file from local PC/server to mainframe.

Username: imtheuser
Password: pwd
Source file: c:myfile.txt
Destination: 'PROD.ABC.DAILY.BATCH'

I've used this workaround, SSIS Script Task but no good.

SQL Server
Feedback Workarounds
281893. SSIS FTP Task - Mainframe
When you try to connect to a mainframe (os / 390) to ftp receive a file you get an error message stating that the path does not begin with a "/".
Active feedback entered 6/7/2007 by EWisdahl

Entered by EWisdahl on 6/7/2007

Add a script task (as follows) to download the desired files...
' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

Public Class ScriptMain
' The execution engine calls this method when the task executes.
' To access the object model, use the Dts object. Connections, variables, events,
' and logging features are available as static members of the Dts class.
' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
'
' To open Code and Text Editor Help, press F1.
' To open Object Browser, press Ctrl+Alt+J.

Public Sub Main()
Try
'Create the connection to the ftp server
Dim cm As ConnectionManager = Dts.Connections.Add("FTP")
'Set the properties like username & password
cm.Properties("ServerName").SetValue(cm, "myServer")
cm.Properties("ServerUserName").SetValue(cm, "myUserName")
cm.Properties("ServerPassword").SetValue(cm, "myPassword")
cm.Properties("ServerPort").SetValue(cm, "21")
cm.Properties("Timeout").SetValue(cm, "0") 'The 0 setting will make it not timeout
cm.Properties("ChunkSize").SetValue(cm, "1000") '1000 kb
cm.Properties("Retries").SetValue(cm, "1")
'create the FTP object that sends the files and pass it the connection created above.

Dim ftp As FtpClientConnection = New FtpClientConnection(cm.AcquireConnection(Nothing))
'Connects to the ftp server
ftp.Connect()
ftp.SetWorkingDirectory("MyFolder.MySubFolder.MySubSubFolder")

Dim files(0) As String
files(0) = "MyfileName"
ftp.ReceiveFiles(files, "C: emp", True, True)

' Close the ftp connection
ftp.Close()

'Set the filename you retreive for use in data flow
Dts.Variables.Item("FILENAME").Value = maxname
Catch ex As Exception
Dts.TaskResult = Dts.Results.Failure

End Try

Dts.TaskResult = Dts.Results.Success

End Sub
End Class


Thanks again.

View 5 Replies View Related

SSIS Is Truncating Data Being Migrated From Mainframe

Feb 13, 2007

I have data on the mainframe that contains special characters such as a copyright symbol (letter c with a circle) when running the SSIS package migration from mainframe database to SQL Server 2005 database the data in the column is truncated at the point it hits this symbol with no errors. I had error reports set up for truncation and did not see an error report for this.

I have tried to change the data type to different data types and re-run nothing changes things.

The symbol is a EBCDIC 'b4' on the mainframe side - I have no idea what that would translate to.

Anyone have any ideas? Am I stuck with substituting something else for the symbol on the mainframe side and re-running the package? Is there anyway to handle this situation in SSIS? I am using the Import/Export Wizard for this one because before I ran into this problem it seemed to be working fine.

Thanks, MaryOS

View 3 Replies View Related

T-SQL (SS2K8) :: Varchar Datatype Field Will Ignore Leading Zeros When Compared With Numeric Datatype?

Jan 28, 2015

Need to know if the varchar datatype field will ingore leading zeros when compared with numeric datatype ?

create table #temp
(
code varchar(4) null,
id int not null
)
insert into #temp

[Code] .....

View 4 Replies View Related

Scrubbing Data In A Data Flow Task

Mar 1, 2007

I can only assume that I either did something very stupid, the examples are completely wrong, or I don't have the slightest clue what it is that I'm doing.

I'm trying to do one of those incredibly complicated things in SSIS that was a brain-dead, point click operation in DTS.  I want to strip characters out of an input stream from a flat file source before loading them into a table.  I've defined my data flow task.  I have my flat file source properly defined.  I have my SQL Server destination properly defined.  For the life of me, I can't figure out how to scrub the data that I want.

My input is a csv file that has 22 columns of data.  Column 22 contains data and if data was not present, there is a - inserted.  So, I want to strip any - from the input before it goes into the column.  So, I grabbed a derived column task and drug it out on the surface of the designer.  I then set the derived column name to Column 22, the derived column to 'Replace Column 22', and tried the following in the expression section:

REPLACE(Column 22, "-", "" )

REPLACE( (DT_WSTR, 50) Column 22, "-", "" )

Both of these threw some really nasty error messages when I clicked the OK button.  The really irritating part is that I exactly followed the examples in BOL.REPLACE(Product, "Bike","")REPLACE((DT_WSTR,8)DaysToManufacture,"6","5") I know I'm not dreaming this, but it absolutely refuses to accept my expression.

View 6 Replies View Related

SSIS Leading Zeros Gone After Migration Of Mainframe To SQL Server

May 22, 2007

I am migrating mainframe data to SQL Server 2005 and have found that from a mainframe character field with leading zeros for example the value of 00023 to a SQL Server column defined as varchar (5) the resulting column value is 23 not 00023. I need the leading zeros because these are account ids, etc. So the value is 00023 not 23. Is this some setting in SQL Server 2005 that needs to be changed or what? This is not a numeric field on the mainframe or a numeric column for SQL Server.

View 11 Replies View Related

FTP Task. / Validation Makes Mainframe Downloads Impossible.

Jul 24, 2007

When attempting to use the FTP task to download a file from a Mainframe system the task fails stating the filename as invalid because it doesn't begin with a "/".

Adding the slash to the front of the file name causes the mainframe to be unable to locate the file.

Commandline version of FTP.exe mimicks this behavior by working perfectly when the filename has no slash, and being unable to find the file when the slash is present.

Why does microsoft force a filename to start with a "/" and is there a way to make SSIS skip the validation phase for the FTP task?

View 7 Replies View Related

Numeric Datatype To Ssis Variable Datatype Conversion Problem

Apr 24, 2008



Good afternoon,

I have an issue with an ssis variable datatype.

The scenario is as follows:

I have a stored procedure:


PROCEDURE [dbo].[sp_newTransaction]



@sourceSystem varchar(50),

@txOut NUMERIC(18,0) OUTPUT

AS

insert into scn_transaction (sourceSystemName) values(@sourceSystem);

SELECT @txOut = @@identity


Whose purpose is to perform an insert into a table and return me the identity value of the inserted record, which I'll then use throughout the rest of my package. The identity column in the inserted table is numeric(18,0).

I execute the stored proc with the following sql with an OLE DB connection manager:

exec sp_newTransaction ?, ?

The first parameter is a string variable from earlier in the package, and the second is the output parameter. I have the following parameter mappings to the execute sql task:

User:ystxId output numeric 1 -1
User:ourceSys input varchar 0 -1

The proc is correctly called, and the row insesrted, however I get a type conversion error when SSIS attempts to map the return parameter to my package variable... I've tried all sorts of combonations, and can't seem to get it to execute.

At one point I wasn't returning a numeric, but rather an int from the stored proc, and all was well until I went to use the variable in a derived column later in the package, and the type was converted quite incorrectly (a 1 was 77799789080 or some such), indicating a type conversion error likely related to the encoding of the number.

I'd like to keep the datatypes as numeric and make ssis use those - any pointers are greatly appreciated as to what type my package variable should be to allow proper assignment of a sql server numeric type to it.

Thanks much,

B

View 6 Replies View Related

Destination File With Multiple Record Types And Sequences - Mainframe-like

Jul 20, 2007

Howdy all,



I've seen several posts about reading and writing files that have different record types with varying column metadata. My particular file has 11 record types plus several header types and looks something like:



<Header1>

<Header2>

<Detail01-#1>

<Subdetail02>

<Subdetail03>

...

<Detail01-#2>

<Subdetail02>

<Subdetail03>

...

...



Since i need to get different detail and subdetail records, i can't really use the technique of 3 dest file connection managers found in http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=87269&SiteID=1



I've tried using an exec sql to get the main detail records and then a forech ADO en umerator that would get the subdetails, but it all seems so kludgy. I'm starting to think that I should just write the bulk of the file creation code in a c# app instead of trying to smush this into SSIS. Opinions? Am I missing some trick in SSIS?



TIA,



-Peter

View 7 Replies View Related

Convert Char Datatype To Datetime Datatype

Sep 17, 2003

Database is SQL Server 2000

I have a field in a table that stores date of birth. The field's datatype is char(6) and looks like this: 091703 (mmddyy). I want to convert this value to a datetime datatype.

What is the syntax to convert char(6) to datetime?

Thank you in advance.

View 1 Replies View Related

Modify Nvarchar Datatype To Datatime Datatype

Mar 14, 2008

Hi,

I imported a table from Accees to SQL 7 with data in it.
I need to modify one of the datatype columns to "datetime" from nvarchar.

I tried to convert it manually, in SQL Server Enterprise Manager tool, but it gave me an error.

I also tried, creating another column "DATE2-datatype:datetime" and updating the column with the old one.

UPDATE users SET DATE2 = DATE.. But it also faild,..

How can I modify the column?

Thank you.

View 10 Replies View Related

Reporting Services :: Exporting Fixed Columns From SSRS For Use In Mainframe Data File

Jun 3, 2015

I need to be able to export a data file asĀ flat file (.txt) with fixed columns for use by Mainframe.

I will be uploaded this file using the Windows File Share Option

Render Format does not have .txt, but does have a data feed option. So I will try that.

But, I do not see an option for fixed column width.

View 7 Replies View Related

Data Scrubbing (Data Cleaning)

May 19, 2008

Hi All,

I am new to SQL Server. I want to know what is data scrubbing (data cleaning). What are the different tools that can be used for data scrubbing.
Where can i find good tutorial or data for data scrubbing?

Thanks

View 2 Replies View Related

Converting INT Datatype To BIGINT Datatype

Dec 15, 2005

HI,I have a table with IDENTITY column with the datatype as INTEGER. Nowthis table record count is almost reaching its limt. that is totalrecord count is almost near to 2^31-1. It will reach the limit with inanother one or two months.In order to avoid the arithmentic overflow error 8115, we would likechange the datatype from INT to BIGINT. we hope this will solve ourproblem.How do I approch this datatype conversion?. Since the data count ishuge, that leads to a long down time of database.we need better approach or solution for this problem?. kindly give mea better solution that will reduce the total downtime of the productiondatabase.?.Regards

View 1 Replies View Related

Text Datatype Vs Nvarchar Datatype

Feb 25, 2008



Hi guys..

i have so doubts in my mind and that i want to discuss with you guys... Can i use more then 5/6 fields in a table with datatype of Text as u know Text can store maximu data... ? acutally i am trying to store a very long strings values into the all fields. it's just popup into my mind that might be table structer would not able to store that my amount of data when u use more then 5/6 text datatypes...

and another thing... is which one is better to use as data type "Text" or "varchar(max)"... ?
if any article to read more about these thing,, can you refere to me...

Thanks and looking forward.-MALIK

View 5 Replies View Related

Which Datatype Should I Use?

Jan 26, 2006

I have a USERS database table with a column called IsValidated.  It identifies if a user has been validated.
I'd like to know which datatype would be the most efficient for this column.
Our DBA made it a varchar(1) column.  Would using int be better?
Please advice.

View 4 Replies View Related

What Datatype??

May 14, 2006

I have a message box or multiline textbox on my form and I am storing into an SQL DB. I currently have it as a VARCHAR, but that doesn't seem to store the carriage returns. Do I have to store them as a nvarchar, text, ntext or something else to make the returns show? I am pulling them out of the DB and showing them in a label where I replace the with a <br>.
 
tempMessage = Dr("Message")
message.Text = tempMessage.Replace("", "<br />")

View 2 Replies View Related

64 Bit Datatype

Nov 23, 1998

Is there a datatype that I can use to store a 64 bit unsigned int? I'm using SQL 6.5. Was thinking of using the standaed Interger datatype.

View 2 Replies View Related

Datatype

Oct 18, 2004

I am trying to convert a NVCHAR datatype to a DATETIME data type is that possible???

View 9 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved