I have thousands of records, which are .txt format, to be loaded to the DB and need to mining them. The only way for that is to write a program and read them to the DB or can I load them directly? And also I have to design the tables based on these data myself, right? Thank you.
HI,
As the source data of the cube is from MySQL, and the source data volume is more than 80M row per month, I have to build multiple partition in the cube, each partition contain only one month data, even though, the time to load data directly from MySQL is still too long, and because the mysql .Net provider and is not mature enough, the connection often break while loading, so I have to try to load from flat file which was exported from MySQL.
The Partition Processing Destination seems support this way, but, even it shows the partition name in component editor, it actually process the partition with partition ID, and there's not any way to change destination partition name for this component.
Unless I have to change the SSIS package every month, looks like it is impossible to make a smart ETL program that can dynamic create new partition with the ID and name as YYYYMM every month and load data directly from the flat csv file exported from big MySQL table.
Does anyone know how to load data from flat text, and also support dynamic destination partition name?
And I also find a bug in Partition Processing Destination, even 2005 SP2, if there's more than one cube in a SSAS database, and if two partition name in different cube is same, in component Editor, you can not set mapping to the second one with the same name, even you point to the second one, and click ok, the next time you open the editor, you will find it high light at the first one. And even it shows name in editor, it actually process with partition ID instead of partition name, this make it is not possible by change the partition name which need to process to a constant name, say currentMonth to force the component process it.
Thanks.
Jun
Hi everyone,
I have to extract, dayly a list of contacts on a exchange server in a table on our EDW on sql server 2005. Is it possible to get the information directly from a dataflow or i will have to developpe a script task ?
Need help desperatly !!!
Hi,
Here I will describe my problem.
1. We are loading large amount of data from database on background thread which is starting on Application_start event in global.aspx.cs file.The data is later cached for subsquent request to improve the performance.
2. Now when we put the application on web farm garden, it is not able to load the application.
3. We are sending the request the servers through Router kind of application.
4 This application is working fine on single server enviornment.
Please help us.
Ajay Kumar Dwivedi
I just have done the SSIS example in the tutorial document included when install SQL 2005 ENT. I have a problem that whenever I test to run, the service load all data from source with out noticing about the data (I mean it load all the data to the destination), I do it several time and it continue to load all without checking. That mean the data is dublicated when the schedule run???
I think there should be a paramete or something like that to help the engine just load the new data to the destination. Could you help please?
Thank
First let me give you a little back ground on me. I'm very new to the ASP, ASP.NET, Visual Studio, Sql Server, Frameworks....thing. I am coming over from a PHP/MySql background of over 5 years. The change over to VBScript and VB has been to tough and I have a basic liking for the Visual Studio 2005 and the ease of putting things together. However, I have come across something that to me seems like it should be relatively simple, but haven't been able to find the documentation or samples to describe what I'm looking to do.
Rough Need:
1. Start with a form will a couple labels and a singe textbox to get the lookup date from an user.
2. Query one table/view based on the users choice of date and select only one field of returned data.(Doing this by itself is not a problem and I can display my results in a Gridview, but this is where it starts getting tricky and the gridviews won't work for me.)
3. If there is something returned, I need to start a HTML table layout or possibly some form of a Gridview(I don't see how I would use the Gridview) and start a loop, adding the first returned row from this query into the first cell.
4. Now, based on that same User Date and the returned row value from the previous query, I need to query another Table/view and return another single field, which might return 1 or multiple rows, which I need to start a loop to display unique items in cells under the first on above.
5. Based on the original User Date and each returned row from the first query, I need to query two other Table/views and get some additional information. 4-5 fields will be returned and be displayed on one row in the number of necessary columns.
6. This would finish the first row from the first query, so I would need to loop back up to see if there were any further results and continue looping until the get to the last of the results from the first query.
7. Finally, I would just need to close up the Table/gridview.
The basic results I'm looking for would be similar to the following:
Results for 10/11/2005
Field name 1
Field name 2
Field name 3
Field name 4
Field name 5
First Result from query 11 of 4 results from Query 2, but only 1 unique result
Info from Query 3
Info from Query 3
Info from Query 3
Info from Query 4
Info from Query 4
Info from Query 3
Info from Query 3.
Info from Query 3
Info from Query 4
Info from Query 4
Info from Query 3
Info from Query 3
Info from Query 3
Info from Query 4
Info from Query 4
Info from Query 3
Info from Query 3
Info from Query 3
Info from Query 4
Info from Query 4
Second Result from query 1First result from Query 2 after looping though the first query Second result from Query 2 after looping though the first query
Info from Query 3
Info from Query 3
Info from Query 3
Info from Query 4
Info from Query 4
Info from Query 3
Info from Query 3.
Info from Query 3
Info from Query 4
Info from Query 4
Info from Query 3
Info from Query 3
Info from Query 3
Info from Query 4
Info from Query 4
Info from Query 3
Info from Query 3
Info from Query 3
Info from Query 4
Info from Query 4
It seems me from what I have read and slowly figuring out, is that I should be able to directly access the DataSet returned by four SqlDataSources, one for each of the above querys and then just write my own VB to handle the necessary looping, table format and such. I can easily add the for SqlDataSources to the page and add a Gridview for each one and get 4 separate chunks of info, but can't see a way with GridViews to intermingle the info like displayed above.
So, if it can be done with Gridviews, then I would love to see how that is done. But, if someone could explain to me how I access the DataSets directly that I get from the 4 SqlDataSources, then that would be ok too. I have figured enough out with VB, that I can write the code to do my looping requirements, if I can just access the information I get back. Thanks for reading all the way through this long post.
Jack
hi
im new to Sql server so this might be a daft question
is there something where i can put data directly into a table (i.e entering data into a grid) or do i have to write the sql every time
Can I connect directly to Outlook using SSIS? If so is there any white paper or walkthroughs I could follow?
TIA
Tom
hi lads,
what i need to do for my project are as following:
a mobile user send data via GPRS to SQL server Database. then i need to have a method to detect while a particular table is being inserted. and then extract data from table construct a report dynamiclly. what should i do to achieve this goal? e.g. window application, store procedure or trigger ?
PS : client side is a mobile application developed using MCL. i don't in which way he will send all data to SQL server Database, so what i need to do is to monitor new data inserting.
2) how to auto generate and print report directly after record been inserted into the table ? Do i need to import report web service API ? if yes, which one? or i can use other methods e.g. predefine report control view in my window application, turn off pop-up menu while printing a report(I guess)
thanks
We have a SQL database that uses Active Directory with Windows Authentication. Can users that are members of the Active Directory group that has read/write access to the SQL database create ODBC connections to access the database directly and update the data? They dont have individual logins on the server. They are only members of the Active Directory group that has a login?
View 1 Replies View Relatedhow I can load the CSV file data into the sql server table. I know there are ways like bulk insert and other to load the csv file data into the table. But in my case the table doesn't exist and has to be created at the run time. With simple insert in temp table we do like select * into #temp from tablename and that creates the temp table. So. I need something like that which create the temp table and load the data into it. because the CSV file would have different number of columns and names so I can not create the table structure in advance. I have to create the table at run time.
View 3 Replies View RelatedI have an source file and i have to load it into the data base by changing datatype of the columns in ssis
View 1 Replies View RelatedHi!
We make complete data load once a week. Now we need to make additional steps to append data on daily basis. We have primary key on the table and it doesn't allow appending duplicate rows.
What steps should we create to append data?
Thank you,
Elena.
Hi everobody!
When I run :
LOAD DATABASE db1
FROM DISK = 'c:mssqldatadb1_backup.dat'
go
I got error message:
Msg 3201, Level 16, State 1
Can't open dump device 'c:mssqldatadb1_backup.dat', device error or device off line. Please consult the SQL Server error log for more details.
How can I fix the problem?
Thank you.
Alona
I'm needing to load data on the client side to load into a DataGrid. I decided to use excel to load the data, but it doesn't need to be. My problem is that it only loads from the server not the client. I browse to find the file and get the path with a control named ctlFindFile. A button labeled ctlLoadData will when pressed display the pathway in label1 and also place the pathway in the function GetDataFromExcel which returns a dataset from the spreadsheet and displays the data in a dataset. Data is returned fine if I’m on the server, but when I'm on a remote machine I receive an error. That is unless I've place a spreadsheet with the same name and pathway on the server as on the client machine, then I’m able to load the file. Now how do I get it to load from on the client machine?
Code below:
Private Sub ctlLoadData_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ctlLoadData.Click
Label1.Text = ctlFindFile.Value
DataGrid1.DataSource = GetDataFromExcel(ctlFindFile.Value, "SampleNamedRange").Tables(0)
DataGrid1.DataBind()
End Sub
Public Function GetDataFromExcel(ByVal FileName As String, ByVal RangeName As String) As System.Data.DataSet
'Returns a DataSet containing information from a named range in the passed Excel worksheet
Try
Dim strConn As String = "Provider=Microsoft.Jet.OLEDB.4.0;" &_ Data Source=" & FileName & ";Extended Properties=Excel 8.0;"
Dim objConn As New System.Data.OleDb.OleDbConnection(strConn)
objConn.Open()
' Create objects ready to grab data
Dim objCmd As New System.Data.OleDb.OleDbCommand("SELECT * FROM " &_ RangeName, objConn)
Dim objDA As New System.Data.OleDb.OleDbDataAdapter
objDA.SelectCommand = objCmd
' Fill DataSet
Dim objDS As New System.Data.DataSet
objDA.Fill(objDS)
' Cleanup and return DataSet
objConn.Close()
Return objDS
Catch ex As Exception
' Possible errors include Excel file already open and locked, et al.
Return Nothing
End Try
End Function
Error on Client:Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.NullReferenceException: Object reference not set to an instance of an object.Source Error:
Line 31: Private Sub ctlLoadData_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ctlLoadData.Click
Line 32: Label1.Text = ctlFindFile.Value
Line 33: DataGrid1.DataSource = GetDataFromExcel(ctlFindFile.Value, "SampleNamedRange").Tables(0)
Line 34: DataGrid1.DataBind()
Line 35: End SubSource File: C:InetpubwwwrootSAI_LoadWebForm1.aspx.vb Line: 33 Stack Trace:
[NullReferenceException: Object reference not set to an instance of an object.]
SAI_Load.WebForm1.ctlLoadData_Click(Object sender, EventArgs e) in C:InetpubwwwrootSAI_LoadWebForm1.aspx.vb:33
System.Web.UI.WebControls.Button.OnClick(EventArgs e) +108
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +57
System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +18
System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +33
System.Web.UI.Page.ProcessRequestMain() +1292
Version Information: Microsoft .NET Framework Version:1.1.4322.2032; ASP.NET Version:1.1.4322.2032
Hi foks!
In MYSQL I can insert a whole text/excel/csv file with the Load Data command. Can I do such a thing for MSSQL?
Thank you!
I have a user that is loading data via an Access load procedure to a table that actually is a SQL Server 2005 table but is linked to the Access database. He is saying the load is extremly slow. How can I monitor what it is doing on the SQL Server side?
Thanks!
duckman2007
Have a great day!
Hi foks!
In MYSQL I can insert a whole text/excel/csv file with the Load Data command. Can I do such a thing for MSSQL?
Thank you!
Hi,I'm extracting data from a mainframe application with a view to loadingit into a MS SQL database. I'm trying to determine the most efficientway to format the mainframe extract file to make loading into thedatabase easier.The problem I have is that the existing record structure includes anarray that can vary between 1 to 50. If I include this array in asingle record the table I use to import the data would need 50 columnsthough not all these would be populated. There is a field in the recordto identify how many occurances of the array there are.Current Record Structure :Account NumberAccount NameOther Account DetailsTotalNumberOfArrayFieldsPopulatedArray :Value1Value2Value3....up to Value50 (if required)i.e.12344,Mr Agent,$29.95,2,BX123,BX12412345,Mr Jones,$14.95,3,XX123,XX124,XX12512345,Mr Jones,$14.00,1,XY12312345,Mr Jones,$15.95,2,XZ124,XZ12512346,Mr Smith,$19.95,3,AX123,AX124,AX12512346,Mr Smith,$19.00,1,BY12312347,Mr Acant,$99.95,7,CX123,CX124,CX125,CX126,CX127,CX128 ,CX129There may be up to 3 records created for each Account Number withdifferent values in the array fields.Am I better to break this file into two files .. one with the corecustomer information and a second file with a row for each array valuewhich has a link to the customer information file.OrIs there a way to efficiently process the original file once it isloaded into the staging tables in the database ?i.e.File 1 - Core Customer Information====================================Current Record Structure :Record NumberAccount NumberAccount NameOther Account DetailsTotalNumberOfArrayFieldsPopulatedFile 2 - Array Information====================================Record NumberArray :Value1Value2Value3....up to Value50 (if required)File 1========================12344,Mr Agent,$29.95,212345,Mr Jones,$14.95,312345,Mr Jones,$14.00,112345,Mr Jones,$15.95,212346,Mr Smith,$19.95,312346,Mr Smith,$19.00,112347,Mr Acant,$99.95,7File 2========================12344,BX12312344,BX12412345,XX12312345,XX12412345,XX12512345,XY12312345,XZ12412345,XZ12512346,AX12312346,AX12412346,AX12512346,BY12312347,CX12312347,CX12412347,CX12512347,CX12612347,CX12712347,CX12812347,CX129At times the individual array values will be used for look ups thoughessentially the Customer Information record will be the primary lookupdata.I'm leaning toward changing my COBOL code and creating the 2nd outputunless someone can suggest a simple way to process the information onceloaded into the table.Any help that could be suggested would be greatly appreciated.
View 1 Replies View RelatedHi,
I have a little experience with SSIS (but a lot with DTS) and none with CRM 3.0.
Is it possible to use SSIS to import flat files into CRM 3.0 ?
And how ?
I read something that CRM 3.0 uses SQL server 2000. Should I upgrade it SQL server 2005 ?
Any suggestion where to read about this ?
Thanks in advance
Constantijn Enders
Hi All,
I am using Bulk Insert task to laod data from .dat file to SQL table but getting an error below.
[Bulk Insert Task] Error: An error occurred with the following error message: "Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.The bulk load failed. The column is too long in the data file for row 1, column 1. Verify that the field terminator and row terminator are specified correctly.".
Any help will be appreciated.
Thanks.
I tried to load data from the table in different server.
If I just want to limit one year data ( using date_key in that table ) then what task do I need to do that?
Please let me know.
Thanks.
Hi,
How to extract and Load Data using ADO.NET in SSIS.i hope to extract data we have DataReader source .but how to load (Insert) data with ADO.NET ?.and is ADO.Net quicker than OLEDB ?
Thanks
Jegan.T
Hi all,
I have some data in a CSV file that looks like this:
<Column Name1>, <Column Name2>, <Column Name3>, <Column NameN>
<Column Value1>, <Column Value2>, <Column Value3>, <Column ValueN>
<Blank Line>
<Column Name1>, <Column Name2>, <Column Name3>, <Column Name4>, <Column NameN>
<Column Value1>, <Column Value2>, <Column Value3>, <Column Value4>, <Column ValueN>
The rub is that I don't know how many columns I am going to have but each <Column Name> has a <Column Value>.
I need to load the data into different tables based on the <Column Name>.
Any ideas????
Thanks for all the information this forum has very helpful!!!
Hello
I am looking for some advise.
I have a process to import flat text files. We are importing data from five vendors. These files are seperated by vendor with each vendor having their own directory. The general file layouts are different for each vendor.
Each vendor may have up to five different types of files to be imported that are their one directory (sales, inventory, transactions etc). Sales file for Vendor 1 is different layout than sales file for Vendor 2.
Each vendor may have multiple instances of each type (store 1 inventory, store 2 inventory, store 1 sales, store 2 sales etc.) There could be up to five hundred files (of the five different types) in a given vendors directory.
I am using an import package. This package has five (.dtsx files) different data flows. Each of these data flows has connection managers that connect to the specific types of files (sales, inventory, transactions etc) for that vendor.
My current play is to have the data flow(.dtsx file) parse the store name (from the file name) for each of the file types. It would load/process each of the available file formats (sales, inventory, transactions etc) for that store in that vendor folder. We want the process to load all data files from a given store. It would then move on to the next available store (same vendor). I would like to set the process to run multithreaded so that I am loading as many of the stores for that vendor as possible (there could be over a hundred stores for each vendor) at the same time.
How do I get each dataflow to run multiple instances (Instance1 for vendor(x).dtsx, instance2 for vendor(x).dtsx etc) for maximum Vendor(x) input.
What is the best way/process/design to track each store name so that each process (instance1, instance2 etc) is loading a distinct store. The files will be moved to a history folder as they are processed. Should I have a different process that gets each store name initially and then saves that information to a SQL table. The dataflow would load the next available store name from the SQL table query and let SQL lock that store?
Any suggestions are welcome and appreciated!
I have to Load Contacts data from different systems into a single SQL Server 2005 table, the scenario is the following :
I have three systems System 1,System 2 and System 3
Step 1 Load Data From System 1
Step 2 Load Data from System 2 and if there are matching contacts with System 1, then match their details, keep System 1 details
Step 3 Load Data from System 3 and if there are matching contacts with System 2, then match their details and if different, flag them, do the same with System details.
The unique ids in the three systems are not the same so only way to match is to do a concatenation of name + address + zip code
Is there any other efficient way to do this...
Please advise
Hi ,
My Input is a flat file source and it has spaces in few columns in the data . These columns are linked to another table as a foreign key and when i try loading them in a relational structure Foreigh key violation is occuring , is there a standard method to replace these spaces .
what approach should i take so that data gets loaded in a relational structure.
for example
Name Age Salary Address
dsds 23 fghghgh
Salary description level
2345 nnncncn 4
here salary is used in this example , the datatype is char in real scenario
what approach should i take to load the data in with cleansing the spaces in ssis
everything is ok.. the connection is on..the page can be loaded... no error....
BUT the data from the database cannot be retrieved... as in.. it just didnt load on screen...
i dont know whats wrong... i have follow all the steps show on a book.. can anyone pls help me?? thks
Hi,
I have problem I'm hoping someone can give me some pointers with.
I need to load data from several text files into one table. The format of the files are simple - each line is comma separated, with double quotes around each element e.g.
"parameter 1","value 1","parameter 2","value 2"....
"parameter 12","value 12","parameter 13","value 13"...
However, the files themselves will have different numbers of columns e.g file 1 may have 8 columns, file 2 may have 16 columns.
I'm going to load the data into a table that has at least as many columns as the longest file. The table columns are all varchar, and are named simply as [Col001] [Col002] [Col003] etc...
The first two columns of this table must be left empty during the load (I use these later on), so the data entry will start at [Col003].
My question is what is the best way to do this? I thought perhaps using a BULK INSERT in a stored procedure might do the trick, but I haven't used it before and haven't got very far. I gather another approach might be to use bcp utility. Someone has also suggested a DTS package, but the filenames will be suffixed with current date/time stamp, so i don't think that will work.
My preferred appraoch would be the BULK INSERT..but i'm open to any pointers.
Many Thanks
Greg
I am extracting data from an SQLServer database to load into Teradata using DTS. The performance is abysmal. The same data in a text file loads quickly via multiload. I can move the data to other DBMSs via DTS quickly as well. Is there some way for me to improve the elapsed time/performance while using DTS? If not, what is the best way to move data from SQLServer into Teradata?
View 1 Replies View RelatedI have a sql server database linked to my application. I have a table that I want one of the columns (Service Perfomed) to load in a list box. When I start the application nothing appears in the list box. What could I be doing wrong?
Here the code the visual studio created:
Private Sub MainForm_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) _
Handles MyBase.Load
Me.ServiceTableAdapter.Fill(Me.MaintenanceRecordsDataSet.Service)
End Sub
I've tried combo box, text box and these seem to work fine but not the list box.
I'm fairly new to SQL and I have been asked to load multiple tables with duplicate and inconsistent data into a single table. Each source (table) has been assigned a rank in terms of the quality and reliability. After some thought, I have come up with this solution to get all of the data loaded based on rank.
Does anyone know of a better way to do this?
Value is a unique key.
SELECT *
FROM table_1
union
SELECT *
FROM table_2
where (value + '_' + yr) in
( select value + '_' + yr
from table_2
except
select value + '_' + yr
from table_1)
union
SELECT *
FROM table_3
where (value + '_' + yr) in
( select value + '_' + yr
from table_3
except
select value + '_' + yr
from table_2
except
select value + '_' + yr
from table_1)
I need to load a lot of data into a sql table as fast as possible. (All data is string data, CSV file). I have read that the "fastest" way to load data is using the "Fast Load" option of the OLEDB Destination but I have also read that the "SQL Server" Destination is fast.
1. Is there a general consensus as to which way is the quicksest load?
2. The file is a CSV file. Would a fixed-format file be faster to read?
TIA,
barkingdog