Real-time Data Imports. How Stable Is It?
Feb 4, 2008
Hi there,
I was wondering about the stability of SSIS when it comes to importing data on a real-time basis. Let's say you have a scenario where flat files, for instance, will be dropped at random intervals ranging from 1 second to 10 seconds apart and the importer has to import these files immediately.
I would imagine that this is done with a package which runs a loop sniffing the directory forever but I stand corrected on the best ways of doing it.
I'm not too sure whether SSIS is a good idea for this as lots of people have had bad comments on SSIS in real-time in my company but they cannot elaborate on why enough to convince me. I have done some pretty cool stuff and must admit that I am a fan of the technology but dont want to defend it into a corner
Please give me your thoughts on this
Regards
Michael
View 13 Replies
ADVERTISEMENT
Jan 25, 2007
I currently have an application which retrieves stats from a SQL database that are updated throughout the day. My application updates every 3 seconds to retrieve these stats but I found that it's quite expensive to the server's CPU when there are several clients running my application. Are there any other methods for extracting the data that won't require so much CPU. The fact that the database is being hit so much isn't good either. I have already written a webservice hoping that this would improve things but I'm not sure it has.
View 3 Replies
View Related
Oct 8, 2007
Hi
I´m a SysAdmin and I´m implementing ASSP (anti-spam) to use with an email server.
But the ASSP requires a txt file with all domains hosted in the server that are in a SQL Table of the server.
My questions is: Is there anything to make a real time TXT file with all the domains ?
maybe use trigger I dont know..
View 4 Replies
View Related
Sep 8, 2006
We have completed the implementation of data mining algorithms.Now we have to build an API for real time data mining.Is that possible in SQL Server?
View 3 Replies
View Related
Sep 27, 2006
I am about to prepare a paper concerning the field of real-time data mining. Real-time here means the process of incremental training of an existing model as soon as the data arrives.
There is a number of papers introducing algorithms for incremental association analysis, incremental clustering etc. Stream mining Ãs a field which is closely related to that. The main reason for the implementation of incremental algorithms is a) the large amount of data to be mined and b) the high rate of new data that is evolving every day.
Using classical batch mining algorithms, models that are outdated for some reason, would have to be re-trained, which could be very time consuming for billions of records. And once the training is completed, the training would have to be restarted once again because a bulk of new data has been arrived.
The question that I would like to discuss now is: For what real world applications would it be a meaningful or even essential to use real-time training of models?
Two main reasons could determine the answer to that question:
You just want to incorporate new data into existing models in order to increase the prediction accuracy of your model or
Your underlying data is subject to more or less massive changes (also refered to as concept drift) and you want to adapt your mining model continuously to that reality.
I'm looking for some examples or ideas where one of these cases apply and it would be a good idea to have incremental mining algorithms involved.
I'm looking forward to inspiring some discussion on that issue.
View 3 Replies
View Related
Jan 10, 2006
I want to build a windows application in Visual Studio 2005 that grabs some data from a SQL DB and displays it in a gridview. That is easy and I already have it done, but I also want to know how to show this data in real-time. For example: right now we have a application that pulls some login information for all the users and displays their phone extension, their name, and their status. They can change their status by clicking on some radios below the datagrid and on change it updates the data and then posts back. However, I dont know how the guy who did it made it so that these local changes are automatically shown on everyone elses computers around the office. In other words, they are shown the live data as it changes, without having to refresh it or anything. How would I go about doing this or does anyone have any places/resources that could help me accomplish this task. Thanks.
Oh and the guy who wrote this is no longer working here and he deleted the source code so i cant look at it.
View 2 Replies
View Related
Jul 19, 2007
I have no idea where to post this kind of question, so here it is!
I have a requirement to retrieve oracle 10 data into SS2000 in as near real-time as possible (stupid users!) and join with resident SS data for on-demand reporting. (We use SS replication to populate some reporting tables from other SS2000 instances and this has spoiled the users as well as the developers! )
I would like to know if there are any clever ways of doing this, or if a plain-old DTS package running in some kind of loop is the practical answer. 1 minute delay is probably too long . . . I don't know if data can be pushed from the oracle side. Or if we need to write a Service and use it to suck and push.
Any suggestions?
thanks!
View 1 Replies
View Related
Mar 15, 2008
I probably posted my question in the wrong forum so please check here:
http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=3009605&SiteID=17
thanks
View 6 Replies
View Related
Jun 11, 2015
Is there a way to keep track in real time on how long a stored procedure is running for? So what I want to do is fire off a trace in a stored procedure if that stored procedure is running for over like 5 minutes.
View 5 Replies
View Related
Apr 18, 2007
Hi,when i create a new user in a .MDF file, the hour of creation in the field "createdate" of table "membership" is exactly 2 hours less than the time of my computer. Probably something to do with different hours between Europe and UK or ...How can i set the creation hour to exact the same hour of my computer?ThanksTartuffe
View 4 Replies
View Related
Oct 28, 2007
Hi all.
I have posted this question in another forum but no one has so far been able to provide a solution to the problem. Since I know Database Journal always has very informative and enlightening posts, I figured I'd post the question here in hopes that some guru can provide an answer.
If you're running SQL 2000 and have any jobs that have been executed, you could perform a query as such:
select last_run_time from msdb.dbo.sysjobsteps
and receive returned values that contain the last time a job was executed "stored in integer datatype" columns. See ->
sp_help sysjobsteps.
In SQL 2005 I believe the concept is the same. I think the intent of Microsoft had it mind for doing this was to store the date separate from the time values which won't work using the datetime datatype and I have read this in documentation in the past.
The challenge is to convert that data into a humanly legible 12 or 24 hour time format like 11:00 AM or 02:45:39.
Does anyone have any suggestions or clues to assist in resolving this problem??? :(
Thanks.
View 6 Replies
View Related
May 6, 2001
We are using SQL 7 at 2 different locations, on 2 different Domains that are connected via a T1 line.
We need to have our 2 SQL servers be identical at all times, meaning when a change is made on one server, it is immediately replicated to the other. The problem is that both servers are always being used, so there is no primary/backup server scenario.
Does anyone know if there is a third party software available, or if this is even possible to do, given the fact that the databases are always open ?
Thanks.
View 1 Replies
View Related
Oct 24, 2000
Hello,
Can anyone guide me. which is the best method for real time synchronisation of my production server. Is it Transactional Replication or Stand By Server?
Thanks In Advance
Sejal
View 2 Replies
View Related
Apr 24, 2008
I have two database, one is for production and another for reporting database, in this Scenario we need to transfer the real time data to reporting database, which replication we need to use and what at the bottelnecks in this..can any one give some suggesstions or give some links...
Thanks in advance
View 4 Replies
View Related
Mar 22, 2007
i have 2 DB in 2 different servers.
db1 and db2
db1 gets 150k records every hour. db2 is like the backup of db1. db2 is used to retrieve records to create reports using reporting services.
what is the best way to synchronize db2 with db1?
is it to create a dts package and set up a job that runs every hour?
off topic question.
what kind of language does reporting services use to create the reports?
View 2 Replies
View Related
Nov 12, 2007
hi guyz ...i m new to sql server so is there anyone who could guide me abt sql server 2005 ....i need some info abt ssis ,ssas ...i have got a proj n have sme ques regd tat so plz let me know ur no or mail id so that i could contact u ..plz its urgent
raghu
View 1 Replies
View Related
Nov 12, 2007
hi guyz ...i m new to sql server so is there anyone who could guide me abt sql server 2005 ....i need some info abt ssis ,ssas ...i have got a proj n have sme ques regd tat so plz let me know ur no or mail id so that i could contact u ..plz its urgent
raghu
View 2 Replies
View Related
Sep 18, 2007
Does anybody know a good way to get a (near) live view of a sqlce database that doesn't involve ResultSets? I tried to use a resultset with the Sensitive option set, but it doesn't like the joins that I have to do in order to have the data make sense to the user.
Thanks
View 1 Replies
View Related
Sep 2, 2006
The ETL processing for my project will be deployed as part of a larger application which is operated by non-technical users. Therefore I want to provide real-time feedback to the user on progress, errors, etc., using windows standard GUI interface objects and style- for example a progress bar. The "designer" of course does some of this but there won't be any designer in our deployed application (and SSIS designer is neither intended to be used by, nor appropriate for, non-technical users anyway).
Now that I have the ETL logic working, it's time to tackle this UI requirement. I am thinking one way to do this is to start a script task to run in parallel with the main ETL processing, open a winform in that task that has various GUI objects whose "state" is based on package scoped variables, update the package level variables via the "on progress" event (or other events as needed) of various SSIS tasks and components, and refresh the winform's display regularly via a timer event.
Does this approach seem like it would be effective? Has anyone tried maintaining an open winform via a script task and updating it's objects via package variables in parrallel with the package is running other tasks?
Thanks,
Ken
View 2 Replies
View Related
Apr 12, 2007
Hi,
We currently have a propriety in memory DB that is used to store the latest transactions in the system and we have a service that copies the data to a SQL Server every couple of seconds - For historical reporting purpose.
We would like to move into a more standard DB as our real time DB, since we have scalability and availability issues. We taught about using SQL Server since this is the DB we know, but I'm not sure it's built to handle real time data.
Does someone has any experience in using SQL Server for "Real Time" applications?
Does someone has any experience in storing the data files on RAM?
Does MS has a solution similar to Oracle's TimesTen, which is their real time DB?
Thanks,
Avi G..
View 3 Replies
View Related
Jun 28, 2007
hello,
I want to create real time replication of my databaser server.I have two database srevers,one is master which having real time data and another slave,where i want to replicate data from master .
plz help me to configure solution
Thanks,
Chetan S. Raut.
View 1 Replies
View Related
Jul 2, 2001
Hi Everybody:
We have a table which needs to be updated 2 million times per day. It hosts all real time transaction. There are 200K records in this table. Would you please to share your experience with me about how to protect/save such table in SQL 2000 from any possible damage?
We plan to use point-in-time backup (every 5 minutes). It still takes at half an hour to recover the whole database. Any new technology from Microsoft or SQL 2000 you can recommend?
Thank you very much.
Joan
View 3 Replies
View Related
Jan 10, 2008
I'm sure just about everyone uses the PRINT command to give feedback as to what their lengthy and involved scripts are doing, as sort of a record.
I cannot figure out how to make the stuff I use in PRINT commands come out in real-time like SELECTs seem to. Does anyone have an answer to this? These are long-running scripts, and I'd rather nip a problem in the bud before the entire script completes if there's a problem I can capture.
___________________________
Geek At Large
View 2 Replies
View Related
Jan 20, 2008
Hi,
I'm would like know if the analysis service data mining enables to detect anomalies from "normal" behavior/patterns of data (1), and alert about such anomalies when detected (2). both above sql server relational DB (3).
Thanks,
Eyal
View 5 Replies
View Related
Nov 29, 2006
Hi,
I would like to find out about SSIS compilation. Can you mention anything regarding this issue or can you point me out to a website for this topic please?
Thanks
View 1 Replies
View Related
Mar 31, 2003
Hi there,
I use Asp.Net Application to upload a Excel file and then a DTS to import data from the file to the SQL2000 and finally to display the read data on the screen.
The DTS starts with setting some variables with the help of Dynamic properties.
On Succes.
DTS rum 2 simultaneous Transform Data task importing data from excel to SQL2000.
This works fine for "most" of the time, but then there are the other times.
One of the TDT(Transform Data task) reads nothing from the excel file. but it can read the data if i upload the same file again right after.
Any kind of input is welcome
Thanx
/bs26
View 1 Replies
View Related
Aug 25, 2001
Hi
There are 2 databases db1 and db2 on SQL server 7 with tables 1 and 2 respectively.
Both these tables have a field called Quantity. IF quantity gets updated in table 1 then that change should be immediately reflected in table2.
I though an update on table1 would serve the purpose but this trigger doesn't seem to be working however there are no errors during the syntax check.
The code for the trigger that i have used on table 1 is as below
CREATE TRIGGER [Qty_UPDATE] ON [Table1]
FOR UPDATE
AS
IF UPDATE (Qty)
BEGIN
UPDATE db2.dbo.table2
SET db2.dbo.table2.qty = qty
END
Please help
Thanks in advance
View 1 Replies
View Related
Sep 7, 2006
how to synchronize and compare two database in sql server 2000 without using any others software in real time.
my purpose is to check the two database in two different server if there has any different. And synchronize these two database continusly.
if one database had change, the other will also change
View 4 Replies
View Related
Sep 7, 2006
how to synchronize and compare two database in sql server 2000 without using any others software in real time.
my purpose is to check the two database in two different server if there has any different. And synchronize these two database continusly.
if one database had change, the other will also change
View 3 Replies
View Related
Apr 28, 2008
I got anywhere from a couple hundred to a hundred thousand records that need to be updated or inserted into their SQL Server 2005 end destination. What are some of the best ways to accomplish this? Right now we are doing it manually through line by line updates and inserts. Would I use BC or some other bulk import tool?
View 5 Replies
View Related
Mar 12, 2007
I'm pretty new to using SQL 2005 Management Studio. Generally speaking, it works pretty much the way I'm used to (using Enterprise Manager) as far as moving data around and designing databases is concerned. But I've been trying to import some data to my local SQL server from an online SQL database and I am getting the most bizarre results.
Basically, it appears to work perfectly, but when the import is finished, there is only one new row in the destination table.
I have tried this with two completely different online databases and I have tried importing using a query and just downloading the table as-is, but whatever happens, I just get one row! All of the databases I'm exporting from or importing into are SQL 2000 - I just happen to be using the SQL 2005 client software.
Have you heard of anything like this?
I'm appending the report to the end of this email. As you can see, it says "success, success" all the way down and it clearly states "71236 rows transferred", but when I get done, there's just one.
Any thoughts?
One explanation is that every new row is overwriting the last new row somehow, but I don't think so because the row that actually gets copied is always the first one in the record set, not the last. Unless they're being transferred in reverse order, I suppose.
I'm pretty stumped and I haven't found any useful blogs or help on the web.
Hope you can think of something because I don't have Enterprise Manager on my computer any more and it's going to be a pain to install it.
Jon
Here's the report:
The execution was successful
- Initializing Data Flow Task (Success)
- Initializing Connections (Success)
- Setting SQL Command (Success)
- Setting Source Connection (Success)
- Setting Destination Connection (Success)
- Validating (Success)
- Prepare for Execute (Success)
- Pre-execute (Success)
- Executing (Success)
- Copying to [inframes].[dbo].[sum_shop_clicks] (Success)
* 71236 rows transferred
Messages
* Information 0x402090df: Data Flow Task: The final commit for the data insertion has started.
(SQL Server Import and Export Wizard)
* Information 0x402090e0: Data Flow Task: The final commit for the data insertion has ended.
(SQL Server Import and Export Wizard)
- Post-execute (Success)
- Cleanup (Success)
Messages
* Information 0x4004300b: Data Flow Task: "component "Destination - sum_shop_clicks" (37)" wrote 71236 rows.
(SQL Server Import and Export Wizard)
View 12 Replies
View Related
Sep 19, 2006
I saw a presentation last week where the speaker created some sorta sql server "watch window" (in Sql Server Management Studio I think) where he could watch all the commands being executed on his sql server database in real-time. For example he could navigate to web pages (that hit the database) and as he pressed buttons you could see the sql commands execute in this "watch window." If other users hit the database at the same time you could see those sql queries execute as well. I didn't think at the time to ask how he did it - does anyone know how to set this up? I have a problem with my sql server right now and it would be useful to see which sql queries (etc) are being executed when. Thanks in advance,J. Shane Kunklejkunkle@vt.edu
View 2 Replies
View Related
Aug 9, 2000
Hi to you all and specially to who will solve my problem!
That is what I want:
I need a database that will serve as an archive db in read-only for day-to-day queries
and a production db that will be kept small by deleting un-needed records
A replication setup would be fine but I didn't find a way to avoid the deletes to keep the production database small, to replicate in the archive database where transaction are replicated.
Is there someting i am missing?
Andre Vigneau
View 2 Replies
View Related
