Scientific Data Table Design
Dec 14, 2007
Hi,
I am new to SQL 2005. I have to design schema for scientific data warehouse. Data is available in 2 or more flat data files recorded at 1 sec interval. At Least 2 of the data files have 100+ columns. I am inclined to create a table per data file type. I want to know If this is correct/optimal for me to do?
I don't think I can create normalize tables based on the headers in these Data files.
Primary Objective of this data warehouse is make it available for reporting services and Analysis Services.
Any suggestions or tutorials for me?
Thanks,
Vinod Kushwaha
View 1 Replies
ADVERTISEMENT
Dec 14, 2007
Hi,
I am new to SQL 2005. I have to design schema for scientific data warehouse. Data is available in 2 or more flat data files recorded at 1 sec interval. At Least 2 of the data files have 100+ columns. I am inclined to create a table per data file type. I want to know If this is correct/optimal for me to do?
I don't think I can create normalize tables based on the headers in these Data files.
Primary Objective of this data warehouse is make it available for reporting services and Analysis Services.
Any suggestions or tutorials for me?
Thanks,
Vinod Kushwaha
View 4 Replies
View Related
Jan 10, 2008
Hi All,
Am I missing something or is there something odd with float data types. i know that float is not the most precise definition but i came across something really odd today.
first let me define the scenario.
this is sql server 2005, standard edition build 3042.
I have a table defined as
CREATE TABLE [dbo].[ASSET](
[Property_Num] [numeric](10, 0) NOT NULL,
[Accrual_Factor_Val] [float] NULL
)
the accrual_factor_val was updated to a value of 0.00005 then the web service failed because the proc returned 5E-05!
i opened the table, and discovered this is the stored value. is this correct?
thoughts and ideas?
thanks for your input!
View 5 Replies
View Related
Apr 25, 2008
Hi ,
I have one problem regarding format of mixed data columns in excel. I am using this query to retieve data from my excel spreadsheet
Select * from OPENROWSET('Microsoft.Jet.OLEDB.4.0',
'Excel 8.0;IMEX=1;IHDR=yes;Database=E: est.xls;HDR=YES',
'SELECT * FROM [original$]')
to retrieve data from excel sheet.
I am able to retreive all data (numaric and text) , but for few cells I am getting wrong values.
like this
15106749310-> 1.51067e+010
70400672657-> 7.04007e+010
48110753143-> 4.81108e+010
registry setting are €œtext and 0€³. I am also using IMEX=1 in connection string.
I have numeric and text data both in same column,
eg:
70400503549
70400672657
KF1080420908
KF1260370908
KF1260880908
KF1260960908
Could you please help me to get the correct value from excel .
thanks in advance ,
View 7 Replies
View Related
Oct 7, 2015
I have a requirement of table partitioning. we have 10 years of data on a table which is 30 billion up rows on 2005 server we are upgrading it to 2014. we have to keep 7 years of data. there is no keys on table or date column. since its a huge amount of data and many users its slow down the process speed. we are thinking to do partition on 7 years for Quarterly based. but as i said there is no date column on table we have to use reference table to get date. is there a way i can do the partitioning with out adding date column on table? also does partition will make query faster?Â
I have think three ways to do it.
1. leave as it is.
2. 7 years partition on one server
3. 3 years partition on server1 and 4 years partition on server2 (for 4 years is snapshot better?)
View 3 Replies
View Related
Jul 30, 2015
I have to tables like given below Landing table "A" (Data load will happen over here, No primary keys mentioned over here) table "B"Â .Now I want to move the data from A to B.I have made use of below query insert into B select * from A...Landing table "A" has huge no of records, MS SQL server is taking huge amount of time.any alternative way to make this insertion process faster?
View 6 Replies
View Related
Jul 21, 2015
I have data that is similar in nature to stock market data. There are about 100 entries per day. I would like to setup a second table to supply chart information. (assuming 1000 pixels per chart max)Â
What is the best way to condense the information in this table so that if I query for a chart, the table returns about 1000 points? I could have several tables setup, one for short term, and one or two for longer term. But I still don't know how to condense them.Â
View 2 Replies
View Related
Sep 30, 2015
I have a very large table that I am trying to partition and use to reduce maintenance overhead as well as improve performance. The table contains about 12 years worth of data but only the most recent years is inserted/updated/deleted from thru the app. I created partitions on a computed(persisted) column which holds the "year" value derived from a date column. I have created the partitions with all the default set options, and the stored procedure which performs the delete against this table also was created with no special set options(basically database/session default). Yet, every time I try to run the proc to delete data thru the app, I get this error:
Msg 1934, Level 16, State 1, Procedure xxxx, Line 118
DELETE failed because the following SET options have incorrect settings: 'ANSI_WARNINGS'. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.
I've tried setting ANSI_WARNINGS on and off when creating the proc, inside the proc etc.., its always the same error whatever I set the option to.
View 4 Replies
View Related
Jun 10, 2015
I have 3 tables (accnt, jobcost, and servic15). all with the same fields (code, jno, ven, date). I need to insert the data from these tables into another table called dummy with the same fields, in one statement or query.
View 3 Replies
View Related
Jul 27, 2015
I never created table on the basis of Data model diagram . I have to create the 3 table on the basis of given Data model diagram. There are 3 tables
1.md_geographyleveltype
2.md_geographylevel
3.md_geographylevelxref
I have tried to create 2 table but unable to create 3rd table.
CREATE TABLEÂ [dbo].[md_geographylevel](
[type_key] [int] NOT NULL,
[geog_key] [int] NOT NULL,
[Type_description] [nvarchar](50) NULL,
[Store_flag] [nvarchar](10) NULL,
[Type_short_desciption] [nvarchar](50) NULL,
[Code] ....
I am getting difficulties to create table script for table
md_geographylevelxref
How to create the table create script for
md_geographylevelxref
View 2 Replies
View Related
May 20, 2015
I normalized the below tables but I am finding it difficult to copy data to the new tables. Â How do I copy data from existing table to the normalized tables? see the table structure below and other supporting information:
SKU_DATA(SKU,SKU_Description,Department,Buyer) Note: this table already has data in it.
CREATE TABLE SKU_DATA (
SKU Â Â
  Integer NOT
NULL,
[code].....
The table structure above have two three determinants( SKU,SKU_Description and Buyer).  SKU and SKU_Description are candidate keys. Primary key is SKU.
Normalization : SKU_DATA(SKU,SKU_Description, Buyer)
 BUYER(Buyer,Department)
View 2 Replies
View Related
Aug 28, 2015
I'm currently stuck with a table that has 350 mil records. Querying this table is insanely slow so I had a better look at existing yearly partitioning. I already managed to partition on a month level which increased the performance/querrying a lot. I did this on the staging table where I used an alter statement to split the 2015 partition by 12 months.
However, in our project we used Data Vault. This means that we have 4 tables (hub, sathub, link, satlink), all carrying 350 mil records. The problem is that altering the partition function does not work. The server cannot handle this action. What the best way is to do this, without having to drop/reload all tables.
View 17 Replies
View Related
Aug 18, 2015
I would like to create a table called product. My objective is to get list of packages available for each product in data grid view column while selecting each product. Each product may have different packages type (eg:- Nos, CTN, OTR etc). Some product may have two packages and some for 3 packages etc. Quantity in each packages also may be differ ( for eg:- for some CTN may contain 12 nos or in other case 8 nos etc). Prices for each packages also will be different that also need to show. Â How to design the table..Â
Product name  : Â
Nestle milk |
Rainbow milk
packages  :
CTN,OTR, NOs |
CTN, NOs
Price:
50,20,5 |
40,6
(Remarks for your reference):CTN=10nos, OTR=4 nos Â
| CTN=8 Nos
View 3 Replies
View Related
Jan 22, 2008
I've got some values stored in nvachar(255) field stored by mistake as scientific notation (eg 7.5013e+006 instead of 7501301) and I need to convert and update the field with normal entry, not scientific notation. Is there a way to do that?
View 6 Replies
View Related
Jul 23, 2005
HiWe've got some numbers stored as Reals which are returning values inscientific notation that we need rounded down to 3 digits to the rightof the decimal.ie 8.7499999E-2 needs to return 8.75.Round, cast, convert, formatnumber in the dts package all fail.Help!Thanks Moe
View 2 Replies
View Related
Dec 5, 2006
Program Descritpion: Our product allows our customers to enter their product catalogs into our ASP.NET pages and we save the data using SQL Server 2005. Customer use GridViews to edit their data and teh data is broken into manageable groupings (called Field Groups) stored as meta-data in SQL Server Tables. Then when a user wants to edit a set of data they choose the Field Group and we dynamically generate a data source and a gridview, bind them together, and our users update their data. A bit more complicated but that's teh gist.
Another thing they can do, and where our problem occurs, is they can print reports or do exports into excel using this data. Since there are so many fields, they use the Field Groups to select which fields they want included so everything needs to be built dynamcially, at runtime, using the Field Group meta-data. Essentially, I want my grid to have these 10 fields.
Problem:For the export we generate an XML data source using FOR XML in SQL Sprocs. The xml is pulled into a xmldatadocument, trasnformed with an xsl file into an Excel XML Spreadsheet. Our problem is that FOR XML is generating our XML real numbers in scientific format. The xsl "format-number" function does not recognize scientific format and returns NaN (not a number) instead of the value in my spreadsheet. If I leave it blank I get the scientific number but Excel doesn't format it correctly, the cell has an error tag that wants me to choose Number stored as text or convert to a number. I need it to show up on teh Excel form already formatted without that message.
I can't change the DataType of the field in SQL, too many other things depend on it and it needs to be a real. I can't use CONVERT(decimal,fieldName) in SQL because the SQL string is dynmically generated using Dynamic SQL and most of the fields are not real. When we build the Field List we just have field names, we can't check anything to add CONVERT functions to only real fields.
Is there any way to force the XML output to not be scientific for the entire document? Or another function in XSL I'm unaware of (pretty new to xsl)? Or perhaps something I can do in the XML Spreadsheet tags to force the conversion?
View 1 Replies
View Related
Oct 16, 2013
We sometimes have small values stored in a column with datatype of float like 0.000644470739403048 which is being converted to -5.8E-05. Perhaps that is OK to be stored in the database however I need the value in decimal format to use. (I'm using longitude values in google maps).
is there anything I can do at the database level. I was looking at the properties which is 53 numeric precision and 8 length.
View 8 Replies
View Related
Jul 20, 2005
I'm trying to find a way to format a FLOAT variable into a varchar inSQL Server 2000 but using CAST/CONVERT I can only get scientificnotation i.e. 1e+006 instead of 1000000 which isn't really what Iwanted.Preferably the varchar would display the number to 2 decimal placesbut I'd settle for integers only as this conversion isn't businesscritical and is a nice to have for background information.Casting to MONEY or NUMERIC before converting to a varchar works finefor most cases but of course runs the risk of arithmetic overflow ifthe FLOAT value is too precise for MONEY/NUMERIC to handle. If anyoneknows of an easy way to test whether overflow will occur and thereforeto know not to convert it then that would be an option.I appreciate SQL Server isn't great at formatting and it would be fareasier in the client code but code this is being performed as adescription of a very simple calculation in a trigger, all stored tothe database on the server side so there's no opportunity for clientintervention.Example code:declare @testFloat floatselect @testFloat = 1000000.12select convert(varchar(100),@testFloat) -- gives 1e+006select cast(@testFloat as varchar(100)) -- gives 1e+006select convert(varchar(100),cast(@testFloat as money)) -- gives1000000.12select @testFloat = 12345678905345633453453624453453524.123select convert(varchar(100),cast(@testFloat as money)) -- givesarithmetic overflow errorselect convert(varchar(100),cast(@testFloat as numeric)) -- givesarithmetic overflow errorAny suggestions welcome...CheersDave
View 3 Replies
View Related
May 15, 2006
Hi
I'm developing a scientific appliation - to forecast the spread of disease. The algorithm has been designed such that different subpopulations reside on separate computers.
The model uses the same set of data repeatedly, and only returns a small signature that reflects the appropriateness of a particular set of parameters.
The model does however change, and I'm looking for a way to roll out this model (which only contains T-SQL, CLR assemblies and service broker code).
I have considered setting up a database on each computer which has the 'data' files which are static, and a separate database which contains the algorithm which can be amended.
Could one then detach the algorithm database making the new version that can then be rolled out (by attaching it to each computer used)?
Is this the best practice? Does anyone have a suggestion as to how to do this?
Go well
Greg
View 1 Replies
View Related
Jan 29, 2005
SELECT membername, outputval
case when choice = 0 then outputval else null end as outputval
from MyDatabase
group by membername, outputval
how to format outputval:
if outputval < 40000
format outputval as:
5 - 5.78 - 6.9 - 6,778 - 4,567.8 - 12,456.78 - etc.
if outputval >= 40000
format it as a scientific.
View 1 Replies
View Related
May 6, 2015
We need to Insert/Update a Fact Table from staging Table. currently we are using a SP which update Fact Table for Each region. this process is schedule, every 5 min job is run and Update fact table.but time of Insert and Update too long from staging to Fact, currently we are using merge statement for Insert and update.in my sp we are looping number how many region we need to update and at a time single Region we are updating using while loop in current SP.
View 7 Replies
View Related
Apr 30, 2008
Hi, I am developing an application to a garment factory. I have a doubt in designing a table.Basic tables:Jobs, JobColors, Material, Units, Currencies ...These tables are designed with normalization rules.I got a problem at PurchaseOrderDetailsMain
table is JobMaterial. It has materialid, jobid, supplierid, description
and TypeFactor(which represents the type of order) means that the
material is ordered based on size or colors or total qty.1 for ByColor, 2 for BySize, 3 for ByQty, 4 for By Contrast colorsThe main problem at the details of the sub table.JobMaterialDetailsIf typefactor is by size, i need to store the details based on sizeex: S - 2000pcs, M - 4000pcs, L - 4000pcs, XL - 2000pcsSo I will have 4 records per each sizeIf it is by color, White - 3000pcs, Portabella - 5000pcs, Black - 2000pcs.If it is by general, Total qty 10000pcsHow
can I design this table. If I take, ColorOrSize column, it will refer
different values for diffrent typefactor. When by size, it will have
Size and when by color, it will refer colorcode.But colors are having referential integrity. So it is violated other than by color as typefactor.What is the best way to design this table?Can anybody suggest?Thanks in advance
View 2 Replies
View Related
Jul 24, 2000
I am desingning a table and i have a column OrderID and another column call Order, is neccessary to use a primary key, because One OrderID may have many Orders?
Thanks.
View 5 Replies
View Related
Aug 21, 2000
Coming from a support background and having to design my first database I have a couple of questions re- table design. Firstly I have set up several tables and included one field (of the same name) in each. This is a primary key in one table with an incremental seed. I would like this info to appear in the other tables although these can be duplicates in the other tables. How is it best to achieve this relationship. From reading it suggest FK in the relationship application but looking at other databases this seems to have been achieved by some other means. Is it more common to use stored procedures to enforce this? If so please add pointers. Secondly, I have set up a couple of master tables to act as looks ups for fields in other tables. Again how do I get this to look up the table - is it done through stored procedures or at the time of writing the front end application?? Sorry if this is all basic stuff but it is new to me.
Thanks for any help
View 1 Replies
View Related
Oct 26, 2004
CREATE TABLE [dbo].[table1] (
[aaa] [bigint] IDENTITY (10000, 1) NOT NULL ,
[bbb] [int] NOT NULL ,
[ccc] [int] NOT NULL ,
[ddd] [bigint] NOT NULL ,
[eee] [int] NOT NULL ,
[ffff] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
[gggg] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[hhh] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[iii] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[jjj] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[kkk] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[lll] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[mmm] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[nnnn] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[ooo] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[ppp] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[qqqq] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[rrrr] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[ssss] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[tttt] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[uuuuu] [varchar] (2000) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[vvvvv] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[wwwww] [varchar] (150) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[xxxxx] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[yyyyy] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[zzzzz] [int] NULL ,
[abc] [varchar] (500) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[def] [datetime] NULL ,
[ghi] [datetime] NOT NULL ,
[jkl] [varchar] (1000) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[mno] [bigint] NULL
) ON [PRIMARY]
I have created a table with above column width. The rowsize is more than 8kb.And the table holds millions of rows of data. So is it a correct way of designing the table?
Or how can I redesign this table.
Thanks.
View 1 Replies
View Related
Mar 16, 2004
I'm currently developing a real estate system to manage order processing and work flow. I'm a little uncertain as to how to design the tables because an order can have N number of applicants, owners, buyers, and properties. There are cases where there are 9 different buyers and some where the number of properties exceeds 20. It seems that normalization might make the situation crazy, but I'm a touch rusty. Thanks.
View 2 Replies
View Related
Apr 16, 2004
I am developing an estimating application and at the "quantity takeoff" level, all products/equipment/sub contracting/etc.... end up on the same table because of the common elements:
Item | Quantity | Unit Price | SubTotal | Total
Now, since the unit price of a product is not determined from the same tables as the unit price of equipment rentals or sub contract work, etc...
how do I direct the attention to the appropriate table when trying to obtain the unit price or edit the unit price etc...
If my question is not clear, let me know, I will try to clarify.
Mike B
View 3 Replies
View Related
May 10, 2008
Hi,
Is there a way to copy and save my Design Table? I want to transfer it to another workstation.
View 8 Replies
View Related
Feb 13, 2006
Hi am trying to figure out my table design need some advice
I have a table that catches the telephone numbers of a customer , there are 5 telephones for each customer. All the numbers are there in the DB
Should I create the design in the following format
CustomerName
Tel1
Tel2
Tel3
Tel4
Tel5
Or Should I have
Customer
Telephone
& then have 5 rows for each customer
Please advice
I even wanted to knw how should the design be , if all the Tel Nos are not captured, I mean what if some customers have 3 tel Nos, while some may just have 1 while some may have all 5
Thanks
View 7 Replies
View Related
Aug 21, 2006
Hi I'm having trouble figuring out this database design and normalising the data. Following are the tables and I am wonder if it can be normalised any further. This is a database of Architects and their work history and building they have designed.
Architects:
FirstName
LastName
Gender
DOB
Bio
Works:
Title
Client
Type_of_Building
Style
Address
Year_Designed
Year_Completed
Comments
Architect_ID
Firm: (this is the various firms that the architect has worked for)
CompanyName
Address
EmploymentHistory: (when the architect worked with a particular firm)
Year_Start
Year_End
Firm_ID
Architect_ID
Publication: (where the architects works were published)
Author
Date_Publication
Title
Source
Publisher
Architect_ID
I would really appreciate some help and suggestions on creating the relationships.
thanks
View 5 Replies
View Related
Sep 17, 2007
I create following tables.
employee table
------------------------
EMP_ID varchar
NAME varchar
DEPARTMENT_CODEint
POSITION_CODEint
COUNTRY_CODEint
department table
------------------
DEPARTMENT_CODEint
DEPARTMENT_NAMEvarchar
position table
---------------
POSITION_CODEint
POSITION_NAMEvarchar
country table
-------------
COUNTRY_CODEint
COUNTRY_NAMEvarchar
In employee table I have to store two country names.
ie. employee currently working in this country
but his home country name (origin country) is another
name. In country table all countries information available.
ie. country_code and country_name.
How I can design the employee table that it get country_name from COUNTRY TABLE ?
but we can not use same country_code in two fields.
OR
otherwise I will have to create employee table like this
employee table
------------------------
EMP_ID varchar
NAME varchar
DEPARTMENT_CODEint
POSITION_CODEint
COUNTRY_CODEint
COUNTRY_NAMEvarchar
ie. during data entries direct insert the country_name in employee
table ?
I want to create separate table for separate data.
department,postion,country.
How I can use unique country_code in employee table's two
fields ?
regards
Martin
View 2 Replies
View Related
Nov 8, 2007
Hi,
I'm struggling coming up with an optimal table design for the following scenario.
I have a table in which each row is recording information about a document - author, publisher, publication, title, abstract and so forth. I also have an image table which contains information about the images which can be associated with the documents.
Each document can have on of three image types associated with it - an author photo, a publisher logo and a publication logo. There aren't actually that many different authors, publishers and publications in the database so it would be common for many documents to have the same author or the same publisher although authors sometimes write for different publications. In the current setup it's possible for a document to have more than one image of each type associated with it, which is undesirable - this needs to eliminated and we need to ensure that each document can only have one image of each type associated with it.
It'd also be a huge boon if I could get all the information about a document and it's attached images with a single database call. In the current setup, trying to do this returns multiple rows so I've ended up making two calls, one for the document and one for it's attached images.
I can see two possible ways of setting this up, neither of which appeal.
The first is to add three columns to my document table to record the image URLs which is simple but inflexible should I want to add more image types later on.
The second is to set up three new tables for author, publisher and publication and associate each of these with the document table through a key and then to the image table through another key. There's also an image type table which is going to come in to play. I'm unwilling to do this because it's going to just explode the number of tables I've got, especially if we add more image types, and play havoc with my XML schemas.
Is there another way of rationalizing this that I've missed?
Cheers,
Matt
View 3 Replies
View Related
Jan 11, 2007
Hi,
I am trying to decide between 2 table designs. I basically have several different types of work items that need to be tracked. Each work item has their own set of statuses, some of which may be the same as other work item's statuses, some of which will be unique for that work item.
Now, should I create one status table for each work item, or would it be better to use one status table, but add a column to track which work item the status belongs to?
I'm not sure which is a better design, or perhaps there is some alternate design that is better. Any help is appreciated.
Thx.
View 1 Replies
View Related
