SQL Server 2014
I have developed a SSIS package to load data to one DW database.
Below is my understanding -
- Create Main table with Clustered index + ColumnStore index + PARTITIONING TABLE BY DATE
- Create Staging table with same Clustered Index and in same File Group
- Load current date data to staging table using SSIS Package job run
- After loading data in staging table, create columnstore index same as Main Table
- Add current date range to Main table, which creates partition for current date
- Using SWITCH command --> Switch Staging table to above created Current Date Partition ( PS : I understand that this command works fine, though Column Store index is enabled on Main Table, there is no need to rebuild the index.)
Now, I think that above is the best practice for partition by date with column store index.
However, there is a requirement to implement Partition by YEAR.
How can I design this structure and implement partition to fulfill below requirements ?
• Partition should be done by YEAR
• SSIS Package will load around 1M rows for each current day to the table.
• Column Store index should be implemented on Main table to query data. (As table size is going to be huge)
• Does staging table require in this case? If yes, then for switching data to partition of year, we have to wait until whole one year of data to be loaded to Staging table ?? I don't see this option feasible.
• There are 4 local disks and total size is around 2TB. How many file groups I should create according to the partitioning by year, so that one active transaction query should perform best ?
• Is it good to have one Clustered columnstore index on the Main table? As I am using SQL Server 2014.