Alright so I am not sure how to go about this, I have files that will be coming in a format like this that I need to read into a SQL Server database:
As you can see, it is "~" delimited and it contains no columns names at all. I will have multiple files like this incoming every couple of hours and I have the entire SSIS set up ready besides the part where I actually need to read the data because I am confused on how to handle this delimiter format that other department came up with. As you can see if I specify the column delimiter just to be "~" it works fine
until it reaches that point where the row ends at which point there is this unnecessary row of "~" that starts and it confuses the connection manager into thinking these are separate columns, creating a bunch of empty columns
I can't simply delete all empty columns because some legit columns can sometimes come in as empty. The only mediocre solution I found so far is to go to advanced options in file connection manager and manually delete all of the columns I don't need. But the reason this will not work at all is because next file I will get might contain more rows than this one and it will still think that "~" after every data row is a column delimiter when in reality it is just a row separator. The number of columns however will always remain static in each file.