Unpivot dynamic table columns into key value rows

Question

The problem that I need to resolve is data transfer from one table with many dynamic fields into other structured key value table. The first table comes from a data export from another system, and has the following structure ( it can have any column name and data): [UserID],[FirstName],[LastName],[Email],[How was your day],[Would you like to receive weekly newsletter],[Confirm that you are 18+] ...

The second table is where I want to put the data, and it has the following structure: [UserID uniqueidentifier],[QuestionText nvarchar(500)],[Question Answer nvarchar(max)]

I saw many examples showing how to unpivot table, but my problem is that I dont know what columns the Table 1 will have. Can I somehow dynamically unpivot the first table,so no matter what columns it has, it is converted into a key-value structure and import the data into the second table.

I will really appreciate your help with this.

Answer 1

You can't pivot or unpivot in one query without knowing the columns.

What you can do, assuming you have privileges, is query sys.columns to get the field names of your source table then build an unpivot query dynamically.

--Source table
create table MyTable (
    id int,
    Field1 nvarchar(10),
    Field2 nvarchar(10),
    Field3 nvarchar(10)
);

insert into MyTable (id, Field1, Field2, Field3) values ( 1, 'aaa', 'bbb', 'ccc' );
insert into MyTable (id, Field1, Field2, Field3) values ( 2, 'eee', 'fff', 'ggg' );
insert into MyTable (id, Field1, Field2, Field3) values ( 3, 'hhh', 'iii', 'jjj' );

--key/value table
create table MyValuesTable (
    id int,
    [field] sysname,
    [value] nvarchar(10)
);



declare @columnString nvarchar(max)

--This recursive CTE examines the source table's columns excluding
--the 'id' column explicitly and builds a string of column names
--like so: '[Field1], [Field2], [Field3]'.

;with columnNames as (
  select column_id, name
  from sys.columns 
  where object_id = object_id('MyTable','U')
    and name <> 'id'
),
columnString (id, string) as (
  select
    2, cast('' as nvarchar(max))
  union all
  select
    b.id + 1, b.string + case when b.string = '' then '' else ', ' end + '[' + a.name + ']'
  from
    columnNames a
    join columnString b on b.id = a.column_id
)
select top 1 @columnString = string from columnString order by id desc

--Now I build a query around the column names which unpivots the source and inserts into the key/value table.
declare @sql nvarchar(max)
set @sql = '
insert MyValuestable
select id, field, value
from
  (select * from MyTable) b
unpivot
  (value for field in (' + @columnString + ')) as unpvt'

--Query's ready to run.
exec (@sql)

select * from MyValuesTable

In case you're getting your source data from a stored procedure, you can use OPENROWSET to get the data into a table, then examine that table's column names. This link shows how to do that part. https://stackoverflow.com/a/1228165/300242

Final note: If you use a temporary table, remember that you get the column names from tempdb.sys.columns like so:

select column_id, name
from tempdb.sys.columns 
where object_id = object_id('tempdb..#MyTable','U')