multipart index in zoo timeseries

Question

I am working with trade dataset with thousands of rows. Every record has a unique key based on a symbol and date. Trade records for a given symbol are irregular, hence using zoo will be natural choice. I need to use lag and merge to create a new dataset. However, I don't know how to setup multi-column index in zoo in order to use lag function. Below is a sample dataset and intended output.

df = data.frame(
    dt = as.Date(c("2015-01-01", "2015-01-05", "2015-01-06",
                   "2015-01-01", "2015-01-02")),
    id = c("i1", "i1", "i1", "i2", "i2"),
    v1 = c(110, 115, 119, 212, 213),
    v2 = c(100, 170, 180, 202, 210),
    v3 = c(11, 13, 16, 22, 24)
)
df$id = as.character(df$id)

And the output should be

2015-01-01, i1, 110, 100, 11, 2015-01-05, i1, 115, 170, 13 
2015-01-05, i1, 115, 170, 13, 2015-01-06, i1, 119, 180, 16 
2015-01-06, i1, 119, 180, 16, NA, NA, NA, NA, NA
2015-01-01, i2, 212, 202, 22, 2015-01-02, i2, 213, 210, 24 
2015-01-02, i2, 213, 210, 24, NA, NA, NA, NA, NA

In SO, there are number of posts accomplishing "grouped" lag operations but for a single column only. I am looking for merging complete row, regardless of number of columns.

Update to this question...

Following is one possible way to solve the "grouped" lag operation based on zoo.

doProcessing = function(df){
  icolnames = colnames(df)
  tt = zoo(df, df$dt)
  tt1 = merge(tt, lag(tt, 1))
  colnames(tt1) = c(icolnames, paste0("lag_", icolnames))
  data.frame(tt1, stringsAsFactors=F)
}
fin_df = do.call(rbind, with(df, by(df, list(id), doProcessing, simplify=F)))

This final output frame has every field as factor. How do I get the output structure right as per input data frame?

---

Based on @Grothendieck's idea of lapply, a possible solution to the above problem is given below.

doProcessing = function(df){
  icolnames = colnames(df)
  tt = zoo(df, df$dt)
  tt1 = merge(tt, lag(tt, 1))
  colnames(tt1) = c(icolnames, paste0("lag_", icolnames))
  data.frame(tt1, stringsAsFactors=F)
}

fin_df = do.call(rbind, with(df, by(df, list(id), doProcessing, simplify=F)))

Still need some help, some how resultant data frame has every column as factors. How do I get original structure back?

original data frame structure

> str(df)
'data.frame':   5 obs. of  5 variables:
 $ dt: Date, format: "2015-01-05" "2015-01-01" ...
 $ id: chr  "i1" "i1" "i1" "i2" ...
 $ v1: num  115 110 119 212 213
 $ v2: num  170 100 180 202 210
 $ v3: num  13 11 16 22 24

resultant data frame looks like

> str(fin_df)
'data.frame':   5 obs. of  10 variables:
 $ dt    : Factor w/ 4 levels "2015-01-01","2015-01-05",..: 1 2 3 1 4
 $ id    : Factor w/ 2 levels "i1","i2": 1 1 1 2 2
 $ v1    : Factor w/ 5 levels "110","115","119",..: 1 2 3 4 5
 $ v2    : Factor w/ 5 levels "100","170","180",..: 1 2 3 4 5
 $ v3    : Factor w/ 5 levels "11","13","16",..: 1 2 3 4 5
 $ lag_dt: Factor w/ 3 levels "2015-01-05","2015-01-06",..: 1 2 NA 3 NA
 $ lag_id: Factor w/ 2 levels "i1","i2": 1 1 NA 2 NA
 $ lag_v1: Factor w/ 3 levels "115","119","213": 1 2 NA 3 NA
 $ lag_v2: Factor w/ 3 levels "170","180","210": 1 2 NA 3 NA
 $ lag_v3: Factor w/ 3 levels "13","16","24": 1 2 NA 3 NA

Answer 1

zoo zoo objects are time series so normally the way this is done so that the result is a time series is to use wide form:

read.zoo(df, split = 2) # zoo object created by splitting on column 2

giving:

           v1.i1 v2.i1 v3.i1 v1.i2 v2.i2 v3.i2
2015-01-01   110   100    11   212   202    22
2015-01-02    NA    NA    NA   213   210    24
2015-01-05   115   170    13    NA    NA    NA
2015-01-06   119   180    16    NA    NA    NA

list or simply split the data frame into a list of zoo objects

L <- lapply(split(df[-2], df$id), read.zoo)

giving:

> L
$i1
            v1  v2 v3
2015-01-01 110 100 11
2015-01-05 115 170 13
2015-01-06 119 180 16

$i2
            v1  v2 v3
2015-01-01 212 202 22
2015-01-02 213 210 24

melt Using the reshape2 package we could create long form:

m <- melt(df, id = 1:2)

giving:

> m
           dt id variable value
1  2015-01-01 i1       v1   110
2  2015-01-05 i1       v1   115
3  2015-01-06 i1       v1   119
4  2015-01-01 i2       v1   212
5  2015-01-02 i2       v1   213
6  2015-01-01 i1       v2   100
7  2015-01-05 i1       v2   170
8  2015-01-06 i1       v2   180
9  2015-01-01 i2       v2   202
10 2015-01-02 i2       v2   210
11 2015-01-01 i1       v3    11
12 2015-01-05 i1       v3    13
13 2015-01-06 i1       v3    16
14 2015-01-01 i2       v3    22
15 2015-01-02 i2       v3    24

In this form it's easy to get various slices. For example,

> subset(m, dt == "2015-01-01")
           dt id variable value
1  2015-01-01 i1       v1   110
4  2015-01-01 i2       v1   212
6  2015-01-01 i1       v2   100
9  2015-01-01 i2       v2   202
11 2015-01-01 i1       v3    11
14 2015-01-01 i2       v3    22

3d array Another possibility is to represent it as a three dimensional array. m is from melt solution above. We could permute the components of the second argument to get variations:

a <- acast(m, ... ~ id ~ dt)

giving:

, , 2015-01-01

    i1  i2
v1 110 212
v2 100 202
v3  11  22

, , 2015-01-02

   i1  i2
v1 NA 213
v2 NA 210
v3 NA  24

, , 2015-01-05

    i1 i2
v1 115 NA
v2 170 NA
v3  13 NA

, , 2015-01-06

    i1 i2
v1 119 NA
v2 180 NA
v3  16 NA

Various slices are easy to get: a[1,,], a[,1,], a[,,1] .

Updates Have added to solution and rearranged and improved some solutions.