Encoding vector length field not adjacent to the vector

Question

I have the following structure I like to encode. I'm aware that I can encode a vector with vector() if the size field is directly in front of the vector data. But here the field encoding the vector size is not adjacent.

case class Item(
    address: Int,
    size: Int,
)
case class Header {
    // lots of other fields before
    numOfItems: Int,
    // lots of other fields after
}
case class Outer(
    hdr: Header,
    items: Vector[]
)

Decoding of Outer is OK:

Header.numOfItems is read from the bit vector and items is created with vectorOfN(provide(hdr.numOfItems, Item.codec))

Encoding of Outer is the problem:

When encoding I would like to have numOfItem be taken from the items.length. I'm aware that I could set numOfItems with additional code when the items Vector is updated or with something like a "before encoding callback".

The question is if there is a more elegant solution. To me Header.numOfItems is redundant with Outer.items.length, so ideally only the Encoder should know about numOfItems.

Answer 1

You could try building a Codec using consume() and start without building the Outer object:

case class OuterExpanded(
  fieldBefore: Int, // Field before number of items in the binary encoding
  fieldAdter: Int,  // Field after number of items in the binary encoding
  items: Vector[Item] // Encoded items
)

// Single Item codec
def itemC: Codec[Item] = (int32 :: int32).as[Item] 

def outerExpandedC: Codec[OuterExpanded] = ( 
  int32 ::                          // Field before count
  int32.consume( c =>               // Item count 
      int32 ::                      // Field after count
      vectorOfN(provide(c), itemC))   // 'consume' (use and forget) the count
    (_.tail.head.length)              // provide the length when encoding
  ).as[OuterExpanded]

As defined above, you get the following when encoding: outerExpandedC.encode(OuterExpanded(-1, -1, Vector(Item(1,2), Item(3,4)))) returns

Successful(BitVector(224 bits, 
     0xffffffff00000002fffffffe00000001000000020000000300000004))
              ^       ^       ^       ^-------^-> First Item
              |-1     |       |-2
                      |Vector length inserted between the two header fields

Afterwards, you can xmap() the Codec[OuterExpanded] to pack the other header fields together into their own object. Ie (adding two conversion methods to Outer and OuterExpanded):

def outerC: Codec[Outer] = 
  outerExpandedC.xmap(_.toOuter,_.expand)

case class OuterExpanded(fieldBefore: Int, fieldAfter: Int,  items: Vector[Item]) {
  def toOuter = Outer(Hdr(fieldBefore,fieldAfter), items)
}

case class Outer(header: Hdr, items: Vector[Item]) {
  def expand = OuterExpanded(header.beforeField1, header.beforeField1, items)
}

_{This can probably be adapted to more complex cases, though I'm not entirely familar with shapeless' heterogeneous lists – or HList – and there might be nicer ways to get to the length of the vector rather than calling _.tail.head.length in the example above, especially if you end up with more than one field after the number of encoded values.}

_{Also, the Codec scaladoc is a nice place to discover useful operators}

Answer 2

Based on the previous answer I came up with something like the code below. I used the consume trick form above and an AtomicInteger to hold the size of the vector.

import java.util.concurrent.atomic.AtomicInteger
import scala.Vector
import org.scalatest._
import scodec._
import scodec.Attempt._
import scodec.codecs._
import scodec.bits._

object SomeStructure {
  case class Item(
    address: Int,
    size: Int)

  def itemC: Codec[Item] = (int32 :: int32).as[Item]

  case class Hdr(
    beforeField1: Int,
    // vectorSize would be here
    afterField1: Int)
  // vectorSize is an "in" param when encoding and an "out" param when decoding
  def hdrC(vectorSize: AtomicInteger): Codec[Hdr] =
    (int32 ::
      int32.consume(c => {
        vectorSize.set(c);
        int32
      })((i) => vectorSize.get)).as[Hdr]

  case class Outer(
    hdr: Hdr,
    var items: Vector[Item])

  def outerC() = {
    // when decoding the length is in this atomic integer
    // when encoding it is set before 
    val c = new AtomicInteger(-1)
    (hdrC(c) :: lazily(vectorOfN(provide(c.get), itemC)))
      .xmapc(identity)((g) => { c.set(g.tail.head.length); g })
  }.as[Outer]
}

import SomeStructure._
class SomeStructureSpec extends FlatSpec with Matchers {
  val bv = hex"ffffffff00000002ffffffff00000001000000020000000300000004".bits
  val v = Vector(Item(1, 2), Item(3, 4))
  val bv2 = hex"ffffffff00000003ffffffff000000010000000200000003000000040000000500000006".bits
  val v2 = Vector(Item(1, 2), Item(3, 4), Item(5, 6))
  val o = Outer(Hdr(-1, -1), v)

  "outerC" should "encode" in {
    o.items = v
    outerC.encode(o) shouldBe Successful(bv)
    o.items = v2
    outerC.encode(o) shouldBe Successful(bv2)
  }
  it should "decode" in {
    o.items = v
    outerC.decode(bv) shouldBe Successful(DecodeResult(o, BitVector.empty))
    o.items = v2
    outerC.decode(bv2) shouldBe Successful(DecodeResult(o, BitVector.empty))
  }
}