Transformer fields set does not match our data

[rolanddb]

Hi,
I'm trying to load the atomic.events data from Snowplow into Spark.
I'd like to do this using the EventTransformer.transform() method.

We observe a difference in the fields in our data, compared to what the transform method requires. Therefore, all events are marked as failure (unable to parse).

This is the mismatch:
Fields in our data: 128
Fields in SDK transformer: 131
In transformer but not in data: Set(derived_contexts, unstruct_event, contexts, refr_device_tstamp)
In data but not in transformer: Set(refr_dvce_tstamp)

• Was refr_dvce_tstamp renamed?
• How come we are missing the contexts and unstruct_event?

I can make it work by forking the SDK and modifying the transform method, but ideally I'd continue to use the main branch..

Thanks!

[alexanderdean]

Hi @rolanddb - hopefully all the answers you need are in #5. It would be great if you could open a PR adding support for your version of the enriched events!

[alexanderdean]

Ah @rolanddb - I read this slightly too fast. Are you trying to parse an extract from Redshift using this SDK? That's not supported - you should be pointing this at your enriched:good:archive instead.

[rolanddb]

Ah! So it wasn't me :)
I can make a PR tomorrow, it's a trivial change.
Thanks.

[alexanderdean]

See follow-up comment! I misunderstood your situation I think...

[rolanddb]

@alexanderdean I'm loading the events from S3, main/shredded/good.
Should I be using 'enriched' instead?

[alexanderdean]

Exactly, yes.

[rolanddb]

Ok, I'll give that a try tomorrow.

[alexanderdean]

Re-opening as @rolanddb's seems to be having an ongoing issue here.

[chuwy]

Hello @rolanddb,

We're about to publish Scala SDK 0.2.0, but still didn't manage to identify any case where transformer could not match enriched TSV data.

I see only one possible cause here - possibly you tried to load events enriched with Snowplow pre-R73 (released in December 2015), which produced more columns than it produces now. If I'm wrong here, would it be possible to provide some details (error message, TSV example) that breaks transformer.

[rolanddb]

Hi @chuwy,
There was some confusion about the difference between enriched / shredded data. So most of the difference between the dataset and the fields defined in the transformer was explained by that.

[alexanderdean]

Okay great, closing and descheduling...