Implement schema_modes config to infer schema fields from document

[MadLittleMods]

MongoDB doesn't have a strict data model, but extracting data should conform to a schema of some kind to help the targets and other downstream processes load the data with appropriate data types.

The Proposed Solution

Two modes: schemaless and strict

These modes are to be specified in the tap's config to associate with collections present in the mongo database, in this form:

config.json

{
   "schema_modes": {
       "users": "strict",
       "events": "schemaless",
       ...
   },
   ...
}

schemaless

This mode will emit a SCHEMA message with an empty object JSON schema associated, {"type": "object"}, as it currently does. This is more loose on which fields should be loaded by the target, and also offloads the data typing logic to the target.

strict

This mode will sample the most recent document in the schema, and use the fields it finds on that document to create the SCHEMA to emit for that table's records. In this case, the target can rely on the SCHEMA message to type the data based on the JSON schema types found in this message.

This isn't 100% to cover all cases, so any methods of making the data typing more accurate (without having to sample every single document) can be applied in the final implementation. As such, I think the design of strict is open to further refinement.

---

As proposed by @dmosorast, #5 (comment)

Implement schema_modes config,

{
    "schema_modes": {
        "users": "strict",
        "events": "schemaless",
        ...
    },
    ...
}

• strict: Infer schema fields from latest document in the collection
• schemaless: No change (backwards compatible)

[timvisher]

It's a little difficult to get up to speed on this issue without reading through and parsing a fairly length conversation on #5.

Can we get a TL;DR here summarizing the issue being addressed, the proposed solution, and finally the details (as you already have). Thanks!

[dmosorast]

Good idea! I can summarize the key points from the other thread to keep the context going.

Summary

The Issue

MongoDB doesn't have a strict data model, but extracting data should conform to a schema of some kind to help the targets and other downstream processes load the data with appropriate data types.

The Proposed Solution

Two modes: schemaless and strict

These modes are to be specified in the tap's config to associate with collections present in the mongo database, in this form:

config.json

{
   "schema_modes": {
       "users": "strict",
       "events": "schemaless",
       ...
   },
   ...
}

schemaless

This mode will emit a SCHEMA message with an empty object JSON schema associated, {"type": "object"}, as it currently does. This is more loose on which fields should be loaded by the target, and also offloads the data typing logic to the target.

strict

This mode will sample the most recent document in the schema, and use the fields it finds on that document to create the SCHEMA to emit for that table's records. In this case, the target can rely on the SCHEMA message to type the data based on the JSON schema types found in this message.

This isn't 100% to cover all cases, so any methods of making the data typing more accurate (without having to sample every single document) can be applied in the final implementation. As such, I think the design of strict is open to further refinement.

[dmosorast]

@MadLittleMods I noticed in your response that the definitions of the two modes are switched from what I wrote. My understanding was that schemaless is the backwards compatible mode, and strict would sample the latest document. Is that not accurate? I'd like to confirm to make sure we're on the same page.

[MadLittleMods]

@dmosorast Same page, I just assumed from the naming that schemaless meant it was fluid and could adapt to the documents (ambiguous). But your way of defining them is good with me (feel free to update description).

[timvisher]

Thanks for this, everyone. Hopefully someone can act on this now. :)