Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add schemars to ZeroVec #4792

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion utils/zerovec/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ all-features = true

[dependencies]
zerofrom = { workspace = true }
serde_json = "1.0"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue: Use the { workspace = true } version of the dependency.


zerovec-derive = { workspace = true, optional = true}

Expand All @@ -37,6 +38,7 @@ serde = { version = "1.0", default-features = false, features = ["alloc"], optio
# and all 0.7 versions, but not further.
yoke = { version = ">=0.6.0, <0.8.0", path = "../yoke", optional = true }
twox-hash = { version = "1.6", default-features = false, optional = true }
schemars = "0.8.16"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: this must be an optional dependency

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and please add it to the top-level Cargo.toml (ideally wait until #4844 is merged)


[dev-dependencies]
bincode = "1.3"
Expand All @@ -48,7 +50,6 @@ rand = "0.8"
rand_distr = "0.4"
rand_pcg = "0.3"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
yoke = { path = "../../utils/yoke", features = ["derive"] }
zerofrom = { path = "../../utils/zerofrom", features = ["derive"] }

Expand Down
34 changes: 34 additions & 0 deletions utils/zerovec/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,19 @@
use core::any;
use core::fmt;

use crate::__zerovec_internal_reexport::boxed::Box;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: just alloc::boxed::Box

use alloc::borrow::ToOwned;
use alloc::format;
use alloc::string::String;
use alloc::vec;
use schemars::gen::SchemaGenerator;
use schemars::schema::InstanceType;
use schemars::schema::Schema;
use schemars::schema::SchemaObject;
use schemars::JsonSchema;

use serde_json::Value;

/// A generic error type to be used for decoding slices of ULE types
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[non_exhaustive]
Expand Down Expand Up @@ -33,6 +46,27 @@ impl fmt::Display for ZeroVecError {
}
}
}
impl JsonSchema for ZeroVecError {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue: I don't think you should need this impl. It should go away when you fix the ZeroSlice impl.

fn schema_name() -> String {
format!("ZeroVecError")
}

fn json_schema(_gen: &mut SchemaGenerator) -> Schema {
Schema::Object(SchemaObject {
instance_type: Some(InstanceType::String.into()),
enum_values: Some(vec![
Value::String("InvalidLength".to_owned()),
Value::String("ParseError".to_owned()),
Value::String("VarZeroVecFormatError".to_owned()),
]),
metadata: Some(Box::new(schemars::schema::Metadata {
description: Some("ZeroVecError is an enum representing errors that can occur during the decoding of slices of ULE".into()),
..Default::default()
})),
..Default::default()
})
}
}

impl ZeroVecError {
/// Construct a parse error for the given type
Expand Down
47 changes: 47 additions & 0 deletions utils/zerovec/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,53 @@ pub use zerovec_derive::make_varule;
mod tests {
use super::*;
use core::mem::size_of;
use schemars::{gen::SchemaGenerator, JsonSchema};
use serde::{Deserialize, Serialize};
use serde_json::Value;

#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(JsonSchema)]
pub struct DataStruct<'data> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: for in-file unit tests, make one test for each of these. For example,

#[test]
fn test_schema_zerovec_u32() {
    let gen = SchemaGenerator::default();
    let schema = gen.into_root_schema_for::<ZeroVec<u32>>();
    // ...
}

As suggested below, use insta for the more complex tests.

#[cfg_attr(feature = "serde", serde(borrow))]
nums: ZeroVec<'data, u32>,

#[cfg_attr(feature = "serde", serde(borrow))]
chars: ZeroVec<'data, char>,

#[cfg_attr(feature = "serde", serde(borrow))]
strs: VarZeroVec<'data, str>,

#[cfg_attr(feature = "serde", serde(borrow))]
nested_numbers: VarZeroVec<'data, ZeroSlice<u32>>,
}

#[test]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: use insta to help generate test cases and check-in the JSON Schema output.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, will look into it 👍

fn check_schema() {
let gen = SchemaGenerator::default();
let schema = gen.into_root_schema_for::<DataStruct>();
let schema_json =
serde_json::to_string_pretty(&schema).expect("Failed to serialize schema");
println!("{}", schema_json);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Please include the generated JSON.

let parsed_schema: Value =
serde_json::from_str(&schema_json).expect("Failed to parse schema JSON");

// Check for the existence of "ZeroVec<Character>" and "ZeroVec<uint32>" in `definitions`
let definitions = parsed_schema
.get("definitions")
.expect("No definitions found in schema");
assert!(
definitions.get("ZeroVec<Character>").is_some(),
"Definition for ZeroVec<Character> not found"
);
assert!(
definitions.get("ZeroVec<uint32>").is_some(),
"Definition for ZeroVec<uint32> not found"
);
assert!(
definitions.get("VarZeroVec<String>").is_some(),
"Definition for VarZeroVec<String> not found"
);
}

/// Checks that the size of the type is one of the given sizes.
/// The size might differ across Rust versions or channels.
Expand Down
35 changes: 35 additions & 0 deletions utils/zerovec/src/varzerovec/vec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,17 @@

use crate::ule::*;

use alloc::borrow::Cow;
use alloc::boxed::Box;
use alloc::format;
use alloc::string::String;
use alloc::vec::Vec;
use core::cmp::{Ord, Ordering, PartialOrd};
use core::fmt;
use core::ops::Deref;
use schemars::gen::SchemaGenerator;
use schemars::schema::{ArrayValidation, InstanceType, Schema, SchemaObject};
use schemars::JsonSchema;

use super::*;

Expand Down Expand Up @@ -232,6 +239,34 @@ impl<T: VarULE + ?Sized, F: VarZeroVecFormat> Deref for VarZeroVec<'_, T, F> {
}
}

impl<'a, T, F> JsonSchema for VarZeroVec<'a, T, F>
where
T: VarULE + ?Sized + JsonSchema,
F: VarZeroVecFormat,
{
fn schema_name() -> String {
format!("VarZeroVec<{}>", T::schema_name())
}

fn schema_id() -> Cow<'static, str> {
Cow::Owned(format!("zerovec::VarZeroVec<{}>", T::schema_id()))
}

fn json_schema(gen: &mut SchemaGenerator) -> Schema {
let items_schema = gen.subschema_for::<T>();

SchemaObject {
instance_type: Some(InstanceType::Array.into()),
array: Some(Box::new(ArrayValidation {
items: Some(items_schema.into()),
..Default::default()
})),
..Default::default()
}
.into()
}
}

impl<'a, T: VarULE + ?Sized, F: VarZeroVecFormat> VarZeroVec<'a, T, F> {
/// Creates a new, empty `VarZeroVec<T>`.
///
Expand Down
32 changes: 32 additions & 0 deletions utils/zerovec/src/zerovec/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,15 @@ mod serde;

mod slice;

use schemars::gen::SchemaGenerator;
use schemars::schema::{ArrayValidation, InstanceType, Schema};
pub use slice::ZeroSlice;

use crate::ule::*;
use alloc::borrow::Cow;
use alloc::boxed::Box;
use alloc::format;
use alloc::string::String;
use alloc::vec::Vec;
use core::cmp::{Ord, Ordering, PartialOrd};
use core::fmt;
Expand All @@ -23,6 +28,7 @@ use core::mem;
use core::num::NonZeroUsize;
use core::ops::Deref;
use core::ptr::{self, NonNull};
use schemars::{schema::SchemaObject, JsonSchema};

/// A zero-copy, byte-aligned vector for fixed-width types.
///
Expand Down Expand Up @@ -114,6 +120,32 @@ impl<'a, T: AsULE> Deref for ZeroVec<'a, T> {
}
}

impl<'a, T> JsonSchema for ZeroVec<'a, T>
where
T: AsULE + JsonSchema,
{
fn schema_name() -> String {
format!("ZeroVec<{}>", T::schema_name())
}

fn schema_id() -> Cow<'static, str> {
Cow::Owned(format!("zerovec::ZeroVec<{}>", T::schema_id()))
}

fn json_schema(gen: &mut SchemaGenerator) -> Schema {
let element_schema = gen.subschema_for::<T>();
SchemaObject {
instance_type: Some(InstanceType::Array.into()),
array: Some(Box::new(ArrayValidation {
items: Some(element_schema.into()),
..Default::default()
})),
..Default::default()
}
.into()
}
}

// Represents an unsafe potentially-owned vector/slice type, without a lifetime
// working around dropck limitations.
//
Expand Down
37 changes: 37 additions & 0 deletions utils/zerovec/src/zerovec/slice.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
// (online at: https://github.com/unicode-org/icu4x/blob/main/LICENSE ).

use super::*;
use crate::alloc::string::ToString;
use ::serde::Serialize;
use alloc::boxed::Box;
use core::cmp::Ordering;
use core::ops::Range;
Expand Down Expand Up @@ -543,6 +545,41 @@ impl<T: AsULE + Ord> Ord for ZeroSlice<T> {
}
}

impl<T> JsonSchema for ZeroSlice<T>
where
T: AsULE + JsonSchema + Serialize, // Ensure T is serializable for accurate schema representation
{
fn schema_name() -> String {
format!("ZeroSlice<{}>", T::schema_name())
}

fn schema_id() -> Cow<'static, str> {
Cow::Owned(format!("zerovec::ZeroSlice<{}>", T::schema_id()))
}

fn json_schema(gen: &mut SchemaGenerator) -> Schema {
// Instead of generating a subschema for T, we generate a schema representing the byte array
let byte_schema = gen.subschema_for::<Vec<u8>>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: why? ZeroSlice is logically an array of items

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, ZeroVec and ZeroSlice should both become arrays of T.

Maybe they can resolve to the same JsonSchema impl.

Copy link
Contributor Author

@ashu26jha ashu26jha Apr 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, expected interaction should be as array not bytes. Lets fallback to implementing as arrays of T.


SchemaObject {
instance_type: Some(InstanceType::Object.into()),
object: Some(Box::new({
let mut object_validation = schemars::schema::ObjectValidation::default();
object_validation
.properties
.insert("data".to_string(), byte_schema);
object_validation
.properties
.insert("error".to_string(), gen.subschema_for::<ZeroVecError>());
object_validation.required.insert("data".to_string());
object_validation
})),
..Default::default()
}
.into()
}
}
Comment on lines +560 to +581
Copy link
Contributor Author

@ashu26jha ashu26jha Apr 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ maintainers I am not certain if this is the right way to move forward


impl<T: AsULE> AsRef<ZeroSlice<T>> for Vec<T::ULE> {
fn as_ref(&self) -> &ZeroSlice<T> {
ZeroSlice::<T>::from_ule_slice(self)
Expand Down
Loading