Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wasmparser: feature gate Wasm simd support #1903

Open
wants to merge 57 commits into
base: main
Choose a base branch
from
Open

wasmparser: feature gate Wasm simd support #1903

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
827b128
add for_each_simd_operator generator macro
Robbepop Nov 12, 2024
5944fb9
define SimdOperator enum
Robbepop Nov 12, 2024
d3e99f0
add SimdOperator::operator_arity impl
Robbepop Nov 12, 2024
9698f97
add VisitSimdOperator trait definition
Robbepop Nov 12, 2024
76557eb
define VisitSimdOperator delegates
Robbepop Nov 12, 2024
6adace1
add benchmark NopVisit impl
Robbepop Nov 12, 2024
2fa11f2
add simd crate feature
Robbepop Nov 12, 2024
baa1de5
add simd_visitor method to VisitOperator trait
Robbepop Nov 12, 2024
bb4a415
add VisitSimdOperator impl to OperatorFactory
Robbepop Nov 12, 2024
df32e79
use VisitOperator::simd_visitor in BinaryReader::visit_operator
Robbepop Nov 12, 2024
de0f049
add lifetime to return value of simd_visitor method
Robbepop Nov 12, 2024
efd1c0c
add VisitSimdOperator impl for OperatorValidator
Robbepop Nov 12, 2024
26cc42d
add Operator::Simd variant
Robbepop Nov 12, 2024
ac2d6a8
remove simd operators from for_each_operator macro
Robbepop Nov 12, 2024
531240e
adjust wasmprinter crate for simd crate feature
Robbepop Nov 19, 2024
5c94737
enable wasmparser's simd feature by default
Robbepop Nov 19, 2024
bfb354a
fix trait impl signature
Robbepop Nov 19, 2024
e257fcb
add docs to wasmparser's simd crate feature
Robbepop Nov 19, 2024
eb3845d
feature gate simd related code in wasmparser
Robbepop Nov 19, 2024
77a33de
update docs for SimdOperator
Robbepop Nov 19, 2024
7439a1f
add missing `simd` crate feature gates
Robbepop Nov 19, 2024
82412a5
Merge branch 'main' into rf-feature-gate-simd
Robbepop Nov 19, 2024
c373909
move simd op validation down in file
Robbepop Nov 19, 2024
267808c
put simd specific operator validation in separate file
Robbepop Nov 19, 2024
28e15df
add docs to for_each_simd_operator macro
Robbepop Nov 19, 2024
faa8317
move for_each_simd_operator macro into a separate file
Robbepop Nov 19, 2024
6eef116
fix docs for VisitSimdOperator
Robbepop Nov 19, 2024
cc88460
allow missing docs again for VisitSimdOperator
Robbepop Nov 19, 2024
a703340
add docs and example to VisitOperator::simd_visitor
Robbepop Nov 19, 2024
e6614e0
move visit_0xfd_operator into separate file
Robbepop Nov 19, 2024
3c9559c
apply rustftm
Robbepop Nov 19, 2024
a8f67fe
wasmprinter: fix remaining simd feature toggles
Robbepop Nov 19, 2024
c9204e6
apply rustfmt
Robbepop Nov 19, 2024
2755180
fix wasmparser benchmarks
Robbepop Nov 19, 2024
accb113
wasm-encoder: add simd support
Robbepop Nov 19, 2024
c87a791
apply rustfmt and avoid formatting some parts
Robbepop Nov 19, 2024
7b0a04b
wasm-mutate: fix compile errors
Robbepop Nov 19, 2024
76d4d6a
apply rustfmt
Robbepop Nov 19, 2024
7c7eb46
fix dev-dependencies for wasm-mutate
Robbepop Nov 19, 2024
73fb3ba
add simd_visitor impl to NopVisitor
Robbepop Nov 19, 2024
a458b3a
add missing VisitSimdOperator impl for WasmProposalValidator
Robbepop Nov 19, 2024
8924923
mark doc example as compile_fail
Robbepop Nov 20, 2024
8884f95
add simd support for VisitConstOperator
Robbepop Nov 20, 2024
0beba9e
use wasmparser/simd in validate tool
Robbepop Nov 20, 2024
351e169
add simd crate feature propagation to wasm-tools CLI tool
Robbepop Nov 20, 2024
04f34ab
wasm-smith: use wasmparser/simd crate feature
Robbepop Nov 20, 2024
c419be7
apply rustfmt
Robbepop Nov 20, 2024
5c1eeb4
feature gate simd_visitor impl
Robbepop Nov 20, 2024
cec6bb7
unconditionally enable simd for wasmprinter
Robbepop Nov 21, 2024
1dfbf6e
unconditionally enable simd for wasm-encoder
Robbepop Nov 21, 2024
a98227f
remove wasm-tools simd feature (enable by default)
Robbepop Nov 21, 2024
efa3d43
use macros generate for_each_operator macros
Robbepop Nov 21, 2024
887a8cf
remove comment out line
Robbepop Nov 21, 2024
cbbc575
put docs on the exported macros
Robbepop Nov 21, 2024
ba48386
fix macro imports
Robbepop Nov 21, 2024
6b9a883
fix bugs in generator macros
Robbepop Nov 22, 2024
91ddc75
improve compile times for macros
Robbepop Nov 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ wasm-metadata = { version = "0.220.0", path = "crates/wasm-metadata" }
wasm-mutate = { version = "0.220.0", path = "crates/wasm-mutate" }
wasm-shrink = { version = "0.220.0", path = "crates/wasm-shrink" }
wasm-smith = { version = "0.220.0", path = "crates/wasm-smith" }
wasmparser = { version = "0.220.0", path = "crates/wasmparser", default-features = false, features = ['std'] }
wasmparser = { version = "0.220.0", path = "crates/wasmparser", default-features = false, features = ['std','simd'] }
wasmprinter = { version = "0.220.0", path = "crates/wasmprinter", default-features = false }
wast = { version = "220.0.0", path = "crates/wast", default-features = false }
wat = { version = "1.220.0", path = "crates/wat", default-features = false }
Expand All @@ -122,7 +122,7 @@ wat = { workspace = true, features = ['dwarf', 'component-model'] }
termcolor = { workspace = true }

# Dependencies of `validate`
wasmparser = { workspace = true, optional = true, features = ['component-model'] }
wasmparser = { workspace = true, optional = true, features = ['component-model', 'simd'] }
rayon = { workspace = true, optional = true }
bitflags = { workspace = true, optional = true }

Expand Down
2 changes: 1 addition & 1 deletion crates/wasm-encoder/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ wasmparser = { optional = true, workspace = true }
[dev-dependencies]
anyhow = { workspace = true }
tempfile = "3.2.0"
wasmparser = { path = "../wasmparser" }
wasmparser = { path = "../wasmparser", features = ["simd"] }
wasmprinter = { workspace = true }

[features]
Expand Down
173 changes: 101 additions & 72 deletions crates/wasm-encoder/src/reencode.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -601,6 +601,7 @@ impl Reencode for RoundtripReencoder {
#[allow(missing_docs)] // FIXME
pub mod utils {
use super::{Error, Reencode};
use crate::Instruction;
use crate::{CoreTypeEncoder, Encode};
use std::ops::Range;

Expand Down Expand Up @@ -1546,98 +1547,126 @@ pub mod utils {
}
}

#[rustfmt::skip]
macro_rules! translate_map {
// This case is used to map, based on the name of the field, from the
// wasmparser payload type to the wasm-encoder payload type through
// `Translator` as applicable.
($reencoder:ident $arg:ident tag_index) => ($reencoder.tag_index($arg));
($reencoder:ident $arg:ident function_index) => ($reencoder.function_index($arg));
($reencoder:ident $arg:ident table) => ($reencoder.table_index($arg));
($reencoder:ident $arg:ident table_index) => ($reencoder.table_index($arg));
($reencoder:ident $arg:ident dst_table) => ($reencoder.table_index($arg));
($reencoder:ident $arg:ident src_table) => ($reencoder.table_index($arg));
($reencoder:ident $arg:ident type_index) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident array_type_index) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident array_type_index_dst) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident array_type_index_src) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident struct_type_index) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident global_index) => ($reencoder.global_index($arg));
($reencoder:ident $arg:ident mem) => ($reencoder.memory_index($arg));
($reencoder:ident $arg:ident src_mem) => ($reencoder.memory_index($arg));
($reencoder:ident $arg:ident dst_mem) => ($reencoder.memory_index($arg));
($reencoder:ident $arg:ident data_index) => ($reencoder.data_index($arg));
($reencoder:ident $arg:ident elem_index) => ($reencoder.element_index($arg));
($reencoder:ident $arg:ident array_data_index) => ($reencoder.data_index($arg));
($reencoder:ident $arg:ident array_elem_index) => ($reencoder.element_index($arg));
($reencoder:ident $arg:ident blockty) => ($reencoder.block_type($arg)?);
($reencoder:ident $arg:ident relative_depth) => ($arg);
($reencoder:ident $arg:ident targets) => ((
$arg
.targets()
.collect::<Result<Vec<_>, wasmparser::BinaryReaderError>>()?
.into(),
$arg.default(),
));
($reencoder:ident $arg:ident ty) => ($reencoder.val_type($arg)?);
($reencoder:ident $arg:ident hty) => ($reencoder.heap_type($arg)?);
($reencoder:ident $arg:ident from_ref_type) => ($reencoder.ref_type($arg)?);
($reencoder:ident $arg:ident to_ref_type) => ($reencoder.ref_type($arg)?);
($reencoder:ident $arg:ident memarg) => ($reencoder.mem_arg($arg));
($reencoder:ident $arg:ident ordering) => ($reencoder.ordering($arg));
($reencoder:ident $arg:ident local_index) => ($arg);
($reencoder:ident $arg:ident value) => ($arg);
($reencoder:ident $arg:ident lane) => ($arg);
($reencoder:ident $arg:ident lanes) => ($arg);
($reencoder:ident $arg:ident array_size) => ($arg);
($reencoder:ident $arg:ident field_index) => ($arg);
($reencoder:ident $arg:ident try_table) => ($arg);
($reencoder:ident $arg:ident argument_index) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident result_index) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident cont_type_index) => ($reencoder.type_index($arg));
($reencoder:ident $arg:ident resume_table) => ((
$arg.handlers.into_iter().map(|h| $reencoder.handle(h)).collect::<Vec<_>>().into()
));
}

#[rustfmt::skip]
macro_rules! translate_build {
// This case takes the arguments of a wasmparser instruction and creates
// a wasm-encoder instruction. There are a few special cases for where
// the structure of a wasmparser instruction differs from that of
// wasm-encoder.
($reencoder:ident $op:ident) => (Instruction::$op);
($reencoder:ident BrTable $arg:ident) => (Instruction::BrTable($arg.0, $arg.1));
($reencoder:ident I32Const $arg:ident) => (Instruction::I32Const($arg));
($reencoder:ident I64Const $arg:ident) => (Instruction::I64Const($arg));
($reencoder:ident F32Const $arg:ident) => (Instruction::F32Const(f32::from_bits($arg.bits())));
($reencoder:ident F64Const $arg:ident) => (Instruction::F64Const(f64::from_bits($arg.bits())));
($reencoder:ident V128Const $arg:ident) => (Instruction::V128Const($arg.i128()));
($reencoder:ident TryTable $table:ident) => (Instruction::TryTable($reencoder.block_type($table.ty)?, {
$table.catches.into_iter().map(|c| $reencoder.catch(c)).collect::<Vec<_>>().into()
}));
($reencoder:ident $op:ident $arg:ident) => (Instruction::$op($arg));
($reencoder:ident $op:ident $($arg:ident)*) => (Instruction::$op { $($arg),* });
}

pub fn instruction<'a, T: ?Sized + Reencode>(
reencoder: &mut T,
arg: wasmparser::Operator<'a>,
) -> Result<crate::Instruction<'a>, Error<T::Error>> {
use crate::Instruction;

macro_rules! translate {
($( @$proposal:ident $op:ident $({ $($arg:ident: $argty:ty),* })? => $visit:ident ($($ann:tt)*))*) => {
Ok(match arg {
$(
wasmparser::Operator::$op $({ $($arg),* })? => {
$(
$(let $arg = translate!(map $arg $arg);)*
$(let $arg = translate_map!(reencoder $arg $arg);)*
)?
translate!(build $op $($($arg)*)?)
translate_build!(reencoder $op $($($arg)*)?)
}
)*
wasmparser::Operator::Simd(simd_arg) => simd_instruction(reencoder, simd_arg)?,
unexpected => unreachable!("encountered unexpected Wasm operator: {unexpected:?}"),
})
};

// This case is used to map, based on the name of the field, from the
// wasmparser payload type to the wasm-encoder payload type through
// `Translator` as applicable.
(map $arg:ident tag_index) => (reencoder.tag_index($arg));
(map $arg:ident function_index) => (reencoder.function_index($arg));
(map $arg:ident table) => (reencoder.table_index($arg));
(map $arg:ident table_index) => (reencoder.table_index($arg));
(map $arg:ident dst_table) => (reencoder.table_index($arg));
(map $arg:ident src_table) => (reencoder.table_index($arg));
(map $arg:ident type_index) => (reencoder.type_index($arg));
(map $arg:ident array_type_index) => (reencoder.type_index($arg));
(map $arg:ident array_type_index_dst) => (reencoder.type_index($arg));
(map $arg:ident array_type_index_src) => (reencoder.type_index($arg));
(map $arg:ident struct_type_index) => (reencoder.type_index($arg));
(map $arg:ident global_index) => (reencoder.global_index($arg));
(map $arg:ident mem) => (reencoder.memory_index($arg));
(map $arg:ident src_mem) => (reencoder.memory_index($arg));
(map $arg:ident dst_mem) => (reencoder.memory_index($arg));
(map $arg:ident data_index) => (reencoder.data_index($arg));
(map $arg:ident elem_index) => (reencoder.element_index($arg));
(map $arg:ident array_data_index) => (reencoder.data_index($arg));
(map $arg:ident array_elem_index) => (reencoder.element_index($arg));
(map $arg:ident blockty) => (reencoder.block_type($arg)?);
(map $arg:ident relative_depth) => ($arg);
(map $arg:ident targets) => ((
$arg
.targets()
.collect::<Result<Vec<_>, wasmparser::BinaryReaderError>>()?
.into(),
$arg.default(),
));
(map $arg:ident ty) => (reencoder.val_type($arg)?);
(map $arg:ident hty) => (reencoder.heap_type($arg)?);
(map $arg:ident from_ref_type) => (reencoder.ref_type($arg)?);
(map $arg:ident to_ref_type) => (reencoder.ref_type($arg)?);
(map $arg:ident memarg) => (reencoder.mem_arg($arg));
(map $arg:ident ordering) => (reencoder.ordering($arg));
(map $arg:ident local_index) => ($arg);
(map $arg:ident value) => ($arg);
(map $arg:ident lane) => ($arg);
(map $arg:ident lanes) => ($arg);
(map $arg:ident array_size) => ($arg);
(map $arg:ident field_index) => ($arg);
(map $arg:ident try_table) => ($arg);
(map $arg:ident argument_index) => (reencoder.type_index($arg));
(map $arg:ident result_index) => (reencoder.type_index($arg));
(map $arg:ident cont_type_index) => (reencoder.type_index($arg));
(map $arg:ident resume_table) => ((
$arg.handlers.into_iter().map(|h| reencoder.handle(h)).collect::<Vec<_>>().into()
));

// This case takes the arguments of a wasmparser instruction and creates
// a wasm-encoder instruction. There are a few special cases for where
// the structure of a wasmparser instruction differs from that of
// wasm-encoder.
(build $op:ident) => (Instruction::$op);
(build BrTable $arg:ident) => (Instruction::BrTable($arg.0, $arg.1));
(build I32Const $arg:ident) => (Instruction::I32Const($arg));
(build I64Const $arg:ident) => (Instruction::I64Const($arg));
(build F32Const $arg:ident) => (Instruction::F32Const(f32::from_bits($arg.bits())));
(build F64Const $arg:ident) => (Instruction::F64Const(f64::from_bits($arg.bits())));
(build V128Const $arg:ident) => (Instruction::V128Const($arg.i128()));
(build TryTable $table:ident) => (Instruction::TryTable(reencoder.block_type($table.ty)?, {
$table.catches.into_iter().map(|c| reencoder.catch(c)).collect::<Vec<_>>().into()
}));
(build $op:ident $arg:ident) => (Instruction::$op($arg));
(build $op:ident $($arg:ident)*) => (Instruction::$op { $($arg),* });
}

wasmparser::for_each_operator!(translate)
}

fn simd_instruction<'a, T: ?Sized + Reencode>(
reencoder: &mut T,
arg: wasmparser::SimdOperator,
) -> Result<crate::Instruction<'a>, Error<T::Error>> {
macro_rules! translate_simd {
($( @$proposal:ident $op:ident $({ $($arg:ident: $argty:ty),* })? => $visit:ident ($($ann:tt)*))*) => {
Ok(match arg {
$(
wasmparser::SimdOperator::$op $({ $($arg),* })? => {
$(
$(let $arg = translate_map!(reencoder $arg $arg);)*
)?
translate_build!(reencoder $op $($($arg)*)?)
}
)*
})
};
}

wasmparser::for_each_simd_operator!(translate_simd)
}

/// Parses the input `section` given from the `wasmparser` crate and adds
/// all the code to the `code` section.
pub fn parse_code_section<T: ?Sized + Reencode>(
Expand Down
6 changes: 3 additions & 3 deletions crates/wasm-mutate/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ workspace = true
[dependencies]
clap = { workspace = true, optional = true }
thiserror = "1.0.28"
wasmparser = { workspace = true }
wasm-encoder = { workspace = true, features = ["wasmparser"] }
wasmparser = { workspace = true, features = ['simd']}
wasm-encoder = { workspace = true, features = ['wasmparser'] }
rand = { workspace = true }
log = { workspace = true }
egg = "0.6.0"
Expand All @@ -24,4 +24,4 @@ anyhow = { workspace = true }
wat = { workspace = true }
wasmprinter = { workspace = true }
env_logger = { workspace = true }
wasmparser = { workspace = true, features = ['validate', 'features'] }
wasmparser = { workspace = true, features = ['validate', 'features', 'simd'] }
8 changes: 4 additions & 4 deletions crates/wasm-mutate/src/mutators/modify_const_exprs.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use crate::{Error, Mutator, ReencodeResult};
use rand::Rng;
use wasm_encoder::reencode::{self, Reencode, RoundtripReencoder};
use wasm_encoder::{ElementSection, GlobalSection};
use wasmparser::{ConstExpr, ElementSectionReader, GlobalSectionReader};
use wasmparser::{ConstExpr, ElementSectionReader, GlobalSectionReader, SimdOperator};

#[derive(PartialEq, Copy, Clone)]
pub enum ConstExpressionMutator {
Expand Down Expand Up @@ -76,7 +76,7 @@ impl<'cfg, 'wasm> Reencode for InitTranslator<'cfg, 'wasm> {
O::RefNull { .. } | O::I32Const { value: 0 | 1 } | O::I64Const { value: 0 | 1 } => true,
O::F32Const { value } => value.bits() == 0,
O::F64Const { value } => value.bits() == 0,
O::V128Const { value } => value.i128() == 0,
O::Simd(SimdOperator::V128Const { value }) => value.i128() == 0,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this again, what do you think of throwing all the simd ops back into Operator? It's already #[non_exhaustive] and would be nice to have a bit less churn as well in theory

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not yet sure if this works as intended due to the split into 2 macros and the problem that you cannot have macro invokation at enum variant definition position but I ll see how it goes. A downside to this approach is that this would result in tons of cfgs.

_ => false,
};
if is_simplest {
Expand All @@ -86,7 +86,7 @@ impl<'cfg, 'wasm> Reencode for InitTranslator<'cfg, 'wasm> {
let ty = match op {
O::I32Const { .. } => T::I32,
O::I64Const { .. } => T::I64,
O::V128Const { .. } => T::V128,
O::Simd(SimdOperator::V128Const { .. }) => T::V128,
O::F32Const { .. } => T::F32,
O::F64Const { .. } => T::F64,
O::RefFunc { .. }
Expand Down Expand Up @@ -131,7 +131,7 @@ impl<'cfg, 'wasm> Reencode for InitTranslator<'cfg, 'wasm> {
} else {
self.config.rng().gen()
}),
T::V128 => CE::v128_const(if let O::V128Const { value } = op {
T::V128 => CE::v128_const(if let O::Simd(SimdOperator::V128Const { value }) = op {
self.config.rng().gen_range(0..value.i128() as u128) as i128
} else {
self.config.rng().gen()
Expand Down
Loading