Dataset generation fails with "cannot mix list and non-list" when using nested arrays in tool metadata

[dtang19]

Description

I encountered a serialization error when generating a dataset using a config that includes nested arrays in the tool _meta field.

Steps to Reproduce

Run generation with the homoglyph.yaml config (included in my recent PR #525):

deepfabric generate examples/mcp-adversarial-suite/configs/homoglyph.yaml --num-samples 3 --tui simple

[dtang19]

Error Output

The generation step completes successfully (e.g. 13/15 samples), but the final save step fails:
Successfully generated 13 samples ⚠️ Failed to generate 2 samples ❌ Dataset generation failed: cannot mix list and non-list, non-null values ❌ Error: Error creating dataset: cannot mix list and non-list, non-null values

Context

This seems to be caused by the _meta field in the tool definition, which contains a list of strings:

    _meta:
      expected_codepoints: ["U+0072", "U+0065", ...]
It appears the dataset serializer struggles when some fields in the dataset are simple types and others (like this metadata) are lists/arrays.

[lukehinds]

hey @dtang19 , my bad I should have reviewed better when the PR was up.

some observations..

There are quite a few config params not known to deepfabric eval_metrics mcp_server are these values that are parsed and used by MCP Adversarial Suite or is the intent that deepfabric would use these values?

The tools you have in the config files is mcp schema, but we don't currently leverage that in the config. There are two approaches at present, we can use the builtin file_read, list_files, write_file , delete_file and populate them with data using scenario_seed

generation:
  tools:
    spin_endpoint: "http://localhost:3000"
    components:
      builtin:
        - read_file
        - write_file
        - list_files
    scenario_seed:
      files:
        "config.json": '{"debug": true, "port": 8080}'
        "data/users.json": '[{"id": 1, "name": "Alice"}]'

Of the other is we load a full version into the spin service which you can do direct from the MCP server instance itself using import-tools command: https://docs.deepfabric.dev/cli/import-tools/

deepfabric import-tools --transport stdio \
  --command "npx -y figma-developer-mcp --stdio" \
  --env "FIGMA_API_KEY=your-api-key" \
  --spin http://localhost:3000

This will load the MCP schema of the server into spin. You can then populate it with data:

curl -X POST http://localhost:3000/mock/add-fixture \
  -H "Content-Type: application/json" \
  -d '{
    "name": "figma_init",
    "match": {"name": "My Design"},
    "response": {"group_id": 55, "public": 0}
  }'

Or we could load a big batch like we do in here: https://github.com/always-further/deepfabric/tree/main/examples/tools-sdk-examples/github

I realise this is a lot of information, i think lets chat I can help get this setup with you - I just need to understand the expected flow