Skip to content

Reproduce the full DACE 84-benchmark suite (FTA paper, OOPSLA'17) #410

Description

@alcides

Goal

Reproduce the full 84-benchmark evaluation suite of the FTA / DACE paper —
Wang, Dillig & Singh, Synthesis of Data Completion Scripts using Finite Tree
Automata
(OOPSLA 2017, arXiv:1707.01469) —
as runnable Aeon FTA programming-by-example tasks under
examples/synthesis/dace/, complementing the worked examples already there.

What already exists

The paper's concretely-specified examples (Fig. 1 running example + the five
§2 motivating examples) are reproduced and machine-verified via -s fta:

Paper example Aeon file StackOverflow source
Fig. 1 (Delta = last − first), MINUS(?1,?2) examples/synthesis/dace/pbe/delta.ae http://stackoverflow.com/questions/30952426
2.1 LOCF +1 pbe/locf.ae http://stackoverflow.com/questions/38100208
2.2 previous same-id (relational) pbe/prev_sameid.ae http://stackoverflow.com/questions/16345583
2.3 "making turns" pbe/turns.ae http://stackoverflow.com/questions/29606616
2.4 group total = COUNT pbe/group_count.ae http://stackoverflow.com/questions/13398218
2.5 previous-else-next (switch) pbe/fallback.ae (variant of q/38100208)

Covered by tests/dace_test.py::test_fta_pbe_completion. The DSL primitives live
in libraries/Dace.ae.

The gap

The paper evaluates DACE on 84 tasks but only the 6 above are individually
published (as footnoted StackOverflow URLs). The other ~78 are not available:

  • Composition (from the evaluation section): 46 data imputation (R / Python
    pandas), 32 spreadsheet computation (Excel / Google Sheets), 6 relational-DB
    virtual columns.
  • How they were collected: StackOverflow search for the keywords
    "data imputation", "missing value", "missing data", "spreadsheet formula".
  • Not enumerated in the paper or its appendices (A–D are scoring, soundness,
    complexity, VSA-vs-FTA — no benchmark list).

Where the original artifact was searched for (came up empty)

So we know whom to ask and what is/isn't public:

  • DACE is a Java tool — not open-sourced.
  • github.com/xwangsd (Xinyu Wang, lead author) — no public repos.
  • utopia-group (Işıl Dillig's group) — no DACE / data-completion repo.
  • MIPL-group (Wang's Michigan lab) — only SQL-synthesis repos.
  • No Zenodo / artifact archive (OOPSLA'17 predates routine PACMPL artifact archival).

Action for maintainer: email the authors (Xinyu Wang xwangsd@umich.edu,
Işıl Dillig, Rishabh Singh) to request the original 84 benchmarks and/or the DACE
source. If obtained, transcribe them directly; otherwise reconstruct (below).

Proposed approach (if the originals can't be obtained)

Reconstruct an equivalent suite by re-scraping StackOverflow with the paper's
documented keywords, then encode each as a pbe/*.ae task (a Column/Grid
global + @example rows + a ?hole function of the cell index), extending
libraries/Dace.ae with any missing navigation/aggregation primitives. Note
clearly that a re-scraped set is DACE-inspired, not the verbatim original.

Checklist

  • Contact the authors for the original 84 benchmarks + DACE source
  • If received: transcribe verbatim into examples/synthesis/dace/
  • If not: re-scrape via the documented keywords and reconstruct
  • Imputation category (target 46): R/pandas LOCF/NOCB/group-fill variants
  • Spreadsheet category (target 32): Excel/Sheets derived-column formulas
  • Relational-DB category (target 6): virtual-column / view computations
  • Extend libraries/Dace.ae with primitives each task needs
  • Wire all into tests/dace_test.py and update examples/synthesis/dace/README.md
  • (Stretch) Whole-Table cell extraction (GetCell navigation) rather than per-Column index functions, to match the paper's DSL more faithfully

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions