Repository navigation
Adding full evals config - #4806
Conversation
| "query": "List all recipients receiving money from Treasury last month", | ||
| "expected_output": { | ||
| "selectedAwardingAgencies": { | ||
| "456_toptier": { | ||
| "id": 456, | ||
| "agencyType": "toptier", | ||
| "toptier_flag": true, | ||
| "toptier_agency": { | ||
| "id": 22, | ||
| "toptier_code": "020", | ||
| "abbreviation": "TREAS", | ||
| "name": "Department of the Treasury" | ||
| } | ||
| } | ||
| }, | ||
| "timePeriodType": "dr", | ||
| "time_period": [{"start_date": "2026-09-01", "end_date": "2026-09-30"}] |
There was a problem hiding this comment.
We will need a better way to handle these types of relative dates other than hardcoding
| "93.00A": {}, | ||
| "93.078": {}, | ||
| "93.226": {}, | ||
| "93.391": {}, | ||
| "93.428": {}, | ||
| "93.440": {}, | ||
| "93.450": {}, | ||
| "93.457": {}, | ||
| "93.691": {}, | ||
| "93.717": {}, | ||
| "93.720": {}, | ||
| "93.801": {}, | ||
| "93.835": {}, | ||
| "93.968": {} |
There was a problem hiding this comment.
We should rework the seleced cfda to just be a list and then convert it to a dictionary later like we do for selectedAwardID and Keyword. I think the llm is getting confused by this shape.
| "F001", | ||
| "F002" | ||
| ], | ||
| "timePeriodFY": ["2025"] |
There was a problem hiding this comment.
It's currently fiscal year 2027, so this is already out of date.
There was a problem hiding this comment.
Brings up a good question about how to keep a JSON like this up to date for items that are dynamic like "last year". We could go the route of some placeholder text that we replace when reading this file.
| "USA_FL": { | ||
| "identifier": "USA_FL", | ||
| "filter": { | ||
| "country": "USA", | ||
| "state": "FL" | ||
| }, | ||
| "display": { | ||
| "entity": "State", | ||
| "standalone": "Florida", | ||
| "title": "Florida" |
There was a problem hiding this comment.
The display that we get back from the llm is slightly different format, so we need to determine the correct format and update the llm response or the expected data. I think all location related questions are failing now.
| "F001", | ||
| "F002" | ||
| ], | ||
| "timePeriodFY": ["2025"] |
There was a problem hiding this comment.
Brings up a good question about how to keep a JSON like this up to date for items that are dynamic like "last year". We could go the route of some placeholder text that we replace when reading this file.
2 participants
Updates to the config.json file for the evals framework.