[DEV-16418] Update agency loader to account for Delta tables

[sethstoudenmier]

Description:

Convert the agency loader to an EMR job that handles updating all relevant Postgres and Delta tables.

Technical Details:

Updates to the agency loader that take into account File D existing now as Postgres and Delta tables. This will handle updating all of the raw / bronze and int / silver tables across Postgres and Delta. The rpt / gold tables are updated by later steps in the pipeline.

Adding locking to the Delta table .execute() method to address what appears to be a race condition by Spark when two jobs complete simultaneously. This will still allow Postgres jobs to process while the Spark jobs are force to run more synchronously.

Requirements for PR Merge:

1. Unit & integration tests updated
2. API documentation updated (examples listed below)
   1. API Contracts
   2. API UI
   3. Comments
3. Data validation completed (examples listed below)
   1. Does this work well with the current frontend? Or is the frontend aware of a needed change?
   2. Is performance impacted in the changes (e.g., API, pipeline, downloads, etc.)?
   3. Is the expected data returned with the expected format?
4. Appropriate Operations ticket(s) created
5. Jira Ticket(s)
   1. DEV-16418

Explain N/A in above checklist: