Add heartbeat endpoint

[hannahwhy]

This is related to #10.

There are many ways that a pipeline can fail its assigned task. Here are some examples:

1. Coding error in the pipeline
2. Unexpected response from a download target (e.g. unhandled 4xx/5xx errors that end up in retry loops, aborts, item failures)
3. Unclean shutdown (warriors on laptops going to sleep, ^C on manually-run pipelines, power failures, ...)

While we cannot know how long an item will take to complete, we can know whether or not an assignee is still working on an item. We can add a heartbeat to the pipeline that periodically reports liveness back to a tracker. The tracker can use this information to fail items that have gone without a heartbeat for too long.

(This could supercede #10 if we're willing to live with failed items eventually going to a failed queue rather than immediately.)

[hannahwhy]

https://github.com/ArchiveTeam/ArchiveBot uses a heartbeat in its pipeline. On the ArchiveBot control node, there's a job that periodically checks the last time a pipeline checked in an item. If the timestamp is too old, a reaper job tells an operator to manually fail the item.

Relevant code:

Pipeline heartbeat: https://github.com/ArchiveTeam/ArchiveBot/blob/efbbc89c2446dd16772e8605744c134c79dba099/pipeline/archivebot/seesaw/monitoring.py
Reaper: https://github.com/ArchiveTeam/ArchiveBot/blob/2132f891e5fd8444d2f41c09eb472d65c5818dcb/cogs/reaper.rb

This system is not perfect. We have had incidents where the pipeline reported a job was alive, but the crawler wasn't making progress. However, for Warrior projects, this could be solved by having the heartbeat check the result that we want: for example, we could check every 30 seconds that the WARC is growing in size, or that the temporary file has changed in some way (last modified time), and only send a heartbeat when that condition has been met.

[hannahwhy]

Some thoughts on API design:

Pipelines can perform POST http://tracker.example.com/project/heartbeat with the following data:

{ "downloader": "foobar",
  "reports": [
    { "item": "a" },
    { "item": "b" },
    { "item": "c" }
  ]
}

This always returns 202 Accepted.

If the downloader is not known to the tracker, the entire packet is ignored.
Claims that are listed in the heartbeat JSON but are not assigned to the downloader are ignored.

We use an array of objects in items because it's not that much more difficult than having an array of strings, and using objects lets us use this endpoint for future monitoring (progress reports, for example).