Description

Post-facto fake news refers to news items or claims that are already known to be false, either by work from organizations like Snopes and Factcheck or by the general public on social media.

For this challenge, we will only consider claims that are outright false or outright true. For example, “eating fruit prevents cancer” — a truth assertion here is dodgy at best, but for headlines like “Hillary has a body double” we can be confident about truth assertion.

We will select headlines for the competition where we can be confident in asserting its veracity.

Task

Input

A claim/headline string with accompanied URL, if any.

• “Pope endorsed Trump”
• “Climate change is a Martian takeover operation”

Output

Four required values

• 1. Boolean fakeness indicator
• 2. Confidence score
• 3. Provenance URL to support
• 4. A confidence threshold for accepting/rejecting

Data

Training Data

We will provide the training data

Development Data

Teams will use part of the training data as a development set

Test Data

Test data is withheld

Auxiliary Data

In addition to training data, you are free to use all of the internet. The solution has to work in real-time.

Evaluation Criteria

Submissions will be evaluated based on a 0-1 loss using the submission-provided threshold. All winning entries must provide a provenance URL for their solutions.

Awards

There is a total of $2000 to be awarded for the top-5 teams.
Each team will get cash awards proportional to their accuracy.
Submissions achieving more than 10% false positive or false negative rates are not considered.
Contesting teams must be willing to open source their systems.