perceptron (FTW!)
Though it does not generate scores suitable for use as probabilties, and
might achieve slightly lower accuracy on some tasks than its
gradient-based counterparts like SGD (a possibility for libpostal)
or LBFGS (prohibitive on this much data), the averaged perceptron is
appealing for two reasons: speed and low memory usage i.e. we can still use
all the same tricks as in the greedy model like sparse construction of
the weight matrix. In this case we can go even sparser than in the
original because the state-transition features are separate from the
state features, and we need to be able to iterate over all of them
instead of simply creating new string keys in the feature space. The
solution to this is quite simple: we simply treat the weights for each
state-transition feature as if they have L * L output labels instead of
simply L. So instead of:
{
"prev|road|word|DD": {1: 1.0, 2: -1.0}
...
}
We'd have:
{
"word|DD": {(0, 1): 1.0, (0, 2): -1.0}
...
}
As usual we compress the features to a trie, and the weights to
compressed-sparse row (CSR) format sparse matrix after the weights have
been averaged. These representations are smaller, faster to load from
disk, and faster to use at runtime (contiguous arrays vs hashtables).
This also includes the min_updates variation from the greedy perceptron,
so features that participate in fewer than N updates are discarded at
the end (and also not used in scoring until they meet the threshold so
the model doesn't become dependent on features it doesn't really have).
This tends to discard irrelevant features, keeping the model small
without hurting accuracy much (within a tenth of a percent or so in my
tests on the greedy perceptron).
implementation for the address parser.
One of the main issues with the greedy averaged perceptron tagger used currently
in libpostal is that it predicts left-to-right and commits to its
answers i.e. doesn't revise its previous predictions. The model can use
its own previous predictions to classify the current word, but
effectively it makes the best local decision it can and never looks back
(the YOLO approach to parsing).
This can be problematic in a multilingual setting like libpostal,
since the order of address components is language/country dependent.
It would be preferable to have a model that scores whole
_sequences_ instead of individual tagging decisions.
That's exactly what a Conditional Random Field (CRF) does. Instead of modeling
P(y_i|x_i, y_i-1), we're modeling P(y|x) where y is the whole sequence of labels
and x is the whole sequence of features. They achieve state-of-the-art results
in many tasks (or are a component in the state-of-the-art model - LSTM-CRFs
have been an interesting direction along these lines).
The crf_context module is heavily borrowed from the version in CRFSuite
(https://github.com/chokkan/crfsuite) though using libpostal's data structures and
allowing for "state-transition features." CRFSuite has state features
like "word=the", and transition features i.e. "prev tag=house", but
no notion of a feature which incorporates both local and transition
information e.g. "word=the and prev tag=house". These types of features are useful
in our setting where there are many languages and it might not make as
much sense to simply have a weight for "house_number => road" because that
highly depends on the country. This implementation introduces a T x L^2 matrix for
those state-transition scores.
For linear-chain CRFs, the Viterbi algorithm is used for computing the
most probable sequence. There are versions of Viterbi for computing the
N most probable sequences as well, which may come in handy later. This
can also compute marginal probabilities of a sequence (though it would
need to wait until a gradient-based learning method that produces
well-calibrated probabilities is implemented).
The cool thing architecturally about crf_context as a separate module is that the
weights can be learned through any method we want. As long as the state
scores, state-transition scores, and transition scores are populated on
the context struct, we have everything we need to run Viterbi inference,
etc. without really caring about which training algorithm was used to optimize
the weights, what the features are, how they're stored, etc.
So far the results have been very encouraging. While it is slower to
train a linear-chain CRF, and it will likely add several days to the
training process, it's still reasonably fast at runtime and not all that
slow at training time. In unscientific tests on a busy MacBook Pro, so far
training has been chunking through ~3k addresses / sec, which is only
about half the speed of the greedy tagger (haven't benchmarked the runtime
difference but anecdotally it's hardly noticeable). Libpostal training
runs considerably faster on Linux with gcc, so 3k might be a little low.
I'd also guess that re-computing features every iteration means there's
a limit on the performance of the greedy tagger. The differences might
be more pronounced if features were pre-computed (a possible optimization).
