Sluice networks: Learning what to share between loosely related tasks.
Year: 2017
Type: preprint
Abstract: Multi-task learning is partly motivated by the observation that humans bringto bear what they know about related problems when solving new ones. Similarly,
deep neural networks can profit from related tasks by sharing parameters with
other networks. However, humans do not consciously decide to transfer knowledge
between tasks (and are typically not aware of the transfer). In machine
learning, it is hard to estimate if sharing will lead to improvements;
especially if tasks are only loosely related. To overcome this, we introduce
Sluice Networks, a general framework for multi-task learning where trainable
parameters control the amount of sharing -- including which parts of the models
to share. Our framework goes beyond and generalizes over previous proposals in
enabling hard or soft sharing of all combinations of subspaces, layers, and
skip connections. We perform experiments on three task pairs from natural
language processing, and across seven different domains, using data from
OntoNotes 5.0, and achieve up to 15% average error reductions over common
approaches to multi-task learning. We analyze when the architecture is
particularly helpful, as well as its ability to fit noise. We show that a)
label entropy is predictive of gains in sluice networks, confirming findings
for hard parameter sharing, and b) while sluice networks easily fit noise, they
are robust across domains in practice.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2617039999