swem

Implementation of the Simple Word Embedding Modell.

`swem.models.swem.SwemConfig`	Configuration for SWEM models.
`swem.models.swem.Swem`	Simple Word Embedding model (see Baselines need more love ).

SwemConfig

class swem.models.swem.SwemConfig(embedding: EmbeddingConfig, pooling: PoolingConfig, pre_pooling_dims: tuple[int, ...] | None = None, post_pooling_dims: tuple[int, ...] | None = None, dropout: float = 0.2)

Configuration for SWEM models.

Parameters

embedding (EmbeddingConfig) –
pooling (PoolingConfig) –
pre_pooling_dims (tuple[int, ...] | None) –
post_pooling_dims (tuple[int, ...] | None) –
dropout (float) –

Return type

None

Swem

class swem.models.swem.Swem(embedding: nn.Embedding, pooling_layer: SwemPoolingLayer, pre_pooling_dims: tuple[int, ...] | None = None, post_pooling_dims: tuple[int, ...] | None = None, dropout: float = 0.2)

Simple Word Embedding model (see Baselines need more love ).

The model consists of an embedding layer, a feed forward network that is applied separately to each word vector, a pooling layer that pools the vectors belonging to the same text into a single vector, and another feed forward network that is applied to this pooled vector.

Parameters

embedding (nn.Embedding) – The embedding layer used by the model.
pooling_layer (nn.Module) – The pooling layer to be used by the model.
pre_pooling_dims (tuple[int, ...] | None) – Intermediate dimensions for the feed forward network applied to the input.
post_pooling_dims (tuple[int, ...] | None) – Intermediate dimensions for the feed forward network applied to the output of the pooling layer.
dropout (float) – Dropout probability after each layer in both feed forward subnetworks.

Shapes:

input: \((\text{batch_size}, \text{seq_len})\)
output: \((\text{batch_size}, \text{enc_dim})\), where \(\text{enc_dim}\) is the last of the post_pooling_dims (if given, otherwise the last of the pre_pooling_dims or failing that the embedding_dimension).

classmethod from_config(config: SwemConfig | dict[str, Any]) → Swem

Construct a SWEM-model from a config.

Instead of a config the user can also provide a dictionary representation of the config.

Parameters: config (SwemConfig | dict[str, Any]) – The config to construct the model from.
Return type: Swem

Examples

>>> config = {
...     "embedding": {
...             "type": "Embedding",
...             "num_embeddings": 10,
...             "embedding_dim": 2
...     },
...     "pooling": {
...             "type": "HierarchicalPooling",
...             "window_size": 5
...     },
...     "pre_pooling_dims": (5, 5),
...     "post_pooling_dims": (6, 6),
...     "dropout": 0.1
... }
>>> Swem.from_config(config)
Swem(
(embedding): Embedding(10, 2)
(pooling_layer): HierarchicalPooling(
    (avg_pooling): AvgPool2d(kernel_size=(5, 1), stride=1, padding=0)
)
(pre_pooling_trafo): Sequential(
    (0): Linear(in_features=2, out_features=5, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.1, inplace=False)
    (3): Linear(in_features=5, out_features=5, bias=True)
    (4): ReLU()
    (5): Dropout(p=0.1, inplace=False)
)
(post_pooling_trafo): Sequential(
    (0): Linear(in_features=5, out_features=6, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.1, inplace=False)
    (3): Linear(in_features=6, out_features=6, bias=True)
)
)

classmethod load(path: str | Path) → Swem

Load a model that was previously saved on disk.

Parameters: path (str | Path) – The directory in which contains the config and weights for the stored model.
Raises: NotADirectoryError – If the given path is not a directory.
Returns: The saved model.
Return type: [Swem]

save(path: str | Path)

Save this model to disk.

The model will be stored to the directory given by the path as two files ‘config.json’ containing the model config and ‘weights.pt” containing the weights of the layers.

Parameters

path (str | Path) – The directory in which to store the model file. Should be empty. If the directory does not exist it will be generated ( this does not apply to intermediate directories).

Raises

FileNotFoundError – If the parent directory of path does not exist.
NotADirectoryError – If path already exists and is not a directory.
FileExistsError – If path is a non-empty directory.