Spacy 101

• variety of tooling to gain insights into the grammatical structure of text being analysed.
  • word types (part of speech), further categorization eg:- nouns into subjects and objects. same words differentiated via their POS ("google" being a verb or a noun…)

Most of the features work independently but some require loading "trained pipelines". They're composed of the following:

• binary weights for part of speech tagger, dependency parser, and NER
• lexical entries : words and their attributes (spelling, length)
• data files: lemmatization and lookup tables
• word vectors: see Text Representation
• configs: metadata to load the pipeline with appropriate configuration

• see Text Representation
• segmenting text into words, punctuation and other similar discrete structures
• this needs to be done smartly and based on context :- each period isn't a full stop ("U.S.A" for instance should be one token without any punctuations)
• a trained pipeline (Language object usually addressed as nlp) when applied to a text, produces an iterable of tokens.
• first step is splitting with whitespace, after which the splits are processed from left to right with the following checks:
  1. check if it is an exception
     • "don't" => "do" and "n't"
     • "U.K." stays the same
  2. check or splittable suffixes, prefixes and infixes
     • commas, periods, hyphens or quotes are candidates for such splits
     • anti-clockwise => "anti" and "clockwise"
• each language needs its own set of extensive hard coded data and exception rules that need to be loaded when using that particular trained pipeline.
• for further details and customization options, see :
  • https://spacy.io/usage/linguistic-features#language-data
  • https://spacy.io/usage/linguistic-features#tokenization

• post tokenization, a Doc can be parsed and tagged.
  • the statistical models come into play at this stage.
• linguistic annotations are available as attributes of token objects.
  • strings are hashed for efficiency, so defaults are integers:
    • tok.pos : integer hash of part of speech tag
    • tok.pos_ : the part of speech tag (string)
    • tok.dep : integer hash of the dependency tag
    • tok.dep_ : the dependency tag as string
  • conventions for attributes:
    • base word is the integer hash
    • _ appended yields the string tag
  • a summary of the attributes of a token is as follows:
    • text : the original text
    • Lemma : base form of the word
    • POS : simple part of speech tags in the format mentioned https://universaldependencies.org/u/pos/
    • Tag : detailed part-of-speech tags
    • Dep : syntactic dependency -> relation between tokens
    • shape : capitalization, punctuation and digits (eg: Apple -> Xxxxx, U.K. -> X.X., 3 -> d)
    • is alpha : predicate on the token being composed of only alphanumeric characters
    • is stop : predicate on the token being a stop word
• spacy.explain("…") can be used to fetch short descriptions of tags and labels
• extra tooling : for visualizations see DisplaCy

• a named real world object…
  • person, country, product, book title, etc..
• a doc object is an iterable of tokens by default
• an iterable of entities can be fetched via doc.ents:

import spacy

# downloading and loading the languge object
nlp = spacy.load("en_core_web_sm") 

# creating the document object
doc = nlp("the quick brown fox jumped over the lazy dog")

for ent in doc.ents:
  print(ent.text, ent.start_char, ent.end_char, ent.label_)

• again, the DisplaCy visualizer
• for a theoretical outlook, see Named Entity Recognition (NER)

• see Text Representation and specifically text-embeddings
• note that small trained pipelines like "en\_core\_web\_sm" (ending in "sm") don't ship with word-vectors
  • instead, use the variant ending in "lg" : "en\_core\_web\_lg"
• given the pipeline ships with vectors, normal objects like token, doc and span now have a vector attribute that defaults to the average of their token vectors.

import spacy
nlp = spacy.load("en_core_web_lg")

doc = nlp("the quick brown fox jumps over the lazy dog")

# do note that this (comments in a paren block) is invalid python code
for token in doc:
    print(token.text, \ # original token text
          token.has_vector, \ # predicate on if this token has a vector representation
          token.vector_norm, \ # L2 norm of the token's vector
          token.is_oov)  # predicate for whether the token is out of vocabulary

• now that token and token spans can be mapped to vectors, a notion of similarity arises ( see cosine similarity for a quick approach )
• every Doc, Span, Token, Lexeme has a .similarity attribute.
• note that non-singular spans have their vectors as the average of their constituent tokens.

import spacy
nlp = spacy.load("en_web_core_lg")

doc1 = nlp("the quick brown fox jumps over the lazy dog")
doc2 = nlp("a lazy dog shouldn't be able jump over a quick brown fox") 

# similarity b/w documents
print(doc1, "<->" doc2, doc1.similarity(doc2))

# similarity b/w tokens and spans
lazy_dog = doc2[1:3]
fox = doc1[3]

print(lazy_dog, "<->", fox, lazy_dog.similarity(fox))

When calling a Language object (named as nlp) on a text, post tokenization, the token sequence is pushed through various steps, collectively referable as a processing pipeline.

briefly explaining the above flow:

• text is input into a language object
• the language object consists of the tokenizer and the pipeline
• the tokenizer creates the Doc object and tokenizes the text
• the pipeline (with trained components from other doc objects (examples)) then processes the doc object accordingly
  • an example is a collection of two doc objects (reference data and predictions)
• all through out, to avoid duplication of strings, we have a vocab that maintains a store of lexemes for the text.
• these are referred to by the tokens and spans in the doc object.
• post processing, the relevant results are extracted from the Doc object

• The default preference is to have a single storage of a unique token in the vocab where it is referred to by multiple documents.
• strings are encoded to hash values
• this also applies to meta-data (tag-strings like "VERB" or "ORG" (entity type)) :- they're also hashed
• All internal communication (for the spaCy lib) is carried out via hashes

• to save stuff, see Serialization
• spacy offers inbuilt functionality for the same, via pickle
  • pretty convenient, check dill for advanced functionalities
• All container classes (Language, Doc, Vocab, StringStore) expose this functionality via the following

read more on saving and loading here

• most pipeline components that spacy uses are statistical (weight based).
  • rule based pipes can also be incorporated but not focusing on that right now
• these weights are decided upon post training that model
  • that requires training data (pairs of text instances and corresponding labels)
  • on generic details for training various components, see Text Representation and Information Extraction
• A generic treaty on training models is explored in the node : Training Loop
• for specific info on how to train models in spacy, see this page

• shared and specific language data are stored in their respective directories when working with multiple languages
• some relevant lingo to this store is as follows