Spark NLP: State-of-the-Art Natural Language Processing & LLMs Library

Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment.

Spark NLP comes with 100000+ pretrained pipelines and models in more than 200+ languages. It also offers tasks such as Tokenization, Word Segmentation, Part-of-Speech Tagging, Word and Sentence Embeddings, Named Entity Recognition, Dependency Parsing, Spell Checking, Text Classification, Sentiment Analysis, Token Classification, Machine Translation (+180 languages), Summarization, Question Answering, Table Question Answering, Text Generation, Image Classification, Image to Text (captioning), Automatic Speech Recognition, Zero-Shot Learning, and many more NLP tasks.

Spark NLP is the only open-source NLP library in production that offers state-of-the-art transformers such as BERT, CamemBERT, ALBERT, ELECTRA, XLNet, DistilBERT, RoBERTa, DeBERTa, XLM-RoBERTa, Longformer, ELMO, Universal Sentence Encoder, Llama-2, M2M100, BART, Instructor, E5, Google T5, MarianMT, OpenAI GPT2, Vision Transformers (ViT), OpenAI Whisper, Llama, Mistral, Phi, Qwen2, and many more not only to Python and R, but also to JVM ecosystem (Java, Scala, and Kotlin) at scale by extending Apache Spark natively.

Model Importing Support

Spark NLP provides easy support for importing models from various popular frameworks:

• TensorFlow
• ONNX
• OpenVINO
• Llama.cpp (GGUF)

This wide range of support allows you to seamlessly integrate models from different sources into your Spark NLP workflows, enhancing flexibility and compatibility with existing machine learning ecosystems.

Project's website

Take a look at our official Spark NLP page: https://sparknlp.org/ for user documentation and examples

Features

• Text Preprocessing
• Parsing and Analysis
• Sentiment and Classification
• Embeddings
• Classification and Question Answering
• Machine Translation and Generation
• Image and Speech
• Integration and Interoperability (ONNX, OpenVINO)
• Pre-trained Models (36000+ in +200 languages)
• Multi-lingual Support

Quick Start

This is a quick example of how to use a Spark NLP pre-trained pipeline in Python and PySpark:

$ java -version
# should be Java 8 or 11 (Oracle or OpenJDK)
$ conda create -n sparknlp python=3.7 -y
$ conda activate sparknlp
# spark-nlp by default is based on pyspark 3.x
$ pip install spark-nlp==6.4.2 pyspark==3.3.1

In Python console or Jupyter Python3 kernel:

# Import Spark NLP
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline
import sparknlp

# Start SparkSession with Spark NLP
# start() functions has 3 parameters: gpu, apple_silicon, and memory
# sparknlp.start(gpu=True) will start the session with GPU support
# sparknlp.start(apple_silicon=True) will start the session with macOS M1 & M2 support
# sparknlp.start(memory="16G") to change the default driver memory in SparkSession
spark = sparknlp.start()

# Download a pre-trained pipeline
pipeline = PretrainedPipeline('explain_document_dl', lang='en')

# Your testing dataset
text = """
The Mona Lisa is a 16th century oil painting created by Leonardo.
It's held at the Louvre in Paris.
"""

# Annotate your testing dataset
result = pipeline.annotate(text)

# What's in the pipeline
list(result.keys())
Output: ['entities', 'stem', 'checked', 'lemma', 'document',
         'pos', 'token', 'ner', 'embeddings', 'sentence']

# Check the results
result['entities']
Output: ['Mona Lisa', 'Leonardo', 'Louvre', 'Paris']

For more examples, you can visit our dedicated examples to showcase all Spark NLP use cases!

Packages Cheatsheet

This is a cheatsheet for corresponding Spark NLP Maven package to Apache Spark / PySpark major version:

NOTE: M1/M2 and AArch64 are under experimental support. Access and support to these architectures are limited by the community and we had to build most of the dependencies by ourselves to make them compatible. We support these two architectures, however, they may not work in some environments.

Pipelines and Models

For a quick example of using pipelines and models take a look at our official documentation

Please check out our Models Hub for the full list of pre-trained models with examples, demo, benchmark, and more

Platform and Ecosystem Support

Apache Spark Support

Spark NLP 6.4.2 has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x

Find out more about Spark NLP versions from our release notes.

Scala and Python Support

Find out more about 4.x SparkNLP versions in our official documentation

Databricks Support

Spark NLP 6.4.2 has been tested and is compatible with the following runtimes:

We are compatible with older runtimes. For a full list check databricks support in our official documentation

EMR Support

Spark NLP 6.4.2 has been tested and is compatible with the following EMR releases:

We are compatible with older EMR releases. For a full list check EMR support in our official documentation

Full list of Amazon EMR 6.x releases Full list of Amazon EMR 7.x releases

NOTE: The EMR 6.1.0 and 6.1.1 are not supported.

Installation

Command line (requires internet connection)

To install spark-nlp packages through command line follow these instructions from our official documentation

Scala

Spark NLP supports Scala 2.12.15 if you are using Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x versions. Our packages are deployed to Maven central. To add any of our packages as a dependency in your application you can follow these instructions from our official documentation.

If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your projects Spark NLP Starter

Python

Spark NLP supports Python 3.7.x and above depending on your major PySpark version. Check all available installations for Python in our official documentation

Compiled JARs

To compile the jars from source follow these instructions from our official documentation

Platform-Specific Instructions

For detailed instructions on how to use Spark NLP on supported platforms, please refer to our official documentation:

Offline

Spark NLP library and all the pre-trained models/pipelines can be used entirely offline with no access to the Internet. Please check these instructions from our official documentation to use Spark NLP offline.

Advanced Settings

You can change Spark NLP configurations via Spark properties configuration. Please check these instructions from our official documentation.

S3 Integration

In Spark NLP we can define S3 locations to:

• Export log files of trainin