Madhukar's Blog

Rediscovering Implicits in Scala 3 - Part 3: Summoning Implicits

Mon, 19 May 2025 00:00:00 +0530

Implicits are one of the most advanced features of Scala. This feature makes many of the meta programming possible in language.

The same power of implicits also brings lot of complexity and confusion to beginners. This complexity often leads to the wrong use of the feature or scares away developers using it altogether.

In Scala 3, the major new version of Scala Language, many features are added to simplify the implicits. These simplifications help the user of the language to clearly understand the different mechanisms available to abstract over different contexts in the code. These simplifications make it easier for beginners to grasp the power of implicits and also make it cleaner code for experts.

In this series of blogs, I will be exploring these new features as if I am rediscovering the implicits all over again. In this third post, I will be discussing about summoning implicits. You can find all the post in this series here.

Passing Implicit Parameter to a Method in Scala 2.x

In Scala 2.x, whenever we want to use an implicit value inside a method, we need to define an additional parameter marked as implicit. This parameter allows the compiler to automatically pass the correct value from the context when the method is called. However, this also means every method that needs an implicit has to explicitly declare it in its signature, which can sometimes add verbosity.

The below code shows an example where we pass an implicit ordering for comparing two integers.

object SummonExample {
  trait Ord[T] {
    def compare(x: T, y: T): Int
  }

  class IntOrdering extends Ord[Int] {
    override def compare(x: Int, y: Int): Int = x.compare(y)
  }

  def main(args: Array[String]): Unit = {
    given intOrdering: Ord[Int] = new IntOrdering()

    def maxWithImplicitParameter(x: Int, y: Int)(implicit intOrdering: Ord[Int]): Int = {
      if (intOrdering.compare(x, y) < 0) y else x
    }

    println(maxWithImplicitParameter(10, 20))
  }
}

In the above code, we first define a trait Ord for comparison logic and provide an implementation IntOrdering for integers. In the maxWithImplicitParameter method, we accept an implicit parameter of type Ord[Int]. At the call site, the compiler automatically fills in the implicit value intOrdering, and the method works without us needing to pass it manually.

Summon in Scala 3.x

In Scala 3, the summon keyword is introduced as a standard way to fetch an implicit value from the surrounding context. Instead of passing an implicit parameter explicitly in every method signature, we can now summon the required context inside the method body itself.This makes the method signatures cleaner and separates the concern of fetching context from defining the method logic.

The same example we discussed earlier can now be improved using summon as shown below.

def maxWithSummon(x: Int, y: Int): Int = {
  // summon the implicit context using surrounding context
  val intOrdering = summon[Ord[Int]]
  if intOrdering.compare(x, y) < 0 then y else x
}

println(maxWithSummon(10, 20))

In the above code, we no longer have to declare an extra implicit parameter in the method definition. Instead, we directly use summon[Ord[Int]] inside the method body to fetch the implicit ordering from the context. This makes the method signature much simpler and keeps the implicit handling localized within the method.

Code

You can find complete code over github.

References

https://docs.scala-lang.org/scala3/reference/contextual/using-clauses.html#summoning-instances-1

Summary

In this post, we explored how Scala 2.x required explicit implicit parameters in method signatures and how Scala 3.x introduced the summon keyword to simplify this process. By using summon, we can now fetch context parameters directly inside the method body, leading to much cleaner and more readable code.

Rediscovering Implicits in Scala 3 - Part 2: Extension methods

Fri, 02 May 2025 00:00:00 +0530

Implicits are one of the most advanced features of Scala. This feature makes many of the meta programming possible in language.

The same power of implicits also brings lot of complexity and confusion to beginners. This complexity often leads to the wrong use of the feature or scares away developers using it altogether.

In this series of blogs, I will be exploring these new features as if I am rediscovering the implicits all over again. In this second post, I will be discussing about extension methods. You can find all the post in this series here.

Extension Methods in Scala 2.x

In Scala 2.x, extension methods are commonly used to add new methods to existing types without modifying their original definitions. This helps in enriching libraries and creating more fluent APIs without needing direct access to the original source code.

The below code shows how to define an extension method for a simple Circle class in Scala 2.x.

class Extensions(circle: Circle) {
  def circumference: Double = circle.radius * math.Pi * 2
}
object Extensions {
implicit def addExtension(c: Circle): Extensions = new Extensions(c)
}
    
import Extensions._

val circle = Circle(0,0,50)
println(circle.circumference)

In the above code, we first create a wrapper class Extensions which adds a new method circumference to the Circle class. Then we define an implicit conversion method addExtension to automatically convert a Circle object to Extensions class. So when we call circumference method on Circle, the Scala compiler automatically inserts the conversion. The import Extensions._ helps the Scala compiler to find right implicit.

Extension Methods in Scala 3.x

In Scala 3, extension methods are made a first-class language feature. This provides a cleaner and more intuitive syntax to add methods to existing types without relying on implicit conversions or wrapper classes. The new syntax makes the intention of extension much more explicit and easy to understand.

The same example we discussed in Scala 2.x can now be rewritten in Scala 3.x as shown below.

extension (circle: Circle) {
  def circumferenceExtension: Double = circle.radius * math.Pi * 2
  }

  val secondCircle = Circle(0, 0, 50)
  println(secondCircle.circumferenceExtension)

In the above code, we define an extension directly on Circle using the extension keyword. Inside the extension block, we add the method circumferenceExtension which can be called directly on Circle instances. There is no need to create an explicit wrapper class or define implicit conversions, making the code much cleaner and expressive.

Code

You can find complete code over github.

References

https://docs.scala-lang.org/scala3/reference/contextual/extension-methods.html.

Summary

In this post, we explored how extension methods were used in Scala 2.x using implicit conversions and how Scala 3 introduced a much cleaner and more expressive syntax for the same. By making extension methods a first-class citizen, Scala 3 greatly simplifies the way we can enrich existing types.

Rediscovering Implicits in Scala 3 - Part 1: Implicit Parameters

Fri, 17 Jan 2025 00:00:00 +0530

Implicits are one of the most advanced features of Scala. This feature makes many of the meta programming possible in language.

The same power of implicits also brings lot of complexity and confusion to beginners. This complexity often leads to the wrong use of the feature or scares away developers using it altogether.

In this series of blogs, I will be exploring these new features as if I am rediscovering the implicits all over again. In this first post, I will be discussing about implicit parameters. You can find all the post in this series here.

Implicit Parameters in Scala 2.x

Implicit parameters is one of the important use case for implicits. In Scala 2.x it can be used as below

 trait Comparator[T] {
     def compare(x:T,y:T):Int
  }

  def max[T](x:T, y:T)(implicit comparator: Comparator[T]):T ={
     if comparator.compare(x,y) <0 then y else x
 }

In above code, we define a trait for comparison. In our user function max, we define an implicit parameter called comparator which is used for comparing.

The above can be invoked as below

class IntComparator extends Comparator[Int] {
   override def compare(x: Int, y: Int): Int = x.compare(y)
}
implicit val intComparator = new IntComparator()
println(max(10,20))

In above code, we create an implementation of the comparator and define an implicit variable to hold the implementation. So when max function is called its resolved from the environment.

Scala 3 Using and Given

In Scala 3, this implicit parameters are made more explicit. These are called given instances. As the name suggests, given an instance of type Scala compiler try to use it appropriately.

So the same code can be now written as

def maxWithUsing[T](x:T,y:T)(using comparator: Comparator[T]):T = {
  if comparator.compare(x,y) <0 then y else x
 }

In above code, we replaced implicit using using keyword, which expresses the intention of parameter that we want use from the environment.This is more explicit and clear compared to just implicit.

To use it, we can now use below code

given intComparatorWithGiven:Comparator[Int] = IntComparator()
println(maxWithUsing(10,20))

Here we are defining a parameter using given keyword, which expresses intention giving or providing the type for the used parameters.

So the combination of using and given pattern makes it much more clear about the usage than generic implicits.

Code

You can find complete code over github.

References

https://docs.scala-lang.org/scala3/reference/contextual/givens.html.

Building Agentic LLM Workflows with LangGraph - Part 1: Hello World

Mon, 13 Jan 2025 00:00:00 +0530

Agentic workflows have revolutionized the way we build LLM applications. Agent gives developers fine-grained control to express complex workflow logic in a structured way. This helps us express workflow logic in a modular manner which is easy to maintain and extend.

LangGraph is one of the frameworks that provide the tools to build agentic workflows. The API of LangGraph is heavily inspired by graph based frameworks that are found in big data frameworks like Hadoop and Spark. Some of the concepts also have similarities to frameworks like actor frameworks like Akka.

So in this series of blogs, I am going to explore different aspects of LangGraph API and show how the core concepts help us to build the different kinds of workflows using graph abstractions. The focus will be not just on LLMs, but also on core concepts of LangGraph which will help the developer to understand different abstractions that are available.

This is the first blog in the series where I introduce LangGraph with the simple hello world example. You can access the posts in the series here.

Installation

You can install LangGraph using below command

pip install -U langgraph

Defining Hello World Agent

In the LangGraph, all the agent logic happens in the node and communication happens over the edge. So we are defining a simple “hello_world_agent” node to return the hello world message.

from langgraph.graph import MessagesState

def hello_world_agent(state:MessagesState):
    return {"messages":["hello world"]}

A node can be defined as a simple Python function. The first parameter is the state which signifies all the activities that are done before this node is invoked. As of now, we can ignore it.

In our function body, we are returning hello world message as a dictionary called messages. This is one of the ways a node communicates with another node.

Building the Graph

Once we have defined our hello world agent node, we are going to build the graph.

from langgraph.graph import END, START, StateGraph

graph_builder = StateGraph(MessagesState)
graph_builder.add_node("hello_world_agent",hello_world_agent)

graph_builder.add_edge(START,"hello_world_agent")
graph_builder.add_edge("hello_world_agent",END)

In our code, first we create a graph builder with some empty state. Then we add our node to the graph builder. After that, we add edges from special node START and END to complete the graph.

Graph Compilation

Once the graph is complete, we are going to compile the graph which will check all the conditions to make sure the graph is valid.

graph = graph_builder.compile()

Visualise the Graph

We can visualise the graph using the below code

from IPython.display import Image, display

try:
    display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
    # This requires some extra dependencies and is optional
    pass

The graph looks as below

Running the Graph

We can run our graph using below code.

for event in graph.stream({"messages": ["hello"]}):
    for value in event.values():
        print(value)

The output will be

{'messages': ['hello world']}

Code

You can find complete code at notebook on github.

Summary

In this post, we have learnt how to express a simple agent using LangGraph framework.

References

https://langchain-ai.github.io/langgraph/tutorials/

Understanding Spark Connect API - Part 5: Dataframe Sharing Across Spark Sessions

Fri, 18 Aug 2023 00:00:00 +0530

In the 3.4 version, Apache Spark has released a new client/server-based API called Spark Connect. This API will help in improving how we develop and deploy Spark applications.

In this series of blogs, we are going to explore various functionalities exposed by spark-connect API. This is the fifth post in the series where we will discuss about sharing dataframe between different spark sessions. You can read all the posts in the series here.

Currently in spark, there is no way to serialize the dataframe and access them in other spark session. This feature is typically useful where we want to rerun the same set of operations on data and we can encode those operations as a logical plan of a dataframe. Currently, we need to use python scripts or other code to encode the same. But with spark-connect API, this is going to change.

Dataframe Plan and Protocol Buffer

In spark-connect, the client uses non resolved dataframe plan as the intermediate format for exchanging information between the server. Having the dataframe plan as the intermediate format makes it very easy for spark to keep the API of dataframe as it is and just change the implementation.

Internally spark uses the protocol buffer for serializing these plans. So we can use this feature to serialize and deserialize the dataframe in the spark and share them across spark sessions.

Serialising the Spark Plan

The below are the steps to create a dataframe and serializing it to a file.

###1. DataFrame Creation

The below spark code creates a simple dataframe and runs filter on it.

val data = List (
  Employee("madhav",26,60000),
  Employee("Raju",30, 80000)
)

val sourceDf = sparkSession.createDataset(data)
val filteredDf = sourceDf.filter("salary > 60000")

###2. Access the Plan Object

We can access the plan of the dataframe using below code.

val plan = filteredDf.plan

On a spark-connect dataframe, there is a field name plan which gives access to the plan object.

###3. Print Plan

Once we created the dataframe, we can print it’s plan using below code

println(plan)

The output will be as below

root {
  common {
    plan_id: 2
  }
  filter {
    input {
      common {
        plan_id: 1
      }
      local_relation {
        data: "\377\377\377\377 # truncated for output here" 
        schema: "{\"type\":\"struct\",\"fields\":
        [{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},
        {\"name\":\"age\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},
        {\"name\":\"salary\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}}]}"
      }
    }
    condition {
      expression_string {
        expression: "salary > 60000"
      }
    }
  }
}

The above is the string representation of protocol buffer plan. As you can it encodes, schema, data and operations on dataframe.

###4. Serialize Plan

The below code serializes the plan to a file.

val file = new File("filterdef.ser")
val fileOutputStream = new FileOutputStream(file)
plan.writeTo(fileOutputStream)
fileOutputStream.close()

The above is a standard Java/Scala code to serialize the object.

Deserialization and Dataframe Recreation

Once we serialized the dataframe, now we can deserialize and recreate the dataframe. The below are steps for the same

Internal API

This portion of code, uses an internal API which is not yet exposed as public API. So make sure this code resides in org.apache.spark.sql package. Otherwise you will be not able run the code.

1. Deserialize the Plan Object

The below code deserializes the file to the plan.

val serializedFile = "filterdef.ser"
val inputStream = new FileInputStream(serializedFile)
val parsedPlan = Plan.parseFrom(inputStream)

2. Create Dataframe the Plan

Once we have the plan, we can create the Dataframe from the same. This is an internal API.

val dataset = new Dataset(sparkSession, parsedPlan, UnboundRowEncoder)
println(dataset.explain(true))

The above code uses new internal Dataset constructor which creates a dataset/dataframe using the parsed plan. The last parameter in the constructor is to define the type of the dataset. If we pass UnboundRowEncoder it will return a Dataset[Row].

3. Print Data

Once we have recreated the dataframe, we can verify the same by outputting the data.

println(dataset.show())

The above code outputs

+----+---+-------+
|name|age| salary|
+----+---+-------+
|Raju| 30|80000.0|
+----+---+-------+

As you can see, we are able to successfully read and recreate dataframe in new spark session.

Code

You can access code for serialize and deserialize on github.

Conclusion

In this post, we saw how to use spark-connect plan serialization to share the dataframe between different spark sessions.

Understanding Spark Connect API - Part 4: PySpark Example

Wed, 16 Aug 2023 00:00:00 +0530

In the 3.4 version, Apache Spark has released a new client/server-based API called Spark Connect. This API will help in improving how we develop and deploy Spark applications.

In this series of blogs, we are going to explore various functionalities exposed by spark-connect API. This is the fourth post in the series where we will discuss PySpark integration. You can read all the posts in the series here.

Spark Connect API Server

Before you can run a program against spark connect, you need to start spark connect server which intern starts a spark driver. You can follow the steps below link for the same.

https://spark.apache.org/docs/latest/spark-connect-overview.html#download-and-start-spark-server-with-spark-connect.

Once the spark connect server is started, it will be running in localhost and you should be able to see a spark UI at http://localhost:4040/.

If we observe the rightmost part of UI, we can observe name as Spark Connect server application UI which confirms its a spark application running for spark connect.

PySpark from Shell

bin/pyspark --remote sc://localhost

We can start the python shell by providing the remote as a parameter for spark session. We have seen this parameter in the earlier scala example.

Spark Connect Spark Session

Once we started the pyspark shell, we can check if we are running against spark-connect API by inspecting the spark variable that points to spark session.

spark

outputs

<pyspark.sql.connect.session.SparkSession object at 0x10a067ac0>

From the above output, it is confirmed that we are running spark-connect based spark session.

Data Frame Code

The below is a simple PySpark code.

df = spark.range(100)
df.count()

As with scala example, we can observe that in sparkUI gRpc calls will show up when we run this example.

Conclusion

In this post, we saw how to write and run a simple PySpark code against the new spark-connect API.

Understanding Spark Connect API - Part 3: Scala API Example

Tue, 30 May 2023 00:00:00 +0530

In the 3.4 version, Apache Spark has released a new client/server-based API called Spark Connect. This API will help in improving how we develop and deploy Spark applications.

In this series of blogs, we are going to explore various functionalities exposed by spark-connect API. This is the third post in the series where we write a simple scala spark code with spark-connect API. You can read all the posts in the series here.

Spark Connect API Server

Before you can run a program against spark-connect, you need to start spark-connect server which intern starts a spark driver. You can follow the steps below link for the same.

https://spark.apache.org/docs/latest/spark-connect-overview.html#download-and-start-spark-server-with-spark-connect.

Once the spark connect server is started, it will be running in localhost and you should be able to see a spark UI at http://localhost:4040/.

If we observe the rightmost part of UI, we can observe the name as Spark Connect server application UI which confirms it’s a spark application running for spark connect.

Adding Dependencies to Project

We need to add the below dependencies in our sbt for using spark-connect API.

libraryDependencies += "org.apache.spark"%  "spark-connect-client-jvm_2.12"%"3.4.0"
libraryDependencies += "org.apache.spark"%  "spark-catalyst_2.12"%"3.4.0"

In above dependencies,

spark-connect-client-jvm is the spark-connect API for Scala and Java.
spark-catalyst is the catalyst API of spark dataframe. As the spark-connect manipulates the logical plans, we require this dependency.

As you can observe from the dependencies, it is minimal and doesn’t include complete spark dependency.

Writing Scala Spark Example using the Spark Connect API

The below are the steps to write a simple scala spark example using spark-connect API.

1. Create Spark Session using Spark Client API

As the first step, we create a spark session.

import org.apache.spark.sql.SparkSession
val sparkSession = SparkSession.builder().remote("sc://localhost").build()

The above spark session API comes from spark-connect library, which has a special method called remote which signifies we are running our code against the spark-connect rather than a standard spark driver.

2. Create DataFrame

val df = sparkSession.range(500)
print(df.count())

Once we created the session, we can create a df using the standard spark API range method. From there all the API’s of dataframe are available.

Even though this look like normal df, we are actually interacting with spark-connect df rather than standard spark-sql df. You can find the docs for the same in github spark-connect Dataset code.

3. Executing the Code

Once you execute the code, you will see the below output on spark UI.

From the output its clear the code is running on gRPC API of spark-connect.

Code

Complete Code on Github.

Conclusion

In this post, we saw how to write and run a simple spark scala code against the new spark-connect API.

Understanding Spark Connect API - Part 2: Introduction to Architecture

Fri, 26 May 2023 00:00:00 +0530

In the 3.4 version, Apache Spark has released a new client/server-based API called Spark Connect. This API will help in improving how we develop and deploy Spark applications.

In this series of blogs, we are going to explore various functionalities exposed by spark connect API. This is the second post in the series where we discuss the architecture of spark connect. You can read all the posts in the series here.

Architecture

The above diagram shows the architecture of the spark-connect. It has below main components.

1.Spark Connect API

Spark-Connect API is a gRPC based API that runs as a server to connect spark client applications with the spark driver and cluster. As you can observe from the diagram spark driver is no more part of the client application. Now it’s on the server side of spark-connect API.

2.Thin Client Side API

As the spark driver is no more part of the client, how do we write the spark applications?. For this, spark now ships a separate client library which wraps the spark-connect API with a nice DataFrame based API. This makes sure that spark client applications do not need full installation and dependency of the spark. We will explore more about this API in our future posts.

3.DataFrame as the Intermediate Protocol

One of the challenges of providing a new API is it forces all the client applications to be rewritten in this new API. But spark connect API smartly solves this issue by exposing the same DataFrame/ Dataset API as in standard spark code and internally converting those API calls to the gRPC calls. This tremendously helps the application developers as they don’t need to learn yet another new API and reuse all their existing code.

The below diagram how the spark code is converted to gRPC calls.

Benefits of Spark-Connect

This section of the post talks about the different benefits of spark-connect API.

1. Lightweight Client

In the last post, we discussed that in spark driver based architecture the client needs to run a full-fledged spark driver for an interactive application. But with thin client API, the actual heavy lifting is done by the spark server rather than the spark client. So the resource requirements for the client application will be very low.

2.Standard Integration for Notebook Systems

As spark-connect is now part of the standard spark, all notebook systems can standardize on this API to provide interactive integration with spark.

3. Smoother Transition with DataFrame based API

As the spark-connect use DataFrame API as it’s user-facing API, transitioning to spark-connect is quite easy. As most of the spark developers are already familiar with DataFrame API there is almost no learning curve.

References

https://spark.apache.org/docs/latest/spark-connect-overview.html.

Conclusion

In this post, we talked about the architecture of the spark-connect and how it avoids some of the shortcomings of the spark driver architecture.

Understanding Spark Connect API - Part 1: Shortcomings of Spark Driver Architecture

Tue, 09 May 2023 00:00:00 +0530

In the 3.4 version, Apache Spark has released a new client/server-based API called Spark Connect. This API will help in improving how we develop and deploy Spark applications.

In this series of blogs, we are going to explore various functionalities exposed by spark connect API. This is the first post in the series where we discuss the shortcoming of current spark driver architecture. You can read all the posts in the series here.

Spark Driver Architecture

The above diagram shows the architecture of a spark program.

To use spark, the user needs to add spark library dependency to their application and run a JVM which will act as the client for the application. This client JVM, known as a driver, holds the spark context, and the spark context intern communicates with other parts of the cluster to actually run the code. The same architecture is used for other language support like Pyspark.

This approach works for batch-based programs which can run with spark-submit but come with many shortcomings for other kinds of applications. We will discuss these shortcomings in below section.

Shortcomings of Spark Driver Architecture

Interactive Spark Applications

If a user wants to run a spark application in an interactive system like a notebook or IDE they need a long-running spark session/context to which this system connects and executes code.

Current spark driver architecture makes building spark applications from systems like Notebook, and IDE very trick as the machine on which these systems run needs to have full-fledged spark installed and has to have enough resources to run the spark driver applications. This becomes an impediment in many applications as stopping the client system will suddenly stop the spark execution also.

Community REST API Solutions

To address the above issue, there are many community based solutions like Apache Livy or Spark Job Server which tries to provide the REST based interface for the Spark. But there few shortcomings in these solutions.

Frameworks like spark-jobserver extends the spark source code, so they always lag behind the official spark release
The API’s exposed by these services are specific to the libraries so there is no inter portability
As this is not part of official spark, these will be not available in managed spark runtime like Databricks.

Conflict Between Spark Libraries and User Libraries

As spark driver needs to have full spark as its dependency, any libraries used in spark will have conflict with the user-added libraries. As an example, if there is a scala library used by the client app and some other version of the library used by spark it often leads to conflict with spark. This is one of the reasons to remove Akka from the spark dependency.

https://issues.apache.org/jira/browse/SPARK-5293

Stability

As the client-server needs to run the driver application for interactive applications, if the driver crashes then the whole application will crash. So if you are running the spark application behind an HTTP server, the crash of the spark application can crash the HTTP server also which is not desired.

So because of the above shortcomings, we need a fully different way to develop spark applications for non-batch scenarios that will not have the above limitations.

References

Spark Connect SPIP Doc.

Conclusion

In this post, we talked short coming of the current spark driver based application development. In the next post in series, we will discuss how spark connect API is going to address some of these issues.

Email Spam Detection using Pre-Trained BERT Model : Part 2 - Model Fine Tuning

Thu, 16 Feb 2023 00:00:00 +0530

Recently I have been looking into Transformer based machine learning models for natural language tasks. The field of NLP has changed tremendously in the last few years and I have been fascinated by the new architectures and tools that come out at the same time. Transformer models are one such architecture.

As the frameworks and tools to build transformer models keep evolving, the documentation often becomes stale and blog posts are often confusing. So for any one topic, you may find multiple approaches which can confuse beginners.

So as I am learning these models, I am planning to document the steps to do a few of the essential tasks in the simplest way possible. This should help any beginner like me to pick up transformer models.

In this two-part series, I will be discussing how to train a simple model for email spam classification using a pre-trained transformer BERT model. This is the second post in the series where I will be discussing fine-tuning the model for spam detection. You can read all the posts in the series here.

Data Preparation and Tokenization

Please make sure you have gone through the first part of the series where we discussed about how to prepare our data using bert tokenization. You can find the same in the below link.

Email Spam Detection using Pre-Trained BERT Model: Part 1 - Introduction and Tokenization.

Model Fine Tuning

Once the tokenization is done, we are now ready to fine-tune the model.

A pre-trained model comes with a body and head. In most of the use cases, we only retrain the head part of the model. So that’s why we call it fine-tuning rather than retraining. You can read more about the head and body of a transformer model at the below link.

https://huggingface.co/course/chapter1/4.

1.Download Model

As we did with the tokenizer, we will download the model using hugging face library.

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased',num_labels=2)

The above downloads a dummy sequence classification model head which needs to be tuned with data.

2. Training Arguments

Training arguments are where you set various options for given model training. For simplicity, we are going to use default ones.

from transformers import TrainingArguments,Trainer
training_args = TrainingArguments(output_dir="test_trainer")

3. Evaluation Metrics

For our training, we are going to use accuracy as an evaluation metric. The below code sets up a method to calculate the same from the model.

import numpy as np
import evaluate
metric = evaluate.load("accuracy")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

In the above code, np.argmax line converts logits returned from model prediction to labels so that, they can be compared with actual labels.

4. Trainer

Let’s create trainer with below code.

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets_train,
    eval_dataset=tokenized_datasets_eval,
    compute_metrics=compute_metrics,
 )

Trainer API of hugging face handles all the batching and looping needed for fine-tuning the model.

5. Run the Train

Once trainer object is created, we can run the train the model using train method call.

trainer.train()

Find Accuracy on Testing Dataset

Once the model is trained, we can find how well our model is doing using accuracy on test dataset.

predictions_output = trainer.predict(tokenizer_datasets_test)

In above code, we are using trainer.predict method to predict on our test dataset.

accuracy_score = compute_metrics((predictions_output.predictions,tokenizer_datasets_test['label']))
print(accuracy_score)

Then we find the accuracy score using same function we defined at the time of train. The output will be

{'accuracy': 0.97}

As you can see we are getting 97% accuracy which is really good.

Code

Complete code for the post is in below google colab notebook.

https://colab.research.google.com/drive/1d3-22HNkuLCqP1ctemMaMw1ETaWL48ET?usp=sharing.

You can also access python notebook on github.

https://github.com/phatak-dev/transformer-models/blob/main/Spam_Detection_using_BERT_Model.ipynb.

Conclusion

In this post, we saw how to fine-tune a pre-trained model using hugging face API. These two posts give you end to end flow of fine-tuning a transformer model.

Madhukar's Blog

Rediscovering Implicits in Scala 3 - Part 3: Summoning Implicits

Passing Implicit Parameter to a Method in Scala 2.x

Summon in Scala 3.x

Code

References

Summary

Rediscovering Implicits in Scala 3 - Part 2: Extension methods

Extension Methods in Scala 2.x

Extension Methods in Scala 3.x

Code

References

Summary

Rediscovering Implicits in Scala 3 - Part 1: Implicit Parameters

Implicit Parameters in Scala 2.x

Scala 3 Using and Given

Code

References

Building Agentic LLM Workflows with LangGraph - Part 1: Hello World

Installation

Defining Hello World Agent

Building the Graph

Graph Compilation

Visualise the Graph

Running the Graph

Code

Summary

References

Understanding Spark Connect API - Part 5: Dataframe Sharing Across Spark Sessions

Sharing DataFrame across Spark Sessions

Dataframe Plan and Protocol Buffer

Serialising the Spark Plan

Deserialization and Dataframe Recreation

Internal API

1. Deserialize the Plan Object

2. Create Dataframe the Plan

3. Print Data

Code

Conclusion

Understanding Spark Connect API - Part 4: PySpark Example

Spark Connect API Server

PySpark from Shell

Spark Connect Spark Session

Data Frame Code

Conclusion

Understanding Spark Connect API - Part 3: Scala API Example

Spark Connect API Server

Adding Dependencies to Project

Writing Scala Spark Example using the Spark Connect API

1. Create Spark Session using Spark Client API

2. Create DataFrame

3. Executing the Code

Code

Conclusion

Understanding Spark Connect API - Part 2: Introduction to Architecture

Architecture

1.Spark Connect API

2.Thin Client Side API

3.DataFrame as the Intermediate Protocol

Benefits of Spark-Connect

1. Lightweight Client

2.Standard Integration for Notebook Systems

3. Smoother Transition with DataFrame based API

References

Conclusion

Understanding Spark Connect API - Part 1: Shortcomings of Spark Driver Architecture

Spark Driver Architecture

Shortcomings of Spark Driver Architecture

Interactive Spark Applications

Community REST API Solutions

Conflict Between Spark Libraries and User Libraries

Stability

References

Conclusion

Email Spam Detection using Pre-Trained BERT Model : Part 2 - Model Fine Tuning

Data Preparation and Tokenization

Model Fine Tuning

1.Download Model

2. Training Arguments

3. Evaluation Metrics