From Wikipedia's definition::

Pair programming is a software development technique in which two programmers work together at one workstation. One, the driver, writes code while the other, the observer or navigator,[1] reviews each line of code as it is typed in. The two programmers switch roles frequently.

A lot of companies these days are adopting the Pair Programming practice. Some have pioneered the way of working using Pair Programming like ThoughtWorks where I used to work before. But, some are new to this and are slowly getting into it.

Pairing is challenging when you start in itself. The real challenge becomes when the manager evaluates performance on the wrong measures. It bothers all the devs in a team. Managers only see one person commit logs in git or one person assigned to JIRA tasks and thus, accounts for work by only one developer.

In such scenarios, the other developer is not visible to the manager. Managers flag developers as “low-performance” team members.

Not all managers have this problem. Eg. In ThoughtWorks, managers know that a pair works on a task. They are well aware of how each person contributes to the complete delivery of the project. But, for the companies where the practices are being adopted, the managers are not aware of how to measure a developer’s performance. They use the same old metrics like Git log and Jira Tasks to check how many tasks are being completed by the developers. As such, one dev suffers if the commits or the tasks are not directly assigned to them while pairing.

This plays badly with the annual reviews too. When the manager shows them that their performance is not meeting the threshold. This becomes a barrier to pair programming. It certainly creates a picture that pairing is not a good thing because they are being evaluated on the wrong grounds. Ultimately the entire team give up the practice of Pair programming.

Git logs and Jira tasks are the wrong metrics to measure a developer’s performance while adopting pair programming. Managers who are using it should avoid such measuring criteria.

You could also talk to your manager if they are friendly and open to new ideas and new learnings. But, not all managers are like that. Giving feedback sometimes backfires and you are not on good terms with them.

So what should one do to be visible for the work they are doing while Pairing? From my own experience, there are three things which I will share with you that would work well in such cases

1. Add Pariee in Jira Task

When selecting a task on Jira or any other project management software. Make sure to have an extra attribute of “Pariee”. Select the person you are paring with you.

This will help when the manager is generating a report on how many tasks each developer is working on. They will have to calculate the report based on aggregated value of tasks which are directly assigned to you as well as tasks which you were part of as a Pariee.

The above context section of Jira shows a sample task. I have configured my JIRA workflow to have Pairee as another attribute for the task.

2. Git Co-Authored Commit

You could attribute a commit to more than one developer using co-authored commit. Both Github and Gitlab support this feature. When working as a pair make sure you follow the below commit message format

  This is your normal commit message
  
  Co-authored-by: Ninad Ingole <myemail@gmail.com>

With this, you will see both authors in the commit history on GitHub. It will show the commits done to the repository by both the developer in their individual GitHub contribution section.

For people using tools like GitHub Desktop. The GUI shows the option to include another author as a contributor eg

For managers who are doing micro-management. This will be your proof. Proof that you were always pairing and that you share equal contributions to the tasks.

Also for PR Workflows, the Reviewer is aware of the people involved in the change. This helps to reach out to any one of the dev for review comments.

3. Pair Names in Commit Message

Not all platforms of Version control support co-authored commits. This last piece of advice is for such platforms. For old commits, having the Pair names in the messages is a quick navigation tool.

I simply like to keep the Pair names in the format [First/Second] in the commit message. This also shows nicely when I am doing git log --oneline

Conclusion

I hope the practices I shared help you to keep developing pair programming skills. It would also help you to avoid any problems with your manager.

I haven’t shared the benefits of Pair programming in this post. Advantages like knowledge sharing, quick reviews, proper planning etc. are a few to list. I would encourage others who are not doing it to try it out.

For any more questions related to pair programming or any problems please reach out to me. I would be happy to help you in the journey of pair programming 😊

To get started, you first need to import the Google UUID library into your Golang project.

go get -u github.com/google/uuid

Once you have imported the library, you could use the uuid.New() method to generate a random UUID. This method takes no arguments and is the easiest one to get started. 🆒

The challenge with using this is how to create a Unit Test for the code 👊. In the Unit Test, we expect to get a constant known behaviour from our code. But, the behaviour of uuid.New() method is to generate random sequences on each execution.

Let me show you what I mean by using a small code. Below, I am writing a very simple test. I will run this test many times and we will check the output. All the generated UUIDs for each different run will be random.

func Test_UUIDGeneration(t *testing.T) {
  uuid := uuid.New()
  
  fmt.Println(fmt.Sprintf("UUID: %s", uuid))
}

------------ OUTPUT ---------------

  UUID: c26a1010-a207-47cb-b399-9de8acad3bba
  UUID: e4c1776f-cd94-4710-b5eb-710c081df916
  UUID: 923c129c-ba0b-4534-aadf-db98470fd99c

Note:: If you run the above test locally with go test make sure to disable the cache. Use --count=1 in the command to disable caching otherwise the test will generate same uuids

Consider you are writing a service layer that generates the UUID and persist it in the DB. You write a Unit Test but cannot assert the value of the generated UUID. Because, on every new execution on the CI, the value will be different. How will you test the code and avoid randomness?

Here is a small code to mimic a service layer and the test::

type Employee struct {
  ID        uuid.UUID `json:"id"`
  FirstName string    `json:"first_name"`
  LastName  string    `json:"last_name"`
}

// Storage interface is used to persist employee to DB
type Storage interface {
    Save(employee Employee) error
}

// Employee service orchestrate the creation of employee
type EmployeeService struct {
    storage Storage
}

// For simplicity the layer only calls the storage layer to persist the value.
// However, in a real world the service layer will do more than this.
func (e *EmployeeService) Create(firstName, lastName string) error {
  employee := Employee{
    ID:        uuid.New(),
    FirstName: firstName,
    LastName:  lastName,
  }
  
  return e.storage.Save(employee)
}

type mockStorage struct {
    mock.Mock
}

// We mock the storage layer to see if the value and type is correctly passed
// to the method
func (m *mockStorage) Save(employee Employee) error {
  args := m.Called(employee)
  return args.Error(0)
}

func TestEmployeeService_Create(t *testing.T) {
  t.Parallel()
  
  expectedEmployee := Employee{
    ID:        uuid.New(), // This will always generate new random uuid
    FirstName: "John",
    LastName:  "Doe",
  }
  
  storage := &mockStorage{}
  employeeService := EmployeeService{storage: storage}
  
  storage.On("Save", mock.MatchedBy(func (x interface{}) bool {
    employee, ok := x.(Employee)
    
    if !ok {
        return false
    }
    
    return assert.EqualValues(t, expectedEmployee, employee)
  })).Return(nil)
  
  err := employeeService.Create("John", "Doe")
  
  assert.NoError(t, err)
  storage.AssertExpectations(t)
}

If I run the above test it will fail 🔴::

=== RUN   TestEmployeeService_Create
=== PAUSE TestEmployeeService_Create
=== CONT  TestEmployeeService_Create
/tests/data_test.go:89:
      Error Trace:    /tests/data_test.go:89
                      /tests/value.go:586
                      /tests/value.go:370
                      /tests/data_test.go:66
                      /tests/data_test.go:58
                      /tests/data_test.go:92
      Error:          Not equal:
                      expected: tests.Employee{ID:uuid.UUID{0xeb, 0x5, 0x6f, 0x11, 0x47, 0x8, 0x42, 0x2d, 0xb5, 0x23, 0x47, 0x2, 0x40, 0x18, 0x85, 0x94}, FirstName:"John", LastName:"Doe"}
                      actual  : tests.Employee{ID:uuid.UUID{0xb4, 0xb7, 0x8f, 0x21, 0x36, 0xc8, 0x4e, 0x3e, 0x9a, 0xa3, 0x96, 0x79, 0xd3, 0xcc, 0xbc, 0xdc}, FirstName:"John", LastName:"Doe"}

                      Diff:
                      --- Expected
                      +++ Actual
                      @@ -2,3 +2,3 @@
                        ID: (uuid.UUID) (len=16) {
                      -  00000000  eb 05 6f 11 47 08 42 2d  b5 23 47 02 40 18 85 94  |..o.G.B-.#G.@...|
                      +  00000000  b4 b7 8f 21 36 c8 4e 3e  9a a3 96 79 d3 cc bc dc  |...!6.N>...y....|
                        },
      Test:           TestEmployeeService_Create

You could see that the UUIDs were generated by the service layer in the logs. To use assert.EqualValues or assert.Equals on a struct, all the attribute values should match. The UUID in the expected employee instance is different.

What would you do to avoid this?

Many people to avoid the problem will think of not writing the test 😅. Let's see the different ways to overcome this problem.

1. Manually check all the struct attributes and avoid the UUID check

Our assumption is that this library is open-source and well-maintained by the community. We can rely on the fact that things will work fine. We skip the UUID check part but check other attributes eg::

storage.On("Save", mock.MatchedBy(func(x interface{}) bool {
  employee, ok := x.(Employee)
  
  if !ok {
    return false
  }
  
  return employee.ID != uuid.Nil && employee.FirstName == expectedEmployee.FirstName && employee.LastName == expectedEmployee.LastName
})).Return(nil)

This is ok if you have a struct with few attributes. But, this will become a headache if you have 15–20 attributes. This indeed is not a scalable approach.

2. Parse the UUID To Check For Error

Parse the UUID to check if the service generates the UUID and if there is no issue. For other attributes do exactly what we did in the previous example. This will also suffer from the same problem as the previous solution

    _, err := uuid.Parse(employee.ID.String())
  
    return err == nil && employee.FirstName == expectedEmployee.FirstName && employee.LastName == expectedEmployee.LastName

3. Pass a UUID Generator Function Type

A separate uuid generator function is passed to the struct to generate the uuid. A test generator is then used in the unit test to mock and pass a static UUID. An assertion on this static value will verify the behaviour of the code. Eg::

type UUIDGenerator func () uuid.UUID

var DefaultUUIDGenerator = func () uuid.UUID {
    return uuid.New()
}

type EmployeeService struct {
  storage       Storage
  uuidGenerator UUIDGenerator
}

func (e *EmployeeService) Create(firstName, lastName string) error {
  employee := Employee{
    ID:        e.uuidGenerator(),
    FirstName: firstName,
    LastName:  lastName,
  }
  
  return e.storage.Save(employee)
}

func TestEmployeeService_Create(t *testing.T) {
  t.Parallel()
  
  expectedID := uuid.New()
  
  expectedEmployee := Employee{
    ID:        expectedID,
    FirstName: "John",
    LastName:  "Doe",
  }
  
  storage := &mockStorage{}
  
  uuidGenerator := func () uuid.UUID {
    return expectedID
  }
  
  employeeService := EmployeeService{storage: storage, uuidGenerator: uuidGenerator}
  
  storage.On("Save", mock.MatchedBy(func (x interface{}) bool {
    employee, ok := x.(Employee)
    
    if !ok {
        return false
    }
    
    return assert.EqualValues(t, expectedEmployee, employee)
  })).Return(nil)
  
  err := employeeService.Create("John", "Doe")
  
  assert.NoError(t, err)
  storage.AssertExpectations(t)
}

This solution is also good. It will allow us to assert the entire entity and don't have to check every field like in the previous two examples. This may become a problem if you have to use it at many places in your code base.

You could create an abstraction that will help to reuse and it will work fine. But what if I tell you there is a * secret* way and you don't have to do all these shenanigans?

Use SetRand 🚀

Let's look at the documentation of the library. There is a way to generate deterministic UUID, using uuid.SetRand(). If we use the correct Seed Source to the SetRand function it will allow us to get deterministic UUIDs.

Let’s see how::

func Test_uuidTest(t *testing.T) {
  uuid.SetRand(rand.New(rand.NewSource(1)))
  
  val1 := uuid.New()
  val2 := uuid.New()
  
  assert.EqualValues(t, uuid.MustParse("52fdfc07-2182-454f-963f-5f0f9a621d72"), val1, fmt.Sprintf("generated %v", val1))
  assert.EqualValues(t, uuid.MustParse("9566c74d-1003-4c4d-bbbb-0407d1e2c649"), val2, fmt.Sprintf("generated %v", val2))
}

I add another test above to show how to use the SetRand method. I pass a new *rand.Rand instance by setting the seed value to the rand.NewSource(1). When you do this the UUIDs generated will be deterministic in nature. They will be exactly the same on each run irrespective of the platform you run on.

To verify this, I ran the same example on the go playground here. You will see below that the generated UUIDs match our assertion value.

So let’s update our existing test to see how to use this new trick.

func TestEmployeeService_Create(t *testing.T) {
  uuid.SetRand(rand.New(rand.NewSource(1)))
  
  expectedEmployee := Employee{
    ID:        uuid.MustParse("52fdfc07-2182-454f-963f-5f0f9a621d72"),
    FirstName: "John",
    LastName:  "Doe",
  }
  
  storage := &mockStorage{}
  
  employeeService := EmployeeService{storage: storage}
  
  storage.On("Save", mock.MatchedBy(func (x interface{}) bool {
    employee, ok := x.(Employee)
    
    if !ok {
        return false
    }
    
    return assert.EqualValues(t, expectedEmployee, employee)
  })).Return(nil)
  
  err := employeeService.Create("John", "Doe")
  
  assert.NoError(t, err)
  storage.AssertExpectations(t)
}

There is one caveat with this approach. The math/rand source that we have used is not thread-safe. As per the documentation::

// NewSource returns a new pseudo-random Source seeded with the given value.
// Unlike the default Source used by top-level functions, this source is not
// safe for concurrent use by multiple goroutines.
// The returned Source implements Source64.

So having multiple tests running parallel would cause a data race issue. I tried to overcome this but there’s no straightforward way. I don’t want to add another layer of complexity to avoid the first one 😅 it’s not productive at all.

I looked into how these libraries did the unit test. I realised that both google/uuid and golang math/rand package do not use t.Parallel() for the tests. So, The simplest way to fix the issue would be to remove t.Parallel() from all such unit tests ✅.

5. Use google/go-cmp

After publishing this post, my awesome readers reached out to me with one more approach. I am really thankful to all those feedback. The google/go-cmp library allows to skip/ignore fields from comparison.

There are two kind of people, one who wants to compare all the fields of the struct and some would prefer to ignore the UUID check because the google/uuid library is battle tested. So based on which category you belong to you could use the previous approach or you could use this approach.

Let’s rewrite our existing test to use the cmpopts.IgnoreFields

func TestEmployeeService_Create(t *testing.T) {
 expectedEmployee := Employee{
  FirstName: "John",
  LastName:  "Doe",
 }

 storage := &mockStorage{}

 employeeService := EmployeeService{storage: storage}

 storage.On("Save", mock.MatchedBy(func(x interface{}) bool {
  actual, ok := x.(Employee)

  if !ok {
   return false
  }

  var diff string
  if diff = cmp.Diff(expectedEmployee, actual, cmpopts.IgnoreFields(Employee{}, "ID")); diff != "" {
   t.Errorf("Save() mismatch (-want +got):\n%s", diff)
  }

  return diff == ""
 })).Return(nil)

 err := employeeService.Create("John", "Doe")

 assert.NoError(t, err)
 storage.AssertExpectations(t)
}

On failure ☠️ the test will show a diff comparison of the fields that didn’t match like below

=== RUN   TestEmployeeService_Create
    /Users/neenadingole/codes/opensource/gotest-ls/main_test.go:275: Save() mismatch (-want +got):
          main.Employee{
                ... // 1 ignored field
        -       FirstName: "Face",
        +       FirstName: "John",
                LastName:  "Doe",
          }
    /Users/neenadingole/codes/opensource/gotest-ls/main_test.go:275: Save() mismatch (-want +got):
          main.Employee{
                ... // 1 ignored field
        -       FirstName: "Face",
        +       FirstName: "John",
                LastName:  "Doe",
          }
--- FAIL: TestEmployeeService_Create (0.00s)
panic:

mock: Unexpected Method Call
-----------------------------

Save(main.Employee)
                0: main.Employee{ID:uuid.UUID{0x4d, 0x8e, 0x29, 0x66, 0x29, 0xdd, 0x4b, 0xbe, 0xa7, 0x30, 0x34, 0xa0, 0x7a, 0xe7, 0xb3, 0x94}, FirstName:"John", LastName:"Doe"}

The closest call I have is:

Save(mock.argumentMatcher)
                0: mock.argumentMatcher{fn:reflect.Value{typ:(*reflect.rtype)(0x12fd9e0), ptr:(unsafe.Pointer)(0xc00010ce40), flag:0x13}}


Diff: 0: FAIL:  (main.Employee={4d8e2966-29dd-4bbe-a730-34a07ae7b394 John Doe}) not matched by func(interface {}) bool [recovered]
        panic:

It becomes easy to navigate the test and fix the issue and the you could ignore multiple fields and from the struct comparison.

I am definitely going to use this library in my projects. This is also a great learning for me by writing and sharing this blog post.

Conclusion

I hope this helps you to write better tests for your code when using the google uuid package. I am very keen on testing code and finding ways to ease testing for developers. Having proper tests helps the team move faster. It also provides a safety net to avoid accidental bug leaks or any dev mistakes.

There are different pros and cons to each of the methods discussed. It will depend on what works for you and what doesn’t. For a small codebase, I don’t like using the generator function approach. I would use the SetRand for such cases. For larger struct or ignoring the field comparison for some then I would also prefer the go-cmp library which does a really good job.

I was curious to see if there is any simple way for unit testing. This post is my research on this topic, but I know there may be some approaches I missed. So, I would like to hear those from others. I will definitely be happy to update my findings with your thoughts 🙏.

Thank you so much for taking the time to read my blog and sharing it with others. Your ❤️ support means a lot to me, and I truly appreciate it. It’s people like you who inspire me to keep writing and sharing my thoughts with the world. Thank you again for your kindness and support.

Click to become a medium member and read unlimited stories!

Supporting Information

A Sample Test From google/uuid package::

// No t.Parallel() used
func TestSetRand(t *testing.T) {
 myString := "805-9dd6-1a877cb526c678e71d38-7122-44c0-9b7c-04e7001cc78783ac3e82-47a3-4cc3-9951-13f3339d88088f5d685a-11f7-4078-ada9-de44ad2daeb7"

 SetRand(strings.NewReader(myString))
 uuid1 := New()
 uuid2 := New()

 SetRand(strings.NewReader(myString))
 uuid3 := New()
 uuid4 := New()

 if uuid1 != uuid3 {
  t.Errorf("expected duplicates, got %q and %q", uuid1, uuid3)
 }
 if uuid2 != uuid4 {
  t.Errorf("expected duplicates, got %q and %q", uuid2, uuid4)
 }
}

A sample test from golang math/rand package::

// No t.Parallel() used
func testReadUniformity(t *testing.T, n int, seed int64) {
 r := New(NewSource(seed))
 buf := make([]byte, n)
 nRead, err := r.Read(buf)
 if err != nil {
  t.Errorf("Read err %v", err)
 }
 if nRead != n {
  t.Errorf("Read returned unexpected n; %d != %d", nRead, n)
 }

 // Expect a uniform distribution of byte values, which lie in [0, 255].
 var (
  mean       = 255.0 / 2
  stddev     = 256.0 / math.Sqrt(12.0)
  errorScale = stddev / math.Sqrt(float64(n))
 )

 expected := &statsResults{mean, stddev, 0.10 * errorScale, 0.08 * errorScale}

 // Cast bytes as floats to use the common distribution-validity checks.
 samples := make([]float64, n)
 for i, val := range buf {
  samples[i] = float64(val)
 }
 // Make sure that the entire set matches the expected distribution.
 checkSampleDistribution(t, samples, expected)
}

// No t.Parallel() used
func TestReadUniformity(t *testing.T) {
 testBufferSizes := []int{
  2, 4, 7, 64, 1024, 1 << 16, 1 << 20,
 }
 for _, seed := range testSeeds {
  for _, n := range testBufferSizes {
   testReadUniformity(t, n, seed)
  }
 }
}

Edits::

• Add google/go-cmp library usage as suggested by readers.

If Yes, I found a small trick in Intellij and DataGrip that could watch your SQL query results. It was always there but I overlooked it. Once I discovered it, it changed my life for debugging or development.

It’s not a trick actually 😅 but a feature built within the IntelliJ products. It could easily go ignored but let’s see how to use it

When you run any query you will see the Result window below::

You would see this clock icon in the toolbar that does all the trick. It was always there but I never thought it would be so useful 😍

When you click on it, the below options would be available. Each choice you select will refresh the query results after the selected delay.

A small tip to remember is to make sure you “Pin” 📌 the result window. When you do it, you could run other queries along with the watch window.

The below video shows a small demo of this feature. In the demo, I am watching the result of a SELECT query. When I update a record it is shown in the result window without running the SELECT query again.

5s seems long? What if I want to keep the watch period lower than 5s? You could use the Custom option provided. With this, you could also monitor your query changes at a 1s rate interval

Well, that's all for this post. I hope next time this small utility will come in handy for you. Don’t run the query every time to see the change. IntelliJ tools are a powerhouse of such small utilities which are very easily ignored by us.

Thanks! and Happy Coding!

In my previous blog post, "A Billion Dollar Go Mistake". I discussed a common but naive mistake that developers make in Golang, which can lead to connection leaks. Although I offered several ways to fix this problem, one issue still bothered both myself and my readers. We can solve this problem by abstracting the transaction mechanism into a different layer. Many people reached out to me to suggest this solution.

So, my first curiosity question I want to reason::

Do we really need a Transaction layer?

I read several other blogs and GitHub codes. I noticed that many of them were great but did not include tests to prove that the abstraction layer works. But, we cannot confirm that the transaction layer works without tests. A connection leak may occur, and we cannot detect it by examining the code.

This led me to another important question::

How can I prove that the new layer have no connection leask?

Testing is the only way to verify whether something works as expected. I decided to start with two straightforward options that came to my mind::

1 - I will run some queries that change the table and verify that the code commits the data to the database. 2 - I will some mechanisms that expose the connection information. I could verify that the code closes the connections at the end.

The first approach is simple, and I have used it many times in my projects 😅. But, the second approach is completely unknown to me 👾. When I face difficulties in understanding how things work, I usually look at the source code of the language.

To resolve my problem, I looked for help in the standard database library code of Golang. I found something that could help me:: the DBStats struct.

// DBStats contains database statistics.
type DBStats struct {
  MaxOpenConnections int // Maximum number of open connections to the database.
  
  // Pool Status
  OpenConnections int // The number of established connections both in use and idle.
  InUse           int // The number of connections currently in use.
  Idle            int // The number of idle connections.
  
  // Counters
  WaitCount         int64         // The total number of connections waited for.
  WaitDuration      time.Duration // The total time blocked waiting for a new connection.
  MaxIdleClosed     int64 // The total number of connections closed due to SetMaxIdleConns.
  MaxIdleTimeClosed int64 // The total number of connections closed due to SetConnMaxIdleTime.
  MaxLifetimeClosed int64 // The total number of connections closed due to SetConnMaxLifetime.
}

This is precisely the solution I was searching for. Upon completion of the transaction, the MaxOpenConnections and InUse counts should be 0. If this is not the case, it indicates a potential leak in the abstraction layer.

I am thrilled to have found the ideal solution to address the second scenario 🎉

To access the DBStats, we can use the Stats() method on the sql.DB instance, as shown below::

db, _ := sqlx.Open("postgres", "postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable")
db.Stats()

The source code for the Stats() method is below. You can read more about it's working and how go code records the information by following the source code here

// Stats returns database statistics.
func (db *DB) Stats() DBStats {
  wait := db.waitDuration.Load()
  
  db.mu.Lock()
  defer db.mu.Unlock()
  
  stats := DBStats{
    MaxOpenConnections: db.maxOpen,
    
    Idle:            len(db.freeConn),
    OpenConnections: db.numOpen,
    InUse:           db.numOpen - len(db.freeConn),
    
    WaitCount:         db.waitCount,
    WaitDuration:      time.Duration(wait),
    MaxIdleClosed:     db.maxIdleClosed,
    MaxIdleTimeClosed: db.maxIdleTimeClosed,
    MaxLifetimeClosed: db.maxLifetimeClosed,
  }
  
  return stats
}

All this is enough for me to implement the Transaction Layer. So, Let's delve into the code now. But before we get started, let me introduce you to some of the libraries that I will be using in the code::

• Sqlx
• dockertest
• testify
• go-sqlmock

Interestingly, it's hard to reproduce the connection leak scenario in a transaction layer. Trust me, I've tried and failed. 😂

I'll show how to test the old code using DBStats assertions. With this example, people who will not abstract the transaction layer could update their tests to avoid any connection leaks. Later on, we'll explore how to extract the transaction layer and test it.

package apptest

import (
  "context"
  "database/sql"
  "github.com/jmoiron/sqlx"
)

type Subscription struct {
  ID         int64        `db:"id"`
  Status     string       `db:"status"`
  CanceledAt sql.NullTime `db:"canceled_at"`
}

// ------------------------------ Repository ------------------------------

type srepo struct {
}

// GetSubscription fetches the subscription by id
func (r *srepo) GetSubscription(tx *sqlx.Tx, id int64) (Subscription, error) {
  var sub Subscription
  err := tx.Get(&sub, "SELECT * FROM subscription WHERE id = $1", id)
  if err != nil {
    return sub, err
  }

  return sub, nil
}

// CancelSubscription cancels a given subscription by setting canceled_at to now()
func (r *srepo) CancelSubscription(tx *sqlx.Tx, id int64) (Subscription, error) {
  var sub Subscription
  err := tx.Get(&sub, "UPDATE subscription SET canceled_at = NOW(), status='canceled' WHERE id = $1 RETURNING *", id)
  if err != nil {
    return sub, err
  }

  return sub, nil
}

// ------------------------------ Service ------------------------------

type Service struct {
  db   *sqlx.DB
  repo *srepo
}

func NewService(db *sqlx.DB, repo *srepo) *Service {
  return &Service{repo: repo, db: db}
}

func (s *Service) CancelSubscription(ctx context.Context, id int64) (*Subscription, error) {
  tx, err := s.db.BeginTxx(ctx, nil)
  if err != nil {
    return nil, err
  }

  defer func() {
    // !!! This would not work if the subscriptions is already canceled 
    // and the error is not returned
    if err != nil {
      _ = tx.Rollback()
      return
    }
  }()

  sub, err := s.repo.GetSubscription(tx, id)
  if err != nil {
    return nil, err
  }

  if sub.Status != "active" {
    return &sub, nil
  }

  if sub.CanceledAt.Valid {
    return &sub, nil
  }

  sub, err = s.repo.CancelSubscription(tx, id)
  if err != nil {
    return nil, err
  }

  err = tx.Commit()

  return &sub, err
}

What's wrong with the above code?

When the subscription is already cancelled it will return without error. When the function returns without error the connection is neither rollback nor commits. This causes the connection leak.

Below is the integration test to see how we can catch the leak.

func Test_ConnectionLeak(t *testing.T) {
  pg, err := apptest.StartTestPostgres(t) // Please use the source code to learn more about this code
  require.NoError(t, err)
  
  _, err = pg.DB.Exec("CREATE TABLE IF NOT EXISTS subscription (id serial PRIMARY KEY, status varchar(25) NOT NULL, canceled_at timestamp NULL)")
  require.NoError(t, err)
  
  _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('active', NULL)")
  require.NoError(t, err)
  
  _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('canceled', '2023-02-02 01:00:00')")
  require.NoError(t, err)
  
  subscription, err := NewService(pg.DB, &srepo{}).CancelSubscription(context.Background(), 2)
  require.NoError(t, err)
  
  stats := pg.DB.Stats()
  require.Equal(t, 0, stats.InUse, "expected no connections in use")
  require.Equal(t, 0, stats.MaxOpenConnections, "expected no max open connection")
  
  require.Equal(t, "canceled", subscription.Status)
  require.Equal(t, "2023-02-02 01:00:00 +0000 +0000", subscription.CanceledAt.Time.String())
}

I am using an integration test here. I connect to a Postgres DB instance in docker and run the test. From the test result below you could see that the code has a transaction problem. The actual count InUse is 1. The connection is not closed at the end of the function call. This is what we will solve using the transaction abstraction layer in the next section.

=== RUN   Test_ConnectionLeak
    service-connection-leak_test.go:27: 
         Error Trace: /pkg/service-connection-leak_test.go:27
         Error:       Not equal: 
                      expected: 0
                      actual  : 1
         Test:        Test_ConnectionLeak
         Messages:    expected no connections in use
--- FAIL: Test_ConnectionLeak (6.69s)

Expected :0
Actual   :1
<Click to see difference>

FAIL

Extracting the Transaction Layer

To address the connection leak issue, one way could be to fix what is failing. Close the connection by manually testing the service and then reviewing the code. But that same issue could creep in again in future. The right way is to extract a transaction layer. The idea behind this extraction is simple:: we provide a public method

func InTx(ctx context.Context, db *sqlx.DB, txFunc func (*TxWrap) error) (err error)

which accepts the transaction-based business logic in the txFunc parameter.

With this approach, the developer no longer has to manually handle transactions, as the InTx method abstracts the transaction mechanism away from the business logic. By passing the txFunc parameter to InTx, the developer can focus on the actual business operations, without worrying about the underlying transaction management.

I am keeping the code simple and avoiding any complications for blogging purposes.

package db

import (
  "context"
  "database/sql"
  "github.com/jmoiron/sqlx"
)

// TxWrap is a wrapper around sqlx.Tx that adds a context
// and redirects calls to methods like Get, Select to GetContext and SelectContext
// with the context it wraps.
type TxWrap struct {
  tx  *sqlx.Tx        // underlying transaction
  ctx context.Context // context to use for all calls
}

// Get is a wrapper around sqlx.Tx.GetContext
// that uses the context it wraps.
func (tx *TxWrap) Get(dest interface{}, query string, args ...interface{}) error {
  return tx.tx.GetContext(tx.ctx, dest, query, args...)
}

// Select is a wrapper around sqlx.Tx.SelectContext
// that uses the context it wraps.
func (tx *TxWrap) Select(dest interface{}, query string, args ...interface{}) error {
  return tx.tx.SelectContext(tx.ctx, dest, query, args...)
}

// IMPLEMENT OTHER RELATED sqlx Methods to use wrapped context

// InTx executes a function in a transaction.
// If the function returns an error, the transaction is rolled back.
// If the function panics, the transaction is rolled back and the panic is re-raised.
// If the function returns nil, the transaction is committed.
func InTx(ctx context.Context, db *sqlx.DB, txFunc func(*TxWrap) error) (err error) {
  ctx, cancel := context.WithCancel(ctx)
  defer cancel()

  tx, err := db.BeginTxx(ctx, nil)
  if err != nil {
    return err
  }

  txWrap := &TxWrap{
    tx:  tx,
    ctx: ctx,
  }

  defer func() {
    if p := recover(); p != nil {
      _ = txWrap.tx.Rollback()
      panic(p)
    }
    if err != nil {
      _ = txWrap.tx.Rollback()
      return
    }
    err = txWrap.tx.Commit()
  }()

  return txFunc(txWrap)
}

The InTx wraps around the actual sqlx.Tx in TxWrap struct and encapsulates the transaction management logic within the method. The InTx takes care of calling Begin, Commit and Rollback.

The TxWrap struct also contains a derived cancel context (ctx), which was discussed in my previous post to ensure the context cancellation happens at the end of the method call. Inside the InTx method, we have a defer block that handles three possible outcomes of the transaction logic::

1 - Panic:: In case of any unhandled exception, the transaction will automatically roll back. 2 - Error:: If any error occurs during the transaction, it will also be rolled back. 3 - Success:: If everything goes as expected, the transaction will be committed.

To provide further safety, we also implement the same Get and Select functions as in the sqlx library, but we proxy the calls to the context version of these functions. This ensures that any in-process requests are cancelled if the context is cancelled, if a client cancels an HTTP request.

Now the real excitement begins !!! TESTS !!! 🚀 🥹

I am doing both approaches, the Unit test and the Integration Test. It is up to you what you want to use. My own preference is the Integration test. They mimic behaviour close to real infrastructure.

Unit Test

For the unit test, I am using sqlmock. I set up the expectation as per the 3 behaviours of our code and assert if the expectations are met. I also check the connection count resets 0 at the end. Unit Tests are pretty fast so we are also doing t.Parallel and initiating a new sqlmock for every test.

// ------------------------------ UNIT Test ------------------------------
func Test_Unit(t *testing.T) {
	t.Parallel()
	
  tests := []struct {
    name      string
    fn        func (tx *TxWrap) error
    setup     func (mock sqlmock.Sqlmock)
    wantErr   bool
    wantPanic bool
  }{
    {
      name: "success path",
      fn: func (tx *TxWrap) error {
          return nil
      },
      setup: func (mock sqlmock.Sqlmock) {
        mock.ExpectBegin()
        mock.ExpectCommit()
      },
    },
    {
      name: "failure path",
      fn: func (tx *TxWrap) error {
          return errors.New("some error")
      },
      setup: func (mock sqlmock.Sqlmock) {
        mock.ExpectBegin()
        mock.ExpectRollback()
      },
      wantErr: true,
    },
    {
      name: "panic",
      fn: func (tx *TxWrap) error {
        panic("some panic")
        return nil
      },
      setup: func (mock sqlmock.Sqlmock) {
        mock.ExpectBegin()
        mock.ExpectRollback()
      },
      wantPanic: true,
    },
  }
  for _, test := range tests {
    test := test
    t.Run(test.name, func (t *testing.T) {
      t.Parallel()
      
      db, mock, err := sqlmock.New()
      require.NoError(t, err)
      
      dbx := sqlx.NewDb(db, "sqlmock")
      
	    if test.setup != nil {
        test.setup(mock)
      }
      
      // Only add this defer when we expect panic to take over the 
      // panic recovery and see if there is a valid error
      if test.wantPanic {
        defer func () {
            require.NotNil(t, recover())
        }()
      }
      
      err = InTx(context.Background(), dbx, test.fn)
      
      require.Equal(t, test.wantErr, err != nil)
      require.NoError(t, mock.ExpectationsWereMet())
      
      stats := dbx.Stats()
      require.Equal(t, 0, stats.InUse)
      require.Equal(t, 0, stats.MaxOpenConnections)
    })
  }
}

Integration Test

For the Integration test below, I am using Postgres DB. I create a dummy Employees table first. Then I execute some inserts and selects statements for different scenario. In the end, check if the layer closes the connection for every test.

A point to note here is that I am not doing t.Parallel. The shared connection between tests in the parallel run would be a problem. The InUse and MaxOpenConnections will never be 0. It is up to you how you want to do it. You can do a similar thing we did in the Unit test to create a separate connection for every test::

// ---------------------------------- INTEGRATION TEST -------------------------------------
type Employee struct {
  ID   int64  `db:"id"`
  Name string `db:"name"`
}

func Test_Integration(t *testing.T) {
  pg, err := apptest.StartTestPostgres(t)
  require.NoError(t, err)

  _, err = pg.DB.Exec("CREATE TABLE IF NOT EXISTS employee (id serial PRIMARY KEY, name varchar(25) NOT NULL)")
  require.NoError(t, err)

  tests := []struct {
    name      string
    txfunc    func (tx *TxWrap) error
    wantErr   bool
    wantPanic bool
  }{
    {
      name: "success path",
      txfunc: func (tx *TxWrap) error {
        _, err := tx.Exec("INSERT INTO employee (name) VALUES ('John Doe')")
        return err
      },
    },
    {
      name: "failure path",
      txfunc: func (tx *TxWrap) error {
        var employee Employee
        err := tx.Get(&employee, "SELECT * FROM employee WHERE id = $1", 100)
        return err
      },
      wantErr: true,
    },
    {
      name: "panic",
      txfunc: func (tx *TxWrap) error {
        panic("some panic")
        
        return nil
      },
      wantPanic: true,
    },
  }
  for _, test := range tests {
    test := test
    t.Run(test.name, func (t *testing.T) {
      // Wrap the function in a defer to catch panics
      // and assert that the panic is not nil.
      defer func () {
        if test.wantPanic {
          require.NotNil(t, recover())
        }
        stats := pg.DB.Stats()
        require.Equal(t, 0, stats.InUse)
        require.Equal(t, 0, stats.MaxOpenConnections)
      }()
      
      err = InTx(context.Background(), pg.DB, test.txfunc)
      
      require.Equal(t, test.wantErr, err != nil)
    })
  }
}

Updating the Service

We have the new transaction layer ready. Let's change our old code to use this new layer.

// ------------------------------ Repository ------------------------------

type txRepo struct {
}

// GetSubscription is a repository method that does not leak connections
// it uses *TxWrap to wrap the transaction
// it uses the context to cancel the transaction if the context is canceled
// but the context is inside the *TxWrap and not exposed to the service
func (r *txRepo) GetSubscription(tx *db.TxWrap, id int64) (Subscription, error) {
  var sub Subscription
  err := tx.Get(&sub, "SELECT * FROM subscription WHERE id = $1", id)
  if err != nil {
    return sub, err
  }
  
  return sub, nil
}

func (r *txRepo) CancelSubscription(tx *db.TxWrap, id int64) (Subscription, error) {
  var sub Subscription
  err := tx.Get(&sub, "UPDATE subscription SET canceled_at = NOW(), status='canceled' WHERE id = $1 RETURNING *", id)
  if err != nil {
    return sub, err
  }
  
  return sub, nil
}

// ------------------------------ Service ------------------------------

type txService struct {
  db   *sqlx.DB
  repo *txRepo
}

// CancelSubscriptionWithoutLeak is a service method that does not leak connections
// it uses InTx helper to wrap the transaction
func (s *txService) CancelSubscriptionWithoutLeak(ctx context.Context, id int64) (*Subscription, error) {
  var sub Subscription
  var err error
  
  // So cool!!!!!!!! 🎸
  err = db.InTx(ctx, s.db, func (tx *db.TxWrap) error {
    sub, err = s.repo.GetSubscription(tx, id)
    if err != nil {
      return err
    }
    
    if sub.Status != "active" {
      return nil
    }
    
    if sub.CanceledAt.Valid {
      return nil
    }
    
    sub, err = s.repo.CancelSubscription(tx, id)
    if err != nil {
      return err
    }
    
    return nil
  })
  
  return &sub, err
}

The Test::

func Test_NoConnectionLeak(t *testing.T) {
  pg, err := apptest.StartTestPostgres(t)
  require.NoError(t, err)
  
  _, err = pg.DB.Exec("CREATE TABLE IF NOT EXISTS subscription (id serial PRIMARY KEY, status varchar(25) NOT NULL, canceled_at timestamp NULL)")
  require.NoError(t, err)
  
  _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('active', NULL)")
  require.NoError(t, err)
  
  _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('canceled', '2023-02-02 01:00:00')")
  require.NoError(t, err)
  
  subscription, err := NewTxService(pg.DB, &txRepo{}).CancelSubscriptionWithoutLeak(context.Background(), 2)
  require.NoError(t, err)
  
  stats := pg.DB.Stats()
  require.Equal(t, 0, stats.InUse, "expected no connections in use")
  require.Equal(t, 0, stats.MaxOpenConnections, "expected no max open connection")
  
  require.Equal(t, "canceled", subscription.Status)
  require.Equal(t, "2023-02-02 01:00:00 +0000 +0000", subscription.CanceledAt.Time.String())
}

-------------

== = RUN   Test_NoConnectionLeak
--- PASS: Test_NoConnectionLeak (5.61s)
PASS

The test above is the exact test we used in our connection leak example. I have to tweak it for the new service and new imports. Everything else in the test is the same.

You could see when we changed the service to use the new transaction layer our same test is green. 🥳🥳🥳

Do We Really Need A Transaction Layer?

It was tough for me to create a connection leak with the transaction layer in the old code. It is close to impossible that anything could go wrong.

Although, I felt that testing would be a challenge. The DBStruct provided an easy way for testing.

With different business operations within a transaction, a single line could cause problems and issues in production. But a separate battle-tested layer would safeguard against any issues. Plus, the same layer could be shared with different flows to avoid code repetition.

So, if you haven't yet extracted that transaction logic please do! You could write your own library or use something off the shelf. I am also going to publish the lib on my GitHub which I would share with you 🚀

Conclusion

With this proof, I am very confident that I will not cause another transaction-related production issue. And I hope this blog post helps you to learn something new about Golang.

Some of my initial assumptions were proven wrong. The tests could allow me to answer that the layer works and there will be no connection leak.

In the first part, we saw how to handle the DB transaction at the service level. What if we still miss the Rollback call? To handle such cases, we could build another layer of safety.

There are two approaches I thought of to handle such cases::

• Using a context
• Server side timeout

Using a context

We could use a cancel context to release the lock at the end of the transaction. Context is very helpful in many use cases.

Let’s see how to do things with context

func (s *service) CancelSubscription(ctx context.Context, id uuid.UUID) (*model.Subscription, error) {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()
    
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return nil, err
    }
	
    defer func() {
        if err != nil {
            tx.Rollback()
        }
    }()
    
    subscription, err := s.fetcher.GetSubscription(ctx, tx, id)
    if err != nil {
        return nil, err
    }

    if subscription.CancelledAt != nil {
        return subscription, nil
    }
	
    subscription.CancelledAt = time.Now()

    err = s.updater.UpdateSubscription(ctx, tx, subscription)
    if err != nil {
        return nil, err
    }

    err = tx.Commit()
    if err != nil {
        return nil, err
    }

    return subscription, nil
}

In the above code, we create a new cancel context from the parent context. If the DB transaction is not released either using Rollback or Commit. A defer cancel() happens at the end when the call returns.

The cancel call will notify the transaction that the operation is complete. The go runtime will close the transaction.

The defer cancel() call is important. If you miss that then the problem will still persist

At the go source code level transaction begin call starts a background go routine. The go routine monitors context Done signal. Let's see how it happens using context as shown here

func (db *DB) beginDC(ctx context.Context, dc *driverConn, release func(error), opts *TxOptions) (tx *Tx, err error) {
     var txi driver.Tx
     keepConnOnRollback := false
     withLock(dc, func() {
      _, hasSessionResetter := dc.ci.(driver.SessionResetter)
      _, hasConnectionValidator := dc.ci.(driver.Validator)
      keepConnOnRollback = hasSessionResetter && hasConnectionValidator
      txi, err = ctxDriverBegin(ctx, opts, dc.ci)
     })
     if err != nil {
      release(err)
      return nil, err
     }

     ctx, cancel := context.WithCancel(ctx)
     tx = &Tx{
      db:                 db,
      dc:                 dc,
      releaseConn:        release,
      txi:                txi,
      cancel:             cancel,
      keepConnOnRollback: keepConnOnRollback,
      ctx:                ctx,
     }
     go tx.awaitDone() // GO WAITS FOR THE CONTEXT TO DONE
     return tx, nil
}

And inside the tx.awaitDone

  func (tx *Tx) awaitDone() {
   // Wait for either the transaction to be committed or rolled
   // back, or for the associated context to be closed.
   <-tx.ctx.Done()
  
   // Discard and close the connection used to ensure the
   // transaction is closed and the resources are released.  This
   // rollback does nothing if the transaction has already been
   // committed or rolled back.
   // Do not discard the connection if the connection knows
   // how to reset the session.
   discardConnection := !tx.keepConnOnRollback
   tx.rollback(discardConnection)  // AT THE END THE TX WILL BE ROLLBACKED
  }

So transactions with a cancel context make more sense. Do not use the context coming from the HTTP request. HTTP context works like a client-side timeout. The service layer has no control over cancelling the HTTP context.

A timeout context could also work but that will not be that reliable. Finding the right timeout value is a difficult task in itself.

Server-side timeout

You set an optimal timeout for the request to complete and if the request is not completed within that time, you cancel the operation and return an error.

server := &http.Server{
    Addr:         ":8080",
    Handler:      router,
    WriteTimeout: 2 * time.Second,
}

In the above code, we are setting a server-side timeout of 2 seconds. If the entire request processing is not completed within 2 seconds, we cancel the operation and return an error to the client. With this approach, you don't need to create a timeout context for each request. But, you would still need to fine-tune the timeout value based on the traces and load testing.

Conclusion

I hope you found this second part of the post useful. Adding a layer of safety is always a good idea, but it comes with a cost. In this case, I would recommend using the context approach along with the defer tx.Rollback() after initiating the transaction.

Cancel context is the best approach for most application types. And would serve most of the cases. The context in general has a lot of use cases and I would recommend you read more about it.

Also, for an extra layer of safety, I would recommend keeping a Server side timeout for HTTP services.

You would be also thinking about the client-side timeout. Those are not reliable enough. For example, a client could keep the timeout for a long duration and with a high load, the server could keep the connection open for a long time. This could again lead to the same problem we are trying to solve.

This post is about a real problem we faced in our project.

I hope after reading this post you will be able to avoid the same mistake we did in our project 😅. This small mistake didn't cost us a million dollars. But, It did cost us a few thousand, and the Prometheus alerts saved us.

But, it could cost you a Million if you are not careful 💸.

The scenario is a simple database transaction. We all have used database transactions at least once in our programming life. If you don't know how transactions work you could read the docs here for Postgres.

Our service handles the management of some Subscriptions in the database. All the changes to the rows happen within a transaction. The service is in Golang and the flow is like this::

• Start the transaction
• Fetch the record by ID
• Confirm if the operation for the change is possible
• If Yes, then update the record
• Commit the transaction
• For any error, revert the transaction

To fetch a record and acquire a lock on the record so that other flows would wait to update, we use SELECT ... FOR UPDATE query to fetch the record from Postgres and lock it.

Note: We use sqlx lib for database access.

The repository code is like this::

func (r *fetcher) GetSubscription(tx *sqlx.Tx, id uuid.UUID) (*model.Subscription, error) {
    var subscription model.Subscription
    err := tx.Get(&subscription, `
        SELECT * FROM subscriptions
        WHERE id = $1
        FOR UPDATE
    `, id)
    if err != nil {
        return nil, err
    }
	
    return &subscription, nil
}

The service code is like this::

func (s *service) CancelSubscription(ctx context.Context, id uuid.UUID) (*model.Subscription, error) {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return nil, err
    }
	
    defer func() {
        if err != nil {
            tx.Rollback()
        }
    }()
    
    subscription, err := s.fetcher.GetSubscription(tx, id)
    if err != nil {
        return nil, err
    }

    if subscription.CancelledAt != nil {
        return subscription, nil
    }
	
    subscription.CancelledAt = time.Now()

    err = s.updater.UpdateSubscription(tx, subscription)
    if err != nil {
        return nil, err
    }

    err = tx.Commit()
    if err != nil {
        return nil, err
    }

    return subscription, nil
}

Problem

All client requests were timing out, and the DB connection spiked over 1.2k connections 😅

Not a single request was able to complete the operation. It was a stop-the-world event 🤣

Why?

The issue happens when we try to cancel a subscription that is already cancelled. If the subscription is already cancelled we return the subscription without doing any changes, but we are not releasing the lock on the record.

The reason is the defer function calls tx.Rollback() only when there is an error. This would cause the lock to be active until the transaction commits or roll back. But since we are not doing any of the two things, the lock is held until the transaction times out.

If the service is having high traffic, the transactions could not time out. This will cause the code to create new connections to the database and exhaust the connection pool.

Fix

There are three ways to fix this issue at the service layer. To keep the post short, I have explained the other ways in a different post.

1. Release the lock in every if condition.

// Error handling is omitted for brevity
if subscription.CancelledAt != nil {
    _ = tx.Rollback() // release the lock

    return subscription, nil
}

This is the simplest way to fix the issue. But this would need you to rollback in every if condition. And if you forget to rollback in any of the if conditions, you would end up with the same issue.

2. Rollback the transaction in the defer function for every case.

// Error handling is omitted for brevity
defer func() {
   _ = tx.Rollback()
}()

This would release the lock in every case. For committed transactions and as per the tracing, we see some milliseconds the code spends to roll back the transactions. In case of committed transactions the rollback does nothing. But this is a better way to fix the issue.

3. Commit the transaction in the defer function for every case.

defer func() {
  if err != nil {
    _ = tx.Rollback()
  }
  
  commitErr := tx.Commit()
  // handle commitErr
}

If there are no changes the transaction will commit without any change. If there is any error only then the rollback would happen and will release the lock.

But, this affects the readability of the code. Your commit is happening in the defer function, and that's an extra pointer you have to keep in mind while reading the code.

Service Layer - Return Error 👾

The problem with the service is that for validations, it is not returning any error. The other way to fix this issue is to return an error from the if condition. This would make the Rollback happen in the defer function.

This is a very clean way. A service should have only one responsibility. Either it completes the operation or it returns an error. It should return error in case of validation failure. This is a good practice and something we missed in our project. We fixed the issue later by returning an error from the if condition.

This change also helps the handler to decide what HTTP status code to return. Which is really helpful as we can return 400 Bad Request for validation errors.

This is how the code would look like after refactoring

var ErrSubscriptionAlreadyCancelled = errors.New("subscription already cancelled")

func (s *service) CancelSubscription(ctx context.Context, id uuid.UUID) (*model.Subscription, error) {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return nil, err
    }
	
    defer func() {
        if err != nil {
            tx.Rollback()
        }
    }()
    
    subscription, err := s.fetcher.GetSubscription(tx, id)
    if err != nil {
        return nil, err
    }

    if subscription.CancelledAt != nil {
        return subscription, ErrSubscriptionAlreadyCancelled
    }
	
    subscription.CancelledAt = time.Now()

    err = s.updater.UpdateSubscription(tx, subscription)
    if err != nil {
        return nil, err
    }

    err = tx.Commit()
    if err != nil {
        return nil, err
    }

    return subscription, nil
}

Conclusion

My personal preference is the second option. But you could choose any of the three options based on your preference. I am choosing the second option because I am sure that whatever happens in the flow at the end, the transaction revert will happen. Yes, it would cost me a few milliseconds in case of a committed transaction, but compared to the loss to the business, it's worth.

Keep in mind that the service layer has only one responsibility. Either it completes the operation or it returns an error.

You may be thinking that a better approach is to have an abstraction that takes care of transactions. I agree, and would also have a separate post on that. But, I also think that we can avoid the complexity of abstraction if the code is stable and not changes too often.

The golang official post for transactions also supports the reasoning. Check the code snippet here. The office post also uses the second option.

// code snippet from the golang official post
func CreateOrder(ctx context.Context, albumID, quantity, custID int) (orderID int64, err error) {

  // Create a helper function for preparing failure results.
  fail := func (err error) (int64, error) {
    return fmt.Errorf("CreateOrder: %v", err)
  }

  // Get a Tx for making transaction requests.
  tx, err := db.BeginTx(ctx, nil)
  if err != nil {
    return fail(err)
  }
  // Defer a rollback in case anything fails.
  defer tx.Rollback()
  
  ... // other code is omitted for brevity
  
  // Commit the transaction.
  if err = tx.Commit(); err != nil {
    return fail(err)
  }
  
  // Return the order ID.
  return orderID, nil
}

This brings us to the end of this post. I hope you enjoyed reading this post. There is a part two to this post, where I will talk about having another layer of protection to prevent this issue.

If you have any questions or suggestions, feel free to reach out to me using the AMA.

I am working with Change Data Capture using Debezium recently and this docker-compose helps me to write the consumer application that will process the changes happening at the DB level.

In this post, I will be setting up the Debezium connector for Postgres with all the changes required to allow Debezium to capture the changes. In addition, I will also provide the details on how to make this configuration on the AWS RDS if you are going to use the Debezium Connector for your next project.

The following services would be needed to make the Debezium and Postgres work locally::

• Kafka Broker - localhost:9092
• Zookeeper - localhost:2181
• Postgres - localhost:5432
• Debezium Connector - localhost:8083
• Schema Registry - localhost:8081
• Debezium UI - localhost:8080
• Rest-Proxy - This is optional, but helps with checking cluster metadata, topics etc - localhost:8082

Starting Docker Compose

Below is the docker-compose file I use to start the stack::

version: '3.9'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    healthcheck:
      test: echo srvr | nc zookeeper 2181 || exit 1
      start_period: 10s
      retries: 20
      interval: 10s
  broker:
    image: confluentinc/cp-kafka:7.3.1
    hostname: broker
    container_name: broker
    depends_on:
      zookeeper:
        condition: service_healthy
    ports:
      - '29092:29092'
      - '9092:9092'
      - '9101:9101'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
    healthcheck:
      test: nc -z localhost 9092 || exit -1
      start_period: 15s
      interval: 5s
      timeout: 10s
      retries: 10
  debezium:
    image: debezium/connect:latest
    restart: always
    container_name: debezium
    hostname: debezium
    depends_on:
      postgres:
        condition: service_healthy
      broker:
        condition: service_healthy
    ports:
      - '8083:8083'
    environment:
      BOOTSTRAP_SERVERS: broker:29092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect_configs
      STATUS_STORAGE_TOPIC: connect_statuses
      OFFSET_STORAGE_TOPIC: connect_offsets
      KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      ENABLE_DEBEZIUM_SCRIPTING: 'true'
    healthcheck:
      test:
        [
          'CMD',
          'curl',
          '--silent',
          '--fail',
          '-X',
          'GET',
          'http://localhost:8083/connectors',
        ]
      start_period: 10s
      interval: 10s
      timeout: 5s
      retries: 5

  schema-registry:
    image: confluentinc/cp-schema-registry:7.3.1
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      broker:
        condition: service_healthy
    ports:
      - '8081:8081'
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: broker:29092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

    healthcheck:
      start_period: 10s
      interval: 10s
      retries: 20
      test: curl --user superUser:superUser --fail --silent --insecure http://localhost:8081/subjects --output /dev/null || exit 1

  rest-proxy:
    image: confluentinc/cp-kafka-rest:7.3.1
    depends_on:
      broker:
        condition: service_healthy
    ports:
      - '8082:8082'
    hostname: rest-proxy
    container_name: rest-proxy
    environment:
      KAFKA_REST_HOST_NAME: rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092'
      KAFKA_REST_LISTENERS: 'http://0.0.0.0:8082'

  debezium-ui:
    image: debezium/debezium-ui:latest
    restart: always
    container_name: debezium-ui
    hostname: debezium-ui
    depends_on:
      debezium:
        condition: service_healthy
    ports:
      - '8080:8080'
    environment:
      KAFKA_CONNECT_URIS: http://debezium:8083

  postgres:
    image: postgres:latest
    restart: always
    container_name: postgres
    hostname: postgres
    ports:
      - '5432:5432'
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: movies_db
    command: ['postgres', '-c', 'wal_level=logical']
    healthcheck:
      test: ['CMD', 'psql', '-U', 'postgres', '-c', 'SELECT 1']
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - ./scripts:/docker-entrypoint-initdb.d

For Debezium to work with Postgres, Postgres needs to have the logical replication enabled and if you observe the line command: ["postgres", "-c", "wal_level=logical"] we are configuring the Postgres DB to start with wal_level as logical.

If we don't do this step, then debezium would not be able to capture the changes happening on Postgres. The default wal_level is replica.

Now we have our docker-compose file ready let's start all the service

 docker compose -f docker-compose.yml up -d
[+] Running 8/8
 ⠿ Network postgres_default   Created                                                                                                                                                                          0.1s
 ⠿ Container postgres         Healthy                                                                                                                                                                         12.8s
 ⠿ Container zookeeper        Healthy                                                                                                                                                                         11.8s
 ⠿ Container broker           Healthy                                                                                                                                                                         22.6s
 ⠿ Container debezium         Healthy                                                                                                                                                                         44.2s
 ⠿ Container rest-proxy       Started                                                                                                                                                                         23.1s
 ⠿ Container schema-registry  Started                                                                                                                                                                         23.1s
 ⠿ Container debezium-ui      Started                                                                                                                                                                         44.6s

As we could see all the containers have started without any errors. If you are running this command for the first time it would take some time to download all the docker images but the later executions will be faster

Configure Postgres Connector

At this point, our Debezium connector is running but it doesn't have any task to read changes happening on the Postgres DB. We need to register the Postgres connector using HTTP API so that debezium could read the transaction logs from the server.

It's easy to register the connector, there is a sample connector config already present in the repo. Execute the below curl and we would see the connector registered

curl -X POST --location "http://localhost:8083/connectors" -H "Content-Type: application/json" -H "Accept: application/json" -d @connector.json

{
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.dbname": "movies_db",
    "database.history.kafka.bootstrap.servers": "kafka:9092",
    "database.history.kafka.topic": "schema-changes.movies",
    "database.hostname": "postgres",
    "database.password": "postgres",
    "database.port": "5432",
    "database.server.name": "postgres",
    "database.user": "postgres",
    "name": "movies-db-connector",
    "plugin.name": "pgoutput",
    "table.include.list": "public.movies",
    "tasks.max": "1",
    "topic.creation.default.cleanup.policy": "delete",
    "topic.creation.default.partitions": "1",
    "topic.creation.default.replication.factor": "1",
    "topic.creation.default.retention.ms": "604800000",
    "topic.creation.enable": "true",
    "topic.prefix": "postgres"
  },
  "name": "movies-db-connector",
  "tasks": [],
  "type": "source"
}

List Kafka Topics

If there was no issue running the above steps we could confirm that our connector is working fine by checking if the topic is created for movies table by the connector.

❯ kafka-topics --bootstrap-server localhost:9092 --list
__consumer_offsets
_schemas
connect_configs
connect_offsets
connect_statuses
postgres.public.movies

As you could see from the above command we have postgres.public.movies a topic created by debezium. The topic name pattern is of a type topic_prefix_config.schema.table so the prefix is what we configured and the other part is fetched by the debezium from the database schema

Reading the data

Now we know that the topic is created next we would check the data available on the topic.

There would be data present in the topic because when the connector starts it takes an initial snapshot of the database table. This is a default config named snapshot.mode which we didn't configure but is set to initial which means that the connector will do a snapshot on the initial run when it doesn't find the last known offset from the transaction log available for the database server, to understand more about this configuration and others read more here.

kafka-console-consumer --bootstrap-server localhost:9092 --topic postgres.public.movies --from-beginning
{
	"schema": {
		"type": "struct",
		"fields": [
			{
				"type": "struct",
				"fields": [
					{
						"type": "int64",
						"optional": false,
						"field": "id"
					},
					{
						"type": "string",
						"optional": false,
						"field": "title"
					},
					{
						"type": "int32",
						"optional": false,
						"field": "year"
					},
					{
						"type": "string",
						"optional": false,
						"field": "director"
					},
					{
						"type": "struct",
						"fields": [
							{
								"type": "int32",
								"optional": false,
								"field": "scale"
							},
							{
								"type": "bytes",
								"optional": false,
								"field": "value"
							}
						],
						"optional": false,
						"name": "io.debezium.data.VariableScaleDecimal",
						"version": 1,
						"doc": "Variable scaled decimal",
						"field": "rating"
					}
				],
				"optional": true,
				"name": "postgres.public.movies.Value",
				"field": "before"
			},
			{
				"type": "struct",
				"fields": [
					{
						"type": "int64",
						"optional": false,
						"field": "id"
					},
					{
						"type": "string",
						"optional": false,
						"field": "title"
					},
					{
						"type": "int32",
						"optional": false,
						"field": "year"
					},
					{
						"type": "string",
						"optional": false,
						"field": "director"
					},
					{
						"type": "struct",
						"fields": [
							{
								"type": "int32",
								"optional": false,
								"field": "scale"
							},
							{
								"type": "bytes",
								"optional": false,
								"field": "value"
							}
						],
						"optional": false,
						"name": "io.debezium.data.VariableScaleDecimal",
						"version": 1,
						"doc": "Variable scaled decimal",
						"field": "rating"
					}
				],
				"optional": true,
				"name": "postgres.public.movies.Value",
				"field": "after"
			},
			{
				"type": "struct",
				"fields": [
					{
						"type": "string",
						"optional": false,
						"field": "version"
					},
					{
						"type": "string",
						"optional": false,
						"field": "connector"
					},
					{
						"type": "string",
						"optional": false,
						"field": "name"
					},
					{
						"type": "int64",
						"optional": false,
						"field": "ts_ms"
					},
					{
						"type": "string",
						"optional": true,
						"name": "io.debezium.data.Enum",
						"version": 1,
						"parameters": {
							"allowed": "true,last,false,incremental"
						},
						"default": "false",
						"field": "snapshot"
					},
					{
						"type": "string",
						"optional": false,
						"field": "db"
					},
					{
						"type": "string",
						"optional": true,
						"field": "sequence"
					},
					{
						"type": "string",
						"optional": false,
						"field": "schema"
					},
					{
						"type": "string",
						"optional": false,
						"field": "table"
					},
					{
						"type": "int64",
						"optional": true,
						"field": "txId"
					},
					{
						"type": "int64",
						"optional": true,
						"field": "lsn"
					},
					{
						"type": "int64",
						"optional": true,
						"field": "xmin"
					}
				],
				"optional": false,
				"name": "io.debezium.connector.postgresql.Source",
				"field": "source"
			},
			{
				"type": "string",
				"optional": false,
				"field": "op"
			},
			{
				"type": "int64",
				"optional": true,
				"field": "ts_ms"
			},
			{
				"type": "struct",
				"fields": [
					{
						"type": "string",
						"optional": false,
						"field": "id"
					},
					{
						"type": "int64",
						"optional": false,
						"field": "total_order"
					},
					{
						"type": "int64",
						"optional": false,
						"field": "data_collection_order"
					}
				],
				"optional": true,
				"name": "event.block",
				"version": 1,
				"field": "transaction"
			}
		],
		"optional": false,
		"name": "postgres.public.movies.Envelope",
		"version": 1
	},
	"payload": {
		"before": null,
		"after": {
			"id": 1,
			"title": "The Shawshank Redemption",
			"year": 1994,
			"director": "Frank Darabont",
			"rating": {
				"scale": 1,
				"value": "XQ=="
			}
		},
		"source": {
			"version": "2.1.1.Final",
			"connector": "postgresql",
			"name": "postgres",
			"ts_ms": 1675103983563,
			"snapshot": "first",
			"db": "movies_db",
			"sequence": "[null,\"26612016\"]",
			"schema": "public",
			"table": "movies",
			"txId": 744,
			"lsn": 26612016,
			"xmin": null
		},
		"op": "r",
		"ts_ms": 1675103983739,
		"transaction": null
	}
}
----- more data ommitted for readability purpose -----

Cleanup

By running the below command all the services will be shut down and the resources would be released

docker compose -f docker-compose.yml -v down

Here we specify -v option to also remove any volumes created by the docker-compose, this will also free the disk space allocated to the containers

Debezium UI

Debezium also provides a control dashboard. You could use it to add a new connector instead of using the REST API. The dashboard also provides the functionality to Pause or Restart the debezium task and also to delete the connector

Go to http://localhost:8080 and you will be able to access the dashboard as shown below

The Dashboard also allows you to add new connectors for other database providers

AWS RDS

Setting up docker-compose is easy because everything is local and we could use the configuration described by the Postgres documentation.

On AWS, it's a different story, on AWS we have to configure rds.logical_replication property in the parameter groups of your Postgres instance. The default value for this property is 0 which means the logical replication is disabled by default, however, setting it to 1 will enable the logical replication. An important thing that you should know is that when you change this property on AWS you would need to restart your AWS RDS Postgres instance. If your application is in production that means you would need some downtime to enable the replication. You could read more about this and the steps to do it in AWS RDS here

This brings us to the end of this small and quick setup of Postgres and Debezium using Docker Compose. Hope, next time you are building any CDC-related consumer this docker-compose setup will help you to run the infrastructure locally without any hassle.

If you like the blog, please don't forget to like the post, follow me on Medium also don't forget to subscribe on Medium.

Code

• Github

I was working with sending some telemetry data from my Golang web application to Prometheus and then create a Grafana dashboard out of it. But to test the flow I needed a local setup of Prometheus + Grafana so that I can check if the metrics are right and that I am building the right PromQL query to create the dashboard.

I am maintaining a Github repo for all the docker-compose setup I require for my local testing and now Prometheus + Grafana is a new addition to it. If you also need a similar setup and save your setup time in future and focus more on building things refer to this post or my Github Repository.

Clone the repo:: https://github.com/ninadingole/docker-compose-stacks

Then go to prometheus-grafana folder and run docker-compose up -d.

This will start Prometheus on http://localhost:9090 and Grafana on http://localhost:3000.

There is also a prometheus.yml configuration file which you can use to add the local apps that you want to scrape, a note:: if your application is running inside a docker then use host.docker.internal as your hostname with the port to scrape the target.

Once you have added the configs to connect Grafana to Prometheus like above image, you are ready with an end-to-end setup on your local.

I hope you will find this docker-compose configuration pretty useful and saves your time. Please subscribe to the blog to get more articles delivered right to your email inbox.

Thanks and Happy Coding!

Reference

• repo

I liked the definition of "Vulnerability" from the Github's documentation:

A vulnerability is a problem in a project's code that could be exploited to damage the confidentiality, integrity, or availability of the project or other projects that use its code. Vulnerabilities vary in type, severity, and method of attack.

When your code depends on a package that has a security vulnerability, this vulnerable dependency can cause a range of problems for your project or the people who use it.

My observation as a developer is when we are developing an internal system or application for any client and if it's not a product we will rarely upgrade the dependencies because most of the time we upgrade those dependencies only to overcome any issue or bugfix.

Other than internal application or services, If we are working on any products we will keep on upgrading the third-party dependencies frequently but not all of them, only those that we are working on most and your application majorly depend on them, or in case there was a bug with the current version which got fixed in a newer version or some additional functionality is added to the library that will benefit the application.

However, what happens when the dependency is at risk, how would we get to know about such information? Will you keep a dedicated person to keep track of it 😝? or you will do it only when you get impacted by some serious exploit to your application? or you might have a static code analysis system in place which checks for any vulnerability in the code. However, what about the dependencies even in that case?

We need some kind of automated way to deal with this issue, and this is the exact issue that Github tried to solve and came up with this cool solution "Dependabot".

Dependabot will alert you whenever there is a vulnerable dependency found in your Github repository. Dependabot will refer the National Vulnerability Database, a combination of machine learning and human review to detect vulnerabilities in public commits on GitHub, Security advisories reported on GitHub, and FriendsOfPHP, and will use information from all these multiple sources to alert you.

As shown below, Dependabot works with a wide range of package managers & support languages that are widely used in the industry. Dependabot is still in beta, however, with future releases, you will see more of the tools support added to Dependabot.

In this post, will see how to enable Dependabot in Github public repo, it's simple and has easy to configure steps.

Let's Go:

1. Go to the security tab of the repository, and you will see the below screen, click on the Enable Dependabot alerts.

You will be shown the below screen which has two independent configurations, enable them based on your requirement.

1. Dependabot alerts:
   It will enable the dependabot to only show an alert on the main page when there's any vulnerability found in your repo.
2. Dependabot security update: ^[1]
   It will raise a PR for upgrading any vulnerable dependencies to a safer version of those dependencies.

Once you enable the above options, it's all done. It's that easy to have this amazing functionality enabled for your repository.

In the below screen you can see I got an alert for security risk in one of my repository in ExpressJS project.

On clicking See Dependabot alert, I found that the mongoose library that I am using is having a security issue with version 4.7.5. The details of the alert show me that the remediation is to upgrade the library to version >=5.7.5 and as my project is a NodeJS project it shows me how to do that in my package.json file.

Below is the remediation you can see the details of the security vulnerability, in this case, its allowing attackers to bypass access control 😝

How to fix this issue now? The details already show how to do it manually, but while enabling dependabot there's the second option that will automatically raise a PR for the version upgrade.

For this repository, I didn't enable the second option ^[1]. However, there's also another way. Click on Create Dependabot security update button in the above page, and it will start creating a PR for you, this is also a manual approach, and as you can see in the below image you will see a notification generated by the dependabot.

If you go to the Pull Request section of the repository you would see a Pull-Request being raised by the dependabot with tag dependencies I have seen similar PR's in a lot of open-source repositories.

And the same PR reference/link you will see in the security alert details now.

Further drilling down into the details of PR you could see more information about the PR, the PR upgrade the mongoose dependency to version 5.9.26 which is what was given in the remediation to upgrade to anything >=5.7.5. It also shows that the severity of this issue is moderate. Dependabot categories issue in four such categories you can read more about them here.

• Low
• Moderate
• High
• Critical

I didn't have any CI setup for this repository. However, if you have a test suite in the repository then this will also run all your test cases to see if this version works or not.

You can also see a Changelog for the recent changes to the library.

And there are some dependabot action commands as shown below that you can use in your comments to trigger actions like if you want to merge this PR instead of clicking on the merge button you can just comment in the PR @dependabot merge and this will merge the PR. You can squash, close, ignore and do all the below-given actions using comments.

This is what the diff of the PR shows, the bot just updated a single line in my package.json file.

Once I merged the PR, this is the file content of my package.json 😀.

Wow, this will be the easiest thing to configure for the next project and comes with so many benefits. I hope this little blog post helps you to get rid of all those vulnerable dependencies in your project and keeps your business and application safe from some level of security threats. Make sure security is always a priority from the start of your project.

For any issues while following the steps or with the content do let me know down in the comments or email me using my contact page.

Don't forget to share the post with your friends & colleagues. Happy Coding!

There are many code scanning tools available some are proprietary licensed products, and some are a free tool, and it requires some efforts to set up with your existing CI/CD setup. The one which I have seen is Veracode, which is a proprietary product and provides information about any security risk or code error by scanning the entire code base, even the dependencies I believe, and then there is SonarQube widely used by many organization and many of you might be aware of, it's a static code analyser and provides information regarding any security risk or code error.

Github, which is the most popular platform for open source development came up with a new service that allows code scanning of the repository for security vulnerabilities and any coding errors. You can use code scanning to find, triage, and prioritise fixes for existing problems in your code. Code scanning also prevents developers from introducing new problems. You can schedule scans for specific days and times, or trigger scans when a specific event occurs in the repository, such as a push.

If code scanning finds a potential vulnerability or error in your code, GitHub displays an alert in the repository. After you fix the code that triggered the alert, GitHub closes the alert. Code Scanning is a new service and is still in beta and lots of scanning rules are work in progress like passwords leaked in java properties file is still not available, however as the service becomes more stable will have all such cases available in CodeQL rules database.

By default, code scanning uses CodeQL, a semantic code analysis engine. CodeQL treats code as data, allowing you to find potential vulnerabilities in your code with greater confidence than traditional static analyzers. You can use CodeQL to find all variants of vulnerability, and remove all the variants from your code.

QL is the query language that powers CodeQL. QL is an object-oriented logic programming language. GitHub, language experts, and security researchers create the queries used for code scanning, and the queries are open source. The community maintains and updates the queries to improve analysis and reduce false positives. For more information, see CodeQL on the GitHub Security Lab website.

In this short tutorial, will see how to configure code scanning for a Java-based project. Code scanning supports both compiled and interpreted languages and can find vulnerabilities and errors in code that's written in the supported languages.

• C/C++
• C#
• Go
• Java
• Javascript/Typescript
• Python

Github Code Scanning is still in beta and access is limited to users on an invitation basis, follow the below steps when you have access to the service

Code Scannning vs SonarQube

I am seeing a lot of people looking for this comparison. Code scanning is a new tool which currently using CodeQL under the hood to scanning code for any security vulnerability. It's new but a lot of security researchers are sharing the findings in CodeQL repo, so you will have your code check against those latest findings. Code Scanning deals more with the Security and coding errors not the static code check analysis what SonarQube provides for every supported language.

SonarQube is used by many organization it has comprehensive set of checks and standards that acts as a quality gate in application development.

SonarQube is very mature product as compared to code scanning, code scanning is new but you never know what the future holds for it.

Let's start 🏃🏻‍♂️

The first thing you need to do is navigate to the security tab of the repository

Then you will see a below screen, and the last option you will see "Code Scanning alerts", click on the Setup code scanning button

Code scanning uses Github Actions, You will see the above screen for CodeQL analysis setup, along with this you will also see many different options from marketplace to setup like "Anchore Container Scan", "OSSAR" etc. For this post will be going with the default action workflow, click on the setup this workflow button for CodeQL analysis, and you will be prompted with the below screen to configure the workflow YAML.

When you see the configuration click on the Start commit and create a commit with the basic configuration will do the modification afterwards.

Take a pull in your repository so that you have the workflow YAML available to modify. Open the file in .github/workflow directory and modify the configuration as per your requirement, below are the configuration that I have done for this post.

Few things I modified from the given workflow like I have disabled the cron scheduled option, this makes the security scan to run on given cron expression, the option push enables the scan to run on every commit push to the remote, and the similar pull_request will happen for every pull request, however in an attempt when I ran a full pack of security rule that took 19min.

In such case, I would recommend setting up the Security scanning as a cron once in a day mostly a nightly build so that your team will get to see the result of the security scan as the first thing at the start of the day and can prioritise fixing the bugs first. Another change I have done is adding configuration to Initialize CodeQL step as below in file .github/codeql-config.yml

The CodeQL actions codeql-action/init@v1 can find the programming language by its auto-detection feature, it is something that is embedded in the action by Github developers, however, I have explicitly added java so that the step doesn't have to do extra work and saves time, and at the same time I like to make things readable instead of the abstraction magic, this helps anyone new joining your team figure out how things are configured.

Then there are queries options, this allows to specify additional scan rules, two are inbuilt in the CodeQL tool which I have specified security-extended and security-and-quality however there are also many additional security rules on the Github's CodeQL repo, you can find it here when I first added the entire repo which was more than 800 rules, it took more time and many of them were not of any use for my experiment purpose so I selected a subset of those.

There is a syntactic way to provide the configuration as shown below, to make the custom queries to run you need to disable the default queries by setting disable-default-queries: true.

queries:
  - name: Use an in-repository QL pack (run queries in the my-queries directory)
    uses: ./my-queries
  - name: Use an external JavaScript QL pack (run queries from an external repo)
    uses: octo-org/javascript-qlpack@master
  - name: Use an external query (run a single query from an external QL pack)
    uses: octo-org/python-qlpack/show_ifs.ql@master
  - name: Use a query suite file (run queries from a query suite in this repo)
    uses: ./codeql-qlpacks/complex-python-qlpack/rootAndBar.qls

In the configurations, I have provided github/codeql/java/ql/src/codeql-suites/java-code-scanning.qls@v1.24.0 this is the path to the actual GitHub link and I have tagged the rules for version 1.24.0 you can also point to the master as shown in the above configuration guide.

After you have done the configuration create a new commit, and push. You will see a new scan action triggered which you can view in the Actions section of the Github repo page.

Clicking on the individual action run will show you additional information about the scan process, in the log which is provided for the step navigate to the Perform CodeQL Analysis section and you can see what all security rules that were checked against the code.

Once the workflow is completed without any error, to check any security vulnerability or error in the code navigate to the Security Tab, the badge also shows the number of security risk that is present in the code, I liked this as the information is right there available on the main page of your repo and this way team can prioritise bugs to be fixed.

If you click on it and move to the code scanning alerts you can see all the vulnerabilities that are available in the code listed

The listing shows the message of the issue found, the file in which the issue was detected with the line number (very cool) and the branch in which the issue was found. Further clicking on any issue you can drill down into the actual code, the UI will highlight the line of code which has the issue, whether its a warning or an actual error and more details like the commit that caused the issue, the rule that failed and tags the issues based on the rules category. In the below image you can see the error are tagged as CWE-798 and security

You can see the details of two errors which are a security risk to my code are listed below(deliberately added for demo purpose 😛). One is because of the password leaked into the source code and the other is directly redirecting to the user-provided URL without first validating the URL.

If you click on the small show more button you can see a good help about the issue and how to avoid it, there are also links to some reference documentation that explains the issue in depth.

Once you have fixed the issue and the code is pushed the issue will be moved to close. However, at times there will be cases of false-positive in such cases there is an option provided at the top right of the issue, to manually close the issue, and has three options showing in below image, select one that suits your use case and close the issues which are not valid.

EDIT:: I was reading some blogs on Github and found this exciting stats::

Since introducing the beta in May, we’ve seen tremendous adoption within the community::

• We’ve scanned over 12,000 repositories 1.4 million times, and found more than 20,000 security issues including remote code execution (RCE), SQL injection, and cross site scripting (XSS) vulnerabilities.
• Developers and maintainers fixed 72% of reported security errors identified in their pull requests before merging in the last 30 days. We’re proud to see this impact, given industry data shows that less than 30% of all flaws are fixed one month after discovery.
• We’ve had 132 community contributions to CodeQL’s open sourced query set.
• We’ve partnered with more than a dozen open source and commercial security vendors to allow developers to run CodeQL and industry leading solutions for SAST, container scanning, and infrastructure as code validation side-by-side in GitHub’s native code scanning experience.

I hope this post helps you, if you don't see the option to enable the security scan on your repository, don't worry the tool is still in beta and access is given on invitation basis by Github. Get an invite for the service if you want to try out and It will be accessible to everyone soon.

The post gives a basic idea of how the scanning will work with repository there are many open-source tools, and their workflows available in the Github marketplace that you can try, however, this post will help you to get started with basic.

For more advance on how different tools can be integrated, what if I work at an organization who has its own set of security rules, how to implement that in CodeQL? I will write about those advance things in another post, and as I explore more of the code scanning tool I will keep on sharing.

For any issue while following the steps please do let me know in the comments below I will try to correct. For anything else please comment and I will try to reply as soon as possible, and please don't forget to share the post.

Happy Coding!

References::

• Github CodeQL
• Sample Code
• Github CodeQL Additional Queries
• CodeQL for GoLang
• https:😕/github.blog/2020-09-30-code-scanning-is-now-available/

Updated on 22/05/2021 :: Add comparison of code scanning vs sonarqube

Recently, I came across the same situation where I was looking for a way to change the response of the service that I was consuming and instead of mocking the entire service using a tool like Postman or wire mock, I wanted to hit the actual service and then change a few fields to see if my error handling works as per the expectations.

Normally, we do write an integration test where we mock the service response using wiremock and then write the test for such edge cases. However, sometimes when we do manual testing and we want a way to mock the service response we need some mechanism to intercept the service response and modify the response. This is where Burp suites will help, Burp suites acts like a proxy that will call the actual service on behalf of you and then provides you with tools that can help to easily modify the response and send it back to your application.

For this post, I will be consuming themoviesdb.org API's to see details of "Fight Club" movie, and then I will write an application in golang which will connect to Burp Suite, there we modify the response using Burpsuite UI and print the modified response in the console which is received by our application.

package main
import (
	"crypto/tls"
	"encoding/json"
	"fmt"
	"net/http"
	"net/url"
	"os"
)

func main() {

	proxyURL, e := url.Parse(os.Getenv("PROXY"))
	if e != nil {
		panic(e)
	}

	client := &http.Client{
		//Timeout: 20 * time.Second,
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxyURL),
			TLSClientConfig: &tls.Config{InsecureSkipVerify:true},
		},
	}

	get, err := client.Get(fmt.Sprintf("https://api.themoviedb.org/3/movie/550?api_key=%s", os.Getenv("KEY")))
	if err != nil {
		panic(err)
	}

	var data map[string]interface{}
	err = json.NewDecoder(get.Body).Decode(&data)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Response: %+v",data)
}

As you can see in the above code, we create an HTTP client with custom Transport set to proxy URL which is configured in the environment setting of the application, for burp suite the URL is http://localhost:8080. The TLSClient config is configured to skip ssl certificate verification using InsecureSkipVerify:true. Then we do a normal GET request to the api which will get the details of the movie id 550.

Let's now start the burp suite and configure the intercept rule to only run when the domain matches themoviedb.org::

Then start the interceptor::

Now, run the go application and you will see that the request is captured by the burp suite as follows::

After you see the above screen click on the Action button and click Do intercept with Response to this request::

After clicking the Forward button, burp suite will call the actual service and will open the editor where you can edit the response, this movie is "Flight Club" movie and I have changed the movie name to Fight Pub and rating from 8.4 to 9.4. Then click the forward button again to forward the response to the golang app.

Once the application receives this response we just print it to the console and you could see that the name of the movie is now Fight Pub and the rating is 9.4 which is what we modified in the response.

Response: map[adult:false
backdrop_path:/mMZRKb3NVo5ZeSPEIaNW9buLWQ0.jpg belongs_to_collection:<nil>
budget:6.3e+07
genres:[map[id:18 name:Drama]]
homepage:http://www.foxmovies.com/movies/fight-club id:550 imdb_id:tt0137523
original_language:en
original_title:Fight Pub
overview:A ticking-time-bomb insomniac and a slippery soap salesman channel primal male aggression into a shocking new form of therapy. Their concept catches on, with underground "fight clubs" forming in every town, until an eccentric gets in the way and ignites an out-of-control spiral toward oblivion.
popularity:58.844
poster_path:/adw6Lq9FiC9zjYEpOqfq03ituwp.jpg
production_companies:[map[id:508 logo_path:/7PzJdsLGlR7oW4J0J5Xcd0pHGRg.png name:Regency Enterprises origin_country:US] map[id:711 logo_path:/tEiIH5QesdheJmDAqQwvtN60727.png name:Fox 2000 Pictures origin_country:US] map[id:20555 logo_path:/hD8yEGUBlHOcfHYbujp71vD8gZp.png name:Taurus Film origin_country:DE] map[id:54051 logo_path:<nil> name:Atman Entertainment origin_country:] map[id:54052 logo_path:<nil> name:Knickerbocker Films origin_country:US] map[id:25 logo_path:/qZCc1lty5FzX30aOCVRBLzaVmcp.png name:20th Century Fox origin_country:US] map[id:4700 logo_path:/A32wmjrs9Psf4zw0uaixF0GXfxq.png name:The Linson Company origin_country:]]
production_countries:[map[iso_3166_1:DE name:Germany] map[iso_3166_1:US name:United States of America]]
release_date:1999-10-15
revenue:1.00853753e+08
runtime:139
spoken_languages:[map[iso_639_1:en name:English]]
status:Released
tagline:Mischief. Mayhem. Soap.
title:Fight Pub
video:false
vote_average:9.4
vote_count:17753]

So, thats how you can connect your application to the burp suite and can intercept the response of the services to manually test the application for different cases.

I hope this post helps you to understand how to configure an HTTP client in your code to connect to Burp suite, I have used golang for this post because I am currently working on it. However, if you are working on any other programming language the HTTP Api's will have some way to configure the proxy as like I have done in my golang code.

Happy Coding!

In short and crisp, tests gets divided into three categories:: unit test, integration/service test, and e2e/UI tests. The unit test validates the functionality of each class and written on per class basis, integration test checks for the functionality of module with external systems like database, web service, and UI/e2e test the entire business flow or UI flow.

As you can see from the test pyramid, we should have more integration test than e2e test and more unit test than the integration test. The reason for having less integration and e2e tests is "time"; Integration test and e2e test interacts typically with other systems which are external or internal like in-memory DB or clusters. These tests take time to execute and are often slower.

And not to mention we have our CD/CI pipelines which run these tests and generate our code artifact. We also want our CI to provide us faster feedback by running these test individually, so we don't have to wait longer if there is an issue at the unit level.

Currently, I am working on a Big Data project which includes frameworks like Spark, HBase, Kafka, HDFS, etc. Some of our unit tests are independent of any spark code, and some unit test does require spark context, but they do not interact with external systems like HDFS and HBase. Moreover, integration test, runs an in-memory HBase cluster, in-memory mini dfs cluster and spark of course :😛. Now, these integration test shares the single running clusters amongst them and executes sequentially.

We have SBT as the build tool and, we have disabled the parallel execution of tests which makes the entire test suite including the unit test to run slower. I am struggling with that sometimes have to wait for 10-15mins to get the feedback about my code changes, running unit tests also runs integration test which takes time for cluster initialization of HBase and HDFS, and I was seriously looking for some alternative to this issue. Seldom I do comment out my integration suite code only to run the unit test which is again a wrong way. We need some mechanism to organize our tests such that we can execute them separately as a suite of unit, integration and e2e tests and, we only have parallel execution disabled for integration and e2e tests. It would provide developers with quick feedback by just running the unit tests.

In this post, I am going to share my findings on how to organize the tests and configure sbt (v1.2.8) such that it enables us to run these different scala test independently without any code hacks or commenting your test cases which is a more of an ugly way.

Integration Test

Integration Test setting comes default in SBT. It's just a matter of configuring the modules with these settings in the build.sbt

lazy val module1 = (project in file("module1"))
  .configs(IntegrationTest)
.settings(
Defaults.itSettings,
  libraryDependencies += scalatest % "it,test"
)

Adding "it, test" at the end of dependencies enables that lib to be available to both standard unit test & integration test packages. After this, you have a new location to write an integration test.

src/it/scala
src/it/java
src/it/resources

I have created an integration test per module and placed in the above location now to execute the integration test use IntegrationTest / testOnly or it:test in SBT console, and the output as::

[info] Compiling 1 Scala source to /Users/neenad/codes/tutorials/scala/sbt-learnings/module1/target/scala-2.11/it-classes ...
[info] Run completed in 9 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
this is a integration test
[info] Module2IntegrationTest1:
[info] - integration test
[info] Run completed in 194 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Done compiling.
module1 integration test
[info] Module1IntegrationTest:
[info] - integration test
[info] Run completed in 134 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 2 s, completed 17 Mar, 2019 5:52:42 PM

End-To-End Tests

Like integration test end-to-end configuration is not provided by SBT. We would need to create them and apply to the module. I have created an E2E object in the project folder and, the contents are as below::

import sbt.{Configuration, Defaults, Test, inConfig}
import sbt._
import sbt.Keys._

object E2E {
  final val E2ETest = Configuration.of("EndToEndTest", "e2e") extend (Test)
  final val e2eSettings =
    inConfig(E2ETest)(e2eConfig)
  lazy val e2eConfig =
      Defaults.configSettings ++ Defaults.testTasks ++ Seq(
    scalaSource in E2ETest := baseDirectory.value / "src" / "e2e" / "scala",
    javaSource in E2ETest := baseDirectory.value / "src" / "e2e" / "java",
    resourceDirectory in E2ETest := baseDirectory.value / "src" / "e2e" / "resources",
  )
}

Now apply these settings to your modules and add e2e to the library dependencies as we did with the integration test

lazy val module1 = (project in file("module1"))
  .configs(IntegrationTest)
  .configs(E2ETest)
.settings(
  E2E.e2eSettings,
  Defaults.itSettings,
  libraryDependencies += scalatest % "it,test,e2e",
    name := "module1",
  version := "0.1"
)

lazy val module2 = (project in file("module2"))
  .configs(IntegrationTest)
  .configs(E2ETest)
  .settings(
    Defaults.itSettings,
    libraryDependencies += scalatest % "it,test,e2e",
      E2E.e2eSettings,
    name := "module2",
    version := "0.1"
  )

lazy val root = (project in file("."))
  .configs(IntegrationTest)
  .configs(E2ETest)
  .aggregate(module1, module2)
  .enablePlugins(Common)
  .settings(
    E2E.e2eSettings,
    Defaults.itSettings,
    libraryDependencies += scalatest % "it,test,e2e"
  )

we can now create our e2e tests inside the following folders

src/e2e/scala
src/e2e/java
src/e2e/resources

To run the e2e test use EndToEndTest / test or e2e:test in sbt. This command runs only the integration test as shown below::

[info] Run completed in 10 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
this is e2e test
[info] Module1E2ETest:
[info] - module1 e2e test
[info] Run completed in 172 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
module2 e2e test
[info] Module2E2ETest:
[info] - module2 e2e test
[info] Run completed in 250 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 1 s, completed 17 Mar, 2019 5:53:46 PM

Unit Test?

Now, what remains are the unit tests and configuring the unit tests is relatively straightforward. You don't have to do anything at all. 😄 Any test under package src/test/scala gets considered as a unit test and to execute it you have to use your standard test task in sbt, this runs all the unit test in your module::

[info] Compiling 1 Scala source to /Users/neenad/codes/tutorials/scala/sbt-learnings/module1/target/scala-2.11/test-classes ...
[info] Run completed in 9 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
module2 sample unit test
[info] Module2UnitTest:
[info] - sample test
[info] Run completed in 202 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Done compiling.
module1 sample unit test
Hello World
[info] Module1UnitTest:
[info] - sample test
[info] Run completed in 142 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 2 s, completed 17 Mar, 2019 6:06:45 PM

Disabling Parallel Execution::

To disable parallel execution of the e2e test in our code add below line in our E2E settings

E2ETest / parallelExecution := false

##Code::

Resources

• SBT Reference

I hope this post helps you to organize your tests such that you get faster feedback by running tests independently and don't have to survive the pain I went through in your project. If you have any doubt, please comment and don't forget to share it.

While I was working on this new platform development one of my colleagues told me about Oracle having vagrant images which could provide Oracle 12c to developers working on mac. Then I checked and found that Oracle has created a GitHub repo with the vagrant files both for Linux and Oracle 11 and 12c. After that, I basically took this vagrant files as a base for my work and installed golden gate setup along with confluent package on the Linux box. But you know it was a very tedious task to manually do the installation and all. And if I do a big blog with all the steps you guys would be like?

So, I enhanced the scripts to do the hard work and install Golden Gate + Confluent + enable some configuration for the Oracle database during vagrant box provisioning. And you know what one command does all the setup and configuration for you. Follow the exact given steps in the post and you will be gifted with the magical power of Oracle database, golden gate and confluent. 😂

1. Clone Vagrant Box Images

Clone my oracle vagrant fork from here. Let's wish my pull request get accepted by the oracle contributors team and this work becomes officially a part of Oracle GitHub repository ( finger crossed 🤞🏼).

2. Download Oracle Database Zip

Download the Oracle database 12c zip file from oracle tech network here or here. Use the Oracle Database 12c Release 2 - Linux x86-64 version and download the single file zip which is around 3.2GB.

3. Download Oracle GG and Oracle GG BD

Now, we need the Oracle Golden Gate and Golden Gate for Big Data zip files. These files can be downloaded from here. Download Oracle GoldenGate for Big Data 12.3.1.1.1 on Linux x86-64 & Oracle GoldenGate 12.3.0.1.4 for Oracle on Linux x86-64.

Keep all the downloaded files in the same vagrant git checkout directory.

4. Start Vagrant

At this point, all the pre-requisites are done. Now run vagrant up and this will start installing Oracle database 12c, oracle golden gate, oracle golden gate for big data and confluent oss package on the Linux box.

Wait till everything finishes. At the very end, the setup will print the password for Oracle database make sure you copy that as this will help you to progress with the next steps. Now, you have your CDC dev environment ready on Mac or on any machine having vagrant. 😎

If you are new to Oracle golden gate and want to know about configuring the golden gate for Oracle database follow the below steps in the post.

Configure Golden Gate On Vagrant Box

There are two things we need to configure, the first is oracle golden gate and then oracle golden gate for big data. Let's begin by first configuring the golden gate for Oracle database. Please follow the exact step to get it running. You can find more information about the following command on oracle golden gate documentation.

1. Load Sample Schema To Oracle Database.

We need a schema present in the installed oracle database so that we can configure the golden gate to listen to the changes happening on this schema. Well, Oracle setup comes with HR schema SQL files present in the installation. We will load the same into our database.

use vagrant ssh command to ssh into the virtual machine

Log in to the database using SQL shell as a sysdba::

sqlplus / as sysdba

Alter the session to allow Oracle scripts to execute::

alter session set "_ORACLE_SCRIPT"=true;

Alter the session to use the PDB database (inside SQL shell)::

alter session set container=ORCLPDB1;

Load the schema and for all the input asked by the below command provide proper responses::

@?/demo/schema/human_resources/hr_main.sql

Execute the below commands as an oracle user, switch the vagrant user to Oracle using

sudo su - oracle

2. Configure Oracle Golden Gate

1. Go to Oracle golden gate installation directory. cd /u01/ogg
2. Open golden gate console. ./ggsci
3. Start manager. > start mgr
4. Log in to the database. > DBLOGIN USERID SYSTEM@localhost:1521/ORCLCDB PASSWORD [password copied while installation]
5. Register Extract. > REGISTER EXTRACT EXT1 DATABASE CONTAINER (ORCLPDB1)
6. Enable schema-level supplemental logging for a table. > ADD SCHEMATRANDATA ORCLPDB1.HR ALLCOLS
7. Create an Extract group. > ADD EXTRACT EXT1, INTEGRATED TRANLOG, BEGIN NOW
8. Create a trail for online processing on the local system and Associate it with an Extract group. > ADD EXTTRAIL ./dirdat/lt EXTRACT EXT1
9. Create EXT1 parameter file and paste the content in the file. > EDIT PARAM EXT1

EXTRACT EXT1
USERID SYSTEM@ORCLCDB, PASSWORD [password copied during installation]
EXTTRAIL ./dirdat/lt
SOURCECATALOG ORCLPDB1
TABLE HR.*;

10. Start Extract EXT1. > start ext1
11. View status of manager and ext1. > info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING
EXTRACT     RUNNING     EXT1        00:00:00      00:00:00

The Oracle golden gate for Oracle 12c is configured. Now, its time to configure Oracle golden gate for big data which will push all the changes/ CDC events to confluent Kafka.

3. Configure Oracle Golden Gate For Big Data

Before following the below steps make sure you add java path to $LD_LIBRARY_PATH environment variable in your .bashrc file. The location of the java is /usr/lib/jvm/java-1.8.0-openjdk-XXXXX/jre/lib/amd64/server. Replace XXXXX with your current version installed on during the installation.

1. Change directory to oggbd cd /u01/oggbd

2. Open golden gate for bd console. ./ggsci

3. Create default directories. > CREATE SUBDIRS After doing this exit the console using exit command, and create the following files in dirprm directory.

• kafkaconnect.properties
• rkafka_handler.props
• rkafka.prm

4. Login to GG console again using step 2 then execute the below command and add the contents to the param file > edit param mgr

   PORT 7801

5. Start the manager. > start mgr

6. Create replicat group rkafka. > add replicat rkafka, exttrail /u01/ogg/dirdat/lt

7. Start the replicat. > start rkafka

8. View status of all if everything is properly configured. > info all

   Program     Status      Group       Lag at Chkpt  Time Since Chkpt
   MANAGER     RUNNING
   REPLICAT    RUNNING     RKAFKA      00:00:00      00:00:09

4. Verify Setup

1. Login to the database and add row to any table like::

   > sqlplus / as sysdba
   SQL> alter session set container=ORCLPDB1;
   SQL> INSERT INTO HR.REGIONS(REGION_ID, REGION_NAME) VALUES(47, 'FOO');
   SQL> COMMIT;

2. Check if the kafka topics are created for the data inserted tables > kafka-topics --zookeeper localhost:2181 --list

   __confluent.support.metrics
   __consumer_offsets
   _confluent-ksql-default__command_topic
   _schemas
   connect-configs
   connect-offsets
   connect-statuses
   ora-ogg-HR-REGIONS-avro

3. To check the data in the kafka topics run - > kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic ora-ogg-HR-REGIONS-avro --from-beginning

You can run both kafka-topics and kafka-avro-console-consumer from the host machine where you are running vagrant. This will help you while coding to connect to Kafka instance present in the vagrant vm. Also, The Oracle database is running on port 1521 to which you can connect from your java code or from your IDE.

This vagrant images is really helpful to me and hope it will also help you guys while developing your CDC related applications. For any issues, during setup or configuring please do comment on the post.

Other Information::

• Zookeeper Port:: 2181
• Kafka Broker:: 9092
• Confluent Schema Registry:: 8081
• Oracle DB Port:: 1521

Don't forget to share the post with your friends and colleagues.

References::

1. Oracle Repository
2. Own Personal Project Repository

The way to design such application is hard because one cannot simply take the entire big complex application and migrated to a new one, it's always better to do it part by part (baby steps). The new application is divided into two parts- read-side and write-side. Considering a part to migrate, the read side of the application is being migrated at first. The consumers will read the data from the new application however, all the write-side operations will still be performed using the legacy application. However, once we have received the certainty that the new application is working as anticipated, the other side of the legacy application will migrate to the new technology stack.

With such decisions in mind, it becomes extremely important for the new application to sync the data from the old one and provide the exact results back to the consumer. For this innovative platform, technology shift in regards to the database is being made though, the older application works with Oracle Database. Now having a CDC (change data capture) solution that puts all the changes happening inside the old database to this new database platform becomes crucial. What could be a better way than to use Kafka as a messaging platform and process it using Kafka Streams?

To explore this idea and getting more understanding of how to manage the data flow I found Debezium which does exactly what I was looking for, a CDC solution to migrate data from source to destination using Kafka and I considered using MySQL and MongoDB for keeping the tutorial simple.

So, this post is all about setting MySQL, MongoDB, Confluent Kafka, Avro Schema Registry and Configuring Debezium. Once everything is setup we will write a simple application using Scala that could capture real-time changes in MySQL and publish those as Document in MongoDB.

1. Installation

MySQL And MongoDB

For this tutorial, both MySQL and MongoDB are needed. Download and install the binaries based on OS version from below given links::

• MySQL download
• MongoDB download

Confluent Kafka

Kafka binaries are needed so that debezium can listen to MySQL changes and push them to Kafka topics. Download the confluent Kafka package from here and extract it. Also, don't forget to set environment variable PATH to point to the binaries.

Debezium

The Confluent Kafka package will come with some default connectors available. However, we will need the debezium MySQL connector for this tutorial, download it from here then extract the jars in a folder and copy the folder at share/java/ inside confluent Kafka directory.

2. Start Confluent Platform

Once you will extract out the confluent binaries and place the debezium jars at the right place you have to execute the following command to start confluent

> confluent up
Using CONFLUENT_CURRENT: /var/folders/y7/j6zgy2w13g1fjljkgb8mdc4r0000gn/T/confluent.2GgyNcVE
Starting zookeeper
zookeeper is [UP]
Starting Kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting Kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
Starting ksql-server
ksql-server is [UP]

As you can see the command start zookeeper, Kafka, schema-registry and Kafka-connect. You can read more about schema-registry, kafka-connect on confuent docs here. Once you see that everything is started properly without any error head to http:😕/localhost::8083/connector-plugins to see if the debezium MySQL plugin is showing in the list as

{
  "class": "io.debezium.connector.mysql.MySqlConnector",
  "type": "source",
  "version": "0.7.5"
}

If the output in the browser is not, as shown above, it means you have missed something or the debezium jars folder is not placed on the right path.

3. Configure MySQL & MySQL Connector

Before the Debezium MySQL connector starts monitoring the changes committed to MySQL database, the server must be set up to use row-level binary logging and have a database user with appropriate privileges. If MySQL is configured to use global transaction identifiers (GTIDs), then the Debezium connector can more easily reestablish connection should one of the MySQL servers fail.

Enabling the binlog

The MySQL server must be configured to use a row-level binary log, which is described in more detail in the MySQL documentation. This is most often done in the MySQL server configuration file, and will look similar to the following fragment::

[mysqld]
server-id         = 223344   # any random number
log_bin           = mysql-bin
binlog_format     = row      # use row or ROW
binlog_row_image  = full     # use full or FULL
expire_logs_days  = 10
gtid_mode                 = on   # (optional)
enforce_gtid_consistency  = on   # (optional)

Create User For MySQL Connector

For the connector to listen to the database changes and monitor the logs it requires USER with SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, RELICATION CLIENT access privileges. Login to your MySQL console using $ mysql -u root -p with the password you have given at the time of installation and run the below command to create a user named debezium with password dbz

mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium' IDENTIFIED BY 'dbz';

Configure MySQL Connector For The Database

I have created a moviesdemo database with two tables movies and sales. The movies tables consist of all the movies and the sales consist of the total number of movie ticket sold. The idea behind using these two tables is to make the tutorial simple and easy and also to show how two related tables get processed in stream processing.

When the movie row is being inserted in the movies table within the same transaction sales table row also gets inserted with a starting value of 0.

To make the debezium Kafka connector to listen to the changes happening on the moviesdemo project we will need to provide configuration settings to the connector. A POST request in the following format to the URL http://localhost:8083/connectors will register the connection settings to the debezium utility. Make sure to provide the same username and password that you used while configuring the mySQL.

{
  "name": "moviesdemo-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "database.hostname": "localhost",
    "database.port": "3306",
    "database.user": "debezium",
    "database.password": "dbz",
    "database.server.id": "184054",
    "database.server.name": "dbserver1",
    "database.history.kafka.bootstrap.servers": "localhost:9092",
    "database.history.kafka.topic": "dbhistory.moviesdemo"
  }
}

Head to http://localhost:8083/connectors/moviesdemo-connector using web browser to see configurations been done properly and if so, your Debezium connector is now ready to listen to any changes happening to MySQL moviedemo database.

4. Kafka Topics

Open MySQL console using $> mysql -u debezium -p, and try to insert one row in both movies and sales table. Make sure auto-commit is being turned off and commit explicitly using mysql> commit; 😊

Note:: Make sure you have created a database named moviesdemo in MySQL along with two tables movies and sales the DDL scripts for both of the tables are here

If you have seen your kafka topic list you will find that there are two topics created as below

$ ./bin/kafka-topics --zookeeper localhost:2181 --list
__confluent.support.metrics
__consumer_offsets
_confluent-ksql-default__command_topic
_schemas
connect-configs
connect-offsets
connect-status
connect-statuses
dbhistory.inventory
dbserver1
dbserver1.moviesdemo.movies
dbserver1.moviesdemo.sales

5. Generating Business Event Using Kafka Streams

I wrote a simple Kafka stream program in Scala that reads from both the two Kafka topics movies and sales, joins the two messages based on movie_id and then create a business event which is published to events Kafka topic.

The program is easy to understand. It works on two business cases Insertion & Updates to both the table. However, I left deletion logic implementation for the readers.

private def buildMovieStream: KStreamS[String, movie.Envelope] = {
    import AppSerdes.movieSerde.consumed
    builder.stream[String, movie.Envelope](Utils.getTopic("movies"))
  }

  private def buildMovieSalesStream = {
    import AppSerdes.movieSalesSerde.consumed
    builder.stream[String, Envelope](Utils.getTopic("sales"))
  }

  val movieStream = buildMovieStream
  val saleStream  = buildMovieSalesStream

  private def filterSalesStreamForCreations = {
    saleStream
      .filter((id, value) => {
        println("filtering sales creation message")
        value.op.equalsIgnoreCase("c")
      })
  }

  def createMovieBusinessEvent = {
    import AppSerdes.movieBEventSerde.{joined, salesSerialized}
    val movieFilteredStream = new MovieCreatedFilter().filter(movieStream)
    val salesFilteredStream = filterSalesStreamForCreations

    val envelopExtractedMovie: KStreamS[Int, Movie] =
      movieFilteredStream.map((id, value) => (value.after.get.movie_id.get, value.after.get))
    val envelopeExtractedSale = salesFilteredStream.map((id, value) => (value.after.get.movie_id.get, value.after.get))

    envelopExtractedMovie.join(envelopeExtractedSale, (movie: Movie, movieSale: MovieSales) => {
      println("Created Business Event")
      val serializer = new KafkaAvroSerializer()
      serializer.configure(schemaConfig, false)
      val movieSerialized = serializer.serialize(Utils.getTopic("movie"), AppSerdes.movieBEventSerde.movieFormat.to(movie))
      val salesSerialized =
        serializer.serialize(Utils.getTopic("movie_sales"), AppSerdes.movieBEventSerde.saleFormat.to(movieSale))

      val map = Map("movie" -> movieSerialized, "sale" -> salesSerialized)
      BusinessEvent(EventTypes.`MOVIECREATEEVENT`, map)

    }, JoinWindows.of(3000))
  }

  def emitMovieBussinessEventToTopic = {
    import AppSerdes.movieBEventSerde.eventProduced
    createMovieBusinessEvent.to("events")
  }

  emitMovieBussinessEventToTopic

  def createMovieUpdateEvent = {
    val updateStream = new MovieUpdateFilter().filter(movieStream)
    updateStream.map((id, envelop) => {
      val before = envelop.before.get
      val after  = envelop.after.get

      val serializer = new KafkaAvroSerializer()
      serializer.configure(schemaConfig, false)

      val beforeMovieSerialized = serializer.serialize("events", AppSerdes.movieBEventSerde.movieFormat.to(before))
      val afterMovieSerialized  = serializer.serialize("events", AppSerdes.movieBEventSerde.movieFormat.to(after))

      (after.movie_id.get,
       BusinessEvent(EventTypes.`MOVIEUPDATEEVENT`, Map("before" -> beforeMovieSerialized, "after" -> afterMovieSerialized)))
    })
  }

  def emitMovieUpdateEvent = {
    import AppSerdes.movieBEventSerde.eventProduced
    createMovieUpdateEvent.to("events")
  }

  emitMovieUpdateEvent

When you run the above program it will start listening to the movies & sales topic and will create event objects for any insert or update operation at MySQL side.

6. Updating MongoDB Based On Business Events

This code that you see below reads the business events from the events topic and inserts or updates MongoDB collection. This code is also written using Kafka Streams in Scala and is pretty straightforward to understand.

It reads the topic, deserialize the message and checks which type of business event is present. Based on the event type it will change the records in the MongoDB movies collection.

def buildEventStream = {
    import AppSerdes.movieBEventSerde.eventConsumed
    builder.stream[Int, BusinessEvent]("events")
  }

  private val eventStreams: KStreamS[Int, BusinessEvent] = buildEventStream

  def filterEventsByType(eventType: String): KStreamS[Int, BusinessEvent] = {
    eventStreams.filter((_: Int, event: BusinessEvent) => event.eventType.equalsIgnoreCase(eventType))
  }

  filterEventsByType(EventTypes.`MOVIECREATEEVENT`).foreach((id, event) => {

    val value = new MovieCreatedEventDeserializer(event).get
    if (value.isDefined) {
      val doc = value.get
      movies.insertOne(document = doc).toFuture().onComplete(_ => println(s"Inserted ${doc}"))
    }

  })

  filterEventsByType(EventTypes.`MOVIEUPDATEEVENT`).foreach((id, event) => {
    val movie: Option[Document] = new MovieUpdateEventDeserializer(event).get
    val movieDocument           = movies.find(Filters.eq("movie_id", movie.get.getInteger("movie_id")))
    movieDocument.toFuture().map(_.head).onSuccess {
      case data => {
        val document = movie.get.toBsonDocument
        println(s"Relpacing Movie Information ${movie.get.get("movie_id").get.asString()}")
        document.put("sales", data.get("sales").get)
        movies
          .replaceOne(Filters.eq("_id", data.getObjectId("_id")), document)
          .toFuture()
          .onSuccess {
            case data => println("Movie Information Updated")
          }
      }
    }

  })

This completes the tutorial on how debezium can help you design change data capture solutions. Hope you guys like it and for any questions or doubt please do comment on this post I will surely get back.

Also, Please don't forget to share the post using the share link and do subscribe to the blog by which you will get the next blog post right into your mailbox. This also helps me to track the users following the blog 😀.

Stay tuned & Happy Coding!

Reference Links::

• Debezium Docs
• Code @Github
• Confluent Docs

I am writing this post because I have to search a lot of things to get it working and there is no single place to get the exact information on how to run spark job on a local kubernetes cluster. I thought writing a post on my findings will enable others not to go through the same pain as I did.

In this post we will take a look on how to setup Kubernetes cluster using minikube on local machine and how to run apache spark job on top of it. We are not going to write any code for this, few examples are already available within Apache Spark distribution and we are going to use the same example jar and run SparkPi program from it onto our kubernetes cluster.

Prerequisites

Before we start please make sure you have the following pre-requisites installed on your local machine.

I am running the tools on Mac High Sierra.

• Apache Spark (v2.3.0)
• Kubernetes (v.1.9.3)
• Docker
• Minikube
• VirtualBox

1. Start Minikube

We can start minikube locally by simply running below command.

Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on a local machine. It enables us to try out Kubernetes or develop with it day-to-day.

When minikube starts it starts with a single node configuration and takes 1Gb of Memory and 2 cores of CPU by default, however, for running spark this requirement will not suffice and the jobs will fail (and they do as I have tried multiple times) so we have to increase both Memory and CPU cores for our minikube cluster.

There are three different ways to do this. However, for me not all worked and that’s how I came to know about the different ways to change the minikube configuration. You can use any one of the below ways to change the configuration.

Pass Configuration To Minikube Start Command

You can directly pass the memory and CPU options to the minikube start command like::

This will start a minikube cluster with 8Gb of memory and 4 cores of CPU. (somehow for me this didn't work)

Modify minikube config

We can change the minikube config with config command in minikube cli. The config command allows setting different config options for minikube like memory, CPUs, vm-driver, disk-size etc. To see all the available options use > minikube config command it will list all the available options that can be modified.

After setting configuration using config command we need to delete our previous running cluster and start a new one. To delete minikube cluster run > minikube delete and rerun the start minikube command.

Change Configuration Using VirtualBox

For me, the above two options didn't work and if you are like me you can use this last option which worked for me and hope it works for you. Open VirtualBox app on your machine and select your VM like the one shown in below image. RightClick -> Settings on your VM, this will open the configuration page for the Minikube VM.

note:: To change configuration using VirtualBox first you need to shutdown the VM if it is already running.

Inside option System -> Motherboard, you can change the memory of the VM using the slider, in my case I have given it 8192MB of memory.

To change the CPU config, go to Processor tab and change it to 4. You can try other options also for me 4 works just fine. Don’t change it to less than 2 otherwise things won’t work. ::p

After the configuration changes are done, start the VM using VirtualBox app or using the above given minikube start command.

Your cluster is running and now you need to have a docker image for spark. Let's see how to build the image next.

Creating Docker Image For Spark

Make sure you have Docker installed on your machine and the spark distribution is extracted.

Go inside your extracted spark folder and run the below command to create a spark docker image

some of the output logs are excluded.

Now if you run > docker image ls you will see the docker build available in your local machine. Make a note of this image name we need to provide the image name to spark-submit command.

There is a push option available to the above command which enables you to push the docker image to your own repository this, in turn, will enable your production kubernetes to pull the docker image from the configured Docker repository. Run the same command without any options to see its usage.

It might happen that the command will not work and will give error like::

This is because of an issue with the docker-image-tool.sh file. I have raised a bug for this in Apache Spark JIRA you can see it here.

The issue is under fix but for you to continue with this post what you can do is open the docker-image-tool.sh file present inside the bin folder and after line no 59 add BUILD_ARGS=(), save the file and run the command once again and it will work.

For those who the above command worked without the workaround the issue might have been fixed at the time you are reading this post and you don't have to do anything hurrah!!!

Submit Spark Job

Now lets submit our SparkPi job to the cluster. Our cluster is ready and we have the docker image. Run the below command to submit the spark job on a kubernetes cluster. The spark-submit script takes care of setting up the classpath with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark supports

options used to run on kubernetes are::

• --class:: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)
• --master:: The master URL for the Kubernetes cluster (e.g. k8s:😕/https://192.168.99.100:8443)
• --deploy-mode:: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default:: client)
• --conf spark.executor.instances=3:: configuration property to specify how many executor instances to use while running the spark job.
• --conf spark.kubernetes.container.image=spark-docker:: Configuration property to specify which docker image to use, here provide the same docker name from docker image ls command.
• local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar:: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:😕/ path or a local:😕/ path that is present on all nodes

The package jar should be available cluster-wide either through HDFS, HTTP or should be available within every packaged docker image so that it will be available to all the executor nodes in our spark program. The local:// address means that the jar is available as a local file on every initialized pod by the spark driver and no IO has to be made to pull the jar from anywhere, this works well when application jar is large which is pushed to every worker node or shared using any shared filesystem.

Luckily, our spark docker image file packages the example jar in the docker container so we can use it. However, how to package our own application code and push it either on HDFS or as a separate docker image I will write it as a separate post.

To check if the pods are started and the spark job is running, open kubernetes dashboard available within minikube.

View Minikube Dashboard & Kubernetes Logs

To check the status of our submitted job we can use either kubernetes dashboard or view kubernetes logs. Minikube comes with the dashboard available as an addon and can be started using the following command.

Navigate to the URL given by above command to view the dashboard. The dashboard provides lots of information about cluster memory usage, CPU usage, pods, services, replica set etc. We can also view service logs directly through the dashboard. However, if you don't want to go to the dashboard you can view the Spark Driver log using > kubectl logs <pod name> command::

If you see in the logs the program has calculated the Pi value and the container are stopped in the end.

Shutdown Cluster

Shutting down cluster is very easy, use > minikube stop and it will stop the cluster.

Hope this helps you try running apache spark on local kubernetes cluster. Please do comment if you didn’t understand any step or getting any errors while following the steps and I will try to add more details to the post. Also, please do let me know if you liked it and help others by sharing.

Happy Coding...

References

1. Apache Spark Documentation
2. Minikube Github

In this post, we'll try to answer all these questions. We'll start with the basic introduction of Kafka, then will see its use-cases, after that will dive into some demonstration and coding of how you can get started with Kafka using Java.

What is Kafka?

Kafka is distributed publish-subscriber message delivery system that is designed to be fast, durable, scalable in very simple terms. But as you go deeper into the design and technicality of the architecture it's a large system. We are not going to discuss the entire design in this post however, you can get more insight about the design of Kafka by reading Kafka Documentation.

Kafka has following 4 core API's::

Producer API - Allows to push messages to one or more topics. Consumer API - Allows to consume messages from one or more topics. Streams API - Allows to consume input stream from one or more topics producing an output stream to one or more topics, effectively converting input streams to output streams. Connector API - Allows building a reusable producers and consumers that connects to Kafka topics to existing applications or data system.

Apache Kafka Architecture

Like other publish-subscribe messaging systems, Kafka maintains feeds of messages in topics. One or more Producers writes to the topic whereas consumers consume from the topic. As Kafka is distributed platform topics are partitioned and replicated across multiple nodes.

Message to the topics are simple byte array and developers can store messages in any form of String, Json etc. Each message is associated with a key and producer guarantees that messages with the same key will arrive at the same partition. Consumers can be organized into consumer groups in which case Kafka makes sure that each consumer within the consumer group will receive exactly one message to the partitioned within the topic it subscribes too i.e each consumer in a consumer group reads from a unique partitioned of topic to which no other consumer within the consumer group will have access, thus all the messages that arrive at the unique partitioned which is identified by a unique key will be delivered to the same consumer always.

Kafka treats each topic as a log(ordered sequence of messages) like a pipeline where messages come from one end and are consumed from another end, each message in a partitioned is assigned a unique offset. Kafka does not maintain any detail of which message is read by each consumer and only retains unread messages rather, Kafka retains messages for a certain amount of time and consumer are responsible to track their location in each log. Kafka can maintain a large amount of data without any issue.

Use Cases::

Website activity tracking:: Web Apps can send events to Kafka such as page clicks, page search for real-time processing and analytics.

Log aggregation:: Logs can be aggregated from multiple servers which can be analyzed in real time or will be available to tools like Hadoop or Apache Solr.

Stream Processing:: Framework like Apache Spark can stream data from a topic, process it and writes the processed data to the new topic. This processed data then can be made available to the customer or another application for further processing.

Operational Metrix:: Alert monitoring and notification can be done by analysing the metadata or logs. Other way application can send stats or counts to the topic which will be then compared and visualized as a dashboard.

Others system may serve the purpose but none of them does all. Like ActiveMQ and RabbitMQ are a very popular messaging system. Google PubSub is also one of the great messaging platforms but does not provide on premises solutions. I have also heard about the performance and throughput of Kafka as oppose to others.

Now, let's get into the coding stuff will see how you can install Kafka on your local machine and perform operations like creating, deleting, listing topics, sending and receiving messages to the topic.

Installing Kafka

For this I am using a windows machine so I will be explaining installing Kafka on windows, however, Unix and Mac's setup is also same. Kafka binaries are packaged into zip folder which needs to be extracted. Download the binaries extract it somewhere on your machine. Execute the following commands to start Kafka on your local machine. For windows machine the bat files are located inside bin\windows.

> bin/zookeeper-server-start.sh config/zookeeper.properties
[2017-09-03 19:00:38,859] INFO Reading configuration from: ..\config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)...
...

You need to first start the Zookeeper instance. There are scripts packaged into the Kafka binaries the above command starts the zookeeper instance.

> bin/kafka-server-start.sh config/server.properties
[2017-09-03 19:03:59,553] INFO starting (kafka.server.KafkaServer)
[2017-09-03 19:03:59,590] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
...

The above command will start Kafka server and the the broker address will be localhost:9092.

Creating a Topic

Creating a topic using cmd is fairly simple. Run the following command to create a "test" topic.

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

To list all the topics::

> bin/kafka-topics.sh --list --zookeeper localhost:2181 test

To delete a Topic::

> bin/kafka-topics.sh --zookeeper localhost:2181 --delete test

Every command given above will work with your console. Let's move to the programming part and see how to use API in java. Will try to do the above operations using Kafka Admin Client API and then will write a simple producer and consumer that sends and messages to the Topic we create.

Project Setup

I will be using maven for the project and the Maven configuration for the project is as follows::

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.kafka.samples</groupId>
    <artifactId>kafka-learning</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.11.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.11</artifactId>
            <version>0.11.0.0</version>
        </dependency>
        <dependency>
	    <groupId>com.esotericsoftware</groupId>
	    <artifactId>kryo</artifactId>
	    <version>4.0.1</version>
	</dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version><4.11</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

The Kryo library in our pom.xml is used to write custom Serializer and Deserializer for Kafka which is explained furture in the post.

Creating a Topic::

You will require properties like bootstrap.server and client.id to configure the admin client. You can read about all the configuration options in Kafka Documentation but for now, as we are not dealing with configurations like secure SSL, bootstrap.server will be enough to connect to the Kafka Server.

public class CreateTopic {

    public static void main(String[] args) throws IOException {
        Properties properties = ConfigUtils.getConfiguration("admin-config");
        AdminClient adminClient = AdminClient.create(properties);
        CreateTopicsResult result = adminClient.createTopics(Arrays.asList(new NewTopic("test1", 1, (short)1)));
        System.out.println("Topic Created: ");
    }

}

The arguments to NewTopic is the new topic name, number of partitioned and replication factor.

Listing All Topics::

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.ListTopicsResult;


public class ListTopics {

    public static void main(String[] args) throws IOException, InterruptedException, ExecutionException {
        Properties prop = ConfigUtils.getConfiguration("admin-config");
        AdminClient admin = AdminClient.create(prop);
        ListTopicsResult topics = admin.listTopics();
        Set<String> topicNames = topics.names().get();
        for(String topic: topicNames) {
            System.out.println(topic);
        }
    }

}

Delete Topic::

public class DeleteTopic {

    public static void main(String[] args) throws IOException, InterruptedException, ExecutionException {
        Properties prop = ConfigUtils.getConfiguration("admin-config");
        ZkClient zkClient = ZkUtils.createZkClient(prop.getProperty("bootstrap.server"), 100, 10000);
        zkClient.deleteRecursive(ZkUtils.getTopicPath("test2"));

    }
}