Integration Tests using a Database and Spring Boot

Author unknown

Integration tests are crucial part in testing most applications. In my experience, some projects even go against the famous pyramid diagram and have more of these, than unit tests. For instance – web systems, where some framework, like Spring Boot, frees us from the burden of implementing input validation, role-based access control, etc. and consequently, good amount of unit testing is done inside the library itself.

When I work on a REST API, I often find that most of acceptance tests, checking various use cases and scenarios, are of integration type. In his article – A Guide to Robust Unit and Integration Tests with JUnit – Joshua Hayden describes how JUnit can be used for writing and running tests. I would like to further it and add more details about testing multiple components of the system.

I like a quote from article by James O Coplien – “Create system tests with good feature coverage (not code coverage) — remembering that proper response to bad inputs or other unanticipated conditions is part of your feature set.”

Let’s assume our project is A REST API for online education system, where user can register, find a course, enroll and submit homework. It would be difficult to test all features of our app without integration tests, as in most cases it would require creating and modifying multiple resources, hence, database access. For instance, to test enrollment, we would need to have a registered and activated user and existing course. In my opinion, mocking a persistence layer for this would be only a waste of time. So, let’s dive in:

To run tests against a database, we have to overcome several challenges:

  • We need a separate DB and corresponding configuration
  • A database should be easy to build from scratch without manual intervention
  • We need isolation for each test or test suite, so they won’t pollute data for other tests.

For the sake of example, I will try to solve them using Spring Boot, however the same idea can be used with other frameworks and technologies.

 

In-Memory Database vs Production

I’ve often received advice to use in-memory databases for my integration tests, but I’m still very skeptical about it. Even though all popular databases follow some standards, they have lots of vendor-specific features. This is what makes writing a fully portable app difficult. For instance, not only in-memory H2 database might act differently than MySQL, but two versions of MySQL might give different results in some cases. It’s error-prone to consider every detail or a deprecated feature, so, for my tests I always prefer to select exact vendor and version, which is used in production environment. (Container technologies, like Docker, make our lives easy in this decision).

Let’s move on with a real database here and decide on configuring it: If we start up a docker container on every `test` command and destroy it afterwards, we can get away with one DB, but if we also need a persistent database for manual testing, which must not be cleared on every test run, then we’ll need an additional temporary DB. In Spring Boot, we can either create a test/resources/application.properties file with connection url of such database or use a test profile. This might vary among frameworks, but a key thing is to solve the task without manual intervention or commenting out stuff for running tests.

 

Rebuilding a Database

To build a database from scratch, one can have a SQL script of structure exported from a DB. But I prefer to go with a nice method, called Database Migrations. This enables us to have a versioning of a database and application code in sync. It also makes rebuilding a DB very easy.
You can also learn about database migrations from this article: Database Migrations: Turning Caterpillars into Butterflies

 

Isolated Tests

Test isolation is good for several reasons: they can be run in parallel and in case database is updated, they can be safe from breaking each others’ data. On the internet you might find a suggestion to use transactions for such tests, which won’t be committed and will be rolled back after test completion. In Spring boot this can be achieved by adding @Transactional annotation on a test method.

I don’t like this solution, because if we actually use transactions inside our business logic, we’ll end up with nested transactions, which might have unexpected results in corner cases. Also I find it very helpful that by running some tests I can quickly populate database with various scenarios and continue manual testing or debugging using persisted data.

Generally, I prefer to do it this way: Dropping and rebuilding the database before running each test is an expensive operation. It might be better to rebuild it with migrations only once before starting testing, so migration scripts will also be tested. Then, I use plain sql to truncate tables before a test.

One of the ways for achieving this:

Add a configuration class, which will run once before all tests and reset migrations there (Flyway is a plugin for database migrations.)

@Configuration
public class TestConfig {
	@Bean
	public FlywayMigrationStrategy clean() {
		return flyway -> {
			flyway.clean();
			flyway.migrate();
		};
	}
}

Add sql script of truncate as an annotation in a base class for integration tests. This will be run before each test method.

@Sql(executionPhase = ExecutionPhase.BEFORE_TEST_METHOD, 
	scripts = "classpath:ge/demo/api/sql/beforeTest.sql")

Sometimes, it might be reasonable to group several tests in a test suite and reset database only before a suit and not a single test.

DevOps Con Tbilisi 2018 and automation revolution

Several months ago, thanks to a common interest, I met two brilliant persons – Erekle Magradze and Vazha Pirtskhalaishvili. Soon the idea was born to make this common interest stronger and find others like us – build community. So we started preparing a tech meetup, which eventually turned into a two-day conference. Anyone reading my blog probably has already attended or heard about it. I just wanted to keep a memory article here :))

In my opinion, Georgian tech world is facing an important decision. DevOps revolution started around 10 years ago and it has just reached our community. There are very few Georgian businesses, which harness the benefits of development process automation. If we don’t catch up, we’ll be out of the global market.

Most of the conference speakers shared experience they had with foreign companies. I’m not exaggerating, when I say that I really enjoyed every talk and listened to them for several times. I’m happy that participants found time and offered us a productive weekend. You can see their video presentations (in Georgian) on our FB page or Youtube: DevOps Con Tbilisi 2018 – Tech talks

This one is mine 🙂 (but in Georgian)

Info about all speakers, organizers, supporters and other details can be found on our event web site or DevOps Georgia Facebook page:
DevOps Con Tbilisi 2018
DevOps Georgia

29-30 სექტემბერს კახა ბენდუქიძის საუნივერსიტეტო კამპუსში გაიმართა ტექნოლოგიური კონფერენცია DevOps Con Tbilisi 2018

Geplaatst door DevOps Georgia op Zondag 7 oktober 2018

My Spring Booklist 2018

As you already know, I love writing book reviews on my blog. 🙂 I read most of them when I’m away on a vacation, so, again, ten days in the forest turned out productive. Here is my list:

 

 

Advanced Microservices

This one turned out a quite practical book. The author describes how good API looks like; how to manage service versions; what options do we have while building our infrastructure; which database should we use – Postgres, Redis, MongoDB, Elasticsearch… how to plan monitoring – Logstash, Kibana; analytics – Grafana, Graphite, StatsD; automatic documentation, Pipeline and CI servers, Service discovery – Consul, Zookeeper.. In short, he makes a reasonable review of currently existing technologies and gives some advice what to consider for automating the build and maintenance of our services.

 

Painless Docker

This book is not printed yet, so you will find lots of typos, but nevertheless, the book was interesting for me. It is based on real cases, problems and solutions (which are not so well documented in product docs). He describes interesting details and containerization technology in depth. I could not memorize everything, as it required practice. If you use docker, I’m sure you will find more than one novelty in the book.

 

Microservices AntiPatterns and Pitfalls

This is a tiny book, which describes problems and antipatterns related to microservices. Despite the popularization, clearly, microservice architecture is not a solution to all problems and has its own place with pros and cons. So, how can we split the system into independent parts without using shared code and shared database, considering that code duplication is evil and also we need accumulated data from many tables simultanously for reporting?

 

Building Microservices: Designing Fine-Grained Systems

Recently, I’ve become a fan of ThoughtWorks company. All the books that I liked, turned out to be written by its employees, including Martin Fowler. This one is by Sam Newman. Lots of good advice and answer to interesting questions; Microservices with all their difficulties and advantages in details. I think the book is mostly for architects, as it does not go into technology usage and configuration details.

 

Building Evolutionary Architectures: Support Constant Change

Probably, this book is an exception from my fav books from ThoughtWorks, because despite its high rating, I didn’t really like it much. In my opinion, it contains too much theory and repetitions, so I left it after 50%. Continuous Delivery still remains my favourite book about evolutionary architecture.

 

Pro REST API Development with Node.js

The book seems normal to me. It describes a REST architectural style, hypermedia, HATEOS, Node principles, several libraries for developing APIs – so, nothing new for me, but novice can deepen their knowledge. Probably, the book will get old soon as it was issued in 2015.

 

The Obesity Code

This book is a gold mine 🙂 and one of the greatest discoveries for me. All the things that I have read in life about food, nutrition, diets, lifestyle, or have heard from famous dietologist, were completely blown apart. This science is full of contradiction. On one year something is healthy and on another it kills. Do you know a ‘healthy’ lifestyle which you can happily follow for your whole life, without feeling guilt after having an ice cream?
A Canadian doctor, Jason Fung has a classic medical education and 20-year experience in nephrology. He chose to be a kidney doctor, as lots of interesting chemical processes take place exactly there and he loves puzzles.

In his book, Mr. Fung plainly explains what happens in our bodies when we take food. He uses only those researches which lasted for many years and had very large number of participants. He argues that obesity is not a caloric, but hormonal dysfunction and ‘eat less, move more’ philosophy is wrong. He started war with type 2 diabetes, which broke out as an epidemy and, for example, based on stats nearly every other person has a type 2 diabetes or prediabetes in the US. Jason Fung has cured diabetes and obesity in his patients within several months. He has not invented anything new and his method exists from the beginning of mankind. It is even included in most religions, we just lost it from our daily life on some stage of development.

I can talk for hours on this topic, so I better stop here. I will only say, that as long as I have a choice, from now on I only believe to the arguments of this doctor. Finally, everything is in place.

As far as the book is concerned, it has some repetitions, but if you start reading, my advice would be to go till the end, as major points are even more after the middle.

 

The Complete Guide to Fasting: Heal Your Body Through Intermittent, Alternate-Day, and Extended

This and the previous book have much in common, but this one seems easier to read with less researches and more success stories. Also, it has some practical tips for increasing a fasting period.

Most expensive code, NASA standards and bold developers

Rover footprints on Mars, starting from nowhere

I was always curious how people write enormous successful applications without bugs, for instance, the Mars Rover “Curiosity”, which flew to Mars, covered 350 million miles in 8 months, landed within 6 mile radius, walks on unknown land and sends data to the Earth.

 

How Rover was sent to Mars

There is one of the rare technical speeches about Mars Rover on the internet. Gerard Holzmann, one of the lead scientists of NASA JPL (Jet Propulsion Laboratory), describes how they were trying to manage software development processes and minimize the problems.

I will shortly mention few facts from the video:

  • Approximately 4 million lines of code, 100+ modules, 120 threads on one processor (+1 backup). Five years and 40 developers. And all of this was for one client and one-time use.
  • The project contained more code, that all previous Mars missions combined. Because of this scale, human code reviews were not effective any more.
  • They created standard to reduce risks and constantly checked code against it using automatic tools. As people used to ignore documents with hundreds of rules, they made a poll and selected out ten most important rules: Power of ten
    E.g. don’t use recursion, goto and other complex flow constructs; variable scope must be minimal; all loops must have fixed edges; pointer usage must be limited to a single dereference, etc.
  • Every night the code was built, checked with static analysis and various automatic tests. It was at night, as analysis took 15 hours.
  • The one who broke the build, would receive a penalty, so they would have to put up a Britney Spears poster in their cubicle. And the one who would leave many warnings in their code, would be printed on the “Wall of Shame”. It seems, that people need motivation, even in NASA 😀
  • Person was not allowed to code until a specific training and certification.
  • They required 100% code coverage. If you have ever done this, you should know how heavy task this is, as they would need to test impossible cases, too.
  • Code review was not a long group meeting. They met only to discuss disagreements. Notes and directions were exchanged via a special application, which also showed the code status after previous night checkings.
  • The warnings from a compiler and static analysis had to be zero. This turned out to be a difficult task and took lots of time. The correlation of this requirement and the project success is unknown, but this was the cleanest code in comparison to their previous missions.
  • In critical parts they followed the MISRA programming standard – which is used in engines and life-critical equipment.
  • They performed logical verification of critical subsystems – mathematically proved correctness of the algorithm.

If you are interested in this mission, there is one more speech (although I liked the first one more): CppCon 2014: Mark Maimone “C++ on Mars: Incorporating C++ into Mars Rover Flight Software”

 

About violating the standard

Of all the cases known on the internet, the most expensive code was written for Space Shuttle – 1000$/line. But in 2013 Toyota lost in court and if we calculate the compensation amount, the cost of one line would turn out as 1200$. The unintended acceleration of Toyta was in news for several times due to car accidents and complaints. They revoked pads, then acceleration pedals, but it was not enough. Then NASA team checked the software of a Toyota car against their standard and even though they found 243 violations, they could not confirm that software caused problems. Court had invited an external expert, who critisized Toyota software because of recursion, stack overflow and much more.

Whole case is here: A Case Study of Toyota Unintended Acceleration and Software Safety

 

We, mere mortal developers

Writing constructor incorrectly in JavaScript

It turns out the we, software developers, risk too much. 🙂 We trust an OS, external libraries, don’t check the validity of value returned from function. Although we filter the user input, do we do the same while communicating to various inner services? In my opinion, this is natural. Checking everything is very timeconsuming and, consequently, expensive. You can look at some Defensive programming strategies.

There are applications, which almost never make mistakes, but when it does, it brings huge loss. Similarly, there are applications, which has more bugs, but correcting them is simple and cheap. Probably the same as with cars – it’s expensive to recover a BMW, which is rarely broken. And during the war, US had Willys MB jeeps, which were recovered very quickly. They would simply disassemble the car to relocate. There even is a video where Canadian soldiers disassemble and assemble Jeep in under 4 minutes.

I think, most of our applications are in the later category. Important thing is to be able to make changes quickly and with minimal expenses.

How to become a better developer effectively

During these 10-11 years, when programming became my profession, I had diverse periods of success and failure. Routines, leaps – the same as with everyone 🙂

I wanted to gather some tips in this article. Would you add some in comments?

 

A Big Picture

V8 Engine

There are various ways to learn the same subject. One might require 5 years, whereas the second way gives the result in one. Programmers are practical people. We want to sit down, write and run the code soon. Probably this is why our most popular learning method looks like roller skating. We fall a lot and bruise a lot. In the end, we might curse its inventor and give up. Instead of this we could have prepared in advance. We could have watched/read how to lace the shoe, when not to raise a hand, how to fall, how to turn. All this is physics and is already known. We will fall a lot anyway, but with less trauma.

Let’s experiment: Think of a langauge, which you have been using for last 1-2 years. I guarantee, that if you read a good book (or documenation) about this language from cover to cover, large part or maybe even half of it will be new for you. I don’t mean that you should stop at every example and run the code, or get into all details and remember. I just say, that when we read the whole information in advance, we see the big picture. We can always google the details later, if we know that such thing exists at all. If we don’t have the base knowledge or maybe, for instance, how memory is managed, we will have memory leaks in our app and we won’t even notice.

After this we can add one more book about corresponding Best practices, which will spare us from even more ‘traumas’. So, we’ll have to rewrite the project for less times.

The book needs 2-3 days, but to get the same information with error and trial, we will need months.

 

Contests and algorithms

I think there is one stage, which every self-respecting programmer should pass. It requires much time, so, it is often passed during the school period, but better late, than never. If someone will feel lazy about it, they might be left behind by competitors.

I’m refering to programming contests and problem-solving. Based on this exercise process, your thinking style and speed will change drastically. Most of the popular mistakes will disappear from code, e.g. accessing wrong or nonexisting index of array, setting wrong limits in loops, allocating unnecessary memory and memory leaks, writing unoptimal code, using wrong data structure. You might even forget when was the last time you used a debugger. You will be able to run application not after writing each function but after writing lots of code and it will have small number of bugs. Unlike the real projects, these testing systems find all your bugs and consequently make you test better.

You may not need these learnt algorithms and you might not find them in real projects, but you will definitely need quick thinking and good skills of writing and testing code. It’s okay, if we don’t win, if we don’t reach into top hundred, even into top thousand best. This is a sport. Some people are born with this mind, some work very hard. Very few fights for an olympic medal, but we all exercise for good health, don’t we?

For me such phase was several years ago, when we created GeOlymp. For me such phase was several years ago, when we created GeOlymp. I didn’t invent the problems there, others did. I had different kind of tasks and also sometimes I was trying to solve them side by side with participants. Even today I feel when I fall out of shape, although I write some kind of code constantly. Then I get embarassed, when I make silly mistakes.

A good list where you can practice: The 10 Best Coding Challenge Websites for 2018

 

Teach

This is a popular method 🙂 When you explain something to other person, you notice completely different details. Generally I think that forming problem as a text is half work done. When you are looking for a bug or don’t know how to do something, try to describe and write the problem statement first. Rubber duck debugging is effective because of this. :))

 

Stress is destroying us

Gravity Glue | Stone Balance by Michael Grab

Programming is one of the very stressfull professions. I’ll tell you why: Every day programmer does what he hasn’t done before. If he does the same as earlier (or repeat what others did) and won’t use it instead, this counts as a big mistake. Also, he is asked to provide time estimation. Clearly, all estimations are approximate. He might come across various hard problems on the way which will cost him days or weeks. Programmers often work overtime to keep given promise.

Every one of them has their methods to cope with stress. It is much harder to recover
from mental tiredness, than from physical. Some practical tips:

  • Use automatizations. The most stressfull processes of our profession are good candidates for automating. This will save you lots of time and nerves.
  • When you are stuck on one problem, change the subject or have a rest. There is a saying – sleep on a problem. If you state the problem well before sleep, your brain will work at night and give you the answer in the morning. This really works.
  • Physical health, walk, meditation, sleep. You might think, that there is no time for exercies, but actually you can work more mentally, when you have physically active lifestyle. On the other hand, code of the sleepless person is full of bugs. Energetic drinks and huge amount of caffeine only make things worse.
  • Only you can take care of yourself. Stress is a nice motivator, but constant stress ruins the body and the brain.

 

Open-source projects

Like books, codes of other people is a good source for development. Nowadays, the most interesting projects are available on the internet and you can even become a contributor. Choose your favourite language, a project and you can participate without leaving home. By the way, this will also look good on your CV.