I have been working in ІТ for almost 15 years, and five years as an Architect. In this article, I continue sharing my experience and point out the most common mistakes when designing and developing application architectures. In the previous article, I wrote about soft skill mistakes that occur in the consulting process. This article is dedicated to typical technical mistakes in architecture design.

Typical mistakes in architecture design

Mistake 1. Disregarding security when designing the architecture

All projects I dealt with had security issues. Even world-famous brands face such problems, not to mention smaller companies.

Most issues occur because of the incorrect design at the very start of the project. How is a project usually designed? Here is an approximate approach: we quickly develop the functionality, check whether it works, and, if everything goes on well, in the end, we add the authorization capability. Later, it turns out that the application is supposed to support several tenants, must be compatible with GDPR, and so on. In order not to start from scratch, Developers gradually add quick but not the best solutions such as attributes that follow several dozens of methods or user role verifications that are roughly added to code right before calling any controller. And there are a lot more of such things that you might have witnessed yourselves.

The application is gradually compiling such quick solutions and turns into a huge code that is hard to handle. You cannot work with microservices anymore because to separate a part of the functionality, you will need to rewrite half of the application. What can you do about it? Nothing will help. There is no magic solution. Most projects facing similar issues came in as start-ups where security was not much of a priority. It led to many problems in the end.

That is an Architect or a Technical Lead who starts designing the system, you need to avoid creating critical connections between your security functionality and the business logic so that you could extend it when needed. If you decide to add the SSO functionality, several tenants, or multi-factor authentication in the future, you can do it with little effort.

If you take up a project with all these connections compiled, I can only wish you to be patient since there is no unique solution in this case, as I mentioned.

Mistake 2. Difficulties with data models

Another big problem is the incorrect usage of the database. And if you have a choice between SQL and NoSQL, you already have two problems.

We use the SQL database as a NoSQL solution. Several times, I came across solutions when the SQL database was used as a NoSQL solution. You might have witnessed such cases as well.

These were tables with 200–250 columns named column1, column2, etc. Of course, these tables are absolutely denormalized with no data consistency. Besides, you can come across combining a couple of such tables.

Another implementation of this anti-pattern looks as follows: there are only several columns, for example, id, refId, values type, and value. And, as you understand, the object is compiled by refId using many table lines, and then each object attribute obtains its value and type.

Of course, during both first and second implementation, there may occur some performance and code support issues.

In most cases, this situation is better than the one with security because such changes are easier to localize and fix. Create a separate code layer responsible for the permissions to this data, and, if you have an opportunity, use the NoSQL database for such cases. It is a perfect option if you do not need many join-operations or generate different types of reports.

If the project is new, it is obvious from the described cases that the best solution is to ask an Architect and Business Analyst to design data models. Otherwise, such minor things may lead to big problems on a project in the future.

And this is the answer to whether it is worth attracting Architects and Business Analysts to further implementation stages after the discovery, or the Developer Team will cope with the task themselves as expected.

We used the NoSQL database, but something went wrong. Sure, what is to think about here? Use the NoSQL database, and you have no issue anymore. It’s scalable and enables working with data structures—everything you put in there is safely stored. That’s what most small projects and start-ups think of. And it could be OK unless they turn into commercial projects with more complicated business logic.

Here are some examples of when Developers and Architects failed to understand that they needed to change the database before it was too late. If you notice this logic in your code, it is high time you thought about it too.

You make several calls to the NoSQL database and then collect the entity at the back end.

To save database data when changing one of the elements on the client’s side, you need to save data in several tables.

You store everything in a single table. User information, user roles and permissions, the organizations they belong to, all their personal data. But if you take a closer look at this table, you’ll see that in addition to users, there are organizations that are only different from users by a single isUser checkbox.

You are trying to implement the logic where the content is not supposed to know anything about the organization you belong to, and only the organization knows what content it possesses. You understand that it is impossible since the organization has references to the content, which, in its turn, has the organization ID. It is a kind of cyclic dependency.

Mistake 3. Our small start-up turned into a successful but huge monolith

Here, we can mention all the above-described issues and mistakes as well as the issues that will be described further in this article because such projects are a manual of highly not recommended practices. But I would like to add more of the psychological than a technical issue here.

You may have noticed that when you are trying to persuade a client to redevelop a part of the project, for example, to add certain logic to a separate service, you face some pushback from the side of an Architect or the company CTO. They seem to understand the need for the change, it seems efficient from the business perspective, but they do not dare make the first step.

It is because they wrote this code themselves three years ago. And how can you “kill” your own child? It is so good and, generally, it works well. All this ends up with “this last time, we’ll add another piece of functionality here, and next time we’ll definitely make a separate service out of it.”

And the story repeats itself every time. Each of such code pieces adds several weeks to the technical debt pool. Each of such solutions makes it more and more complicated and expensive to support the existing functionality and develop the new one.

You may have seen projects that attract up to twenty people for the single purpose of providing support and bug fixing. A Developer spends up to a week setting up the development environment from scratch. To find a bug and fix it, they need no less than another week. Thus, the start-ups with the “let’s add another piece of code here” principle turn into such monsters.

So, what are you supposed to do about it? You have at least two options here.

For small projects and teams—when the moment comes to change the project design, on a Friday evening the Technical Lead instructs the whole team not to touch the code before Monday, takes a pizza, some coffee, and changes the project significantly during the weekend. I remember doing so myself.

For big commercial companies—you need to have a pool of projects with technical debts that will list all the necessary changes in architecture and design. The team then will spend part of their working time (say, 10–15%) closing the issues on this list.

Mistake 4. We are not yet in production but are already facing performance issues

It is a typical mistake. I wouldn’t have believed that if I hadn’t seen similar cases myself.

We once got a request to enhance project performance. The project wasn’t yet in production, but it was tested by 200 users, and some performance issues were detected. The project launch was planned in several months, and another 5–10 thousand users were expected in a few more months.

We started working. At once, we detected many performance issues, and we could tell by analyzing the code that no one really tried to write it for some reasonable workload. It was a surprise since the people who developed the code could have done a much better job. We had to figure out the reason why that happened.

It turned out that the code was developed for proof of concept, and it was not designed for any commercial projects use. However, the customer had access to the code as all work was done in the client’s environment. So, the customer decided to accomplish it and launch it in production. After they’d tested it using little workload, they realized that their approach did not work and decided to enhance the performance.

All wise books tell you that if you work with some PoC or Spike, you need to have a code trunk. You should throw it away and never use it in a commercial project.

Here is another test case. Once upon a time, before New Year’s Eve, my colleague and I were writing a high-quality code for a commercial project. It took us both two weeks to work on it. When we started testing it, we realized that the performance was twice lower than expected. Upon giving it some of our analysis, we found a better performance solution. Our mistake was that we hadn’t done a quick РoС and did not test the performance before writing the code. But the take-away message here is different.

We were not ashamed to admit that the approach was not good: we made a mistake and wasted one month of work. We could have hidden it, added some enhancement, and nobody wouldn’t notice right away.

It would have become an issue in production with the heavy workload. But what would it cost?

So, don’t be ashamed of throwing away the code that either wasn’t intended for a commercial project or wasn’t good enough and might have led to issues in the future. It’s much better than to doom the Developers supporting it to endless torture.

Mistake 5. We are in production, but we still do not know why we have performance issues

In such cases, everything (or almost everything) is written well, just as it should be in a commercial project. Still, there occur performance issues. It worked OK when there were 500 users. But when there appeared 200 thousand users, the performance slowed down for no reason. Have you come across such situations?

The primary mistake of such projects is that they start thinking about the performance too late. The following questions should be asked at the very start of the development:

What is the expected workload?
How do you expect the workload to change within a year, two, or three?
From the performance perspective, how do you know that your code is bad?
What is the maximum workload for the application?
How do you detect bugs when you face performance issues?

Here are my recommendations for such cases. It is, of course, better to implement them at the development stage, but they will also be handy with the ready-made project.

When you only start developing an application, think about how you are going to measure the performance metrics on the database level, services, infrastructure, etc. There are some commercial solutions like New Relic, Datadog, etc., which will help you set everything up in a short time. If you do not want to pay for them, you can deploy similar systems yourself, but this will take more time. You can use the ELK (Elasticsearch, Logstash, Kibana) or EFK (Elasticsearch, Fluentd, Kibana) stack.

Set up static code analyzers like SonarQube. They will help you find typical mistakes related to the code performance, recursive calls, maybe, endless cycles, etc. Of course, you will have to search for the most intricate mistakes manually, but the analyzer’s function is to make your code cleaner and better, at least partially.

Do not forget about performance and load testing. Make these tests as realistic as possible. If your task is to test a thousand simultaneous users, do not try to make your life easier and make a thousand calls by one user, it is not the same thing. If you have already done performance testing, check different scenarios so you could have some spare capacity. It may work OK for a thousand users but may cause severe performance issues with more than a thousand users.

In our project, we had a situation when we forgot to disable the load tests for the weekend. On Monday morning meeting, the Tech Lead told us there are good news and bad news. The bad one was that we spent an extra $400 on Amazon resources, and the good news was that our application could tolerate ten times as much load as we potentially expected to have in production.

Think when writing the code and always perform code reviews since it is always a human who finds the most intricate mistakes. We can still see cases of dragging all 100500 entities out of a table and, afterward, performing front-end pagination.

Mistake 6. Clients detect errors themselves and point them out to the development team

This last piece of advice may sound trivial, sad, and funny at the same time, but trust me, it can hardly be overestimated.

Be responsible when you perform logging. It is an easy thing to do. It’s not difficult and can be performed everywhere: on an old project, a new project, a monolith, or microservices. There are plenty of commercial solutions that can help you with logging.

Do you think it is easy? Well, maybe, but according to my experience, about 20–30% of projects still face logging issues.

I’ll tell you about this process based on the example of a project. All errors were caught by a single handler, which tried to send them to the administrator’s email. The email address was written in the code itself. Of course, most dev, stage, and test environments were not configured. However, it seemed that even for the production environment, this approach did not work well. It happened because the error notification and fixing process looked as follows: a client faced an application issue, called the technical support, and informed them about the problem. The Developer explained to the client how they should open the developer dashboard in a browser and take a screenshot so it could display errors both in the application graphical user interface and in the developer console. The client sent this screenshot to the support team for further investigation. So, isn’t it easier to set up logging?

This article touches upon topics that deserve separate articles or even books and cannot be fully described on several pages. However, I did my best to show you the most common mistakes you face on the way to project success from both architecture design and business perspectives. It’s a sort of a checklist of the things you must pay attention to.

I hope you will learn from the mistakes of others and be successful in running your own projects.

Typical mistakes in architecture design

Mistake 1. Disregarding security when designing the architecture

Mistake 2. Difficulties with data models

Mistake 3. Our small start-up turned into a successful but huge monolith

Mistake 4. We are not yet in production but are already facing performance issues