Hey there! My name is Daniel Restrepo Hincapie. I am a Big Data Engineer at SoftServe Colombia and have been in this field for over four years. In this article, I would like to lead you through my journey from on-premises technologies with the Apache Hadoop environment to Cloud technologies with AWS (Amazon Web Services). I will talk about the knowledge and skills I had to master for it and share a few handy insights that may be useful for those planning to start this journey to a Big Data career. So, let's go!

How to become a Big Data Engineer: my path from Apache Hadoop to AWS?

I started this adventure 4 years ago when Apache Hadoop was still a strong technology in the field, especially in the financial industry, where there are enormous amounts of data from varied sources. Among my responsibilities as a Big Data Engineer there, I had the opportunity to learn how a distributed architecture with the ability to be scalable is designed, used, and implemented.

Overall, I learned the importance of knowing and exploiting your data as much as possible to generate more excellent value in any operation. It sounds very casual for any Big Data Engineer's responsibilities. Still, at that time, it was a huge step, considering that when you start working as a Big Data Engineer, you'll encounter as many technologies as you can use, like Data ingestion tools, ETL tools, and Data Marts for data storage.

You will learn that, although many resources are available, sometimes they are not enough to ingest the data from certain sources, leading the team to think creatively to find efficient solutions. It was challenging but quite fun!

Later, I encountered myself working with the cloud, specifically with AWS. I focused more on data processing. The golden rule is that every implementation must be meticulously calculated and designed to get the most out of the services without burning out the entire year's budget in a single execution.

Without hesitation, I would say that "budget" is the most important word I learned in my first days on the cloud. Any cloud, in theory, gives you unlimited resources for the business. Still, it is expensive, so everything implemented in the cloud must be carefully designed to be as efficient as possible.

What does a Big Data Engineer do?

Seeking to venture into AWS, I was introduced to the opportunity to work with other services oriented toward modular computing, infrastructure, and automation. This led me to fully use the knowledge I learned about Big Data and distributed systems and broadened my horizons.

With an insatiable curiosity, I started working on this new opportunity. I had to face data challenges and implement solutions involving the entire infrastructure behind the data solutions, such as integrating efficient monitoring of services, resources, and data pipelines. Also, I had challenges establishing granular access policies on pipeline nodes to limit their communication, access, and integration to the minimum necessary design architectures to integrate the Data Lake and Data Warehouse to migrate and ingest data in real-time pipelines.

Currently, I am working more on the data side, focusing on converting it into valuable information for the business (without neglecting the other concepts I have learned). I still manage copious amounts of data, so I must consider the efficient use of limited resources when building SQL queries. Additionally, for other types of solutions, I must also ingest the data into the Data Lake since, for these cases, it is necessary to consume the data from that layer. So, I must build the pipelines to ingest data from SQL and non-SQL sources, which involves automating the deployments of those pipelines, so I apply CI/CD technologies in conjunction with Git repositories. The new thing that I am learning is more about the data consumption side, how to analyze it, how to prepare it, and how to transform it according to the requirements proposed by the business.

What have I learned on my career path?

The next thing I have told you about based on my experience is the following question: how to meet Big Data job requirements, and what's next? Searching blogs and courses, I found my answer: a DataOps Engineer (I leave the manifesto in case you want to have a look).

A DataOps Engineer combines all the concepts of a Big Data Engineer with concepts from DevOps and Data Analytics to streamline the delivery of analytics to the point of creating manufacturing lines on existing pipelines. This means the architecture must be designed to be as modular as possible. It should be that way to allow each node or layer to be consumed by other nodes that will be added during the lifecycle of the initial architecture but in a continuous integration manner. This involves not only the design of the Big Data solution. Integration solutions and continuous deployments must be included in the components both involved in the ETL/ELT that transport the data to DataMarts and the analytical models that will add value. Thus, leaving the bases for the consumers (ML/IA Models, reports, etc.) of the final data must be easily integrated into the manufacturing line.

I know what you are thinking about - that is too much! But that is where the interesting thing comes in. Imagine taking data from any source until the analyzed data is delivered to the business or creating a hub where every party within the organization can consume from that system and feed it back at the same time. The potential is enormous! Bearing this in mind, if you are still interested in continuing to the end of the rabbit hole, here is what I have learned so far to get into the DataOps realm, and I wish to hear it from someone in the past.

Skills you need to learn

Python will become your best ally. Yes, there are many other programming languages, more powerful and faster than Python, but this language offers something that I have seen little in other languages. And I mean versatility. The author and the Python community have endeavored to make it a language to solve any problem. If you need to implement an API, you can use FastAPI, Django, and Flask, among others; if you need Analytics and Data Processing, there is PySpark (Apache Spark in Python), pandas, NumPy, pyramid, among others. If you need distributed applications, there's PySpark, Celery, distributed, and endless libraries for endless applications.

Never skip Linux Programming, despite this command line interface's time in the market (30+ years). This is the most widely used programming language across the IT community. Therefore, the most likely scenario is that you will need it to address or complement the solution to a challenge. That's why, believe me when I say this is a MUST.

Learn SQL, but I am not discussing a vendor or a DB engine. I am talking about the general concept of SQL (Structured Query Language). It will allow you to understand better the data you have in front of you once you work with any relational database or data mart. Besides that, it will also allow you to manipulate, transform, query, filter, and the list goes down. You can learn No-SQL, which is the next level, but other databases do not depend on a pre-established relational model for storing data. Having a broad experience with them is nice, but just being able to identify them and knowing how each type of these databases works is enough.

Learn how distributed systems work and how communication works in Web Systems. This is vital to know at least the general concept because when working with solutions that need to be distributed, you must understand how everything works underneath, how the nodes are communicating with each other, and how they manipulate the data according to the defined code. A better understanding of all the machinery behind the scenes, applications, and processes can lead to a better design that adjusts to the available resources and specifications, which means doing more with less.

Finally, remember, it never hurts to learn other programming languages, like JavaScript, NodeJS, or Scala, for applications that require the best performance in a more streaming/real-time manner.

Let's keep forward

Necessary to learn, Data Modeling is the process of diagraming software systems and how they interact with each other using distinct levels of technical complexity according to the stakeholders that are going to be involved in the project.

Now let's move into the cloud. You can learn any cloud, but just one is enough. It is even much better to specialize in one cloud and, if the opportunity arises, to learn from another. The good thing is that all clouds are similar and offer the same services under different names, so the learning curve is flattened when starting with a new cloud.

To continue the DataOps way, you must remember that data analytics is necessary to deliver the data in its final state. I recommend you study this field and start putting it into practice with exercises, for example, here.

Besides, to start implementing manufacturing lines of products from data analytics, it is highly recommended to learn CI/CD. The easiest and practically free way is to learn by practicing with GitLab CI or Github actions. Both platforms offer free Tiers for implementing CI pipelines if you do not exceed the monthly fee.

Having a clear concept of what CI/CD is, it is good to learn about Data Pipelines and Pipeline Automation. In particular, how to create an Architecture to process data in a series of steps where each is the input of the next or many others.

Finally, the most important thing is to study the use and cases where this architecture is implemented. Although likely, you will not find the exact solution from these texts in any challenge that arises. The intention is to internalize the concept behind these architectures.

Read our previous blog about Big Data use cases: https://career.softserveinc.com/en-us/stories/4-big-data-use-cases-from-softserve-how-to-help-client-companies-thrive

For those who want to dive deeper

In recent years one of the practices that have won more traction in the IT industry is DevOps. It combines philosophies, techniques, and technologies for delivering applications and services at optimal speed, constantly evolving and improving them on the fly to provide a better experience. Translating this to the data world, offering services using DevOps technologies like Airflow or Jenkins, automating, and using CI/CD for continuously evolving those products over time, plus other benefits this practice can offer.

So, the first steps at this point:

Architectures of Data Marts, Data Warehousing, and Data Lakes. Learn these types of data stores, how to differentiate them, why to implement them, and how to use them optimally.
Data Governance will teach you how to successfully organize your data when dealing with ridiculous amounts of it and solve conflicts with inconsistencies, misuse, or uncompliant data.
Data Lineage. This topic goes hand in hand with Data Governance, so it is also good to learn it. Its focus is to uncover the life cycle of data. Hence, it aims to show the complete data flow to understand where to begin and where to finalize and identify improvement opportunities over the different points of the life cycle.
Last but not least, FinOps, is a practice mainly focused on the cloud, specifically in the management of the finances of your Cloud ecosystem. In a few words, this will help you to avoid running out of money or burning out the entire budget when dealing with high-end and highly scalable architectures.

Conclusions

To conclude, I must mention that the journey to reach the DataOps career path realm is long and requires varied knowledge and many hours of study. But keep forward that it is one of the most enriching journeys that I have had.

If you have gotten this far in reading, it is because my message made you curious about what DataOps is. So, regardless of your role or knowledge related to these topics, I urge you to start this journey since I can promise you that you will enjoy it. Without further ado, Happy Coding!