NGOPI IT Explores Big Data Fundamentals: Building a Strong Foundation for the Data Engineering Era
Bandung, July 6, 2026 – Amid the accelerating pace of digital transformation, data has become one of the most valuable strategic assets for organizations across every industry. Yet before discussing Artificial Intelligence, Machine Learning, or advanced analytics, there is one essential foundation that every technology professional should understand: Big Data. Continuing its commitment to fostering a culture of continuous learning, PT Tech Mayantara Asia (TMA) once again hosted its regular NGOPI IT (Ngobrol Pintar IT) knowledge-sharing session, this time featuring the theme "Big Data Fundamental." The session was presented by Akhmad Sofyan, affectionately known as Mang Made, who guided participants through the fundamental concepts of Big Data, its characteristics, architecture, technology ecosystem, and real-world industry applications.
The event was held on Monday, July 6, 2026, starting at 4:00 PM WIB, in a hybrid format with participants joining both offline at the Magna Office in Bandung and online via Discord. This hybrid approach has become a hallmark of TMA's internal learning culture, supporting the company's flexible work environment. The session attracted software engineers, data engineers, data analysts, interns, university students, and professionals from various departments eager to deepen their understanding of the technologies that power today's data-driven world.
From the very beginning, the atmosphere reflected the signature style of NGOPI IT—relaxed, interactive, and highly engaging. Mang Made opened the session with a simple but thought-provoking question: "Why did Big Data emerge?" Rather than immediately diving into technical definitions, he invited participants to reflect on how dramatically the digital world has changed over the past decade. He explained that Big Data did not arise simply because data volumes became larger, but because of the explosive growth of digital information generated every second by smartphones, social media platforms, IoT devices, industrial sensors, cloud services, and online transactions. Platforms such as Google Search, TikTok, Shopee, Gojek, and countless digital services continuously generate millions of new records every minute, creating datasets far beyond the capabilities of traditional databases or spreadsheet applications like Microsoft Excel.
Mang Made then introduced the concept of Big Data in a straightforward and practical way. Big Data is not merely "large amounts of data." Instead, it refers to datasets that have become too large, too complex, or too fast-moving to be efficiently processed using conventional technologies. This shift requires entirely new approaches to data storage, processing, and analysis through distributed computing architectures. His explanation helped participants understand that Big Data represents a paradigm shift rather than simply an increase in storage capacity.
The discussion continued with one of the most fundamental concepts in Big Data: the well-known 5V model, which has evolved into 7V in many modern interpretations. Mang Made carefully explained each characteristic using relatable examples. Volume represents the enormous scale of today's data, ranging from terabytes to zettabytes. Velocity refers to the speed at which data is generated and processed in real time. Variety highlights the diversity of modern data, including structured databases, text documents, images, videos, audio, and semi-structured formats. Beyond these, he discussed Veracity, emphasizing data quality and reliability; Value, which reminds organizations that data only becomes useful after being transformed into meaningful insights; Variability, referring to changing data patterns depending on context; and Visualization, which focuses on presenting complex information in forms that decision-makers can easily understand.
The session became even more engaging as Mang Made compared Big Data systems with traditional data management approaches. Conventional databases typically handle gigabytes or a few terabytes of structured information stored on a single server using relational database management systems (RDBMS) and SQL queries. In contrast, Big Data platforms are designed to process terabytes, petabytes, and even exabytes of structured, semi-structured, and unstructured data distributed across multiple servers. Technologies such as Apache Hadoop and Apache Spark enable parallel processing at scales that traditional database systems simply cannot support. This comparison gave participants a clear understanding of why conventional approaches are no longer sufficient for modern enterprise analytics.
One of the highlights of the session was the introduction to the Big Data technology ecosystem. Mang Made explained that Big Data is not a single software product but rather a comprehensive ecosystem composed of specialized technologies working together. For data storage, platforms such as HDFS, Amazon S3, and Google Cloud Storage provide scalable repositories. Processing is handled by engines including Apache Hadoop MapReduce, Apache Spark, and Apache Flink. Data ingestion is supported by technologies such as Apache Kafka, Apache Flume, Sqoop, and Apache NiFi, while analytical queries can be executed through Hive, Presto, Spark SQL, and Impala. Finally, business insights are presented using visualization tools such as Apache Superset, Tableau, Power BI, and Looker. This overview helped participants appreciate how each technology fulfills a distinct role while collectively forming a complete Big Data platform.
For TMA, the discussion was particularly relevant because the company is actively developing its own TeMA BigData Platform. Many participants naturally connected the concepts introduced during the presentation with TMA's internal technology stack, which includes Apache Spark, Apache Iceberg, Trino, Apache Superset, Apache Airflow, Apache NiFi, OpenMetadata, and other open-source technologies. The conversation expanded toward how these components integrate into a Modern Lakehouse Architecture, enabling enterprises to manage large-scale analytical workloads efficiently while maintaining governance, scalability, and performance.
The presentation then explored the fundamental architecture of a Big Data platform, describing the complete lifecycle of data. The journey begins with Data Sources, including databases, application logs, IoT devices, APIs, and external systems. Data is then collected through the Ingestion Layer, utilizing tools such as Apache Kafka, Apache Flume, Sqoop, and Apache NiFi. Once ingested, the information is stored in distributed storage systems like HDFS or Amazon S3 before entering the Processing Layer, where engines such as Apache Spark and Hive perform transformations and analytics. The processed information is subsequently analyzed through visualization platforms including Apache Superset and Tableau, ultimately generating actionable insights that support strategic business decisions. This simple architectural flow helped participants understand how raw data is transformed into meaningful business intelligence.
To demonstrate that Big Data extends far beyond theoretical concepts, Mang Made presented several real-world industry use cases. In e-commerce, Big Data powers recommendation engines that suggest products based on customer behavior. Within the transportation sector, companies leverage real-time data to calculate dynamic pricing according to traffic conditions and demand. In healthcare, Big Data helps predict disease outbreaks by analyzing social media activity and search engine trends. The banking industry relies on Big Data to detect fraudulent transactions in real time, while government institutions utilize it for national data integration initiatives and social assistance analysis. These practical examples illustrated how Big Data has quietly become an essential part of everyday life across multiple industries.
Beyond discussing opportunities, Mang Made also addressed the major challenges associated with Big Data adoption. On the technical side, organizations require sophisticated infrastructure, distributed computing platforms, and engineers with specialized expertise. Non-technical challenges include data privacy, cybersecurity, governance, and compliance with regulations such as Indonesia's Personal Data Protection (PDP) Law. Perhaps the greatest challenge, however, lies in developing human talent. Successful Big Data initiatives require highly skilled Data Engineers, Data Scientists, Data Analysts, and data architects capable of designing and managing complex analytical platforms. According to Mang Made, investing in people is just as important as investing in technology itself.
As with previous NGOPI IT sessions, the discussion concluded with an energetic Q&A session. Participants exchanged experiences related to data warehouses, Apache Spark optimization, data lake implementation strategies, and the evolution from Hadoop-based systems toward modern Lakehouse architectures. The atmosphere became even livelier with humorous engineering stories about troubleshooting complex data pipelines, debugging distributed systems, and handling billions of records in production environments. Laughter frequently accompanied the discussion, yet every conversation remained rich with practical technical insights.
Discord once again proved to be an effective collaboration platform, allowing online participants to engage seamlessly with colleagues attending the event in person. Engineers from different locations shared questions, ideas, and experiences in real time, demonstrating how TMA's flexible work culture extends naturally into its learning ecosystem.
Through NGOPI IT: Big Data Fundamental, PT Tech Mayantara Asia reaffirmed its commitment to cultivating a culture of lifelong learning and technological excellence. At TMA, mastering technology is not simply about adopting the latest tools—it begins with understanding the underlying principles that enable organizations to build sustainable, scalable, and impactful solutions. Big Data serves as one of the most critical foundations for digital transformation, supporting advanced technologies such as Artificial Intelligence, Machine Learning, predictive analytics, and intelligent decision-making systems.
More than just another technical seminar, NGOPI IT has evolved into a collaborative learning platform that strengthens TMA's engineering culture. It brings together software engineers, developers, data specialists, product enthusiasts, interns, and students to exchange ideas, share practical experiences, and grow together. This spirit of continuous learning and open collaboration continues to drive TMA's innovation journey, empowering the company to build next-generation data platforms while preparing digital talent for the challenges of tomorrow's technology landscape.