Commonly used terms in Hadoop / Big Data are described in this post.
VMWare:
A hypervisor is installed on the physical server using VMware server virtualization to allow many virtual machines (VMs) to run on the same server. Each virtual machine (VM) can run its own operating system (OS), allowing many operating systems to coexist on a single physical server. The networking and RAM resources are shared among all VMs on the same physical server.
Virtual Machines:
A virtual machine is a software computer that runs an operating system and applications just like a physical computer. The virtual machine is made up of a set of specification and configuration files that are backed up by a host’s physical resources.
Shell Scripting:
A shell script is a computer program that runs on the Unix shell, which is a command-line interpreter. Shell scripts come in a variety of dialects, which are referred to as scripting languages. Shell scripts can execute a variety of tasks, including file manipulation, program execution, and text output.
Apache Hadoop:
Apache Hadoop is an open-source software platform that uses the MapReduce programming model to share storage and processing of large data collections. It’s made up of commodity-hardware-based computer clusters. All of Hadoop’s modules are built on the idea that hardware failures would occur frequently and should be handled automatically by the framework.
Apache Hadoop is made up of two parts: a storage system called Hadoop Distributed File System (HDFS) and a processing system called MapReduce. Hadoop divides files into large blocks and distributes them among cluster nodes. The packaged code is subsequently transferred to nodes, which process the data in parallel.
HDFS:
HDFS, or Hadoop Distributed File System, is a distributed file system that runs on commodity hardware. HDFS, or Hadoop Distributed File System, is a distributed file system that runs on commodity hardware. It has a lot in common with other distributed file systems. However, there are considerable distinctions between it and other distributed file systems. HDFS is meant to run on low-cost hardware and is highly fault-tolerant. HDFS is a file system that allows high-throughput access to application data and is well-suited to applications with huge data collections.
MapReduce:
The Apache Hadoop software architecture includes MapReduce as a major component. Hadoop allows for the resiliency and distributed processing of large unstructured data sets over commodity computer clusters, with each node having its own storage. MapReduce does two key tasks: it distributes work among nodes in a cluster or map, and it organizes and reduces the output from each node into a unified response to a query.
Apache Hive:
Apache Hive is built on top of Apache Hadoop, which is a distributed, fault-tolerant, and open source data warehouse platform for reading, writing, and handling massive datasets stored directly in HDFS or other data management structures such as Apache HBase. Hive is characterized by the ability to query massive datasets using Apache Tez or MapReduce.
Apache Impala:
Impala is a massively parallel query engine that runs on hundreds of servers in existing Hadoop clusters. Unlike standard relational database management systems, where query processing and the underlying storage engine are tightly coupled, it is decoupled from the latter.
Impala improves the efficiency of SQL queries on Apache Hadoop while maintaining a familiar user interface. Impala allows you to query data in real time, whether it’s stored in HDFS or Apache HBase, using SELECT, JOIN, and aggregate functions. Impala also shares Apache Hive’s metadata, SQL syntax (Hive SQL), ODBC driver, and user interface (Hue Beeswax), making it a familiar and coherent platform for batch and real-time queries.
Apache Pig:
Pig is a high-level scripting language used in Apache Hadoop. Pig allows to write dynamic data transformations without the awareness of Java. Pig’s basic SQL-like scripting language is called Pig Latin and appeals to developers already acquainted with SQL and scripting languages.
Apache Pig is a data analysis platform that comprises of a high-level language for defining data analysis algorithms and infrastructure for assessing these programs. Pig programs are notable for their structure, which allows for significant parallelization and, as a result, the handling of very large data sets.
Pig’s infrastructure layer now consists of a compiler that generates Map-Reduce program sequences for which large-scale parallel implementations already exist (e.g., the Hadoop subproject). Pig’s language layer currently consists of Pig Latin, a textual language with the following basic characteristics:
Apache Spoop:
SQL-to-Hadoop is a Hadoop ecosystem component and an ETL tool that offers the capability to extract data from various structured data stores such as relational databases with the help of map-reduce. This command-line interpreter works efficiently to transfer a huge volume of the data to the Hadoop file system (HDFS, HBase, and Hive). Similarly, it exports the data from Hadoop to SQL.
Apache Zookeeper:
Apache ZooKeeper is an open-source server that provides a highly reliable distributed coordination service for managing distributed frameworks with many hosts, such as Hadoop, Hbase, and others. In short, it is a tool to manage the cluster. It’s fail-safe synchronization approach is used to handle race conditions and deadlocks. And data inconsistency is handled with atomicity.
Apache Kafka:
Apache Kafka is a distributed data storage designed for real-time ingest and processing of streaming data. Streaming data is the data that is continuously generated by thousands of data sources, all of which transmit data records in at the same time. A streaming platform must be able to cope with the constant inflow of data and process it sequentially and progressively.
Kafka is used by businesses for a number of tasks, including building ETL pipelines, data synchronization, real-time streaming, and more.
Apache Hue:
Hue is an open source Hadoop user interface with the most intelligent auto-completes and Query Editor components, allowing you to interact with data warehouses.
Apache Oozie:
Apache Oozie is a Hadoop task management solution that runs on a server. Workflows in Oozie are defined as a directed acyclic graph with a collection of control flow and action nodes. The beginning and conclusion of a workflow, as well as a mechanism to control the workflow execution path, are defined by control flow nodes.
Oozie assists in the building of complex data transformations from numerous component tasks.
Apache Flume:
Apache Flume is a distributed, reliable, and flexible open-source system for collecting, aggregating, and transferring huge volumes of log data effectively. Its design is simple and flexible, based on streaming data flows. Flume is a tool that allows you to reliably stream log and event data from web servers into Apache Hadoop.
Apache Solr:
Solr is a scalability and fault tolerance distributed search and index replication solution. Solr is a prominent open-source enterprise search and analytics use cases with a thriving development community and regular updates. Full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, NoSQL features, and rich document (Word, PDF) handling are some of its key features.
Apache Spark:
Apache Spark is a free and open-source unified analytics engine for processing large amounts of data. Spark provides a programming interface for entire clusters with implicit data parallelism and fault tolerance. It is used for large-scale data workloads. For fast analytic queries against any size of data, it uses in-memory caching and optimized query execution. Java, Python, Scala, and R are among the programming languages supported by Spark. Spark allows application developers and data scientists to query, analyze, and transform data at scale quickly and easily.
Apache Spark SQL:
As stated above, SQL is one of the language supported by Apache Spark. It is used to work with structured data, either within Spark programs or through standard JDBC and ODBC connectors.
Machine Learning (R, Mahout):
Machine learning is a type of data analysis that automates the creation of analytical models. It’s a field of artificial intelligence based on the premise that computers can learn from data, recognize patterns, and make judgments with little or no human input. It is used in internet search engines, spam filters, banking software that detects suspicious transactions, speech recognition, etc.
NoSQL (HBase, MongoDB, Cassandra, Riak, CouchBase, etc):
A NoSQL database allows data to be stored and retrieved using methods other than tabular relations. NoSQL is a non-relational database management system (DBMS) that does not require a fixed schema, eliminates joins, and is scalable. For distributed data repositories with large data storage requirements, NoSQL DB is utilized. Big data and real-time web apps both employ NoSQL.
NoSQL is divided into four categories: Column Based (BigTable, Cassandra, Hbase, etc.), Graph (Neo4J, InfoGrid, Flock DB, etc.), Document (MongoDB, CouchDB, RavenDB, etc.), and Key-Value (Riak, Memcached, Redis Serveamr, etc.), to deal with structured, semi-structured, unstructured, and polymorphic data.
I hope you found this post to be informative.
By subscribing to this site, you will receive regular updates on new articles that will cover practically all aspects of the Hadoop ecosystem with various examples.
Big Data and SQL 
2 comments