Internals of Spark 2.0 is explored in great detail. Programing paradigm of spark is given due importance. RDDs are compared to Dataframes and Data set. High performance programming patterns are explored with complex problems.

• Interfaces

• Hadoop Filesystems

• The Design of HDFS

Using Hadoop Archives

• Limitations

Parallel Copying with distcp

• Keeping an HDFS Cluster Balanced

• Hadoop Archives

Data Flow

• Anatomy of a File Write

• Anatomy of a File Read

• Coherency Model

The Command-Line Interface

• Basic Filesystem Operations

The Java Interface

• Querying the Filesystem

• Reading Data Using the FileSystem API

• Directories

• Deleting Data

• Reading Data from a Hadoop URL

• Writing Data

Data Integrity

• ChecksumFileSystem

• LocalFileSystem

• Data Integrity in HDFS

Serialization

• Implementing a Custom Writable

• Serialization Frameworks

• The Writable Interface

• Writable Classes

• Avro

ORC Files

• Large size enables efficient read of columns

• New types (datetime, decimal)

• Encoding specific to the column type

• Default stripe size is 250 MB

• A single file as output of each task

• Split files without scanning for markers

• Bound the amount of memory required for reading or writing.

• Lowers pressure on the NameNode

• Dramatically simplifies integration with Hive

• Break file into sets of rows called a stripe

• Complex types (struct, list, map, union)

• Support for the Hive type model

ORC File:Footer

• Count, min, max, and sum for each column

• Types, number of rows

• Contains list of stripes

ORC Files:Index

• Required for skipping rows

• Position in each stream

• Min and max for each column

• Currently every 10,000 rows

• Could include bit field or bloom filter

ORC Files:Postscript

• Contains compression parameters

• Size of compressed footer

ORC Files:Data

• Directory of stream locations

• Required for table scan

Parquet

• Nested Encoding

• Configurations

• Error recovery

• Extensibility

• Nulls

• File format

• Data Pages

• Motivation

• Unit of parallelization

• Logical Types

• Metadata

• Modules

• Column chunks

• Separating metadata and column data

• Checksumming

• Types

File-Based Data Structures

• MapFile

• SequenceFile

Compression

• Codecs

• Using Compression in MapReduce

• Compression and Input Splits

Apache Tez

• Apache Tez: A New Chapter in Hadoop Data Processing

• Data Processing API in Apache Tez

• Writing a Tez Input/Processor/Output

• Runtime API in Apache Tez

• Apache Tez: Dynamic Graph Reconfiguration

Apache YARN

• Agility

• global ResourceManager

• per-node slave NodeManager

• Scalability

• Support for workloads other than MapReduce

• Compatibility with MapReduce

• per-application Container running on a NodeManager

• Improved cluster utilization

• per-application ApplicationMaster

HDFS-2

• High Availability for HDFS

• HDFS-append support

• HDFS Federation

• HDFS Snapshots

• GraphX

• MLlib

• Spark SQL

• Data Processing Applications

• Spark Streaming

• What Is Apache Spark?

• Data Science Tasks

• Storage Layers for Spark

• Spark Core

• Who Uses Spark, and for What?

• A Unified Stack

• Cluster Managers

• Lazy Evaluation

• Common Transformations and Actions

• Passing Functions to Spark

• RDD Operations

• Creating RDDs

• Actions

• Transformations

• Scala

• Java

• Persistence

• Python

• Converting Between RDD Types

• RDD Basics

• Basic RDDs

• Expressing Existing Programming Models

• Fault Recovery

• Interpreter Integration

• Memory Management

• Implementation

• MapReduce

• RDD Operations in Spark

• User Applications Built with Spark

• Google's Pregel

• Console Log Minning

• Iterative MapReduce

• Behavior with Insufficient Memory

• A Fault-Tolerant Abstraction

• Support for Checkpointing

• Evaluation

• Spark Programming Interface

• Job Scheduling

• Advantages of the RDD Model

• Understanding the Speedup

• Leveraging RDDs for Debugging

• Iterative Machine Learning Applications

• Explaining the Expressivity of RDDs

• Representing RDDs

• Applications Not Suitable for RDDs

• Sorting Data

• Determining an RDD’s Partitioner

• Operations That Affect Partitioning

• Grouping Data

• Motivation

• Aggregations

• Data Partitioning (Advanced)

• Actions Available on Pair RDDs

• Joins

• Creating Pair RDDs

• Operations That Benefit from Partitioning

• Transformations on Pair RDDs

• Example: PageRank

• Custom Partitioners

• File Formats

• Hadoop Input and Output Formats

• Local/“Regular” FS

• Text Files

• Java Database Connectivity

• Structured Data with Spark SQL

• Elasticsearch

• File Compression

• Apache Hive

• Cassandra

• Object Files

• Comma-Separated Values and Tab-Separated Values

• HBase

• Databases

• Filesystems

• SequenceFiles

• JSON

• HDFS

• Motivation

• JSON

• Amazon S3

• Working on a Per-Partition Basis

• Accumulators

• Optimizing Broadcasts

• Custom Accumulators

• Accumulators and Fault Tolerance

• Numeric RDD Operations

• Piping to External Programs

• Broadcast Variables

• Scheduling Within and Between Spark Applications

• Spark Runtime Architecture

• A Scala Spark Application Built with sbt

• Packaging Your Code and Dependencies

• Launching a Program

• A Java Spark Application Built with Maven

• Hadoop YARN

• Deploying Applications with spark-submit

• The Driver

• Standalone Cluster Manager

• Cluster Managers

• Executors

• Amazon EC2

• Cluster Manager

• Dependency Conflicts

• Apache Mesos

• Which Cluster Manager to Use?

Spark:YARN Mode

• Resource Manager

• Node Manager

• Workers

• Containers

• Threads

• Task

• Executers

• Application Master

• Multiple Applications

• Tuning Parameters

Spark:LocalModeSpark Caching

• With Serialization

• Off-heap

• In Memory

Running on a Cluster

• Scheduling Within and Between Spark Applications

• Spark Runtime Architecture

• A Scala Spark Application Built with sbt

• Packaging Your Code and Dependencies

• Launching a Program

• A Java Spark Application Built with Maven

• Hadoop YARN

• Deploying Applications with spark-submit

• The Driver

• Standalone Cluster Manager

• Cluster Managers

• Executors

• Amazon EC2

• Cluster Manager

• Dependency Conflicts

• Apache Mesos

• Which Cluster Manager to Use?

Spark SerializationStandAlone Mode

• Task

• Multiple Applications

• Executers

• Tuning Parameters

• Workers

• Threads

Effective Transformations

• Shared Variables

• Using Smaller Data Structures

• Narrow Versus Wide Transformations

• Reducing Setup Overhead

• Deciding if Recompute Is Inexpensive Enough

• Accumulators

• Implications for Fault Tolerance

• LRU Caching

• Space and Time Advantages

• What Type of RDD Does Your Transformation Return?

• Cases for Reuse

• Implications for Performance

• Minimizing Object Creation

• Alluxio (nee Tachyon)

• Interaction with Accumulators

• Noisy Cluster Considerations

• Reusing Existing Objects

• Types of Reuse: Cache, Persist, Checkpoint, Shuffle Files

• Broadcast Variables

• Set Operations

• Reusing RDDs

• The Special Case of coalesce

Working with Key/Value DataThe Powerful Iterator-to-Iterator transformation

• Reduce to Distinct on Each Partition

• A Different Approach to Key/Value

• Straggler Detection and Unbalanced

• Sort on Cell Values

Secondary Sort SolutiongroupByKey Solution

• Multiple RDD Operations

• Choosing an Aggregation Operation

• Sorting by Two Keys with SortByKey

• Preserving Partitioning Information Across Transformations

• Dictionary of OrderedRDDOperations

• Leveraging Co-Located and Co-Partitioned RDDs

• Leveraging repartitionAndSortWithinPartitions for a Group by

• Key and

• Range Partitioning

• Dictionary of Mapping and Partitioning Functions

• PairRDDFunctions

• Sort Values Function

• Partitioners and Key/Value Data

• Co-Grouping

• Custom Partitioning

• Secondary Sort and repartitionAndSortWithinPartitions

• Using the Spark Partitioner Object

• Hash Partitioning

• Dictionary of Aggregation Operations with Performance

• Considerations

Iterative Solution

• What’s So Dangerous About the groupByKey Function

• How to Use PairRDDFunctions and OrderedRDDFunctions

• Actions on Key/Value Pairs

How Not to Sort by Two Orderings

Spark SQL—Structured Queries on Large Scale

• UserDefinedAggregateFunction—User-Defined Aggregate

• Functions (UDAFs)

• User-Defined Functions (UDFs)

• Schema—Structure of Data

• Dataset Operators

• Encoders—Internal Row Converters

• Joins

• StructField

• StructType

• InternalRow—Internal Binary Row Format

• Builder—Building SparkSession with Fluent API

• Aggregation—Typed and Untyped Grouping

• Column Operators

• Standard Functions—functions object

• Window Aggregate Operators—Windows

• Caching

• Data Types

• Row

• Datasets—Strongly-Typed DataFrames with Encoders

• DataFrame—Dataset of Rows

• SparkSession—The Entry Point to Spark SQL

• RowEncoder—DataFrame Encoder

DataSource API—Loading and Saving Datasets

• DDLStrategy

• Custom Formats

• CSVFileFormat

• BaseRelation

• DataFrameWriter

• JoinSelection

• Query Execution

• DataSource—Pluggable Data Sources

• SparkPlanner—Default Query Planner (with no Hive Support)

• ParquetFileFormat

• QueryPlanner—From Logical to Physical Plans

• DataSourceStrategy

• DataSourceRegister

• Structured Query Plan

• BasicOperators

• FileSourceStrategy

EnsureRequirements Physical Plan Optimization

• AlterViewAsCommand Runnable Command

• SparkPlan—Physical Execution Plan

• Join Logical Operator

• CoalesceExec Physical Operator

• LocalTableScanExec Physical Operator

• ExplainCommand Logical Command

• ClearCacheCommand Runnable Command

• ExecutedCommandExec Physical Operator

• CreateViewCommand Runnable Command

• BroadcastHashJoinExec Physical Operator

• ShuffledRowRDD

• ExchangeCoordinator and Adaptive Query Execution

• Debugging Query Execution

• LocalRelation Logical Operator

• CheckAnalysis

• BroadcastNestedLoopJoinExec Physical Operator

• Logical Query Plan Analyzer

• InMemoryRelation Logical Operator

• InMemoryTableScanExec Physical Operator

• LogicalPlan—Logical Query Plan

• ShuffleExchange Physical Operator

• DeserializeToObject Logical Operator

• WindowExec Physical Operator

Joins (SQL & Core)

• Core Spark Joins

• Spark SQL Joins

• Choosing a Join Type

• DataFrame Joins

• Choosing an Execution Plan

Datasets vs DataFrames vs RDDs

• Catalog

• SQL Parser Framework

• Eliminate Serialization

• Combine Typed Filters

• Predicate Pushdown / Filter Pushdown

• CatalystSerde

• Nullability (NULL Value) Propagation

• SessionState

• Logical Query Plan Optimizer

• Column Pruning

• Constant Folding

• Vectorized Parquet Decoder

• SessionCatalog

• Propagate Empty Relation

• SQLExecution Helper Object

• Simplify Casts

• SQLConf

• GetCurrentDatabase / ComputeCurrentTime

• SparkSqlAstBuilder

• ExternalCatalog—System Catalog of Permanent Entities

Tungsten Execution Backend (aka Project Tungsten)

• Catalyst—Tree Manipulation Framework

• Spark SQL CLI - spark-sql

• Whole-Stage Code Generation (CodeGen)

• Attribute Expression

• Hive Integration

• Expression TreeNode

• SparkSQLEnv

• CacheManager—In-Memory Cache for Cached Tables

• Settings

• (obsolete) SQLContext

• TreeNode

• Generator

• DataSinks Strategy

• Thrift JDBC/ODBC Server—Spark Thrift Server (STS)

DataFrames, Datasets & Spark SQLQuery Optimize

• Debugging Spark SQL Queries

• Large Query Plans and Iterative Algorithms

• JDBC/ODBC Server

• Logical and Physical Plans

• Code Generation

Data Loading and Saving Functions

• DataFrameWriter and DataFrameReader

• Save Modes

• Partitions (Discovery and Writing)

• Formats

Datasets

• Easier Functional (RDD “like”) Transformations

• Compile-Time Strong Typing

• Grouped Operations on Datasets

• Extending with User-Defined Functions and Aggregate

• Functions (UDFs, UDAFs)

• Multi-Dataset Relational Transformations

• Interoperability with RDDs, DataFrames, and Local Collections

• Relational Transformations

Spark SQL Dependencies

• Managing Spark Dependencies

• Basics of Schemas

• Avoiding Hive JARs

DataFrame API

• Multi-DataFrame Transformations

• Data Representation in DataFrames and Datasets

• Transformations

• Plain Old SQL Queries and Interacting with Hive Data

Getting Started with the SparkSession (or HiveContext or SQLContext)

• Checkpointing

• Output Operations

• Stateless Transformations

• Receiver Fault Tolerance

• Core Sources

• Worker Fault Tolerance

• Stateful Transformations

• Batch and Window Sizes

• Performance Considerations

• Architecture and Abstraction

• Streaming UI

• Driver Fault Tolerance

• Multiple Sources and Cluster Sizing

• Processing Guarantees

• A Simple Example

• Input Sources

• Additional Sources

• Transformations

Kafka Core Concepts

• brokers

• Topics

• producers

• replicas

• Partitions

• consumers

Operating Kafka

• P&S tuning

• monitoring

• deploying

• Architecture

• hardware specs

Developing Kafka apps

• serialization

• compression

• testing

• Case Study

• reading from Kafka

• Writing to Kafka

• Driver and Executor Logs

• Memory Management

• Finding Information

• Key Performance Considerations

• Configuring Spark with SparkConf

• Components of Execution: Jobs, Tasks, and Stages

• Spark Web UI

• Hardware Provisioning

• Level of Parallelism

• Serialization Format

Memory Management Driver and Executor Logs Components of Execution: Jobs, Tasks, and Stages Key Performance Considerations Hardware ProvisioningMetrics and Debugging

• Evaluating spark jobs

• Monitoring tool for spark

• Spark WebUI

• Memory consumption and resource allocation

• Job metrics

• Debugging & troubleshooting spark jobs

• Monitoring Spark jobs

Level of ParallelismMonitoring Spark

• Logging in Spark

• Spark History Server

• Spark Metrics

• Exploring the Spark Application UI

Finding Information

Spark Administration & Best Practices

• Estimating cluster resource requirements

• Estimating Drive/Executer Memory Sizes

Serialization Format

• Adapting Perf for cluster with NOSQL

• Understanding Hardware performance counters

• Making perf JVM aware

• Understanding Perf