Interfaces Hadoop Filesystems The Design of HDFS Using Hadoop Archives Limitations Parallel Copying with distcp Keeping an HDFS Cluster Balanced Hadoop Archives Data Flow Anatomy of a File Write Anatomy of a File Read Coherency Model The Command-Line Interface Basic Filesystem Operations The Java Interface Querying the Filesystem Reading Data Using the FileSystem API Directories Deleting Data Reading Data from a Hadoop URL Writing Data

Data Integrity ChecksumFileSystem LocalFileSystem Data Integrity in HDFS Serialization Implementing a Custom Writable Serialization Frameworks The Writable Interface Writable Classes Avro ORC Files Large size enables efficient read of columns New types (datetime, decimal) Encoding specific to the column type Default stripe size is 250 MB A single file as output of each task Split files without scanning for markers Bound the amount of memory required for reading or writing. Lowers pressure on the NameNode Dramatically simplifies integration with Hive Break file into sets of rows called a stripe Complex types (struct, list, map, union) Support for the Hive type model ORC File:Footer Count, min, max, and sum for each column Types, number of rows Contains list of stripes ORC Files:Index Required for skipping rows Position in each stream Min and max for each column Currently every 10,000 rows Could include bit field or bloom filter ORC Files:Postscript Contains compression parameters Size of compressed footer ORC Files:Data Directory of stream locations Required for table scan Parquet Nested Encoding Configurations Error recovery Extensibility Nulls File format Data Pages Motivation Unit of parallelization Logical Types Metadata Modules Column chunks Separating metadata and column data Checksumming Types File-Based Data Structures MapFile SequenceFile Compression Codecs Using Compression in MapReduce Compression and Input Splits

GraphX MLlib Spark SQL Data Processing Applications Spark Streaming What Is Apache Spark? Data Science Tasks Storage Layers for Spark Spark Core Who Uses Spark, and for What? A Unified Stack Cluster Managers

Lazy Evaluation Common Transformations and Actions Passing Functions to Spark RDD Operations Creating RDDs Actions Transformations Scala Java Persistence Python Converting Between RDD Types RDD Basics Basic RDDs

Expressing Existing Programming Models Fault Recovery Interpreter Integration Memory Management Implementation MapReduce RDD Operations in Spark User Applications Built with Spark Google's Pregel Console Log Minning Iterative MapReduce Behavior with Insufficient Memory A Fault-Tolerant Abstraction Support for Checkpointing Evaluation Spark Programming Interface Job Scheduling Advantages of the RDD Model Understanding the Speedup Leveraging RDDs for Debugging Iterative Machine Learning Applications Explaining the Expressivity of RDDs Representing RDDs Applications Not Suitable for RDDs

Sorting Data Determining an RDD’s Partitioner Operations That Affect Partitioning Grouping Data Motivation Aggregations Data Partitioning (Advanced) Actions Available on Pair RDDs Joins Creating Pair RDDs Operations That Benefit from Partitioning Transformations on Pair RDDs Example: PageRank Custom Partitioners

File Formats Hadoop Input and Output Formats Local/“Regular” FS Text Files Java Database Connectivity Structured Data with Spark SQL Elasticsearch File Compression Apache Hive Cassandra Object Files Comma-Separated Values and Tab-Separated Values HBase Databases Filesystems SequenceFiles JSON HDFS Motivation JSON Amazon S3

Scheduling Within and Between Spark Applications Spark Runtime Architecture A Scala Spark Application Built with sbt Packaging Your Code and Dependencies Launching a Program A Java Spark Application Built with Maven Hadoop YARN Deploying Applications with spark-submit The Driver Standalone Cluster Manager Cluster Managers Executors Amazon EC2 Cluster Manager Dependency Conflicts Apache Mesos Which Cluster Manager to Use?

Spark:YARN Mode Resource Manager Node Manager Workers Containers Threads Task Executers Application Master Multiple Applications Tuning Parameters Spark:LocalModeSpark Caching With Serialization Off-heap In Memory Running on a Cluster Scheduling Within and Between Spark Applications Spark Runtime Architecture A Scala Spark Application Built with sbt Packaging Your Code and Dependencies Launching a Program A Java Spark Application Built with Maven Hadoop YARN Deploying Applications with spark-submit The Driver Standalone Cluster Manager Cluster Managers Executors Amazon EC2 Cluster Manager Dependency Conflicts Apache Mesos Which Cluster Manager to Use? Spark SerializationStandAlone Mode Task Multiple Applications Executers Tuning Parameters Workers Threads

Working on a Per-Partition Basis Optimizing Broadcasts Accumulators Custom Accumulators Accumulators and Fault Tolerance Numeric RDD Operations Piping to External Programs Broadcast Variables

Checkpointing Output Operations Stateless Transformations Receiver Fault Tolerance Core Sources Worker Fault Tolerance Stateful Transformations Batch and Window Sizes Performance Considerations Architecture and Abstraction Streaming UI Driver Fault Tolerance Multiple Sources and Cluster Sizing Processing Guarantees A Simple Example Input Sources Additional Sources Transformations

User-Defined Functions Long-Lived Tables and Queries Spark SQL Performance Apache Hive Loading and Saving Data Performance Tuning Options Parquet Initializing Spark SQL Caching SchemaRDDs JSON From RDDs Linking with Spark SQL Spark SQL UDFs Using Spark SQL in Applications Basic Query Example

Driver and Executor Logs Memory Management Finding Information Key Performance Considerations Configuring Spark with SparkConf Components of Execution: Jobs, Tasks, and Stages Spark Web UI Hardware Provisioning Level of Parallelism Serialization Format Memory Management Driver and Executor Logs Components of Execution: Jobs, Tasks, and Stages Key Performance Considerations Hardware ProvisioningMetrics and Debugging Evaluating spark jobs Monitoring tool for spark Spark WebUI Memory consumption and resource allocation Job metrics Debugging & troubleshooting spark jobs Monitoring Spark jobs Level of ParallelismMonitoring Spark Logging in Spark Spark History Server Spark Metrics Exploring the Spark Application UI Finding Information Spark Administration & Best Practices Estimating cluster resource requirements Estimating Drive/Executer Memory Sizes Serialization Format

Kafka Core Concepts brokers Topics producers replicas Partitions consumers Operating Kafka P&S tuning monitoring deploying Architecture hardware specs Developing Kafka apps serialization compression testing Case Study reading from Kafka Writing to Kafka

Cassandra in a cluster Replication Strategies Seed Nodes Adding Nodes to a Cluster Node Configuration Cassandra Cluster Manager Creating a Cluster Dynamic Ring Participation Snitches Partitioners The Cassandra Query Language Data Types CQL1 The Relational Data Model CQL3 CQL Types Cassandra’s Data Model Secondary Indexes CQL2 Performance Tuning Memtables Commit Logs Caching Compaction Hinted Handoff JVM Settings Concurrency and Threading SSTables Networking and Timeouts Managing Performance Using cassandra-stress Cassandra Introduction A Quick Review of Relational Databases Beyond Relational Databases Web Scale What’s Wrong with Relational Databases? The Cassandra Elevator Pitch The Rise of NoSQL Where Did Cassandra Come From? Is Cassandra a Good Fit for My Project? The Cassandra Architecture System Keyspaces Partitioners Data Centers and Racks Staged Event-Driven Architecture (SEDA) Lightweight Transactions and Paxos Rings and Tokens Compaction Queries and Coordinator Nodes Caching Consistency Levels Hinted Handoff Bloom Filters Gossip and Failure Detection Anti-Entropy, Repair, and Merkle Trees Snitches Virtual Nodes Managers and Services Replication Strategies Memtables, SSTables, and Commit Logs Tombstones Data Modeling Evaluating and Refining Conceptual Data Modeling Defining Database Schema Defining Application Queries Logical Data Modeling RDBMS Design Physical Data Modeling Monitoring and Maintenance Logging Cassandra’s MBeans Backup and Recovery Maintenance Tools SSTable Utilities Basic Maintenance Adding Nodes Handling Node Failure Health Check Monitoring with nodetool Monitoring Cassandra with JMX

Indexing and query optimization Replication Sharding Sharding Starting the Servers Adding a Shard from a Replica Set Splitting Chunks Chunk Ranges Configuring Sharding Sharding Data Understanding the Components of a Cluster When to Shard The Balancer Config Servers Adding Capacity The mongos Processes How MongoDB Tracks Cluster Data Monitoring MongoDB Applications False Positives Seeing the Current Operations Documents Collections Calculating Sizes Finding Problematic Operations Preventing Phantom Operations Using mongotop and mongostat Killing Operations Using the System Profiler Databases Seeing What Your Application Is Doing Durability What Journaling Does Replacing Data Files Durability with Replication Sneaky Unclean Shutdowns Planning Commit Batches Checking for Corruption What MongoDB Does Not Guarantee Repairing Data Files Setting Commit Intervals Turning Off Journaling The mongodlock File Advanced Sharding Controlling Data Distribution Location-Based Shard Keys Hashed Shard Keys for GridFS The Firehose Strategy Shard Key Limitations Ascending Shard Keys Shard Key Strategies Picturing Distributions Randomly Distributed Shard Keys Hashed Shard Key Choosing a Shard Key Shard Key Cardinality Manual Sharding Shard Key Rules and Guidelines Taking Stock of Your Usage Multi-Hotspot Using a Cluster for Multiple Databases and Collections

Intoduction Clients Concepts Hbase vs RDBMSLog Structures Merge Trees Compaction Limitations of B+ Trees Limitations of Binary Trees LogStructured Merge tree as the back bone of storage HBase Storage Architecture MemStore Read and Write Path Physical Architecture HFile WAL HFile Format HMaster and HRegionServer How Data is Store in Hfile Root Table and Meta Table Key Format Role ofZookeeper Future Directions MMap for bloom filters and Block indexes Exploring OFF-Heap Storage Introduction Common Advantages Dynamo and Bigtable Table, Column Families,Rows and Columns Data Mode BigTable and HBase(C + P) What am I giving up? Schemaless Key/Value Stores HBase OperationsAccess Patterns Batching Filters Put Gets Caching Scanning Desiging HBase Tables and Schemas Time-Ordered Relations Pagination Concepts Key Design Partial Key Scans Tall-Narrow Versus Flat-Wide Tables Time Series Data Secondary Indexes Advanced Schemas