My Documentation

• Pyspark
  • Basic Pyspark documentation
    • Installation of Spark (3.0.1) on Ubuntu machine
    • Certifications
    • Spark-submit tasks
    • Killing YARN applications:
    • Simple HDFS commands
    • Importing Pyspark modules
    • Creation of Pyspark objects
    • RDDs
    • Dataframes
    • Partitions in pyspark
    • Concerning partition skewness problem
    • Spark executor/cores and memory management: Resources allocation in Spark
    • Spark UI
    • Basic commands
    • Reading/writing data
    • User-Defined Schemas
    • Random sampling, stratified sampling
    • Aggregating in Pyspark
    • Joins
    • Window functions
    • Generate a column with dates between 2 dates
    • Generate an array of dates between 2 dates
    • explode operation
    • Fill forward or backward in spark
    • Arrays: Create time series format from row time series (ArrayType format)
    • Revert from time series (list) format to traditional (exploded) format
    • Converting dates in Pyspark
    • Maps/dictionaries in pyspark
    • NaN/Null/None handling
    • Saving a table in Hadoop
    • Filtering data in Pyspark
    • Opening tables from Data Warehouse
    • User-defined functions (UDF)
    • Pandas UDF
  • ETL in Spark
    • Normalizing data
    • Imputing Null or Missing Data
    • Deduplicating Data
    • Pivot in pyspark
  • Machine Learning using the MLlib package
    • The Random Forest
    • Kernel Density Estimation (here 1-D only)
  • Machine Learning using the ML package
    • Convert a string column to indexed labels
    • Assembling multiple columns in one “features” columns using VectorAssembler
    • One-Hot encoding
    • Grouping data using Bucketizer
    • The Random Forest
    • Evaluation of accuracy
  • Parallelization of scikit-learn into spark
  • Text analysis in Pyspark
    • Dealing with text: Tokenizer, Hashing, IDF
  • Spark Streaming
  • Scala Spark
  • Petastorm library
    • Conversion of Spark data to Petastorm dataset
• Python, Jupyter and Pandas
  • Exercises on Python (Pandas, Numpy,…)
  • Basic Jupyter infos
  • Getting Notebooks work on server and access them using ssh
  • Python environments: pip & conda
    • pip: Installing through proxy
    • pip, venv & setup.py: create a simple virtual environment for model development
    • pip: How does pip install -e . work?
    • Create a package out of your code (wheels)
    • Conda packages
    • Conda environments
    • Pyenv & pipenv python environments
  • Python linting, static code analysis
    • Pylint
    • Flake8
  • Visual Studio Code set-up
  • Python basic info
    • Formats for printing
    • How many cores in the edge node?
    • Basic dates in Python
    • Docstrings in functions and classes:
    • PEP - Code Refactoring - Autopep8
  • Unit tests in Python: pytest
    • Coverage (of unit test): pytest-cov
  • How to package an application in python
  • Numpy basic documentation
  • Basic Pandas documentation
    • Good Pandas links:
    • Loading Pandas dataframe from file
    • Creation of some data in a Pandas dataframe
    • Creating dataframe with several objects per cell
    • Stacking of dataframes in Pandas
    • How to shuffle the columns of a dataframe?
    • Pandas and memory
    • Re-setting of index in Pandas dataframes
    • Iterating over Pandas dataframe rows:
    • Check number of nulls in each feature column
    • Identify which columns are categorical and which are not (important for some ML algorithms)
    • Deleting a column, or list of columns:
    • Displaying dataframes to screen
    • Reading very big files using chunk
    • Reading JSON blobs (from command line)
    • Retrieval of data from SQL data warehouse
    • Exporting data to SQL warehouse
    • Transform format of dataframe: collapse multiple columns into one
    • Apply function to all rows (axis=1) or to all columns (axis=0):
    • Dataframe containing column of lists
    • Exploding a dataframe of lists of items (with ID column) into exploded ID-item column
    • Group by operations in Pandas
    • Ranking inside groups
    • Apply vs transform operations on groupby objects
    • Comparison SQL-Pandas
    • Merging and Concatenation operations
    • Pivot operations
    • Melting operation
    • Pandas Cheatsheet
    • Assigining values to dataframe
    • Percentiles - quantiles in Pandas
    • Saving of Pandas dataframe to LIBSVM file format and inverse
    • Check that 2 dataframes are equal
    • Pandas and memory
    • Cutting a dataframe into train-test-validation sets
  • Useful plots
    • The Swarbee plot of seaborn
    • Computation of PDF AND CDF plots (having only PDF)
    • Matplotlib: main functions
    • Saving objects in Python
  • Dask, or parallel Pandas
  • Python API’s:
    • Flask
    • Streamlit
• Scikit-learn
  • Basic Scikit-learn documentation
  • Data Preparation
    • Scaling Data
    • Encoding of data
    • Stratified sampling
  • Decision Tree
  • Random Forest
  • Neural Networks
    • Introduction to Keras package
  • Parameters Tuning
  • How to measure the Feature importance in RF (or other algo)
    • Mean decrease accuracy (i.e. how is reduced the accuracy if we permute the values of some feature?)
  • Score explanation for individual observation (using LIME)
    • Classification examples
  • Score explanation for individual observation (using tree-interpreter)
  • Very important update in the field of interpretation:
  • Ensemble Classification
  • Clustering
    • Gaussian Mixture Model (GMM)
• Text Mining in Python
  • Libraries and useful links
    • Basic functions
  • Intro to regular expressions (REGEX)
  • Bag of Words (BOW)
  • TF-IDF (Term Frequency - Inverse Document Frequency)
  • Cosine distance, Cosine similarity
  • Word2Vec
  • GloVe
  • FastText
  • BERT (Bidirectional Encoder Representation from Transformers)
  • Chatbot
• Deep Learning
  • Introduction
    • Good general introduction links:
    • Terminology
    • Activation functions
    • Back Propagation
    • Loss / Cost functions
    • Convolutional Neural Networks (CNN)
    • Convolution
    • Pooling
    • Batch Normalization
    • Dropout
    • Zero-Padding
    • Fully Connected Networks, or MultiPerceptron
  • Keras
    • Loss functions
    • Optimizers
    • Conversion the label to categorical: One-Hot-Encoding
    • RNN: Recurrent Neural Networks
    • LSTM: Long Short-Term Memory
    • Autoencoders
    • Variational autoencoders
    • Genetic Algorithm hyperparameters tuning
    • GAN: Generative Adversarial Networks
    • Simple NN examples
    • Petastorm
  • Tensorflow
    • Installation
    • Main Tensorflow outline
    • Graph, Session
    • Data types
    • Regression
    • Classification
• Reinforcement Learning
  • Definitions
  • Value Function
  • Discounting Factor
  • How to learn? Monte-Carlo vs Temporal Difference Learning
  • Exploration/Exploitation trade off
  • RL approaches
  • The Policy Function
  • The Q-function
  • Q-Learning
• Time Series in Python
  • Basic TS concepts
    • Forecast quality metrics
    • Augmented Dickey-Fuller test
  • Main forecasting methods
  • Basic Pandas Time Series
    • TS reading from CSV file
    • Filtering times and time ranges
    • Reindexing
    • Resampling
    • Rolling mean (moving average)
    • exponential smoothing, Double exponential smoothing
    • Interpolation
    • df.str method: how does it work?
  • Time series decomposition
  • Tsfresh: extracting features from time series
  • Seasonal ARIMA: an example
• Algorithms
  • Supervised Machine Learning
    • Association rules: Apriori algorithm
    • Implementations
    • Collaborative filtering
    • Naive Bayes
    • Decision Tree
    • Random Forest
    • Gradient Boosting:, XGBoost
    • Isolation Forest (IF)
    • Algorithm interpretability
    • Performance metrics (for SL)
  • Unsupervised Machine Learning - Clustering
  • Deep Learning
  • Machine Learning Ranking
    • RankNet
    • LambdaRank
    • LambdaMart
• Statistics and Probabilities
  • Statistics
    • Central Limit Theorem (CLT)
    • Q-Q plot and Kolmogorov-Smirmov test (KS test)
    • Box-Cox
  • Probabilities
• Great plot libraries
  • Matplotlib
    • Matplotlib: main functions
    • The Swarbee plot of seaborn
    • Computation of PDF and CDF plots (having only PDF)
  • Plotly and Dash
    • Installation
• SQL Server documentation
  • Basic SQL documentation
    • Basic query
    • Joining tables
    • Keys in SQL
    • Import data from another server
    • Data warehouse tables authorizations (works only for some tables)
    • Rank function - Exercise
    • Joining on KNID and earlier than some dates
  • Posgresql
• Useful Bash commands (or batch)
  • Find 10 largest files in a directory:
  • Find previous command matching some word in bash
  • Find RAM type in windows:
  • Top
  • Finding files and folder in linux
  • Symbolic link between folders, files
• Useful GIT commands
  • Define name globally
  • Github: how to create a repo locally and then push it to remote?
  • Bitbucket/Github: how to create ssh-keys and connect to bitbucket server
  • Pull requests: how-to
  • How to create a local (empty) branch and upstream it to remote?
  • What to do when your local dev branch is behind remote master?
  • A branch exists in remote and not in local, how to get on it?
  • My branch “feature” is based on branch “developement” and I wish to bring SPECIFIC new files from “development” into “feature”. How?
  • How to compare 2 branches from the command line?
  • Avoiding git pull
  • Git aliases
  • Other useful commands
  • Git push configuration: matching vs simple
  • Versioning in git: git tag
  • What is a detached head?
  • Adoption of a git flow
  • After updating my windows password, cannot connect push, authentication error… (stash-bitbucket)
  • Book & Cheatsheets
  • Screen commands
  • TMUX commands
• Useful VIM commands
• Data types
• Interesting business models
  • Transaction data analytics
    • The RFM segmentation
• Sphinx
  • Basic Sphinx commands
    • Subject Subtitle
    • Inline Markup
• DevOps
  • Airflow
    • Airflow CLI
  • Docker
  • Docker-compose
  • Kubernetes
    • Minikube
    • Azure Kubernetes Service
    • Spark on Kubernetes
  • Openshift
    • Persistent storage
    • Deletion
    • Additional Openshift doc
  • CICD developement
    • GitLab
    • Jenkins
    • Azure DevOps
    • Tests
    • Git Flow
• Infrastructure as a code (IaC)
  • terraform
• MLOps - Machine learning life cycle
  • MLFlow
    • 1. MLFLow Tracking: https://www.mlflow.org/docs/latest/tracking.html
    • 2. MLFLow Projects: https://www.mlflow.org/docs/latest/projects.html
    • 3. MLFlow Models: https://www.mlflow.org/docs/latest/models.html
    • 4. MLFlow Model registry: https://www.mlflow.org/docs/latest/model-registry.html
    • How to create an open-source MLFlow server on Docker or Kubernetes
    • SonarQube (or SonarCloud, SonarLint): static code analysis
  • Kubeflow
• DataOps
  • Feature Store
    • Hopsworks
    • FEAST
• Monitoring of ML models
• Azure Cloud, Databricks
  • Azure Cloud
    • Azure Key Vault
  • Azure Databricks
    • DBX - Databricks Extended CLI (supersedes Databricks-connect)
    • Databricks-connect
    • Databricks CLI
    • Centralized Databricks workspace
    • Delta Lake
    • Azure Data Factory (ADF)
    • Azure ML
• Data Science Interviews Questions
  • Data Science
  • Data Engineering
  • MLOps
• Ray

Indices and tables

• Index

• Module Index

• Search Page