DevOps

Terraform: Configuration Language

Terraform uses a declarative configuration language to define the desired state of your infrastructure. This configuration language is designed to be human-readable and easy to understand, making it accessible to both developers and operations teams.

Declarative vs. Imperative

Terraform's configuration language is declarative, meaning that you define the desired state of your infrastructure without specifying the exact steps needed to achieve that state. This is in contrast to imperative languages, where you specify the exact sequence of steps needed to achieve a desired outcome.

For example, in an imperative language, you might write a script that creates a virtual machine by executing a series of commands to provision the necessary resources. In a declarative language like Terraform, you would simply define the desired state of the virtual machine (e.g., its size, image, and network configuration) and let Terraform figure out the steps needed to achieve that state.

Functions

try

The try function allows you to provide a default value in case an expression returns an error. This can be useful when working with optional values that may or may not be present.

variable "optional_value" {
  type    = string
  default = null
}

locals {
  value = try(var.optional_value, "default_value")
}

Debugging Terraform

You can use the TF_LOG environment variable to enable debug logging in Terraform. This can be useful when troubleshooting issues with your infrastructure or understanding how Terraform is executing your configuration.

export TF_LOG=DEBUG
terraform plan

TOu can use the following decreasing verbosity levels log: TRACE, DEBUG, INFO, WARN or ERROR

To persist logged output logs in a file:

export TF_LOG_PATH="terraform.log"

To separare logs for Terraform and provider, you can use the following environment variables TF_LOG_CORE and TF_LOG_PROVIDER respectively. For example, to enable debug logging for both Terraform and the Azure provider, you can use the following environment variables:

export TF_LOG_CORE=DEBUG
export TF_LOG_PATH="terraform.log"

or

export TF_LOG_PROVIDER=DEBUG
export TF_LOG_PATH="provider.log"

To disable debug logging, you can unset the TF_LOG environment variable:

unset TF_LOG

References

• Terraform Configuration Language
• Terraform Functions
• Terraform Debugging

Terraform: Set your local environment developer

I wil use Ubuntu in WSL v2 as my local environment for my IaC project with Terraform. I will install the following tools:

• vscode
• Trunk
• tenv
• az cli

az cli

I will use the Azure CLI to interact with Azure resources from the command line. I will install the Azure CLI using the following commands:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

vscode

I will use Visual Studio Code as my code editor for my IaC project with Terraform. I will install the following extensions:

• Terraform
• Azure Terraform
• Azure Account
• Trunk

tenv

tenv is a tool that allows you to manage multiple Terraform environments with ease. It provides a simple way to switch between different environments, such as development, staging, and production, by managing environment variables and state files for each environment.

Installation

You can install tenv using go get:

LATEST_VERSION=$(curl --silent https://api.github.com/repos/tofuutils/tenv/releases/latest | jq -r .tag_name)
curl -O -L "https://github.com/tofuutils/tenv/releases/latest/download/tenv_${LATEST_VERSION}_amd64.deb"
sudo dpkg -i "tenv_${LATEST_VERSION}_amd64.deb"

Usage

To create a new environment, you can use the tenv create command:

# to install the latest version of Terraform
tenv tf install 

References

• tenv

Starting my IaC project with terraform

This is not my first IaC project but I want to share with you some key considerations that I have in mind to start a personal IaC project with Terraform based on post What you need to think about when starting an IaC project ?

1. Define Your Goals

These are my goals:

• Automate the provisioning of infrastructure
• Improve consistency and repeatability
• Reduce manual effort
• Enable faster deployments

2. Select the Right Tools

For my project I will use Terraform because I am familiar with it and I like its declarative configuration language.

3. Design Your Infrastructure

In my project, I will use a modular design that separates my infrastructure into different modules, such as networking, compute, and storage. This will allow me to reuse code across different projects and make my infrastructure more maintainable.

4. Version Control Your Code

I will use Git for version control and follow best practices for version control, such as using descriptive commit messages and branching strategies.

5. Automate Testing

Like appear in Implement compliance testing with Terraform and Azure, I'd like to implement:

• Compliance testing
• End-to-end testing
• Integration testing

6. Implement Continuous Integration/Continuous Deployment (CI/CD)

I set up my CI/CD pipelines with Github Actions.

7. Monitor and Maintain Your Infrastructure

I will use Azure Monitor to monitor my infrastructure and set up alerts to notify me of any issues. I will also regularly review and update my infrastructure code to ensure that it remains up to date and secure.

Of course I don't have all the answers yet, but I will keep you updated on my progress and share my learnings along the way. Stay tuned for more updates on my IaC project with Terraform!

What you need to think about when starting an IaC project ?

Infrastructure as Code (IaC) is a key practice in modern software development that allows you to manage your infrastructure in a declarative manner. With IaC, you can define your infrastructure using code, which can be version-controlled, tested, and deployed automatically. This approach brings several benefits, such as increased consistency, repeatability, and scalability.

When starting an IaC project, there are several key considerations you need to keep in mind to ensure its success. In this article, we will discuss some of the key things you should think about when embarking on an IaC project.

1. Define Your Goals

Before you start writing any code, it's essential to define your goals for the IaC project. What are you trying to achieve with IaC? Are you looking to automate the provisioning of infrastructure, improve consistency, or increase scalability? By clearly defining your goals, you can ensure that your IaC project is aligned with your organization's objectives.

Some examples:

• Automate the provisioning of infrastructure
• Improve consistency and repeatability
• Increase scalability and reduce manual effort
• Enhance security and compliance
• Enable faster development and deployment cycles

2. Select the Right Tools

Choosing the right tools is crucial for the success of your IaC project. There are several IaC tools available, such as Terraform, Ansible, and AWS CloudFormation, each with its strengths and weaknesses. Consider factors such as ease of use, scalability, and integration with your existing tools when selecting an IaC tool.

Some examples:

• Terraform: A popular IaC tool that allows you to define your infrastructure using a declarative configuration language.
• Ansible: A configuration management tool that can also be used for IaC.
• AWS CloudFormation: A service provided by AWS that allows you to define your infrastructure using JSON or YAML templates.
• Azure Resource Manager (ARM) templates: A service provided by Azure that allows you to define your infrastructure using JSON templates.
• Bicep: A domain-specific language for defining Azure resources that compiles to ARM templates.
• Pulumi: A tool that allows you to define your infrastructure using familiar programming languages such as Python, JavaScript, and Go.
• Chef: A configuration management tool that can also be used for IaC.

3. Design Your Infrastructure

When designing your infrastructure, think about how you want to structure your code. Consider using modular designs that allow you to reuse code across different projects. Define your infrastructure in a way that is easy to understand and maintain, and follow best practices for code organization.

4. Version Control Your Code

Version control is a fundamental practice in software development, and it is equally important for IaC projects. By using version control systems such as Git, you can track changes to your infrastructure code, collaborate with team members, and roll back changes if needed. Make sure to follow best practices for version control, such as using descriptive commit messages and branching strategies.

Some examples:

• Use Git for version control
• Use branching strategies like Microsoft Flow or GitHub Flow to manage your codebase
• Use pull requests for code reviews
• Automate your CI/CD pipelines to run tests and deploy changes
• Use tags to mark releases or milestones

5. Automate Testing

Testing is an essential part of any software development project, and IaC is no exception. Automating your tests can help you catch errors early in the development process and ensure that your infrastructure code is working as expected. Consider using tools such as Terraform's built-in testing framework or third-party testing tools to automate your tests.

Some examples:

• Use Terraform's built-in testing framework to write unit tests for your infrastructure code
• Use tools like Terratest or Kitchen-Terraform to write integration tests for your infrastructure code
• Use static code analysis tools to check for common errors and best practices in your infrastructure code, like Terraform's terraform validate command, or tools like tfsec or checkov.
• Use linting tools to enforce coding standards and style guidelines in your infrastructure code, like Terraform's terraform fmt command, or tools like tflint or checkov.
• Use security scanning tools to identify potential security vulnerabilities in your infrastructure code.

6. Implement Continuous Integration/Continuous Deployment (CI/CD)

CI/CD pipelines are a key component of modern software development practices, and they are equally important for IaC projects. By implementing CI/CD pipelines, you can automate the testing, building, and deployment of your infrastructure code, reducing the risk of errors and speeding up the development process. Consider using tools such as Github or Azure DevOps to set up your CI/CD pipelines.

Use tools like Terraform Cloud, Azure DevOps, or Github Actions to automate your CI/CD pipelines.

7. Monitor and Maintain Your Infrastructure

Once your IaC project is up and running, it's essential to monitor and maintain your infrastructure. Implement monitoring solutions that allow you to track the health and performance of your infrastructure, and set up alerts to notify you of any issues. Regularly review and update your infrastructure code to ensure that it remains up-to-date and secure.

By keeping these key considerations in mind when starting an IaC project, you can set yourself up for success and ensure that your infrastructure is managed efficiently and effectively. IaC is a powerful practice that can help you automate and scale your infrastructure, and by following best practices, you can maximize the benefits of this approach.

Trunk

What is Trunk ?

Trunk is a tool that runs a suite of security and best practice checks against your code. It is designed to be used in CI/CD pipelines, but can also be used as a standalone tool.

Support for the following languages is currently available:

Installing Trunk

curl https://get.trunk.io -fsSL | bash

code --install-extension Trunk.io  

Trunk checks

Trunk checks cli

Trunk detects checks to enable in function of the files in the current directory, but you can also enable and disable checks manually.

• trunck check list: list all available checks
• trunck check enable checkname: enable a check
• trunck check disable checkname: disable a check
• trunck check: run all enabled checks

For example, to enable the Terraform check:

trunk check enable terraform 
1 linter was enabled:
  terraform 1.1.0

Examples:

trunk check   

Checking 68% [====================================================================================================================================================================>                                                                              ]  38/56  9.4s 
 ↳ checkov                                                                                                                                                                                                                                                                      
   ↳ modules/webapps/linux_function_app/private_endpoint.tf [lint] ⠧                                                                                                                                                                                                            
   ↳ modules/webapps/linux_function_app/variables.tf [lint] ⠧                                                                                                                                                                                                                   
 ↳ terrascan                                                                                                                                                                                                                                                                    
   ↳ modules/webapps/linux_function_app/locals.tf [lint] ⠧                                                                                                                                                                                                                      
   ↳ modules/webapps/linux_function_app/main.tf [lint] ⠧                                                                              

Trunk checks vscode

In the case of the VSCode extension, you can review your checks in your IDE:

And you can disable checks from quick fix menu:

Trunk updates

Trunk updates cli

Trunk is updated regularly with new checks and improvements. You can update Trunk by running the following command:

trunk update

Trunk updates vscode

In the case of the VSCode extension, it will be updated automatically:

References

• Trunk
• Trunk VSCode extension

Enhance your mkdocks.yml

In the previous post I explained how to create a blog with MkDocs and mkdocs-material theme.

mkdocs.yml is the configuration file for MkDocs. In this file we can configure the theme, the plugins, the pages, etc.

In this post I am going to explain you how to create a blog with MkDocs and mkdocs-material theme, add some plugins and configure it.

Minimal configuration for mkdocs.yml with mkdocs-material

site_name: My Site
theme: 
  name: material
#plugins:

#markdown_extensions:

Theme

I only change the palette for now.

theme: 
  name: material
  palette:
    primary: blue
    accent: white  

Plugins for mkdoc

glightbox

glightbox add image zoom functionality to your documentation.

mkdocs-glightbox

plugins:
  - glightbox

Example:

Imagen de marymarkevich en Freepik

mkdocs-minify-plugin

An MkDocs plugin to minify HTML, JS or CSS files prior to being written to disk.

mkdocs-minify-plugin

Extensions

Material for MkDocs

MkDocs supports a large number of Python Markdown extensions

mermaid

mermaid2 is a plugin for MkDocs that allows you to embed diagrams written in mermaid.js in your Markdown documentation.

  - pymdownx.superfences:
      custom_fences:
        - name: mermaid
          class: mermaid
          format: !!python/name:pymdownx.superfences.fence_code_format

  ```mermaid
  graph LR
      A[Square Rect] -- Link text --> B((Circle))
      A --> C(Round Rect)
      B --> D{Rhombus}
      C --> D
  ```

graph LR
    A[Square Rect] -- Link text --> B((Circle))
    A --> C(Round Rect)
    B --> D{Rhombus}
    C --> D

You can find more information about mermaid.js in https://mermaid-js.github.io/mermaid/#/

Admonitions

Admonitions is a markdown extension of materials for MkDocs that allows you to add admonition blocks to your Markdown documentation.

markdown_extensions:
  - admonition
  - pymdownx.details
  - pymdownx.superfences

Example:

!!! Example
    Example
!!! Error
    Error
!!! Warning
    Warning    
!!! Success
    Success
!!! Info
    Info    
!!! Tip
    Tip
!!! Question
    Question
!!! Quote
    Quote

Icons, Emojis

With material you can use more than 10000 icons and thousand of emojis in your documentation.

markdown_extensions:  
  - attr_list
  - pymdownx.emoji:
      emoji_index: !!python/name:material.extensions.emoji.twemoji
      emoji_generator: !!python/name:material.extensions.emoji.to_svg

Example:

:smile:
:man_head:
:face_with_monocle:
:jack_o_lantern:

Annotations

One of the flagship features of Material for MkDocs is the ability to inject annotations – little markers that can be added almost anywhere in a document and expand a tooltip containing arbitrary Markdown on click or keyboard focus.

markdown_extensions:
  - attr_list
  - md_in_html
  - pymdownx.superfences

Examples:

This is a paragraph with a annotation(1).
{ .annotate }

1.  :man_raising_hand: I'm an annotation! I can contain `code`, __formatted
    text__, images, ... basically anything that can be expressed in Markdown.

This is a paragraph with a annotation(1).

1. I'm an annotation! I can contain code, formatted text, images, ... basically anything that can be expressed in Markdown.

This is a paragraph with a annotation(1).
{ .annotate }

1.  :man_raising_hand: I'm an annotation! with a nested annotation(1)
    { .annotate }

    1. I'm a nested annotation!

This is a paragraph with a annotation(1).

1. I'm an annotation! with a nested annotation(1)

   1. I'm a nested annotation!

Buttons

markdown_extensions:
  - attr_list  

Examples:

[This is a button](#)
{ .md-button }

This is a button

[This is a button](#)
{ .md-button .md-button--primary }

This is a button

[Send :fontawesome-regular-face-laugh-wink:](#){ .md-button }

Send

Content tabs

markdown_extensions:
  - pymdownx.superfences
  - pymdownx.tabbed:
      alternate_style: true 

Example:

=== "azcli"

    ``` azcli    
    az group create --name myResourceGroup --location westeurope
    ```

=== "pwsh"

    ``` pwsh    
    New-AzResourceGroup -Name myResourceGroup -Location westeurope    
    ```

az group create --name myResourceGroup --location westeurope

New-AzResourceGroup -Name myResourceGroup -Location westeurope    

Footnotes

markdown_extensions:
  - footnotes

Example:

This is a paragraph with a footnote[^1].

[^1]: And here is the definition.

This is a paragraph with a footnote¹.

Formatting

markdown_extensions:
  - pymdownx.critic
  - pymdownx.caret
  - pymdownx.keys
  - pymdownx.mark
  - pymdownx.tilde

Example:

- ~~Mistaken text.~~
- ^^Superscript^^
- ==Marked text.==

• Mistaken text.
• Superscript
• Marked text.

mkdocs.yml complete

site_name: My Site
site_description: A blog about Azure, DevOps and other stuff
site_author: Rafael Fernández
site_url: https://rfernandezdo.github.io

theme: 
  name: material
  palette:
    primary: blue
    accent: white
  features:
    - navigation.tabs
    - navigation.expand
    - navigation.sections
    - toc.integrate
    - toc.nested
    - toc.smoothscroll
    - footer
    - content.code.copy
    - content.code.annotate
    - content.tooltips
extra:
  social:
    - icon: fontawesome/brands/linkedin
      link: https://www.linkedin.com/in/rafaelfernandezd/
      name: LinkedIn
    - icon: fontawesome/brands/github
      link: https://github.com/rfernandezdo
      name: GitHub
    - icon: fontawesome/solid/square-rss
      link: https://rfernandezdo.github.io/feed_rss_created.xml
      name: RSS feed
copyright: Copyright © 2023-now Rafael Fernández

plugins:
  - search  
  - mermaid2
  - blog  
  - tags:
      tags_file: tags.md    
  - rss:
      match_path: blog/posts/.* 
      date_from_meta:
        as_creation: date
      categories:
        - categories
        - tags
  - minify:
      minify_html: true
      minify_js: true
      minify_css: true
      htmlmin_opts:
          remove_comments: true
      cache_safe: true
  - glightbox:
      zoomable: true
      draggable: true
      skip_classes:
        - skip-lightbox
    #- meta in insiders, review in next release
  - social
markdown_extensions:
  - admonition
  - pymdownx.details
  - pymdownx.superfences:
      custom_fences:
        - name: mermaid
          class: mermaid
          format: !!python/name:pymdownx.superfences.fence_code_format
  - md_in_html
  - attr_list
  - pymdownx.emoji:
      emoji_index: !!python/name:material.extensions.emoji.twemoji
      emoji_generator: !!python/name:material.extensions.emoji.to_svg
  - pymdownx.tabbed:
      alternate_style: true
  - pymdownx.highlight:
      anchor_linenums: true
      line_spans: __span
      pygments_lang_class: true
  - pymdownx.inlinehilite
  - pymdownx.snippets
  - footnotes
  - pymdownx.critic
  - pymdownx.caret
  - pymdownx.keys
  - pymdownx.mark
  - pymdownx.tilde
  - def_list
  - pymdownx.tasklist:
      custom_checkbox: true

urls for reference

• Font Awesome
• Emojis ...

Create a blog with MkDocs,mkdocs-material, mkdocs-rss-plugin and GitHub Pages

A few time ago I maintained a blog with Wordpress. I was happy with it, but I wanted to try something new.

I tried Jekyll but it didn't convince me, I discovered mkdocs so I decided to use MkDocs and mkdocs-material. I was happy with the result, so I decided to write this post to explain how to create a blog with MkDocs, mkdocs-material and some plugins.

These is the first post of a serie of posts to create a blog with MkDocs, mkdocs-material and GitHub Pages and some customization.

Some knowledge:

• MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file.

• Material for MkDocs is a theme for MkDocs, a static site generator geared towards (technical) project documentation. It is built using Google's Material Design guidelines. Material for MkDocs provides a polished and responsive experience out of the box, and it is as easy to use for the beginner as it is for the seasoned developer.

• GitHub Pages is a static site hosting service that takes HTML, CSS, and JavaScript files straight from a repository on GitHub, optionally runs the files through a build process, and publishes a website. You can see more information about GitHub Pages here.

• This plugin generates an RSS feed for your MkDocs site. You can see more information about mkdocs-rss-plugin here.

Steps to deploy

Create a new repository

Create a new repository on GitHub named username.github.io, where username is your username (or organization name) on GitHub. If the first part of the repository doesn’t exactly match your username, it won’t work, so make sure to get it right.

Enable GitHub Pages on your repository

Go into the repository settings and, if you are not using GitHub Pages already, enable GitHub Pages on the gh-pages branch.

Clone the repository

Go to the folder where you want to store your project, and clone the new repository:

git clone ssh://github.com/username/username.github.io
cd username.github.io

Create requirements.txt in root folder for mkdocs, mkdocs-material and plugins

mkdocs==1.5.3
mkdocs-material==9.4.6
mkdocs-rss-plugin==1.8.0

Create a Python Virtual Environment and install requirements.txt

In username.github.io$ path:

sudo apt update
sudo apt install libcairo2
sudo apt install python3.10-venv
python3 -m venv mysite
source mysite/bin/activate
pip install -r requirements.txt

Initialize your site

mkdocs new .

Add configuration to mkdocs.yml in root folder

For this post I am going to add the following configuration:

• basic configuration
• configuration for theme mkdocs-material
• some native plugins of mkdocs-material and some ones that I like

site_name: My Site 
site_description: A blog about Azure, DevOps and other stuff
site_author: Rafael Fernández

theme: 
  name: material
  features:
    - navigation.tabs
    - navigation.expand
    - navigation.sections
    - toc.integrate
    - toc.nested
    - toc.smoothscroll
    - footer

plugins:
  - search  
  - blog
  - tags:
      tags_file: tags.md      

  - rss:
      match_path: blog/posts/.* 
      date_from_meta:
        as_creation: date
      categories:
        - categories
        - tags

Add a new post

In blog/post folder create a new folder with the name of the post and create a new file with the name of the post and the extension .md. For example: welcome.md

---
date: 2023-10-18
categories:
  - Hello
  - World
---

# "Hello world!!!" from mkdocs-material

...

Check your site

In username.github.io$ path:

mkdocs serve

You can check your site in http://127.0.0.1:8000/ and make live changes in your site and see the results in your browser.

Publish your site

In username.github.io$ path:

mkdocs gh-deploy

After a seconds, you can check your site in https://username.github.io/

Automate deploy with GitHub Actions

name: ci # (1)!
on:
  push:
    branches:      
      - main
permissions:
  contents: write
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with:
          python-version: 3.x
      - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV # (3)!
      - uses: actions/cache@v3
        with:
          key: mkdocs-material-${{ env.cache_id }}
          path: .cache
          restore-keys: |
            mkdocs-material-
      - run: pip install -r requirements.txt # (4)!
      - run: mkdocs gh-deploy --force

1. You can change the name to your liking.

2. At some point, GitHub renamed master to main. If your default branch is named master, you can safely remove main, vice versa.

3. Store the cache_id environmental variable to access it later during cache key creation. The name is case-sensitive, so be sure to align it with ${{ env.cache_id }}.

   • The --utc option makes sure that each workflow runner uses the same time zone.
   • The %V format assures a cache update once a week.
   • You can change the format to %F to have daily cache updates.

   You can read the [manual page] to learn more about the formatting options of the date command.

4. Add [MkDocs plugins] or Markdown extensions with pip to requirements.txt to be used during the build.

In the next post I will explain how to customize your site with mkdocs-material and some plugins writing mkdocs.yml.

That's it folks

urls for reference

• https://www.mkdocs.org/
• https://pages.github.com/
• https://squidfunk.github.io/mkdocs-material/setup/setting-up-a-blog/
• https://guts.github.io/mkdocs-rss-plugin/ ...