1 - Intro to DevOps
Today I'm excited to start a DevOps bootcamp, which marks the start of a career transition. I'm targeting backend development (Python for now), and cloud engineering (at least AWS, Docker, K8S, Terraform...).
For now I'm not planning to do too much frontend dev, that's why I'll stick with Google Sites for now!
The bootcamp is from TechWorld with Nana.
Nov 22. 2023
2 - Operating Systems & Linux Basics
Hypervisor, Type 1 & Type 2 virtual machines..., a great introduction to virtualization. I've downloaded Oracle VirtualBox, created a Linux VM, and installed Ubuntu 22.04.3. I also installed the VirtualBox Extension Pack and enabled bidirectional copy/paste between host and guest.
Users & Groups
Linux has 3 types of users:
The root user
User accounts
Service accounts (for server distributions)
List all users:
cat /etc/passwd
The format of each line is:
username:password:UID:GID:GECOS:HOMEDIR:SHELL
List all groups:
cat /etc/group
Changing password:
passwd <user>
Switch user
su - <user>
Switch to root user
su -
Adding a group
sudo groupadd <group>
Adding a user
sudo useradd <user>
sudo useradd -G <group> <user>
Adding user to group:
sudo usermod -g <group> <user>
Adding several groups for a user
overwrite existing ones
sudo usermod -G <group1>,<group2> <user>
append to existing ones
sudo usermod -aG <group1>,<group2> <user>
Deleting a group
sudo delgroup <group>
List all groups for a user
groups <user>
Remove user from group
sudo gpasswd -d <user> <group>
Ownership & Permissions
Every file (and folders since folders are files in Linux) is owned by
an owner user (usually the creator of the file)
an owner group (usually the primary group of the owner user)
Changing ownership:
sudo chown <user>:<group> <file>
sudo chown <user> <file>
sudo chgrp <group> <file>
drwxrwxrwx
d for directory or - for files
rwx permissions for owner user
rwx permissions for users of owner group
rwx permissions for all other users
Different ways to add, remove, and set permissions:
sudo chmod u+w script.sh
sudo chmod g-x script.sh
sudo chmod o+r script.sh
sudo chmod g=r-x script.sh
sudo chmod 740 script.sh
That last one is simply the decimal value of the 3-digit binary value rwx:
--- 0
--x 1
-w- 2
-wx 3
r-- 4
r-x 5
rw- 6
rwx 7
Bash scripting
sh: Bourne shell: /bin/sh.
bash: Bourne Again shell: /bin/bash. Improved version of sh and the new default in most distributions.
All shell scripts have a .sh extension. The shebang line at the top of the file tells the OS which intepreter to use:
#!/bin/bash or #!/bin/sh
Variables
declaration:
var = "whatever"
storing the output of a command in a variable:
var = $(ls <directory>)
using the variable:
echo "the value is $var"
Conditional
if [ -d "dir" ] (checks if dir is a directory)
then
...
elif [...]
then
...
else
...
fi
Testing numeric variables:
if [ "$var" -eq 10 ] (other operators: -lt -le -gt -ge -ne)
Testing strings:
if [ "$var" = "test" ] (= for Posix, == for Bash)
Brackets:
if [...] for Posix, if [[...]] for Bash
Arguments
var = $1 : Getting the first argument
var = $* : Getting all arguments
var = $# : Getting the number of arguments
User input
read -p <prompt> <var>
Loops
for <var> in <list of values, for example $*>
do
echo $var
done
while true
do
if [...]
then
break (exits the loop)
...
done
Addition vs. concatenation
var = $var1 + $var2 : concatenation
var = $(($var1 + $var2)): addition
Functions
function myfunction () {
var = $1 (to get first argument passed to function)
...
return $var (optional, to return a value)
}
When calling the function:
myfunction arg1
result = $! (stores what the function returned inside result)
Environment variables
All uppercase: USER, HOME,...
To list all environment variables:
printenv
To see only env. variable VAR:
printenv VAR
echo $VAR
We can create our own, useful for authentification.
For use only in current terminal session:
export VAR = myvalue
Persistent but only for current user, add export function in ~/.bashrc
source .bashrc to apply changes
System-wide:
/etc/environment
Adding a directory to PATH but only for current user:
PATH = $PATH:/home/user/directory inside .bashrc
Networking
LAN - Router - WAN: the router IP address is the Gateway
Subnet: IP address start + Subnet mask
For example: 192.168.0.0 + 255.255.255.0 -> the first 24 bits are fixed, also noted 192.168.0.0/24.
IP addresses of LAN are not visible to the outside of the subnet. Instead, the router pulic IP address is sent.
NAT: Network Address Translation: adds security + allows reuse of IP addresses
Firewall: which IP addresses are accessible
Ports are like doors in a building.
Typical ports: 80 (web servers), 3306 (mySQL), 5432 (PostgreSQL)
1 application per port.
DNS - Domain Name Service
Root domain .
TLD (Top-Level Do.) .edu, .com, .org...
Geograpic domains .fr, .us, .de...
Other domains .biz, ...
Regulated by ICANN: Internet Corporation for Assigned Names and Numbers
Fully qualified URL: 'course.ebersoft.com.'
DNS entries are cached for efficiency
SSH: Secure Shell
Secure way to access remote servers.
2 types of Authentification:
Username + password
SSH key pair (more secure)
Key pair: private key + public key
Services can also have their own username / key pair, not just people.
Typical port for SSH is 22.
Powerful -> should be restricted to specific IP addresses!
Quick List of Essential Linux commands
File&Directory
ls
pwd
cd
mkdir
rm
cp
mv
touch
file
zip, tar
nano, vi, vim
File Manipulation
cat
grep
sed
head, tail
awk
sort
cut
diff
tee
locate
find
User&Permission
sudo
su
chmod
chown
useradd
userdel
System
df
du
top, htop
ps
uname
hostname
time
systemctl
watch
jobs
kill
shutdown
Networking
traceroute
nslookup
dig
ping
wget
curl
scp
rsync
ifconfig
netstat
Miscellaneous
history
cal
man
echo
ln
alias, unalias
Git
After dedicating December 2023 to Linux, I am starting 2024 with Git. As a LabVIEW developer, I had always stuck with Subversion since we dealt mostly with large binary files. I am excited to tame the great features Git has to offer.
Windows Subsystem for Linux
Since I want to use Git command line, I want to stay in a Linux terminal so I can keep working with Linux commands.
Until now I was using the VirtualBox virtual machine, but it does add extra steps to start and stop the VM, and I wanted the same files to be visible from both OS. So I was excited to find and activate the WSL.
Visual Studio Code
VS Code is my tool of choice for programming and eventually I might perform Git operations directly from its integrated environment. For now I will use the Git command line until I am familiar with it enough.
I also installed the WSL extension which allows VS Code to access files stored under the Linux files system.
Linux file system
Windows and Linux still have their own file system. To access the Windows files from Linux, the /mnt/ syntax is used. For the other way around, the screenshot above shows how straightforward it is!
For performance reasons, it is not recommended to access files of an OS from the other. So I will work mostly in the Linux file system.
Commands
git init: initializes a local repo from existing local code (creates .git folder)
git add <files>: adds files to staging area
git add . : stages all files
git commit: commits all changes in the staging area to the local repo
git commit -a: to also commit changes not yet added to staging area
git commit -m "message": to add the commit message
git stash: temporarily hides changes that we are not ready to commit yet
git stach pop: restores the hidden changes
git log: displays the commit history of the current branch
git push [remote][branch]: Updates remote repo with recent commits from local copy
git pull: Updates local copy with recent commits from remote repo
git reset: discards a commit. HEAD~1 for last commit.
By default (--soft), changes are still there. --hard to also delete the changes. If the commit was already pushed to the remote repo, it can also be discarded with git push --force after the local reset. This should only be done on branches where we are the single developer since other developers like bugfix and feature branches. For master branch, use git revert.
git revert <hash>: creates a new commit that reverts the changes made by a previous commit, without removing that commit from the log.
git rebase: takes the changes that were committed on one branch and replay them on a different branch.
git checkout: switches the local copy to a different branch or older commit of the current branch.
git branch: lists 'local' branches, inside .git/refs/heads
git branch -r: lists 'remote-tracking' branches, inside .git/refs/remotes
git branch "name": creates the new branch "name"
git branch -d <branch>: deletes the local branch
git merge <source>: merges commits from source branch into current branch inside local repo (then git push to remote repo)
git clone <url>: clones a repository into a new directory
git remote add <name> <url>: adds a remote <name> for the repository at <url>
git fetch <remote>: updates remote-tracking branches under .git/refs/remotes/<repo>