Linux

A collection of notes on using Linux systems. Notes here are for Ubuntu but should work on similar debian derivative distros.

Basic Terminal Commands

List

ls

-l shows long format
-a shows all files including hidden files, current directory ., and parent directory ...
- -A omits . and ..
-h human readable file sizes
-s shows blocks taken up by the file (i.e. size on disk)

There are also other commands like lsblk, lscpu, lshw.

Disk Space

du Disk Usage
- du -sh Show size of current directory
- du -h --max-depth=1 Show size of files and folders in current directory. I have du aliased to this.
- Flags:
  - -h human readable (adds M or G)
  - --max-depth depth to recurse. Default is N.
df Disk Filesystems
- Shows usage, total space available, and mount position
- df -Ph . See free space in current directory

If looking to free up space, I recommend installing ncdu.

Monitoring

htop - basic terminal system monitor, enhanced version of top
watch -n 0.5 <program> - repeatedly call <program> every 0.5 seconds

Standard Streams

| will pipe stdout to the stdin of another process
> will redirect stdout to a file
2>&1 will redirect stderr (2) to stdout (1)
tee will redirect stdout to multiple files and show it in the terminal

Shutdown

shutdown -h [now | -t <time>]

-h poweroff, the default
-t time schedule a shutdown in time seconds
-r restart
-c cancel pending shutdown

Package Management

See DigitalOcean: Package management basics

apt

# List all installed packages
apt list --installed

# Search repos for package
apt search libdpkg-dev

Repositories

Repository sources are saved in

A line in /etc/apt/sources.list
A file in /etc/apt/sources.list.d/

Application desktop icons are stored in /usr/share/applications/.

The update notifications are in /etc/apt/apt.conf.d/99update-notifier. Comment these out to disable them.
Unattended-updates are in /etc/apt/apt.conf.d/50unattended-upgrades.

dpkg

# List everything
sudo dpkg -l

# List things with apache in the name
sudo dpkg -l | grep apache

yum

# Update package lists, typically not necessary
yum check-update

# Upgrade packages
yum update

SSH

SSH Keys

Generate an ssh-key for every client

ssh-keygen -t ed25519 [-C "comment your client name"] [-f output_path]

Some older software such as Solid file explorer require RSA keys in PEM key format

ssh-keygen -t rsa -b 4096 -m PEM [-C "comment your client name"] [-f output_path]

You can also convert existing keys to PEM format

ssh-keygen -p -m PEM [-C "comment your client name"] [-f output_path]

If you want to change the comment on your key

ssh-keygen -c -C "New comment" -f path_to_key

Manage ssh keys

# On the client
ssh-copy-id <host>

# On the server
vim ~/.ssh/authorized_keys

Notes:

According to this you should avoid using ECDSA and DSA keys.

Disable password authentication

Edit /etc/ssh/sshd_config
Set PasswordAuthentication to no
Set ChallengeResponseAuthentication to no
Test by ssh'ing into the machine using -o PreferredAuthentications=password -o PubkeyAuthentication=no

Port Forwarding

Also known as: SSH Tunneling, SSH Proxy, SSH Reverse Proxy

If you need to access a port on the remote computer, you can use the -L option to forward ports from the remote to the local machine.

ssh -L <localport>:localhost:<remoteport> <remoteurl>
# E.g. ssh -L 8080:localhost:80 [email protected]

You can also do the reverse, giving the remote access to a local port using -R

ssh -R <localport>:host:<remoteport> <remoteurl>
# E.g. ssh -R 8080:localhost:80 [email protected]

Notes

You can also run this without creating a shell using -N. This will block your shell. See SE Answer.
Adding -f pushes ssh to the background.
- This will implicitly add -n which redirects stdin from /dev/null.
- If you want to be able to foreground this again, the use & or Ctrl+z instead.

alias

You can create aliases in your .ssh/config

Host my_alias
  User my_username
  Hostname my_server@my_domain.com
  Port 52

VNC

x11vnc

Reference

I recommend not exposing VNC. Set it to localhost only and use ssh port forwarding.

Remmina

If using a wired connection, you can save a preset to localhost:5901 or similar.

Note that the Remmina which ships with Ubuntu 18.04 is outdated and buggy. You can upgrade it by adding the Remmina PPA. See https://remmina.org/how-to-install-remmina/ for details.

sudo apt-add-repository ppa:remmina-ppa-team/remmina-next
sudo apt update
sudo apt install remmina remmina-plugin-rdp remmina-plugin-secret

Nvidia

Driver Installation

Run ubuntu-drivers list to get a list of drivers
Install the latest driver
- E.g. sudo apt install nvidia-driver-460
If you have secure boot enabled, you will be asked for a password during installation
- This is because the driver is a DKMS module.
- After installation, reboot your computer and select "Enroll MOK" and enter that password in.
- Note Failure to do this will result in the driver not working
Validate your installation by running nvidia-smi.
- nvidia-smi shows the latest cuda version supported by the driver, not the cuda version installed.

Cuda Installation

Download cuda from the nvidia website or add the cuda repo to your apt sources.

Switching between Nvidia and Intel

Reference

Make sure the Nvidia graphics drivers are installed. Then you can select between Nvidia and Intel GPUs using the Nvidia X Server Settings application nvidia-settings. Alternatively, you can use the following commands in the terminal.
To switch to the Nvidia GPU:

sudo prime-select nvidia

To switch back to the Intel GPU:

sudo prime-select intel

prime-select query will print either nvidia or intel to stdout.

Fix tearing on laptops

Reference

Add options nvidia-drm modeset=1 to /etc/modprobe.d/nvidia-drm-nomodeset.conf
Run sudo update-initramfs -u

Environment Variables

Ubuntu Help Reference

Tmux

Tmux cheat sheet

Tmux, or Terminal Multiplexer is an alternative to screen.
Use it to keep terminals open and tasks running after you disconnect your SSH connection.
Getting Started:

# Make a new session
tmux
# Make a new named session
tmux new -s my_session
# Rename a session
# Keybinding: Ctrl + b, $
tmux rename-session [-t current-name] [new-name]
# Detach from a session
# Keybinding: Ctrl + b, d
tmux detach
# List windows
tmux ls
# Attach to a session
tmux attach -t my_session
# Renumber windows
:movew

Mouse scrolling

Set set -g mouse on in your ~/.tmux.conf

File Manager

The default file manager in Ubuntu is Nautilus

Add to context menu

AskUbuntu

22.04

See https://github.com/harry-cpp/code-nautilus

20.04

sudo add-apt-repository universe
sudo apt update
sudo apt install filemanager-actions nautilus-actions nautilus-extension-fma

Etcher

Github
Installing etcher

echo "deb https://deb.etcher.io stable etcher" | sudo tee /etc/apt/sources.list.d/balena-etcher.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 379CE192D401AB61
sudo apt update
sudo apt install balena-etcher-electron

Logs

Logs are stored under /var/log. These can end up taking up a lot of space.
You can delete logs in the journal folder Reference

Default gcc/g++ version

See https://askubuntu.com/questions/26498/how-to-choose-the-default-gcc-and-g-version.

# Install
sudo update-alternatives --remove-all gcc 
sudo update-alternatives --remove-all g++
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 10
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 20
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 10
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 20
sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30
sudo update-alternatives --set cc /usr/bin/gcc
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30
sudo update-alternatives --set c++ /usr/bin/g++

# Select
sudo update-alternatives --config gcc
sudo update-alternatives --config g++

Power Management

tlp

Website
Battery power management

Virtual Machines (VM)

Guest VMs

Using Ubuntu as a guest:

Install open-vm-tools-desktop

KVM

Docker

Services and Scheduling

crontab

The following will open a list of cron jobs you have.

crontab -e

The default editor is nano. You can change it to vim using VISUAL=vim env variable or with select-editor.

systemd service

See debian/systemd Services
manual

System-wide services are in /etc/systemd/system/
User services are in ~/.config/systemd/user/

A basic systemd service file

# Contents of /etc/systemd/system/myservice.service
[Unit]
Description=My Service
After=network.target

[Service]
Type=simple
Restart=always
WorkingDirectory=/usr/local/bin
ExecStart=/usr/local/bin/myservice

[Install]
WantedBy=multi-user.target

Enable with sudo systemctl enable myservice

Usage

sudo systemctl enable <my_service>
sudo systemctl status <my_service>
sudo systemctl start <my_service>
sudo systemctl stop <my_service>
sudo systemctl restart <my_service>
sudo systemctl disable <my_service>

Notes

Type should be forking if your service runs and then ends
See service log with sudo journalctl myservice

File Management

rsync

Documentation

Use this to sync folders between directories of across networks

Common Flags

-a, --archive archive mode; equals -rlptgoD
--info=progress2 show progress

See ArchWiki: rsync to learn how to use rclone for incremental backups (a la time machine).

rclone

Similar to rsync but for cloud services such as Dropbox and Google Drive.
I recommend installing from their website to get the latest version.

scp

Usage

scp [source_machine]:[source_file] [target_machine]:[target_file]

Flags

-r recursive, needed to scp directories
-P [port]

Notes

The machine can be an alias or user@domain

7z

7zip CLI
Install with sudo apt install p7zip-full

# Archive
7z a <output_file> <input_file/folder>
# Archive with password
7z a <output_file> <input_file> -p -mhe=on

# Extract 
7z x <file> [-o{dir}]

-mhe=on hides file stuctures

zip/unzip

Note that p7zip-full also includes the ability to zip/unzip .zip files.

Zip a folder

zip -r file.zip folder

Unzip an archive

unzip file.zip [-d destination]

diff

diff examples

Important flags

--strip-trailing-cr Ignores \r

tar

Extraction

tar xzvf archive.tar.gz

Archive

tar czpvf archive.tar.gz files

find

Find files by their filename

find <folder> [args] -name <name>

-maxdepth <num>

grep

Find files containing a pattern

grep -r <pattern> *

Dual Booting

Fix time difference between Windows

Reference

timedatectl set-local-rtc 1 --adjust-system-clock

Recover GRUB after installing Windows

Ubuntu Help
If you install windows after installing Ubuntu

GrubReboot

GrubReboot
Allows you to reboot into an OS one time.
i.e. If you are ssh'd into linux and want to boot into Windows one time.

Encryption

https://www.mikekasberg.com/blog/2020/04/08/dual-boot-ubuntu-and-windows-with-encryption.html

Users and Groups

Users

# Make a new user
adduser <user>

# Add user to admins
usermod -aG sudo <user>

# Change the password of a user
passwd
passwd <user>

# Delete a user
# -r will also delete their home directory
userdel -r <user>

Groups

# Make a group
groupadd <group>

# Delete a group
groupdel <group>

# List members in groups
getent group <group>

# Add user to group
usermod -a -G <group> <user>

# Remove user from group
gpasswd -d <user> <group>

Permissions

In unix filesystems, files and folders have individual permissions.
You can set permissions for each file/folder independently and for the following sets of users:

User/Owner u
Group g
Other o

You can also set permissions for all of the above with:

All a

Each file and folder can have the following permission for each set of user:

Read r
Write w
Execute x

The above totals 9 bits (3 sets of users times 3 permissions).

In addition to the above, there are 3 special bits:

Sticky bit t - only allow the owners of subfiles/subfolders to modify them
- Useful for shared folders such as /tmp
Setuid - automatically elevate execution of this file to the owner's priviledges
Setgid - automatically elevate execution of this file to the group's priviledges

In total, permissions for each file and folder can be stored in 16 bits or 2 bytes.

chmod

change mode

chown

change owner

chown [-r] <user>[:<group>] <item>

chgrp

Access Control Lists (ACL)

Display Scaling (HiDPI)

See Arch Wiki HiDPI

Fractional scaling is natively available in Ubuntu 20.04+.

Ubuntu 18.04

Xorg

# Find your display
xrandr
xrandr --output <display> --scale 1.25x1.25

Wayland

gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer']"

I have the following script run at startup

#!/bin/bash

gsettings set org.gnome.desktop.interface scaling-factor 2
gsettings set org.gnome.settings-daemon.plugins.xsettings overrides "{'Gdk/WindowScalingFactor': <2>}"
xrandr --output DP-2 --scale 1.3x1.3

Clock

See Ubuntu Time Synchronization

# Install chrony
sudo apt install chrony
# Synchronize time
sudo chronyd -q
# Check time synchronization
sudo chronyd -Q

Notes

Syncing over the internet will be off by a few milliseconds (e.g. 0.003 seconds).
Syncing with another computer over lan

Syncing with another computer

See askubuntu

On the server

Add the following to /etc/chrony.conf

# make it serve time even if it is not synced (as it can't reach out)
local stratum 8
# allow the IP of your peer to connect (192.168 subnet)
allow 192.168
# Or
# allow all

On the client

Add the following to /etc/chrony.conf

# set the servers IP here to sync to it
server <Server_IP> iburst
# remove the default servers in the config

`/dev/`

See Wikipedia: Device file#Pseudo-devices

`null`

Discards all input.
Produces EOF.

`random`

See stackexchange
See Myths about urandom

TLDR: Use /dev/urandom instead of /dev/random

`urandom`

Produces random numbers.

On my system, it's limited to about 60 MB/s. If you need faster randomness, you can encrypt from /dev/zero to get 2.7 GB/s.
See reference.

# Using urandom
pv < /dev/urandom > /dev/ull

# Using encryption
openssl enc -pbkdf2 -iter 100000 -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero | pv > /dev/null

# Create a 4 GB file.
dd if=/dev/zero bs=4M count=1024 | openssl enc -pbkdf2 -iter 100000 -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt | pv > random.bin

Gnome

Tweaks

sudo apt install gnome-tweaks
sudo apt install chrome-gnome-shell

Auto Reboot

reference

Auto Reboot Script

Auto reboot if no internet is detected:

#!/bin/bash

TMP_FILE=/tmp/inet_up

# Edit this function if you want to do something besides reboot
no_inet_action() {
    if [ "$1" -eq 1 ]; then
        systemctl restart network-manager
    elif [ "$1" -ge 2 ]; then
        rm -f $TMP_FILE
        shutdown -r now "No Internet"
    fi
}

increment_tmp_file() {
    if [ ! -f $TMP_FILE ]; then
       echo 0 > $TMP_FILE
    fi
    oldnum=$(cut -d ',' -f2 $TMP_FILE)
    newnum=$(("$oldnum" + 1))
    sed -i "s/$oldnum\$/$newnum/g" $TMP_FILE
}

if ping -c5 google.com; then
    echo 0 > $TMP_FILE
    date > /tmp/inet_up_last_check
else
    increment_tmp_file
    oldnum=$(cut -d ',' -f2 $TMP_FILE)
    no_inet_action "$oldnum"
fi

Add to sudo's crontab to run every 10 minutes

*/10 * * * * /home/david/bin/check_inet.sh

Encryption

For encrypting entire drives, I recommend LUKS.
If you want encrypt a directly, you can use fscrypt (ext4 only).

Note that ecryptfs is deprecated and shouldn't be used.

Encrypt Home After Install

See Archwiki: Fscrypt#Encrypt_a_home_directory.
See https://tlbdk.github.io/ubuntu/2018/10/22/fscrypt.html.

This uses the newer fscrypt and requires Ubuntu 18.10+.

Install fscrypt and do setup

sudo apt-get install fscrypt libpam-fscrypt
sudo fscrypt setup
sudo fscrypt setup /
sudo tune2fs -O encrypt /dev/<yourdevice>
# E.g. sudo tune2fs -O encrypt /dev/sda5

Create a new temp sudo user and login to it

Create the encrypted home folder

export USERNAME=david
# Move old home folder
sudo mv /home/$USERNAME /home/$USERNAME.bak

# Create a new home folder and encrypt it
mkdir /home/$USERNAME
chown $USERNAME:$USERNAME /home/$USERNAME
fscrypt encrypt /home/$USERNAME --user=$USERNAME

# Copy files to the new home folder using cp or rsync
# cp -a -T /home/$USERNAME.bak /home
rsync -aHX --info=progress2 /home/$USERNAME.bak/ /home/$USERNAME/

Test the encrypted home folder by logging into your user
Cleanup by removing the temporary user and deleting the old home folder
```
shred /home/$USERNAME.bak/
```

Notes and Caveats

systemd will no longer have access to your home so all startup apps should be placed elsewhere
- E.g. Move all startup scripts in your ~/.local/bin to /usr/local/bin
ssh will not work until home has been decrypted since the authorized keys are in ~/.ssh/authorized_keys

SSH Workaround

Getting SSH to work with an encrypted home dir is a giant pain.
Also things like tmux still won't work.
Overall I do not recommend doing this on a server.

Move ssh keys elsewhere such as /etc/ssh/authorized_keys/<user>.
- Add /etc/ssh/authorized_keys/%u to the AuthorizedKeysFile line in /etc/ssh/sshd_config.
Create a sudo user with and unencrypted home directory.
After every restart, ssh into the unencrypted sudo user and decrypt your home directory:
- sudo fscrypt unlock /home/david --user=david
Then ssh into your account.

SFTP

You can create a specific user with a chroot to limit SFTP to specific folders.
See Archwiki: SFTP chroot for details. /etc/ssh/sshd_config

Subsystem sftp /usr/lib/ssh/sftp-server

Match Group sftponly
  ChrootDirectory %h
  ForceCommand internal-sftp
  AllowTcpForwarding no
  X11Forwarding no
  PasswordAuthentication no

Hardware Info

Benchmarking

Basic CPU benchmark

sysbench cpu --threads=2 run

MOTD

Message of the day is the text you see when you login via SSH.
Ubuntu stores its MOTD in /etc/update-motd.d/. Other distros use /etc/motd/.
You can disable the Ubuntu news motd in /etc/default/motd-news.

System Administration

Installing Binaries

Copy your binary to /usr/local/bin/ or ~/.local/bin/
Copy your man page to /usr/local/share/man/man1/ or ~/.local/share/man/man1/

Network Troubleshooting

On one of my OptiPlex 5060 servers, the network adapter would reset on git ssh clones.
This would appear in /var/log/syslog as:

Feb  8 22:22:01 optiplex5060-2 kernel: [ 4378.992607] e1000e 0000:00:1f.6 eno1: Reset adapter unexpectedly

This was resolved by disabling TCP Segmentation Offload:

sudo ethtool -K eno1 tso off

# Verify tso is disabled
ethtool -k eno1 | grep tcp

To make this persist across reboots:

Script

If you're using netplan (default for Ubuntu):
Reference

output_path=/usr/lib/networkd-dispatcher/routable.d/10-disable-offloading
sudo tee $output_path <<EOF> /dev/null
#!/bin/bash
ethtool -K eno1 tso off
EOF
sudo chmod +x $output_path

If using ifupdown:

output_path=/etc/network/if-up.d/disable-tso
sudo tee $output_path <<EOF> /dev/null
#!/bin/bash
ethtool -K eno1 tso off
EOF
sudo chmod +x $output_path

Cloning to a new disk

The easiest way is to use gparted.

Terminal Guide

To do this in the terminal:

OLD_DRIVE=/dev/sda
NEW_DRIVE=/dev/sdb

# Show old drive partitions in sectors
parted $OLD_DRIVE unit s print free

# Apply GPT
parted $NEW_DRIVE mklabel gpt
# Copy new EFI partition with start 1024s and end 1050623s
parted $NEW_DRIVE mkpart primary fat32 2048s 1050623s
# Apply boot and esp flags.
parted $NEW_DRIVE set 1 boot on
parted $NEW_DRIVE set 1 esp on
parted $NEW_DRIVE name 1 'EFI System Partition'
# dd the old to the new
dd if=${OLD_DRIVE}1 of=${NEW_DRIVE}1 bs=4k

# Make a new partition. Make sure start and end sectors are aligned.
# i.e. start % 8 == 0 and end % 8 == 7 if your physical sector size is 4096 bytes, typical for new HDDs and SSDs.
parted $NEW_DRIVE mkpart primary btrfs 1050624s 488396791s
parted $NEW_DRIVE align-check opt 2

# Copy the filesystem
mkfs.btrfs ${NEW_DRIVE}2
mkdir /media/${NEW_DRIVE}
mount -t btrfs -o compress=zstd /media/${NEW_DRIVE}2
rsync -axHAWXS --numeric-ids --info=progress2 /media/${NEW_DRIVE}2

rsync reference

rsync options

-a archive mode
-x one file system
-H preserve hard links
-A preserve ACLs
-W copy whole files instead of deltas
-X preserve extended attributes
-S handle sparse files efficiently
--numeric-ids use id instead of uid/gid

To copy a root partition, make sure you change the following on the new drive:

Update the UUID and mount options in /etc/fstab
Update the UUID in /boot/grub/grub.cfg and run update-grub
Update the UUID in /boot/EFI/ubuntu/grub.cfg
Run boot-repair from a live disk if you run into any issues.

Ubuntu

Ubuntu-specific notes

Disable ESM message

Reference

# Disable MOTD
sudo chmod -x /etc/update-motd.d/88-esm-announce
sudo chmod -x /etc/update-motd.d/91-contract-ua-esm-status

# Disable APT check
sudo sed -Ezi.orig \
  -e 's/(def _output_esm_service_status.outstream, have_esm_service, service_type.:\n)/\1    return\n/' \
  -e 's/(def _output_esm_package_alert.*?\n.*?\n.:\n)/\1    return\n/' \
  /usr/lib/update-notifier/apt_check.py
sudo /usr/lib/update-notifier/update-motd-updates-available --force