ATAD #22 – File Archives on Linux

The two main formats for file archiving are tar and cpio. Both a used in tandem with compression utilities like the gzip and bzip. The archive formats were initially used for tape and other sequential access devices for backup purposes, it is now commonly used to collate collections of files into one larger file, for distribution or archiving, while preserving file system information such as user and group permissions, dates, and directory structures.

An important point you need to consider while using these archives to create backups is that, in the unfortunate event where a portion of the archive gets corrupted, tar will skip a corrupted archive portion and proceed to the next whereas cpio is going to quit with an error.

Tar accepts explicit file names to make the archive whereas cpio seems to give better control on the files that need to be archived.

Eg

# find . -type f -name '*.txt' -print | cpio -o | gzip >all_my_txt.cpio.gz

A useful way to use archives and transfer multiple files and directories to a target directory/machine is to create an archive in stdout and then, in the target directory/machine, extracting the tar file from the piped stdin.

Eg

# using cpio

# find . -type f -name '*.txt' -print | cpio -o | ssh target_machine "cd /target_dir && cpio -idum"

# using a tarpipe

# tar -cf - "source_dir" | ( cd "target_dir" && tar -xvf - )

Many software vendors like RedHat and Oracle ship their products in cpio file format. Eg, RPM uses cpio and can be extracted using the rpm2cpio utility

# rpm2cpio rpm_name | cpio -ivd

__tipped__
–vinaydeep

ATAD #21 – Linux process priority range

The Linux kernel implements two separate priority ranges.

The first is the nice value, a number from –20 to 19 with a default of zero. Larger nice values correspond to a lower priority (you are being nice to the other processes on the system). Processes with a lower nice value (higher priority) run before processes with a higher nice value (lower priority). The nice value also helps determine how long a processor timeslice the process receives. A process with a nice value of –20 receives the maximum timeslice, whereas a process with a nice value of 19 receives the minimum timeslice. Nice values are the standard priority range used in all Unix systems.

The second range is the real-time priority (rtprio). By default, it ranges from zero to 99. All real-time processes are at a higher priority than normal processes. Linux implements real-time priorities in accordance with POSIX.

In order to select a process to run, the Linux scheduler must consider the priority of each process. Actually, there are two kinds of priority:

Static priority
This kind is assigned by the users to real-time processes and ranges from 1 to 99 (highest). It is never changed by the scheduler. The sys_sched_get_priority_max( ) routine returns the static priority of the process, it returns 0 for non-realtime processes.

Dynamic priority
This kind applies only to conventional processes (non realtime processes)

Processes have an initial priority specified as the nice value. This value is stored in the static_prio member of the process’s task_struct. The value is called the static priority because it does not change from what the user specifies. The scheduler, in turn, bases its decisions on the dynamic priority that is stored in prio. The dynamic priority is calculated as a function of the static priority and the task’s interactivity

The static priority of a real-time process is always higher than the dynamic priority of a conventional one: the scheduler will start running conventional processes only when there is no real-time process in a TASK_RUNNING state.

.

Do experiment with the ps command

~$ ps -eo pid,rtprio,ni,pri,comm
PID RTPRIO  NI PRI COMMAND
1      -   0  19 init
2      -  -5  24 kthreadd
3     99   - 139 migration/0
4      -  -5  24 ksoftirqd/0

.

This scale could probably illustrate priorities better

HIGH PRIORITY – – – – – > – – – – – > – – – – – > – – – – – > – – – – LEAST PRIORITY
……..real time priority (static priority)…….| …. nice value (dynamic priority) …..
99 ……………………….. 50 ……………………… 1 | -20 …….. -10 …….. 0 …….. 10 ……. 19

.

Further Reading: Linux Process Scheduler, Linux Process Scheduling

__tipped__
–vinaydeep

ATAD #19 – Linux login program

The login program is used when signing onto a system. It can also be used to switch from one user to another at any time (most modern shells have support for this feature built into them, however).

If an argument is not given, login prompts for the username.

If the user is not root, and if /etc/nologin exists, the contents of this file are printed to the screen, and the login is terminated. This is typically used to prevent logins when the system is being taken down.

If special access restrictions are specified for the user in /etc/usertty, these must be met, or the log in attempt will be denied and a syslog message will be generated.

If the user is root, then the login must be occurring on a tty listed in /etc/securetty. Failures will be logged with the syslog facility. If the file does not exist, root is allowed to log in on any tty.

After these conditions have been checked, the password will be requested and checked (if a password is required for this username). Ten attempts are allowed before login dies, but after the first three, the response starts to get very slow. Login failures are reported via the syslog facility. This facility is also used to report any successful root logins.

If the file .hushlogin exists, then a “quiet” login is performed (this disables the checking of mail and the printing of the last login time and message of the day). Otherwise, if /var/log/lastlog exists, the last login time is printed (and the current login is recorded).

Random administrative things, such as setting the UID and GID of the tty are performed. The TERM environment variable is preserved, if it exists (other environment variables are preserved if the -p option is used). Then the HOME, PATH, SHELL, TERM, MAIL, and LOGNAME environment variables are set. PATH defaults to /usr/local/bin:/bin:/usr/bin for normal users, and to /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin for root. Last, if this is not a “quiet” login, the message of the day is printed and the file with the user’s name in /var/spool/mail will be checked, and a message printed if it has non-zero length.

The user’s shell is then started. If no shell is specified for the user in /etc/passwd, then /bin/sh is used. If there is no directory specified in /etc/passwd, then / is used (the home directory is checked for the .hushlogin file described above).

Further Reading: nologin, /etc/passwd, /etc/shadow, PAM Pluggable Authentication Modules

__tipped__

ATAD #18 – Unix Standardization

Software is probably the fastest thing that undergoes change, and its standardization is an important method for both software development and maintenance.

Since its development in 1969, Unix has undergone a lot of changes and has evolved over the years. Here are a few of the efforts towards standardizing Unix.

Probably the first step was in 1980 when BSD 4.1 was developed which included vi, job control and improvements in signals. Subsequently came AT&Ts draft standards which also included TCP/IP networking with BSD4.2 in 1983.

In 1985, AT&T released the System V Interface Definition (SVID) whose description included system calls, C libraries, available programs and devices.

In 1985 Institute of Electrical and Electronic Engineers (IEEE) committee developed a series of standards generally known as POSIX. These went beyond describing merely systems calls and C library facilities; they specified detailed semantics of a shell and a minimum command set, and also detailed bindings for various non-C programming languages. The first release in 1990 was followed by a second edition in 1996.

Key POSIX standards include the following:

• 1003.1 (released 1990): Library procedures. Described the C system call API, much like Version 7 except for signals and the terminal-control interface.
• 1003.2 (released 1992): Standard shell and utilities. Shell semantics strongly resemble those of the System V Bourne shell.
• 1003.4 (released 1993): Real-time Unix. Binary semaphores, process memory locking, memory-mapped files, shared memory, priority scheduling, real-time signals, clocks and timers, IPC message passing, synchronized I/O, asynchronous I/O, real-time files.

And then in 2001, the X/Open (The Open Group) framed the Single UNIX Specification which specified four important things.

• Base Definitions – a list of definitions and conventions used in the specifications and a list of C header files which must be provided by compliant systems
• Shell and Utilities – a list of utilities and a description of the shell, sh
• System Interfaces – a list of available C system calls which must be provided
• Rationale – the explanation behind the standard

Further Reading: Unix Standards and Specs, Unix Standards

__tipped__

ATAD #16 – escape sequence in bash

Certain characters are processed by the shell, and can be escaped (\) to tell the shell to interpret that literally. Of note are the characters '$', '`', '\', and '"'

echo "Hello Word"          #output: Hello Word
echo "\"Hello World\""     #output: "Hello World"

With certain utilities like echo and sed, escaping a character may have the opposite effect – it can toggle on a special meaning for that character.

\n means newline means return
\t means tab
\v means vertical tab
\b means backspace
\a means “alert” (beep or flash)
\d the date in “Weekday Month Date” format
\e an ASCII escape character (033)
\H the hostname
\j the number of jobs currently managed by the shell
\s the name of the shell, the basename of $0
\t the current time in 24-hour HH:MM:SS format
\@ the current time in 12-hour am/pm format

echo -e "\v\v\v\v"     # Prints 4 vertical tabs.
# The $'\X' construct makes the -e option unnecessary.
echo $'\n'             # Newline.

The behavior of \ depends on whether it is itself escaped, quoted, or appearing within command substitution

# escaping and quoting
echo \hoo              # hoo
echo \\hoo             # \hoo
echo '\hoo'            # \hoo
echo '\\hoo'           # \\hoo
echo "\hoo"            # \hoo
echo "\\hoo"           # \hoo

# Command substitution
echo `echo \hoo`       # hoo
echo `echo \\hoo`      # hoo
echo `echo \\\hoo`     # \hoo
echo `echo \\\\hoo`    # \hoo
echo `echo \\\\\\\hoo` # \\hoo
echo `echo "\hoo"`     # \hoo
echo `echo "\\hoo"`    # \hoo

# Here document
cat <<EOF
\hoo
EOF                    # \hoo

cat <<EOF
\\hoo
EOF                    # \hoo

Source [ via Advanced Bash ]

__tipped__

ATAD #12 – Securing your DNS

DNS exploits broadly fall in the category of cache poisoning (DNS spoofing where incorrect information is introduced in the DNS cache), client flooding (attacker floods false responses pretending to be a nameserver), dynamic update vulnerability (exploits weak authentication during zone updates between distributed nameservers), information leakage (attacker can extract internal information during zone transfers), and compromise of the DNS server’s authoritative database (attacker obtains administrative previlidges).

The most recent of the DNS expliots whose code can be found here, implements an intelligent cache poisoning algorithm.

The following topics could be of interest while considering DNS security

1. Carefully written named.conf
   – have DNS keys in a separate file and use the include statement to reference it.
   – backup your named.conf
   – use view and the match-clients statements to restrict information to limited clients.
2. Better designed zone updates
   – Use of TSIG (Transaction SIGnatures) or TKEY which allows a transfer from master to slave only after verifying that a shared secret key exists on both nameservers.
   – Use of Access Control Lists (ACLs) provided by BIND which allow simple IP address protection.
3. Cryptographically Signed Zones
   – Use of DNSSEC which allows for zones to be cryptographically signed with a zone key.This way, the information about a specific zone can be verified as coming from a nameserver that has signed it with a particular private key, as long as the recipient has that nameserver’s public key. This indeed would add considerable overhead but would prevent cache poisoning exploits.

If there is way to secure a system, some smart guy will invariably come up with a way to exploit an undiscovered vulnerability. These exploits, if not prevented, can atleast be curtailed by good system administration practices, regular system monitoring and updates.

Furthur Reading: DNS Security Extensions, DNS and BIND

__tipped__

ATAD #11 – Domain Name System (DNS)

The Domain Name System that dates back from the era of the ARPAnet can be understood as a distributed database (like a “phone book”) for any network that translates hostnames to IP addresses, and also vice versa translation called reverse lookup.

The DNS namespace/data stored in a nameserver is divided into manageable sets of data called zones. Zones contain name and IP address information about one or more parts of a DNS domain. A server that contains all of the information for a zone is the authoritative server for the domain. The namespace information is stored in zone files that may contain directives and resource records. Directives (optional) tell the nameserver to perform tasks or apply special settings to the zone. Resource records (mandatory) define the parameters of the zone and assign identities to individual hosts.

A nameserver can take one or more of the following roles:

master

Stores original and authoritative zone records for a namespace, and answers queries about the namespace from other nameservers.

slave

Answers queries from other nameservers concerning namespaces for which it is considered an authority. However, slave nameservers get their namespace information from master nameservers.

caching-only

Offers name-to-IP resolution services, but is not authoritative for any zones. Answers for all resolutions are cached in memory for a fixed period of time, which is specified by the retrieved zone record as the TTL value.

forwarding

Forwards requests to a specific list of nameservers for name resolution. If none of the specified nameservers can perform the resolution, the resolution fails.

Furthur Reading: BIND,  named.conf, rndc

__tipped__

ATAD #10 – Managing access to network services

Managing access to networking services on the system would be an important step in managing the system’s security. If you do not need to provide a network service, its best to turn it off. These services are usually managed by xinetd which offers a more secure extension to inetd

xinetd the eXtended InterNET Daemon is an open source daemon that is started on system bootup, and listens on ports designated in /etc/xinetd.conf for incoming network connections. When a new connection is made, xinetd starts up the corresponding network service.

Both the services managed by xinetd and the services in the /etc/rc.d/ directories (also known as SysV services) can be configured to start or stop using the following tools.

On Fedora

1. chkconfig

This is a command line utility that allows you to turn services on and off for the different runlevels. Non-xinetd services can not be started, stopped, or restarted using this utility (use service program instead).

2. Services Configuration Tool

This is a graphical application that displays a description of each service, displays whether each service is started at boot time (for runlevels 3, 4, and 5), and allows services to be started, stopped, and restarted.

3. ntsysv

This is a text-based application that allows you to configure which services are started at boot time for each runlevel. Non-xinetd services can not be started, stopped, or restarted using this program (use service program instead). ntsysv can also be used to configure runlevels.

On Debian

4. update-rc.d

update-rc.d automatically updates the System V style init script links under /etc/rcrunlevel.d/

5. rcconf

Text based GUI Runlevel configuration tool to add or remove services
Note: It would be prudent to modify firewall settings and policies while enabling/disabling services.

Furthur Reading: hosts_access, runlevels

Sources: [RedHat deployment guide, Wikipedia]

__tipped__

ATAD #9 – Linux init and runlevels

A runlevel is a state or mode of operation of the operating system. Runlevels define what services or processes should be running on the system which are listed in the /etc/rc.d/rc(x).d (where x is the number of the runlevel) directory in linux. After init is invoked as the last step of the kernel boot sequence, it looks for the file /etc/inittab file for the default runlevel.

# Default runlevel. The runlevels used by RHS are:

#   0 – halt (Do NOT set initdefault to this).
#   1 – Single user mode. Only the root user can log in.
#   2 – Multiuser, without NFS (The same as 3, if you do not have networking)
#   3 – Full multiuser mode with networking
#       This is the most common runlevel for server based systems that do not require GUI.
#   4 – unused or user defined.
#   5 – X11
#       Networked, multi-user state with X Window System capability
#   6 – reboot (Do NOT set initdefault to this)
#
id:3:initdefault:

The runlevel program can be used to determing to the current runlevel of the system and the telinit program can be used to change the runlevel of the system on the fly. Note that its also possiblesupply init with the desired runlevel by specifying it as a kernel command line parameter from the grub bootloader (kernel /vmlinuz ro root=/dev/hda1 5)

__tipped__

ATAD #8 – Package Management Systems

A Software Package is usually a software program that provides some functionality bundled with metadata that contains information about the package. And a Package Management System as defined by wikipedia is a collection of tools to automate the process of installing, upgrading, configuring, and removing software packages from a computer. The package management system is more common in unix and unix-like operating systems to operate with multiple packages. A few significant benefits of such a system are

– maintains a Repository of packages available for the running system (OS and Architecture specific) and resolves dependencies among various packages.
– looks up the repository and automatically downloads the requested packages and its dependencies when installation is triggered.
– can be used to easily upgrade the system to a defined release or level.
– system administrators can create repositories of packages which can be pulled by users, thus allowing easy maintenance of systems across the network and centrally stored packages to reduce memory requirements.

Common Package Management Systems are yum (Yellowdog Update Modifier) that is distributed with Fedora and apt (Advanced Packaging Tool) that’s distributed with Ubuntu.

__tipped__