If you’ve installed Shiny Server Pro as a user other than shiny, you might have experienced difficulty adding R packages. This is because Shiny Server Pro runs R as the shiny user, and running R -e “install.packages(‘foo’)” will install packages to the local user’s files only.
The solution to this is to su to the shiny user:
su - shiny
R -e "install.packages('foo', repos='http://cran.rstudio.com')"
Alternatively, this script will parse an R file looking for require statements and install the necessary packages. It isn’t very smart, so be careful.
This post provides examples of everyday Linux commands. I wrote this as a quick orientation to the CLI for newcomers. Keep in mind that there are usually several ways to accomplish a task, and other command combinations or programs might be better suited to your needs. Don’t be afraid to google it.
Get Help With Commands By Using MAN
The examples on this page represent a small fraction of the possible uses for each command. Many commands have tons of flags and arguments that allow them to adapt to many scenarios. Since it’s impossible to remember how each program works, its arguments, etc., most come with a manual page, or man page. The man command retrieves the manual for the program and displays it on the screen. While sometimes verbose, man pages are typically your best source of initial information. This is the Manual in “RTFM“.
The man system has several different sections, each providing documentation on a specific aspect of the program:
Library functions (Particularly the C STL)
Special files and drivers (Typically in /dev)
File Formats & Conventions
Games and Screensavers
System Administration Commands and Daemons
Typing man <command> will generally display section 1, if it exists. Calling man on something else — for example, the C function pthread_join, will display page 3 by default. To view other sections, type man <page> <command>. Note that not all programs have manual pages. Of those that do, most don’t have manual pages in each section. On some systems, you can type “man <page>” and press TAB to view a list of pages available for that section.
To view the manual page for man itself, type man man.
Finding the right Linux command with apropos
The Unix Tools Philosophy aims for tools that serve a specific purpose and can be chained together and results in a seemingly-endless variety of tools to accomplish the same job. The apropos command will search the manual pages for terms and return a list of possible commands. This “search” feature isn’t full-featured and works mostly on keyword matching.
apt-ftparchive (1) - Utility to generate index files
ftp (1) - Internet file transfer program
netkit-ftp (1) - Internet file transfer program
netrc (5) - user configuration for ftp
pam_ftp (8) - PAM module for anonymous access module
pftp (1) - Internet file transfer program
sftp (1) - secure file transfer program
smbclient (1) - ftp-like client to access SMB/CIFS resources on servers
Sometimes the number of commands can be unwieldy (try “apropos user” or “apropos ip“). Since the apropos command doesn’t do well with terms like “add user” or “ftp upload”, it’s sometimes useful to join the output with grep. You can also pipe the output to more or less.
apropos user | grep add
adduser.conf (5) - configuration file for adduser(8) and addgroup(8) .
addgroup (8) - add a user or group to the system
adduser (8) - add a user or group to the system
pam_issue (8) - PAM module to add issue file to user prompt
useradd (8) - create a new user or update default new user information
List directories with ls
ls is the command for listing a directory. Useful flags include:
-a (–all), which shows dotfiles
-l which provides a long listing that includes file size and type,
-h (–human-readable) shows filesizes in normal units instead of bytes (1.23 GB instead of 1234921293)
-S to sort by file size
Default ll command, plus human-readable sizes
alias ll=”ls -alh”
List all tar files in current directory:
List tar files in long format:
ls -l *.tar
List the largest10 files in a directory and output size in human readable form:
ls -lhS | head -n 10
Navigate the filesystem with CD and MC
Navigating around the filesystem is done with the change directory (cd) command. Instead of merely listing the contents of the /tmp directory (ls -l /tmp), you can move into the /tmp directory and list the contents of the current directory:
cd /tmp && ls -l
To return to your home directory, simply type cd with no arguments. You can also use the shortcut ~ to refer to files from your home directory. The two paths below describe the same location on the filesystem, assuming that the second command is run by the cmattoon user.
If you aren’t sure which directory you’re in, you can use the pwd (Present Working Directory) command to tell you.
Midnight Commander (mc) is a third-party application that some people find useful for navigating the filesystem, copying and moving files, etc. You can find more information on their site.
Move, Rename and Copy files with mv and cp
Copy a config file to a backup:
cp config.ini config.ini.bak
Copy the entire config directory and it’s contents:
cp -r config/ config-backup
Remove files with rm and shred
There is no “undo” command for rm.
The rm command removes files basically forever, so be careful.
There is no “undo” command for rm. Some people choose to edit their ~/.bashrc or ~/.bash_aliases and add the following:
alias rm="rm -i" # Ask for confirmation before deleting files.
Linux makes the process of deleting a file forever deceptively simple:
To indiscriminately remove everything in the “/tmp” directory:
rm -rf /tmp/*
Note: Either “rm -rf /tmp/” or “rm -rf /tmp” would delete the “/tmp” directory itself.
rm My\ Document
For private information, you might consider using shred.
Read large files one screen at a time with more and less
More and less are two programs that filter text output. They’re commonly used to page through large files, but can also be used to buffer output from programs. It’s a helpful habit to read config files with more or less, rather than open it with a text editor. In both programs, you can find help by pressing h and exit by pressing the q key.
cat README.md | less
Writes the output of the install process to the screen for later
./install.sh 2>&1 | less
Although both programs are very similar, less is the newer one with more features. Specifically, less allows forward and backward navigation (via arrow keys and PgUp/PgDn) and doesn’t have to read the entire file into memory. This makes it more efficient on large files than it’s predecessor, more.
Use the slash key (/) to begin a search. While a search is in progress, the “n” key will move to the next result.
Download files with curl and wget
Since both curl and wget support HTTP/HTTPS and FTP, they are especially useful for interacting with web-based services like API’s and HTML forms. Both programs use the HTTP GET method by default, but are capable of others as well (POST, HEAD, PUT, etc..), and both support SSL. cURL supports even more protocols including Telnet, SCP, SFTP, POP3, IMAP, SMTP and LDAP, and a number of other features.
Generally speaking, I prefer wget for downloading files and cURL for
Note: Ubuntu comes with wget, but you’ll need to install curl. CentOS and OS X are the opposite. You’ll probably need to download one or the other.
To download a file with wget:
If the URL contains special characters, or is pointing to a script, it’s sometimes better to wrap the URL in quotes and use the -O flag to specify an output file.
If you still want to see error output, but no progress bar, you can use -sS in curl. The lowercase -s is for silent mode, the uppercase -S for “show errors”.
For more details, type “man wget” or “man curl”.
Check disk usage with du and df
To check the amount of disk space available, use the df command (think “disk free space”). The du command will show you the amount of disk space used in the specified directory. Like the ls command, both df and du can output the human-readable filesize by using the -h flag.
Output of df -h will show the disk space for all mounted drives by default:
To see how much space the current directory is taking up, use du -sh. The -s flag means “summary”, and prints the total usage of all subdirectories. WIthout the -s flag, du will generate a report for each subdirectory. This feature can be useful for finding the largest n files in a directory. The following command finds the 10 largest subdirectories of the current directory. By piping the output of du into sort (-h sort by human-readable filesize, -r reverse), we can sort the files from largest to smallest. That output is then piped into head to retrieve the top 10 only.
du -h . | sort -rh | head
Of course, you could pipe this output to more or less and peruse the entire list of directories, but there’s already a better tool for this:ncdu. (The “nc” alludes to the ncurses library used to render the user interface.) As you can see in the screenshot below, ncdu provides an easy way of tracking down large files.
Get file contents with cat and strings
The cat and strings commands are used to write file contents to stdout. The cat command will dump the raw file contents (in whatever form), while strings will print only printable characters. This feature makes the strings command a useful choice in identifying a file format or other initial discovery tasks.
Print raw binary data of /bin/true to stdout:
See all human-readable strings in the “true” binary:
Zero a file with cat:
cat /dev/null > README.md
Find what you’re looking for with grep and find
Grep (Globally search a Regular Expression and Print) is useful for finding strings in files (or stdout). The find utility is used for searching by file name, size, etc.
To find all PHP files with the string “@todo” (case insensitive) in the src/ directory:
grep -i "@todo" src/*.php
Recursively search the src/ directory for files containing the string “@todo” (case-insensitive):
grep -ri "@todo" src/
This uses the -r(–recursive) and -i (–ignore-case) flags. As you may suspect, the –recursive flag searches the directory recursively, while the –ignore-case flag ignores the difference between uppercase and lowercase characters.
Grep is also useful to filter output from commands or stdout:
If you have multi-line output in the log, grep will cut off all but the first line. If you want to see lines on either side of the target line use the -A (–after-context) or -B (–before-context). For example, consider grepme.txt, a file with “This is Line #n” from 0-30. Both commands produce the same output:
grep 20 grepme.txt -A 5 -B 3
cat grepme.txt | grep "20" -A 5 -B 3
This is Line #17
This is Line #18
This is Line #19
This is Line #20
This is Line #21
This is Line #22
This is Line #23
This is Line #24
This is Line #25
Other useful examples include -v, which inverts the match and the -L/-l flags that show filenames of lines matched instead of lines matched.
Show all lines in access log that don’t include “GoogleBot”:
This information can now be used to log in to MySQL’s command-line interface:
mysql -u username -p
Leaving the “-p” parameter empty will trigger MySQL to prompt you for a password. On a *NIX server, it will look like you’re not typing anything — this is by design. While you may specify the password in the same line, this can leave your plaintext password in your command history, which is easily readable. If you want to use this format anyway (i.e., in a script), note that you cannot put a space between the “-p” flag and your password:
mysql -u username -ppassword
Once you’ve logged in, you can view available databases with the show databases; command. To use your wordpress database, take the value from DB_NAME (above) and use the use command: use wordpress;. To see available tables in the selected database, run show tables;.
The TrackMan mouse has four (physical) buttons which include a large left and right button (1, 3), that serve as the primary mouse buttons, and two smaller left and right buttons (8, 9) that trigger your browser’s “back” and “forward” buttons. To replace this action with “Ctrl+Click” to scroll, insert the following lines in your ~/.bashrc (or anywhere else that can call some commands):
I recently stumbled upon a copy of RedStar OS, which appears to be a RHEL-based server distribution developed by North Korea. Version 2.5 was initially purchased and reviewed by a Russian student studying abroad, and a user by the name of slipstream uploaded version 3.0 (server) to TPB in mid-2014.
Several reports portray it as a tool to monitor web usage by the regime, and while I don’t doubt that, it seems unnecessary to repackage an operating system to do so. It seems more likely that it’s a symbol of sovereignty and independence from Windows (made in USA). Since North Korea’s internet is a giant class A network (10.76.1.0/22), any reporting software would likely try to report to an otherwise “internal” network. For example, the browser packaged with the OS has its homepage set to 10.76.1.11. A quick Wireshark analysis didn’t reveal anything immediately suspicious, but I’ve yet to dig into that fully.
On the surface, it’s a pretty hollow clone of RHEL using KDE desktop. The directory structure is a cross between OSX and *nix, as is the overall feel of the desktop environment.
It comes with a couple of standard applications – a calculator, notepad, contact book, etc., as well as QuickTime and Naenera Browser (a Firefox clone). As Naenera (“my country”) is also the name of the official web portal, and that most citizens can’t access the “international internet”, the two might as well be synonymous.
You can see the public-facing Naenera at http://www.naenara.com.kp/en/, but be aware that they’ve been known to inject malware on some of their public-facing sites.
It’s also interesting to note there’s a CHM (compiled HTML) viewer. This is typically used for software documentation, and very well may be the case here. I’d be interested to see if this is utilized for something akin to Cuba’s Paquetes, downloading parts of the Kwangmyong, or something altogether different. (There is an empty “Sites” folder in the user’s home directory)
There’s an OpenOffice clone, called Sogwang Office.
It also has this music composition program, UnBangUI:
The mail program doesn’t have any clear way to add an email account, but does prevent you from checking mail until you’ve added one.
The software center only allows importing from /media. There is a repository of extra applications that’s offered on a second CD (the Russian site says the extra CD costs about twice what the original OS costs), and I haven’t started to dig through that yet.
In the “System Update” area, the Settings dialog shows a location for a URL and proxy, but I’m not sure it’s usable.
Interestingly, the user isn’t added to sudoers and the root account is disabled. Fortunately, this is trivial to bypass, since someone “overlooked” the permissions in /etc/udev/rules.d. If you’re looking for a terminal shortcut, you won’t find it – you’ll have to press Alt+F2, then run konsole to get a shell.
Once you’ve done that, fire up vi and create /tmp/freedom, or whatever you’d like to call it.
Now, open up that file in /etc/udev/rules.d and call /tmp/freedom via a RUN expression:
Now that that’s taken care of, you’ll need to refresh the udev rules. In VirtualBox, this worked simply by taking a snapshot, but you might have to reboot.
Enabling English on RedStar OS
Once you’re back up and running, you’ll likely want to enable a language other than Korean. While some reports state that Korean is the only language on the system, this isn’t true. It’s just that Korean is selected by default. Now that you have sudo superpowers, this can be done easily with sed: (obviously,for a language other than US English, use the appropriate locale code)
sed -i 's/ko_KP/en_US/g' /etc/sysconfig/i18n
sed -i 's/ko_KP/en_US/g' /usr/share/config/kdeglobals
Log out, and you should see the login screen in English:
That’s it! You should now be able to browse around the OS relatively easily. I’ll post some interesting findings later, once I’ve had an opportunity to dig through it more.
Here’s a quick Python script to visualize binary data. In the grayscale example, each pixel is the color of the bit value (0x00 – 0xFF). The same method is used for colorization, except the bit value is used to provide hue and value values for HSV colorspace (saturation is fixed at 0.99).
The cols parameter is the width of the image to be generated (in pixels). By default, the script generates a couple of different sizes. The height is calculated based on the width. Patterns tend to be clearer when the column width is a multiple of 8 (16, 32, 64, 128…), though that could depend on the format and type of data in the file.
As an example, here are some images from a 256-byte file generated with the following Python program:
with open('foo.txt', 'wb') as fd:
for i in range(256):
./process_dir.py <dirname> <cols>
The program will generate images for each of the binaries in the specified directory, create an “index.html” file and attempt to launch it in the browser.
The generated image on the left is from a PNG file. A dark patch in the beginning with a mostly-uniform distribution is consistent with file headers followed by image data.
The image to the right is an OpenOffice Writer file. The striped area indicates a repeating pattern of bytes, which often separates the metadata header and content in word processor files. The example screenshot shows an image generated from a compiled binary.
This can also be used to visually approximate the amount of entropy in a file. A high-entropy file would have a uniform byte distribution, thus occupying all of the available colorspace. I’ll include a histogram function later. This would show the frequency distribution of the bytes as well.
I recently started using TomatoUSB on a Cisco/Linksys E1200 router and noticed that I had an extremely slow download speed (around 0.5 Mb/s) and an acceptable upload speed (5.5 Mb/s). After checking QoS settings and the “Bandwidth Limiter” tab, I found a forum post indicating the WMM settings severely affected the speed. Disabling WMM increased the speed to around 1.5 Mb/s.
To disable WMM, go to Advanced > Wireless and look for the WMM field.
This, along with a call to Comcast, has increased my WiFi speed to around 7 Mb/s (25+ wired).
Dovecot: Error reading configuration: Invalid settings: postmaster_address setting not given
status=deferred (temporary failure. Command output: lda: Error: user <user>@<domain>: Error reading configuration: Invalid settings: postmaster_address setting not given lda: Fatal: Internal error occurred. Refer to server log for more information. )
While the error message itself is quite clear (the postmaster_address setting is missing), some of the highly-ranked answers didn’t quite work for me.
First, check the output of:
dovecot -a | grep postmaster_address
Expect no results (the setting isn’t given, after all). If you do have results, check that the setting is declared correctly.
The ‘postmaster_address’ Setting
On Ubuntu 12.04, I found the postmaster_address setting is defined in two places:
I found the setting at the top of /etc/dovecot/conf.d/15-lda.conf to be commented out. Intuitively, you might uncomment this line and provide a setting, but if that doesn’t work, open /etc/dovecot/dovecot.conf and search for the following section:
I got tired of installing web-mode.el all the time, so I scripted it. It’s available on GitHub Gists. (If you’re not familiar with Web Mode, it’s a much better replacement for PHP mode (IMO) that can handle multiple languages in the same file. I highly recommend it if you edit web templates in Emacs. )