SSH: Transferring large files between hosts

I need to transfer several 10+ gigabyte files between two internal Linux hosts. The easiest way is to use either the scp or sftp. This will encrypt the transfer which can slow things down. There are several ciphers available that you can use to speed things up. Using modern OSs (Fedora 27, CentOS 7, FreeNAS 11) I wanted to find the best cipher to standardize on. The fastest cipher supported by all of my operating systems is aes128-gcm@openssh.com.

You can use aes128-gcm@openssh.com with scp and sftp like this:

scp -c aes128-gcm@openssh.com user@domain.com
sftp -c aes128-gcm@openssh.com user@domain.com

To use an alternate cipher with rsync use this command:

rsync -avP --rsh="ssh -Tc aes128-gcm@openssh.com" /source/dir user@domain.com:/destination/dir

Honorable mention goes to aes128-ctr as the second place contender. If for whatever reason aes128-gcm@openssh.com isn't available it would make a good alternate choice.

Leave A Reply

PHP: Convert an array to a hash

I have a flat array that I want to convert to a hash so I can use it as a lookup table. There isn't an easy or clear way to do that in PHP so I wrote my own function:

function array_to_hash(array $array, $val = 1) {
    $ret = array_fill_keys($array, $val);

    return $ret;
}
Leave A Reply

Linux: Fedora 27 major package versions

Fedora 27 has these versions of some core packages:

Package Version
Apache 2.4.29
GCC 7.2.1
Kernel 4.13.3
Perl 5.26.1
PHP 7.1.11
Vim 8.0.1187
Git 2.14.3
Leave A Reply - 1 Reply

Perl: redirect STDOUT and STDERR to a file

I need to redirect STDOUT and STDERR to a log file in my script. I didn't find really conclusive documentation on the best way to do this so here is what I came up with.

my $file = "/tmp/debug.log";
open(my $stdlog, ">", $file) or die("Cannot open $file");

*STDOUT = $stdlog;
*STDERR = $stdlog;
Leave A Reply

Reddit on Soap

Reddit user amaranth1977 made a great post about soap:

Most modern liquid "soaps" are not technically soaps at all, which is to say they're not produced from mixing vegetable or animal fats with a strongly alkaline solution such as lye or potash. Instead they're a blend of (usually) petroleum-derived surfactants such as sodium lauryl sulfate with other chemicals to produce a detergent that matches the desired use..

Shampoo is designed to be gentle on the keratin which forms hair, have strong foaming properties to be more easily worked through the fine strands, remove common hairstyling products, and - especially for those of us with more than a couple inches of hair - have specific effects on the texture of the hair. It has a fairly low concentration of surfactants so that it rinses out quickly and you're not in the shower forever trying to get it all out of your hair.

Hand and body wash is usually formulated with a mild surfactant to avoid skin irritation, plus various ingredients that can moisturize the skin, add scent, improve lather, etc. Lathering agents are generally surfactants as well, so there's a careful balance here between getting a nice lather and not drying out skin. Hand washes are usually less foamy since they don't need to cover much surface area and are used frequently throughout the day, while body washes tend towards more foam since they need to cover a lot more surface area and are used less frequently. They're both a bit more concentrated than shampoo, since it's easier to rinse soap off of skin than hair, and in the case of body wash, most consumers pour it onto a sponge/pouf/washcloth/etc. before applying it to the skin, which spreads it out thinner than applying it directly. Face washes are their own magical category and can include all sorts of fun chemistry like ceramides and multivesicular emulsions, alpha hydroxy acids, benzoyl peroxide or salicylic acid, etc. - and as an acne sufferer, it definitely makes a difference.

Bar soaps are one area where true soaps are still relatively common. Moisturizing ingredients can also be added, and the naturally occurring glycerine is also somewhat moisturizing. However, traditional soaps also have a lot of limitations. They have a fairly narrow range of environments in which they're effective, needing hot water with a low mineral content to function, and must be rinsed a second time with clean water to avoid deposits.

Laundry detergents are commonly formulated these days with specific surfactants designed to work well with cold and hard water, both of which decrease the effectiveness of traditional soaps. They're also highly concentrated, since they're going to be diluted by the large volume of wash-water, which is why just a few drops of liquid detergent on your hands will take much longer to rinse off than an equal amount of hand soap. This is also why ideally you should fill the washer with soap and water so they can mix, then add the clothes.

Dishwasher detergents aren't worried about gentleness, since they don't come in contact with skin or organic fibers, so they can use harsher detergents and often include abrasives, but do rely on hot water to be effective. For similar reasons to laundry detergents, they're highly concentrated, but they use surfactants that are more effective on metal and ceramic.

Dish detergents meant for hand-washing dishes have to balance removing grease, starches, sugars, etc. from food with not completely stripping the natural oils from skin, which is a bit tricky - the oils that keep your skin nice and pliable aren't any different from the oils of any other animal, chemically speaking. They're also pretty highly concentrated for that "grease-fighting" effect, and so that your washcloth/sponge/etc. doesn't need more soap on it after every dish you wash.

Leave A Reply

Total possible glyphs using UTF-8

UTF-8 is an encoding method for representing large amount of glyphs. UTF-8 will use one, two, three, or four bytes to encode a given glyph depending on the given code point needed. Wikipedia has a good table that explains how UTF-8 breaks out:

Number of bytes Code point bits First code point Last code point Byte 1 Byte 2 Byte 3 Byte 4
1 7 U+0000 U+007F 0xxxxxxx
2 11 U+0080 U+07FF 110xxxxx 10xxxxxx
3 16 U+0800 U+FFFF 1110xxxx 10xxxxxx 10xxxxxx
4 21 U+10000 U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

There are 1,114,112 (17 x 2^16) total code points available. BableStone reports that 276,337 (approximately 24.8%) code points are in use, which leaves 837,775 still available. That's a lot of room left for emojis.

Leave A Reply

Linux: Fedora 26 major package versions

Fedora 26 has these versions of some core packages:

Package Version
Apache 2.4.26
GCC 7.1.1
Kernel 4.11.8
Perl 5.24.1
PHP 7.1.6
Vim 8.0.662
Git 2.13.0
Leave A Reply

Perl: Variable scoping

Perl allows you scope variables in several ways.

The most common is my which scopes the variable to the surrounding {} (or the entire script/package if there are no {}).

Perl also supports our which is used inside of packages to make a variable global. This allows you to access that variable from a calling script by accessing $Package::variable_name.

Finally there is the local keyword which takes a global variable and makes a locally scoped copy of the variable. Changes to the variable will not be reflected in the global scope.

Name Scope
my Local
our Global
local Temp copy of global
Leave A Reply

PHP: Calculate the percentage difference between two numbers

I need to compare two numbers and see if they're close to each other. Specifically I wanted to see if two numbers were within 3% of of each other. I wrote this simple function to calculate the percentage difference between two numbers, and optionally (with the third parameter) return true or false if they're within a given range. This should allow me to do a "fuzzy compare" on two numbers.

function percent_diff($a, $b, $ok_per = null) {
    $per_diff = abs((1 - ($a / $b)) * 100);

    if (is_numeric($ok_per)) {
        $ok = $per_diff < $ok_per;

        return $ok;
    }

    return $per_diff;
}
Leave A Reply

Linux: Debian 9 major package versions

Debian 9 ships with these versions of some core packages:

Package Version
Apache 2.4.25
GCC 6.3.0
Kernel 4.9.0
Perl 5.24.1
PHP 7.0.19
Vim 8.0.550
Git 2.11.0
Leave A Reply

MySQL: Using if statements for data conversion in a SELECT

I have an integer field in my database that I'd like to display as Yes/No in a report. You can do this conversion in your front end code, or you can use a MySQL if statement to do the conversion for you.

SELECT IF(CustPrimary = 1,'Yes','No') AS CustPrimary FROM Customer;
Leave A Reply

Perl: hashes in list syntax and arrays in hash syntax

Today I learned that Perl list and hash syntax can be used interchangeably. If you use list syntax but assign to a hash Perl will convert the pairs in to hash key/values.

my @array = ("one", "two", "three", "four");      # Create a standard array
my %hash  = ("apple", "red", "banana", "yellow"); # Create a hash using list syntax
my %hash  = ("apple" => "red", "banana" => "yellow"); # Create a standard hash
my @array = ("one" => "two", "three" => "four");      # Create an array using hash syntax

This is also why qw() is able to create hashes. Perl automagically converts lists to hashes if they're being assigned to a hash (and they have an even number of elements):

my %turtles = qw(Donatello Purple Raphael Red Michelangelo Orange Leonardo Blue);
Leave A Reply

Vim: Plugins written in Perl

Vim has it's own internal scripting language called Vimscript, which is complicated and only appropriate in Vim. Most versions of Vim ship with Perl support. I taught myself how to write a simple Vim script in Perl. The following will define a Vim function named CommaToggle, that calls a perl function named comma_toggle. This will toggle spaces after commas on/off.

function! CommaToggle()
perl << EOF

# Get the current line number, and line text
my ($line_num,$column) = $curwin->Cursor();
my $line               = $curbuf->Get($line_num);

if ($line =~ /,/) {
    my $fixed = comma_toggle($line);
    $curbuf->Set($line_num,$fixed);
}

sub comma_toggle {
    my $line = shift();

    if ($line =~ /, /) {
        # Remove spaces after commas
        $line =~ s/, /,/g;
    } else {
        # Add a space after commas
        $line =~ s/,/, /g;
    }

    return $line;
}

EOF
endfunction

Other Vim/Perl commands are available from the documentation. Then you can map a key combination to call that function:

nnoremap <Leader>, :call CommaToggle()<cr>
Leave A Reply

Linux: AC3 is now legally free (patent unencumbered)

The Dolby Surround (aka AC3) audio codec is now legally free, along with MP3.

Leave A Reply

Git: Find the branch that a given commit landed on

If you need to find the branch that a commit was checked in to use the following command:

git branch --contains 26495cfd4ab17d4d685d0d352ed333f73d6d1b96

This should show a list of branches, with the respective branched highlighted and preceed with an asterisk.

Leave A Reply