Archive for Month: September 2015
Linux + C – Your New Friend, Awk
In modern internet lingo, awk is a synonym for awkward or abnormal. However, the Linux tool awk (and its brethren, gawk and nawk) are anything but awkward.
Awk – the programmable filter
So far, we’ve looked at a set of filters that perform fairly singular tasks. LESS and MORE are designed to filter output to the screen for easier viewing. SED is designed to use regular expressions to find and replace values in a stream. CAT is designed to print a file to the terminal, and ECHO is designed to print some text to the terminal.
What we’ve lacked so far is a filter whose operations are almost entirely under our control. While we had a lot of power with sed, we could only reliably use it to replace simple patterns in a stream; it wasn’t designed to, say, print a set of fields from a table to our terminal, then create a running sum of those field values.
This is where awk comes into play.
AWK (named after its developers, Aho, Weinberger, and Kernighan) is a pseudo-C language designed to manipulate files (and streams, but that’s not it’s best use), allowing us to create complex filtering operations with simple syntax and operations. There are a number of things we can teach the filter to do:
- Find and replace in a similar manner to sed
- Create multi-dimensional arrays
- Print individual fields out of a file (extremely useful for filtering log files and raw data)
- Print only specific lines out of a file, based on length or contents
- Perform arithmetic operations, including increment and decrement
Perhaps there are even more applications for this fascinating filter, but so far I’ve not had need for them.
Three Stages of Awk
There are three basic phases of an awk program: the beginning, middle, and end. (Ed. Whoa, what a crazy concept)
The beginning phase occurs before awk begins to operate on the file. We can set up variables here, print lines of text, and perform system operations (terminal commands) exactly once, before we start processing any data.
The middle phase is performed on every line of the input file(s). This is where we do most of our filtering.
The end phase occurs exactly once, when the middle phase has finished reading all inputs. Generally, we use this phase to print out some results, or provide a message, or call some system operation.
An awk script basically looks like this:
Soon we’ll look at some of the nifty features that make awk such a powerful tool.
Linux + C – Brief Overview of Sed
Sed is a powerful editing tool, but difficult for most modern computer users to grasp. We’ll cover some of the basics here so that you can understand it when you see it in a script. Continue Reading…
Linux + C – Some Useful Filters
Soon we’ll talk about perhaps the most powerful programmable filter ever conceived (the scripting language AWK), but first let’s look at a few other common filters and their uses.
The filter “more” is a program that lets us print output one screen at a time. If you’ve ever run a diff between two files or a recursive ls, you might know that the output can be thousands of lines long. Normally, these lines will be presented in one long stream, and you will never see more than the hundred or so lines that the terminal stores in memory by default.
The solution is simple enough. more stores all these lines in memory and prints them out one screen at a time. This way, you can see all the output without piping it to a file and opening it in vi.
The most common usage of more is as follows:
cat file_name | more
Note: cat is a filter that prints a file out to the terminal. However, any program that creates output can be piped to more.
Unfortunately for more, it has no history. That means that you can only ever scroll down, not up.
The solution? less is a program that acts like more, but it allows you to scroll up and down as well. The wikipedia page on less is more useful (in my mind) than the manual page, but basically you can scroll down with the Space Bar (just like more) and scroll back up with b. You can also scroll by one line using j and k (j down, k up). To quit this program (just like with more), you hit the q key.
We usually call less exactly like we call more:
cat file_name | more
Sed is a stream editor (as opposed to a file editor). What this means is that the tool takes a stream of input, performs some operation on it, and moves on.
Sed is less powerful (and generally less useful) than awk, but I do have one excellent application: removing tabs from the end of a line.
The following operations will extract the tabs and spaces (respectively) from the end of lines in a file. Very useful for someone like me, who tends to accidentally leave a tab or two in some files.
sed -i ‘s/[\t]*s//’ File_Name
sed -i ‘s/[ ]*s//’ File_name
While we’re talking about printing lines on a screen, it’s worth noting that the program “clear” will print a number of blank lines sufficient to clear your terminal window. This makes it look somewhat like you just started the terminal, but with all the history and memory of the terminal still intact.
If you’re done with all the output on your screen, just run clear once.
Hypocrisy, God’s Goodness, and Right Judgment
Grace and Peace to you from God our Father and the Lord Jesus Christ.
Today we’ll be looking at another famous section of the “Sermon on the Mount.” Here, Jesus gives us counsel on justice between men and with God.
Reading: Matthew 7:1-23
Judge not, that ye be not judged.
Behold, one of the most misunderstood and misused verses in all of Scripture!
2 For with what judgment ye judge, ye shall be judged: and with what measure ye mete, it shall be measured to you again.
It was common among the Pharisees to pronounce judgment and sentencing on those around them. Much like the Westboro Baptist Church today, they would make all manner of declarations, occasionally violating Roman law in order to pass the sentences they saw fit.
However, many of these leaders were every ounce as wicked as those they condemned. In the name of Justice, the measure cannot be different for one man as another. Thus, Jesus warns us against posing harsh sentences against others.
This is NOT to say that we may not speak that which the Lord has spoken. When he condemns homosexuality, adultery, blasphemy, or false testimony, it is our duty to speak out what He has declared. However, when we move beyond His word into our own, we open ourselves up to the full measure of justice.
3 And why beholdest thou the mote that is in thy brother’s eye, but considerest not the beam that is in thine own eye?
4 Or how wilt thou say to thy brother, Let me pull out the mote out of thine eye; and, behold, a beam is in thine own eye?
5 Thou hypocrite, first cast out the beam out of thine own eye; and then shalt thou see clearly to cast out the mote out of thy brother’s eye.
Hypocrites are notoriously unaware of their own flaws, whether through sheer narcissism or studious denial.
Tony Robbins, for all that he’s an excellent motivational speaker and success coach, has been divorced a number of times; therefore, it is foolish to listen to his advice on maintaining a lasting marriage. Steve Harvey is in a similar boat, as are many other “relationship experts” we could name.
Why take advice on kicking your drug addiction from someone who is in perpetual relapse? Or why take career advice from a man losing money hand over fist?
If you wish to help others, you must first correct the problems in yourself.
6 Give not that which is holy unto the dogs, neither cast ye your pearls before swine, lest they trample them under their feet, and turn again and rend you.
Don’t throw bad money after good. Don’t adhere to a diet that has never worked for you, or with which you cannot be happy. Don’t pine after a relationship that only brought you pain and heartbreak.
Basically, cut your losses and start over, rather than continue working on the wrong things.
7 Ask, and it shall be given you; seek, and ye shall find; knock, and it shall be opened unto you:
8 For every one that asketh receiveth; and he that seeketh findeth; and to him that knocketh it shall be opened.
This is the way the Lord cares for us. Keep this verse in your heart, for it is a source of great comfort in trials and tribulation.
9 Or what man is there of you, whom if his son ask bread, will he give him a stone?
10 Or if he ask a fish, will he give him a serpent?
11 If ye then, being evil, know how to give good gifts unto your children, how much more shall your Father which is in heaven give good things to them that ask him?
Most Western atheists and a number of Christians hold a deep-seated belief that, if there is a God, he is a cruel master who delights in our pain. However, Jesus knows differently.
If God is Good, and God is our Father, how can he delight in our pain and send misery into our lives? Even a mediocre parent provides for his or her children in some way, even if that provision is lacking in some critical areas. How much better must a truly Good Father provide for his children, if they only ask?
12 Therefore all things whatsoever ye would that men should do to you, do ye even so to them: for this is the law and the prophets.
In another passage, Jesus is asked which commandment is greatest. He provides the greatest and its close second:
Thou shalt love the Lord thy God with all thy heart, and with all thy soul, and with all thy mind. This is the first and great commandment.
And the second is like unto it: Thou shalt love thy neighbour as thyself.
Twice, then, we see Jesus delivering to us the same law regarding proper behavior in our society: whatever you would have them do to you, do so unto them. The “self improvement” corollary is to treat them as they want to be treated, so that they will treat you as you want to be treated.
Any law which conforms not to the first and second great commandments (the “Platinum” and “Golden” rules, as they are often known) has nothing to do with justice or goodness. If you wish to rebel against society or the common law, you should strive to only violate those societal laws which either oppose or disregard the morality of these two great laws. In this, that which Sun Tzu called the “moral authority” remains with you and against that which you oppose.
13 Enter ye in at the strait gate: for wide is the gate, and broad is the way, that leadeth to destruction, and many there be which go in thereat:
14 Because strait is the gate, and narrow is the way, which leadeth unto life, and few there be that find it. (Matthew 22)
Oprah and the other spiritual leaders of this day would have you believe that every lifestyle is a path to God (except, maybe, a devout Christian life). Jesus knew these leaders would come, and so gave us this teaching.
There is but one Way, Truth, and Life, and that is Jesus Christ. He is the Bread of Life, the Alpha and Omega, and the Name above all names. There is no other name under Heaven given unto men by which we must be saved (Acts 4).
15 Beware of false prophets, which come to you in sheep’s clothing, but inwardly they are ravening wolves.
16 Ye shall know them by their fruits. Do men gather grapes of thorns, or figs of thistles?
If you wish to judge a man’s teaching and his life, look at what he produces.
Marx performed no labor in the course of his life, and his teachings are followed with poverty, hatred, violence, and censorship. One has only to look at the former USSR, the history of China and the Vietnamese, and the modern Social Justice movements to see the vitriol and mindless rage that such teachings evoke.
The most devout followers of Islam become murderers, liars, rapists, and socially disobedient. A cursory glance at the supposed “golden years” of Islam demonstrate that they only ever produced a repackaging of the ideas of their predecessors (Hospitals, Algebra, and even their architectural achievements were derived from the Romans and Byzantines).
The Progressives in the West have set up a financial system whose coming collapse (which is inevitable) will rock the world. They have also produced entire classes of “victims” who live on the wealth extracted by force from the greatest producers in our country, and who have never developed the knowledge or skills required to rise above that status.
If a philosophy or religion has not produced good in the past, why would you expect it to produce good in the future?
17 Even so every good tree bringeth forth good fruit; but a corrupt tree bringeth forth evil fruit.
18 A good tree cannot bring forth evil fruit, neither can a corrupt tree bring forth good fruit.
19 Every tree that bringeth not forth good fruit is hewn down, and cast into the fire.
20 Wherefore by their fruits ye shall know them.
Those ideas which produce only bitter and violent fruit should be cast away immediately. As listed above, Marxism and its derivatives have produced naught but evil in our world. So too have modern Feminism (which is filled with man-hatred, socialism, and lack of personal responsibility for a woman’s actions), Islam (which has produced devastation only rivaled by the Socialist dictators of Marxism), and Progressivism (which has produced cancerous thoughts and laws that will likely cripple the West in the next few years).
To a lesser degree, modern television programming has tended to produce laziness and intellectual sloth, while emasculating men and ridiculing Christianity at every opportunity. This too should be avoided, because it does not inspire any degree of greatness.
The greatest advice is that of Paul: “Whatsoever things are honest, just, pure, lovely, of good report, excellent, virtuous or praiseworthy, think on such things.”
21 Not every one that saith unto me, Lord, Lord, shall enter into the kingdom of heaven; but he that doeth the will of my Father which is in heaven.
22 Many will say to me in that day, Lord, Lord, have we not prophesied in thy name? and in thy name have cast out devils? and in thy name done many wonderful works?
23 And then will I profess unto them, I never knew you: depart from me, ye that work iniquity.
Don’t be a “churchian” – one who knows all the words and attends all the services, but who does not do that which the Lord commands. Strive to be a Christian, one who unashamedly serves the Lord and preaches His commands.
Let us Pray
Righteous Lord whose hand covers over all of Creation, we thank you that You are Good and Just. Without you, Lord, we know that we would have to rely on our personal feelings to create morality, and we would summarily lose ourselves in pettiness and evil.
Reveal to us our weaknesses and wickedness, and direct us so that we may “remove the plank from our own eyes”. Then, Lord, we trust that you will guide us to help others in the same way You have helped us.
We know that we are impure creatures, Lord, and we know that we lack discipline to discern righteousness from wickedness. Grant us a double measure of the Gift of Discernment, and let us Prophesy boldly against that wickedness with which we struggle in this world.
We ask that you do all things, in accordance with Your Will, in the name of your Son, Jesus, who is the Christ.
Go in the peace and power of Almighty God.
“Nothing is Neutral”
An excellent article by Caleb Jones of Sublime Your Time.
Utter neutrality is not a condition that exists in human life. Everything you do or do not do either harms you or helps you. Everything either gives or takes away. Everything either adds or lessens. There isn’t any action (or inaction) that results in “nothing.” Nothing is neutral.
Just by adopting the “nothing is neutral” mindset, you can’t not improve your daily conditions, at least by a little.
As they say at the gym, “You’re either moving forward or you’re moving back.”
Linux + C – Regular Expressions (REGEX)
Many valuable programs in Linux rely on the concept of the Regular Expression (REGEX). It’s important for us to understand the basics of this “language” so that we can better use the tools we’re provided. Continue Reading…