stdin/stdout/stderr a primer

When building console apps you are going to hear a lot about three little entities: stdin, stdout and stderr.
In the Linux, Windows and OSX world any time you launch an application three file descriptors are automatically opened and attached to the application.
I refer to these three file descriptors as 'the holy trinity'. If you are going to do Command Line Interface (CLI) programming then it is imperative that you understand what they are and how to use them.
This primer discusses the origins, the structure and finally how to interact with the holy trinity in CLI apps.
stdin/stdout/stderr are not unique to dart. Virtually every langue supports them.
In the beginning
Let's take a little history lesson.
Way back in the dark ages (circa 1970) the computer gods got together and created Unix.
And Dennis said let there be 'C'. And Denis looked upon 'C' and said it was good and the people agreed.
But Dennis did not rest on the seventh day, instead he called upon Kenneth and over lunch they doth created Unix.
Dennis Ritchie ; 9th Sept 1944 - 12th Oct 2011
Kenneth Lane Thompson February 4, 1943
My first bible.
Unix is the direct ancestor of Linux, OSX and to a lesser extent Windows. You might more correctly say that 'C' is the common ancestor of all three OSs as their kernels are all written in C.
The concept of stdin/stdout and stderr proliferated across the OS world as C was taken up as the primary language for writing Operating Systems.
The result is today that a large no. of operating systems support stdin/stdout and stderr in all CLI applications.
The majority of people reading this primer will be working with Linux, OSx or Windows and in each of these cases the Holy Trinity (stdin/stdout/stderr) are available in every CLI app they use or write.
The following examples are presented using the Dart programming language but the concepts and even most of the details are correct across multiple OSs and languages.
When you have a hammer, everything's a snail
In the Unix world EVERYTHING is a file. Even devices and processes are treated as files.
If you know where to look, processes and devices are actually visible in the Linux/OSx directory tree as files.
So if everything is a file, does that mean we can directly read/write to a device/process/directory ....?
The simple answer is yes.
If we want to read/write to a file we need to open the file. In the Unix world (and virtually every other OS) when we open a file we get a 'file descriptor' or FD for short. Once we have an FD we can read/write to the file. The FD may be presented differently in your language of choice but under the hood its still an FD. (In Dart we have the File class that wraps an FD).
The terms 'file descriptor' and 'file handle' are often used interchangeably.
So what exactly is an FD? Under the hood an FD is just an integer that acts as an index to an array of open files. The FD array contains information such as the path to the file, the size of the file, the current seek position and more.
The Holy Trinity
So now we understand that in Unix everything is a file, you probably won't be surprised when I tell you that stdin/stdout/stderr are also files.
So if stdin/stdout/stderr are files how do you open them?
The answer is you don't need to open them as the OS opens them for you. When your app starts, it is passed one file descriptor (FD) for each of stdin/stdout/stderr.
If you recall we said that an FD is just an integer indexing into an array of structures, with one array entry for each open file. Each application has its own array. When your app starts that array already has three entries, stdin, stdout and stderr.
The order of those entries in the array is important.
[0] = stdin
[1] = stdout
[2] = stderr.
If you open any additional files they will appear as element [3] and greater.
The tower of Babel
If you have done any Bash, Zsh or Powershell programming you may have seen a line similar to:
1
find . '*.png' >out.txt 2>&1
Copied!
You can't get much more obtuse than the above line, but now we know about FD's it actually makes a little more sense.
Bash was not created by the gods. I think the other bloke had a hand in this one.
The >out.txt section is actually a shorthand for 1>out.txt . It instructs Bash to take anything that find writes to FD =1 (stdout) and re-write it to the file called 'out.txt'.
The 2> &1 section instructs Bash to take anything find writes to FD=2 (stderr) and re-write it to FD=1. i.e. anything written to stderr (FD=2) should be re-written to stdout (FD=1).
The result of the above command is that both stdout and stderr are written to the file called 'out.txt'.
It would have been less obtuse to write:
1
find . '*.png' 1>out.txt 2>out.txt
Copied!
But of course we are talking about Bash here and apparently more obtuse is always better :)
Many other shells use a similar syntax.
Most languages provide a specific wrapper for each these file handles. In Dart we have the global properties:
stdin
stdout
stderr
The 'C' programming language has the same three properties and many other languages use the same names.
And on this rock I will build my app
I like to think of the Unix philosophy as programming by Lego.
Unix was all about Lego - build lots of little bricks (apps) that can be connected.
In the Unix world (and the dart world) every CLI app you write contributes to the set of available Lego bricks. But Lego bricks would be useless unless you can connect them. In order to connect bricks the 'pegs' on each brick must match the 'holes' on other bricks and that's where stdin/stdout/stderr come in.
In the Unix world every brick (app) has three connection points:
stdin - a hole for input
stdout - a peg for normal output
stderr - a peg for error output
Any peg can go into any hole.
You might now have guessed that you can connect stdout from one program to stdin on another program:
(myapp -> stdout) -> (stdin -> yourapp)
If you are familiar with Bash you may have even seen one of the common ways to connect two apps.
1
ls "*.png" | grep "pengiuns"
Copied!
The '|' pipe operator connects the stdout of 'ls' to the stdin of 'grep'.
If you like, the 'pipe' command is the plumbing and Bash is the plumber.
Any data ls writes to it's stdout, is written to 'grep's stdin. The two apps are now connected via a 'pipe'.
A 'pipe' is just a program that reads from one FD and writes to another. When Bash sees the '|' character it takes it as an instruction to launch the two applications (ls and grep) read stdout from ls and write that data to stdin of grep.
A couple of other interesting things happened here.
1) stdin of ls is still connected to the terminal (ls is just ignoring it)
2) stdout of grep is still connected to the terminal anything that grep writes to its stdout will appear on the terminal.
Implement a shell
To make the whole concept a little more concrete we are going to implement a toy shell replacement for Bash.
Bash, Powershell and every other shell implement a read–eval–print loop (REPL).
Read input from the user, Evaluate the input (execute it), Print the results, Loop and do it again.
It turns it that its really easy to implement your own toy shell. So let's do it in just 50 lines of code.
1
#! /usr/bin/env dcli
2
​
3
import 'dart:async';
4
import 'dart:io';
5
​
6
import 'package:dcli/dcli.dart';
7
​
8
void main(List<String> args) {
9
  // Loop, asking for user input and evaluating it
10
  for (;;) {
11
    var line = ask('${green(basename(pwd))}${blue('>')}');
12
    if (line.isNotEmpty) {
13
      evaluate(line);
14
    }
15
  }
16
}
17
// Evaluate the users input
18
void evaluate(String command) {
19
  var parts = command.split(' ');
20
  switch (parts[0]) {
21
    case 'ls':
22
      ls(parts.sublist(1));
23
      break;
24
    case 'cd':
25
      Directory.current = join(pwd, parts[1]);
26
      break;
27
    default:
28
      if (which(parts[0]).found) {
29
        command.run;
30
      } else {
31
        print(red('Unknown command: ${parts[0]}'));
32
      }
33
      break;
34
  }
35
}
36
/// our own implementation of the 'ls' command.
37
void ls(List<String> patterns) {
38
  if (patterns.isEmpty) {
39
    find('*',
40
            root: pwd,  recursive: false,
41
            types: [Find.file, Find.directory])
42
        .forEach((file) => print('  $file'));
43
  } else {
44
    for (var pattern in patterns) {
45
      find(pattern, root: pwd,  recursive: false, types: [
46
        Find.file,
47
        Find.directory
48
      ]).forEach((file) => print('  $file'));
49
    }
50
  }
51
}
Copied!
If you save and run the above Dart script you will get an interactive shell. Here is a sample session:
1
./dshell.dart 
2
example> ls
3
  dshell.dart
4
example> mkdir tmp
5
example> cd tmp
6
tmp> touch me
7
tmp> ls
8
  me
9
tmp> cd ..
10
example> ls
11
  dshell.dart
12
  tmp
13
example>
Copied!
Implement a pipe
Now we have a basic shell let's extend it to implement a pipe.
Revelations
You take the red pill—you stay in Wonderland, and I show you how deep the rabbit hole goes.
So let's just stop for a moment and consider this fact; the terminal you are using is actually an app!
Like every other app it has stdin/stdout/stderr.
Writing to stdout
When you run an app in a console/terminal window your app's stdout is automatically piped to the terminal's stdin.
[print('hellow') -> stdout] -> [stdin -> terminal -> font -> graphics card -> eye -> brain]
When you call print('hello') your app writes 'hello' to stdout, this arrives in the terminal app via its stdin.
The terminal app then takes the ASCII characters you sent (hello) and sends little blobs of pixels to your graphics card. These blobs of pixel form, what many people like to call, a 'font'. Somehow, rather magically, your brain translates this little pixels into characters and you see the word 'hello'.
In the beginning was the Word, and the Word was 'hello world'.
The above example uses print to write to stdout. Print is a common function for writing to stdout and print or similar exists in most languages. Under the hood print literally writes to stdout:
So where does stderr fit in?
1
void print(String message)
2
{
3
    stdout.write('message\n');
4
}
Copied!
Well yes, I did, but it was a morally sound lie. I really wanted to avoid melting your brain.
And you will know the truth, and the truth will set you free.”
When your app is launched from the CLI (command line interface) your app is actually connected to the shell that launched. It doesn't matter if the shell was Bash, Powershell or Zsh.
In case you skipped the class, a command line interface (CLI) is a type of application referred to as a shell. A shell is designed to take keystrokes from a user, echo those keystrokes to the screen and when the user hits the enter key, try to interpret those keystrokes as a command. Often the command will be the name of an application, in which case the shell will start that application. Examples of shells are: Bash, Zsh, Powershell, cmd, Ash, Bourne, Korn, Hamilton............. and of course you could build your own.
So lets look at what actually happens when you launch a terminal window or connect to a console.
When the terminal window launches it creates a canvas to display text on and starts listening to keystrokes. If the terminal window has the focus then the OS will send keystrokes to it, otherwise it gets nothing. It then launches your default shell as a child process. Let's call this shell Bash but it could be called Powershell.
When Bash is launched it of course receives three file descriptors stdin/stdout and stderr.
The terminal window, being an app, also has its own stdin/stdout and stderr, but it essentially ignores them and they don't play a part in our process.
Reading from stdin
1
import 'dart:io';
2
​
3
void main() {
4
  stdout.writeln('Type something');
5
  String input = stdin.readLineSync();
6
  stdout.writeln('You typed: $input');
7
}
Copied!
So by default 'find''s stderr is also connected to 'greps' stdin. The pipe ''|' is doing this for use by interleaving stdout and stderr from 'find' into 'grep's stdin.
If you recall earlier we mentioned that most classes provide a wrapper for each of the holy trinity.
You can however separate stderr and stdout and read them independently.
So when we call:
1
printerr('An error occured');
Copied!
A program reading our stderr can process this separately.
Stdout
Stdout is the easiest to understand so let's start here.
In the classic hello world program, exactly how is the 'hello world' displayed to the user?
To put it simply; when you 'print' the string 'hello world', the print function writes 'hello world' to the file handle 'stdout'.
1
print('hello world');
Copied!
Stdin
If stdout is used to send data to the terminal, then stdin is used to receive data from the terminal.
More correctly we say that we write data to stdout and read data from stdin.
So if we want to capture what the user is typing into the terminal then we need to 'read' from stdin.
1
 user -&gt; hello -&gt; terminal -&gt; stdin -&gt; read
Copied!
Dart actually provides low level methods to directly read from stdin but their a little bit painful to work with.
As DCli likes to make things easy we provide the 'ask' function which does the hard work of reading from stdin.
1
String username = ask('username:');
Copied!
The 'ask' function prints 'username:' to stdout, then sits in a loop reading from 'stdin' until the user hits the enter key. When the user hits the enter key we return the anything they typed (and we read from stdin) and it is assigned to the variable 'String username';
Stderr
So stderr is both simple and complex.
Its simple in that by default it works just like stdout. If you write to stderr then it will appear on the console just the same as stdout. If fact a user can't tell the difference.
DCli provides a function to let you write to stderr:
1
printerr('hello world');
Copied!
So if it looks the same to the user why do we have both stdout and stderr?