| 1 | ---
|
| 2 | in_progress: yes
|
| 3 | ---
|
| 4 |
|
| 5 | The Unix Shell Process Model - When Are Processes Created?
|
| 6 | =============
|
| 7 |
|
| 8 | OSH and YSH are both extensions of POSIX shell, and share its underlying "process model".
|
| 9 |
|
| 10 | Each Unix process has its **own** memory, that is not shared with other
|
| 11 | processes. (It's created by `fork()`, which means that the memory is
|
| 12 | "copy-on-write".)
|
| 13 |
|
| 14 | Understanding when a shell starts processes will make you a better shell
|
| 15 | programmer.
|
| 16 |
|
| 17 | As a concrete example, here is some code that behaves differently in
|
| 18 | [bash]($xref) and [zsh]($xref):
|
| 19 |
|
| 20 |
|
| 21 | $ bash -c 'echo hi | read x; echo x=$x'
|
| 22 | x=
|
| 23 |
|
| 24 | $ zsh -c 'echo hi | read x; echo x=$x'
|
| 25 | x=hi
|
| 26 |
|
| 27 | If you understand why they are different, then that means you understand the
|
| 28 | process model!
|
| 29 |
|
| 30 | (OSH behaves like zsh.)
|
| 31 |
|
| 32 | ---
|
| 33 |
|
| 34 | Related: [Interpreter State](interpreter-state.html). These two docs are the
|
| 35 | missing documentation for shell!
|
| 36 |
|
| 37 | <div id="toc">
|
| 38 | </div>
|
| 39 |
|
| 40 | ## Shell Constructs That Start Processes
|
| 41 |
|
| 42 | ### Simple Command
|
| 43 |
|
| 44 | ls /tmp
|
| 45 |
|
| 46 | ### Pipelines `myproc | wc -l`
|
| 47 |
|
| 48 | Affected by these options:
|
| 49 |
|
| 50 | - `shopt -s lastpipe`
|
| 51 | - `set -o pipefail`
|
| 52 |
|
| 53 | Note that functions Can Be Transparently Put in Pipelines:
|
| 54 |
|
| 55 | Hidden subshell:
|
| 56 |
|
| 57 | { echo 1; echo 2; } | wc -l
|
| 58 |
|
| 59 | A `SubProgramThunk` is started for the LHS of `|`.
|
| 60 |
|
| 61 | ### Command Sub `d=$(date)`
|
| 62 |
|
| 63 | d=$(date)
|
| 64 |
|
| 65 | ### Process Sub `<(sort left.txt)`
|
| 66 |
|
| 67 | diff -u <(sort left.txt) <(sort right.txt)
|
| 68 |
|
| 69 | ### Async - `fork` or `sleep 2 &`
|
| 70 |
|
| 71 | ### Explicit Subshell - `forkwait` or `( echo hi )`
|
| 72 |
|
| 73 | Explicit Subshells are Rarely Needed.
|
| 74 |
|
| 75 | - prefer `pushd` / `popd`, or `cd { }` in YSH.
|
| 76 |
|
| 77 |
|
| 78 | ## FAQ: "Subshells By Surprise"
|
| 79 |
|
| 80 | Sometimes subshells have no syntax.
|
| 81 |
|
| 82 | Common issues:
|
| 83 |
|
| 84 | ### shopt -s lastpipe
|
| 85 |
|
| 86 | Mentioned in the intro:
|
| 87 |
|
| 88 | $ bash -c 'echo hi | read x; echo x=$x'
|
| 89 | x=
|
| 90 |
|
| 91 | $ zsh -c 'echo hi | read x; echo x=$x'
|
| 92 | x=hi
|
| 93 |
|
| 94 | ### Other Pipelines
|
| 95 |
|
| 96 | myproc (&p) | grep foo
|
| 97 |
|
| 98 | ## Process Optimizations - `noforklast`
|
| 99 |
|
| 100 | Why does a Unix shell start processes? How many processes are started?
|
| 101 |
|
| 102 | Bugs / issues
|
| 103 |
|
| 104 | - job control:
|
| 105 | - restoring process state after the shell runs
|
| 106 | - `sh -i -c 'echo hi'`
|
| 107 | - traps
|
| 108 | - not run - issue #1853
|
| 109 | - Bug with `set -o pipefail`
|
| 110 | - likewise we have to disable process optimizations for `! false` and
|
| 111 | `! false | true`
|
| 112 |
|
| 113 | Oils/YSH specific:
|
| 114 |
|
| 115 | - `shopt -s verbose_errexit`
|
| 116 | - crash dump
|
| 117 | - because we don't get to test if it failed
|
| 118 | - stats / tracing - counting exit codes
|
| 119 |
|
| 120 |
|
| 121 | ## Process State
|
| 122 |
|
| 123 | ### Redirects
|
| 124 |
|
| 125 | ## Builtins
|
| 126 |
|
| 127 | ### [wait]($help)
|
| 128 |
|
| 129 | ### [fg]($help)
|
| 130 |
|
| 131 | ### [bg]($help)
|
| 132 |
|
| 133 | ### [trap]($help)
|
| 134 |
|
| 135 | ## YSH
|
| 136 |
|
| 137 | ### io.captureAll()
|
| 138 |
|
| 139 | Gets stdout, stderr, status all at once. Other shells can't do this.
|
| 140 |
|
| 141 | ### Ideas
|
| 142 |
|
| 143 | - Rich history: this feature may fork a process for each interactive line, with
|
| 144 | a PTY (rather than a pipe) connecting the processes
|
| 145 | - The "pipecar" process to turn process completion events into pipe events?
|
| 146 | - Or perhaps we need general coroutines, like async/await
|
| 147 |
|
| 148 | ## Appendix
|
| 149 |
|
| 150 | ### Non-Shell Tools
|
| 151 |
|
| 152 | These Unix tools start processes:
|
| 153 |
|
| 154 | - `init` - the process supervisor
|
| 155 | - `find -exec`
|
| 156 | - has a mechanism for batching, e.g. with `find . -exec echo {} +` vs. `\;`
|
| 157 | - `xargs -P` starts parallel processes
|
| 158 | - but it doesn't do anything with stdout
|
| 159 | - GNU parallel is enhanced version of `xargs -P`
|
| 160 | - a Perl program in one big 16K line file!
|
| 161 | - has many interesting man pages `man parallel_design`, `man
|
| 162 | parallel_alternatives`, ...
|
| 163 | - many integrations: SSH, SQL, fish shell, etc.
|
| 164 | - `make`
|
| 165 | - `make -j` starts parallel processes (but doesn't buffer output)
|
| 166 | - there is the "job server protocol", which works across child processes,
|
| 167 | e.g. grandchildren and more
|
| 168 | - `ninja` (buffers output)
|
| 169 | - `docker build` (BuiltKit I think) streams logs from parallel processes
|
| 170 | - in a way that's more fine-grained than ninja
|
| 171 | - Start servers and wait for them to respond: <https://github.com/zombocom/wait_for_it>
|
| 172 | - Not parallel, but could be - we want parallel FD events
|
| 173 |
|
| 174 | Another concurrent programming problem:
|
| 175 |
|
| 176 | - concurrent HTTPS client
|
| 177 | - it's possible use `openssl_client`, and do our own HTTP parsing, if we
|
| 178 | insist on controlling all languages/protocols
|
| 179 | - although if we're using Python, it already has SSL. It doesn't have
|
| 180 | a built-in async HTTPS client, but there are many third party ones
|
| 181 | - we will probably use Python's async HTTP servers anyway
|
| 182 |
|
| 183 | <!-- TODO
|
| 184 | - Do most of these in Python async/await
|
| 185 | - And node.js with async/await
|
| 186 | - can Deno and Bun runtimes do them?
|
| 187 | - Lua has "symmetric coroutines"
|
| 188 | - same with Julia - https://docs.julialang.org/en/v1/manual/asynchronous-programming/
|
| 189 |
|
| 190 | - Shell philosophy
|
| 191 | - Bootstrap YSH, and then YSH can download these tools, with the right
|
| 192 | runtime?
|
| 193 | - A problem is that you can't serialize YSH closures, but that might be OK
|
| 194 | - or we can probably figure out a way to do it
|
| 195 | -->
|
| 196 |
|
| 197 | ### Related
|
| 198 |
|
| 199 | - When reading from multiple pipes with `select()`, we may have to solve the
|
| 200 | [Framing Problem](framing.html)
|
| 201 |
|