logo

blog

My website can't be that messy, right? git clone https://anongit.hacktivis.me/git/blog.git/

bootstrapping.shtml (16188B)

    

  1. <!DOCTYPE html>
  2. <html xmlns="http://www.w3.org/1999/xhtml">
  3. 	<head>
  4. <!--#include file="/templates/head.shtml" -->
  5. 		<title>Bootstrapping — lanodan’s cyber-home</title>
  6. 	</head>
  7. 	<body>
  8. <!--#include file="/templates/en/nav.shtml" -->
  9. 		<main class="section-count">
  10. 			<h1>Bootstrapping</h1>
  11. 			<ul>
  12. 				<li><a href="https://bootstrappable.org/">Bootstrappable Builds</a> (GNU Guix focus)</li>
  13. 				<li><a href="https://bootstrapping.miraheze.org/wiki/Main_Page">bootstrapping wiki</a></li>
  14. 				<li><a href="https://dwheeler.com/trusting-trust/">David A. Wheeler’s Page on Fully Countering Trusting Trust through Diverse Double-Compiling (DDC) - Countering Trojan Horse attacks on Compilers</a> (Note: Requires trustworthy bootstrap compiler(s) as starting point)</li>
  15. 				<li><a href="https://www.quora.com/What-is-a-coders-worst-nightmare/answer/Mick-Stute?srid=tQ46&amp;share=1">Mike Stute's answer to What is a coder's worst nightmare?</a></li>
  16. 			</ul>
  17. 			<h2>Reasons</h2>
  18. 			<dl>
  19. 				<dt>Security</dt>
  20. 				<dd>See the "<a href="#devtools-backdoors">Backdoors inserted into dev tools</a>" section</dd>
  22. 				<dt>Portability</dt>
  23. 				<dd>Binary executables have much higher <a href="https://en.wikipedia.org/wiki/Software_rot">bitrot</a> than source code and keeping obsolete binary interfaces often means keeping known security issues.</dd>
  25. 				<dt>Maintainability</dt>
  26. 				<dd>By making sure someone else can actually continue maintaining the software, canonical versions or forks</dd>
  28. 				<dt>Reproducibility's other side of the coin</dt>
  29. 				<dd>One of <a href="https://reproducible-builds.org/">reproducibility</a>'s effect is allowing to audit source code instead of binaries, but said source code needs to be actually used.</dd>
  30. 			</dl>
  32. 			<h2 id="devtools-backdoors">Backdoors inserted into dev tools</h2>
  33. 			<p>This is by no means an exhaustive list, mostly because it happens way too regularly on npm.</p>
  35. 			<h3>Ken Thompson “Reflections on Trusting Trust” Compiler</h3>
  36. 			<p>In Chronological order:</p>
  37. 			<ol>
  38. 				<li>1983-10: <a href="https://dl.acm.org/doi/10.1145/358198.358210">Reflections on Trusting Trust, Ken Thompson</a></li>
  39. 				<li>2022-07-27: <a href="https://niconiconi.neocities.org/posts/ken-thompson-really-did-launch-his-trusting-trust-trojan-attack-in-real-life/">Ken Thompson Really Did Launch His "Trusting Trust" Trojan Attack in Real Life</a></li>
  40. 				<li>2023-03-12: <a href="https://www.youtube.com/live/kaandEt_pKw?t=3284">Q&A section after Ken Thompson talk on keeping his music collection</a> (<a href="https://www.socallinuxexpo.org/scale/20x/presentations/keynote-ken-thompson">Talk page</a>)</li>
  41. 				<li>2023-10-25: <a href="https://research.swtch.com/nih">research!rsc: Running the “Reflections on Trusting Trust” Compiler</a>: This notably contains the code that Ken Thomspon used together with explainations</li>
  42. 			</ol>
  44. 			<h3>CVE-2024-3094: Jia Tan backdoor in xz-utils</h3>
  45. 			<p>
  46. 				This one is noteworthy for primarily being an insertion of a payload
  47. 				into a <code>./configure</code> script generated by autotools rather
  48. 				than a binary; being a near-successful attack on OpenSSH when patched
  49. 				to link with systemd-journald; having been detected pretty much
  50. 				by a combination of sheer curiosity and lucky pre-existing benchmark.
  51. 			</p>
  52. 			<ol>
  53. 				<li><a href="https://www.openwall.com/lists/oss-security/2024/03/29/4">Discovery by Andres Freund</a></li>
  54. 				<li><a href="https://tukaani.org/xz-backdoor/">https://tukaani.org/xz-backdoor/</a></li>
  55. 				<li><a href="https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27">Detailed FAQ on the xz-utils backdoor</a> by Sam James</li>
  56. 			</ol>
  58. 			<h3>Proof of Concepts</h3>
  59. 			<ul>
  60. 				<li><a href="https://manishearth.github.io/blog/2016/12/02/reflections-on-rusting-trust/">Reflections on Rusting Trust</a>: Backdooring The One True Rust Compiler.</li>
  61. 			</ul>
  63. 			<h2>Definitions</h2>
  64. 			<dl>
  65. 				<dt>Source Code</dt>
  66. 				<dd>
  67. 					<p>
  68. 						As shared by GPLv3,
  69. 						the <a href="https://www.gnu.org/philosophy/free-sw.html#make-changes">Free Software Definition</a>,
  70. 						and the <a href="https://opensource.org/osd">Open-Source Definition</a>.
  71. 						Each with little differences, reproduced the latter below as it explictly exclude
  72. 						obfuscation and codegen:
  73. 					</p>
  74. 					<blockquote>
  75. 						[The] Source code must be the preferred form in which a programmer would
  76. 						modify the program.  Deliberately obfuscated source code is not allowed.
  77. 						Intermediate forms such as the output of a preprocessor or translator
  78. 						are not allowed.
  79. 					</blockquote>
  80. 				</dd>
  81. 			</dl>
  83. 			<h2 id="tools">Tools</h2>
  84. 			<dl>
  85. 				<dt><a href="https://hacktivis.me/projects/deblob">deblob</a></dt>
  86. 					<dd>Remove known binary executable formats (including bytecode), designed to be fast enough to barely impact distro-scale package building performance, cannot detect all blobs</dd>
  87. 				<dt>Debian's <a href="https://salsa.debian.org/debian/devscripts/-/blob/master/scripts/suspicious-source">suspicious-source</a> script</dt>
  88. 					<dd>Lists what isn't present in a list of source code formats, good for manual audits. Python+<code>magic(5)</code> means it is quite slow.</dd>
  89. 			</dl>
  91. 			<h2>Disclaimers</h2>
  92. 			<p>
  93. 				Unless you plan to make a system language, bootstrapping doesn't
  94. 				have to be done from C or a low-level language.
  95. 				It's fine to use any language as long at it's also bootstrappable
  96. 				and wouldn't introduce circular dependencies.
  97. 			</p>
  98. 			<p>
  99. 				For example: Go and Lua are fine.
  100. 				Perl and Python are a bit involved but live-bootstrap got both.
  101. 				Any language supported by GCC is also okay, although a simpler backend such as
  102. 				<a href="https://c9x.me/compile/">QBE</a>
  103. 				could be more interesting.
  104. 			</p>
  106. 			<h2>Non-Problematic / Praise</h2>
  107. 			<h3 id="go">Go</h3>
  108. 			<p>
  109. 				<a href="https://golang.org/doc/install/source">Installing Go from source</a>
  110. 				in the official Go documentation details it, both GCCGO and a branching
  111. 				out of Go 1.4 are supported.
  112. 			</p>
  113. 			<p>
  114. 				More recent versions do require a bootstrap chain but as Go's own
  115. 				toolchain is standalone it's fine.
  116. 			</p>
  118. 			<h2>Historically problematic</h2>
  120. 			<h3 id="firefox_python2">Firefox &gt;=68 &lt;=78</h3>
  121. 			<p>Firefox would bundle python2 and refuse to build if removed. See <a href="https://salsa.debian.org/mozilla-team/firefox/-/commits/esr78/master/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin">Debian firefox-esr source history</a></p>
  123. 			<h2>Potentially problematic</h2>
  125. 			<h3>OCaml</h3>
  126. 			<p>Has binary seeds in <code>./boot</code>, there is <a href="https://github.com/Ekdohibs/camlboot">camlboot</a> but it seems to be pretty inefficient (takes hours to compile when regular ocaml takes minutes to compile)</p>
  128. 			<h2>Problematic software</h2>
  130. 			<h3 id="zig">Zig</h3>
  131. 			<p>
  132. 				<a href="https://ziglang.org/news/goodbye-cpp/">Threw out the C++ implementation in favor of a <strong>large</strong> WASM binary seed</a> in 0.10.0,
  133. 				for now it's chained-bootstrapping.
  134. 				Hopefully an alternative compiler written in a bootstrapped language
  135. 				will appear, because keeping versions of LLVM all the way to 13
  136. 				working properly like Guix does just doesn't seems reasonable.
  137. 			</p>
  138. 			<p>
  139. 				<a href="https://jakstys.lt/2024/zig-reproduced-without-binaries/">Zig Reproduced Without Binaries</a> 
  140. 				(<a href="https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74217">related debbugs.gnu.org entry</a>):
  141. 				Successfully reproducing Zig binaries within Guix, sadly inpractical as it
  142. 				uses 53+ intermediate versions between 0.9.1-ish (inclusive) and 0.13.
  143. 			</p>
  145. 			<h3 id="erlang">Erlang</h3>
  146. 			<p>Documented as originally implemented in prolog, now version <i class="math">n</i> requires binaries version <i class="math">n-1</i> or <i class="math">n</i> to build. No alternative compiler known so far.</p>
  148. 			<h3 id="rust">Rust</h3>
  149. 			<p>
  150. 				There is <a href="https://github.com/thepowersgang/mrustc">mrustc</a>
  151. 				(packaged in Guix and Gentoo)
  152. 				but it tends to lag behind by about ten 1.x versions,
  153. 				which sadly you each need to compile as intermediary steps.
  154. 				Rustc also vendors several other projects like LLVM and rust crates
  155. 				(enjoy non-installable libraries), similarly to other rust software.
  156. 			</p>
  157. 			<p>
  158. 				GCC Rust Frontend is also not ready yet (2023-03) for userland,
  159. 				as <a href="#cargo">cargo</a> doesn't bootstraps…
  160. 			</p>
  162. 			<h3 id="cargo">Cargo</h3>
  163. 			<p>
  164. 				As if rustc being a bootstrapping problem wouldn't be enough, cargo,
  165. 				the buildsystem+dependency-installer for Rust software depends on
  166. 				<a href="https://github.com/rust-lang/cargo/blob/master/Cargo.toml">~60 direct libraries</a>,
  167. 				notably including 2+ git libraries, HTTP Authentication, and OpenSSL.<br />
  168. 			</p>
  169. 			<p>
  170. 				Cargo isn't a buildsystem, it's a full blown package manager
  171. 				and a troublemaker when it comes to dependency management due to
  172. 				<a href="https://drewdevault.com/2022/05/12/Supply-chain-when-will-we-learn.html">designed-vulnerable crates.io</a>.
  173. 			</p>
  174. 			<p>
  175. 				It really ought to be replaced by something which only
  176. 				takes care of building code (or even just generating
  177. 				a <code>Makefile</code> or a <code>build.ninja</code> file),
  178. 				as was done in the C ecosystem many times in the past
  179. 				(pkg-config ⇒ <a href="https://gitea.treehouse.systems/ariadne/pkgconf">pkgconf</a>,
  180. 				ninja ⇒ <a href="https://github.com/michaelforney/samurai">samurai</a>,
  181. 				…).<br />
  182. 				This isn't a system that scales, this is just creating a gigantic blob
  183. 				of software that cannot be reasonably audited, right in the toolchain.
  184. 			</p>
  186. 			<h3 id="java">Java</h3>
  187. 			<p>Requires compilers abandonned ~10 years ago, currently doesn't builds to OpenJDK for me.</p>
  189. 			<h3>Free-Pascal Compiler / Object Pascal</h3>
  190. 			<p><a href="https://bootstrapping.miraheze.org/wiki/Aesop">Aesop</a> seems to still be at the vaporware stage, no code is available.</p>
  192. 			<h3 id="nim">Nim</h3>
  193. 			<p>
  194. 				The transpiled C non-source code used for bootstrapping contained in <code>./c_code/</code> is pretty much what you would get with C++ mangled symbols auto-decompiled to C.<br />
  195. 				<a href="https://bootstrapping.miraheze.org/wiki/Bootstrapping_Nim">Bootstrapping Nim via historical releases</a> would need a bootstrap path for Object Pascal, which doesn't exists (yet?), another way would be to have a minimal Nim compiler written in another language which is capable of compiling the current compiler.
  196. 			</p>
  198. 			<h3 id="qemu">QEMU</h3>
  199. 			<p>
  200. 				QEMU 7.0 <a href="https://github.com/gentoo/gentoo/commit/11c7bca43160b3d893dc8d846d8da2838332123c">needs a quick fix on the <code>pc-bios/meson.build</code> file</a> so you can choose to not use the binaries it ships, fixed in QEMU 7.1.<br />
  201. 				They are still required so it means identifying the source of all of them and having proper from-source packaging, it's already done in gentoo for Seabios and EDK2-OVMF (UEFI) which is enough to boot machines but not for full-x86 support, non-x86 being even more problematic (ie. which upstream is used for OpenBIOS/OpenFirmware as used for sparc32, sparc64 and ppc32).
  202. 			</p>
  204. 			<h3 id="wine-mono">wine-mono</h3>
  205. 			<p>In gentoo it's a collection of binaries. Upstream repository is at <a href="https://github.com/madewokherd/wine-mono">https://github.com/madewokherd/wine-mono</a> but still includes binaries…</p>
  207. 			<h3 id="mono">mono / .NET</h3>
  208. 			<p>
  209. 				Source-only building is unsupported and nearly impossible (massive chain + intermediary unstable versions).<br />
  210. 				Should also be noted that Mono started itself with the Microsoft C# compiler (<a href="https://www.mono-project.com/docs/about-mono/history/">History | Mono</a>) instead of <a href="https://www.gnu.org/software/dotgnu/">dotGNU</a> (which is dead since 2012).
  211. 			</p>
  212. 			<p>2024-11-30 Update: unmush managed to build mono all the way to 6.12.0 on Guix: <a href="https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74609">[PATCH] Adding a fully-bootstrapped mono</a></p>
  213. 			<ul>
  214. 				<li><a href="https://issues.guix.gnu.org/55026">potential prebuilt binaries in the Mono package</a></li>
  215. 				<li><a href="https://github.com/mono/mono/issues/7445">Cannot build without binary-reference-assemblies</a></li>
  216. 				<li><a href="https://github.com/dotnet/source-build/issues/1930">Full source bootstrap · Issue #1930 · dotnet/source-build</a></li>
  217. 				<li><a href="/notes/mono-6.12.0.199_deblob.log">mono-6.12.0.199_deblob.log</a>, <a href="/notes/mono-6.12.0.199_deblob.json">mono-6.12.0.199_deblob.json</a>: Automatically generated list of blobs via deblob on mono-6.12.0.199.tar.xz</li>
  218. 			</ul>
  220. 			<h3 id="chez">Chez Scheme</h3>
  221. 			<p>Requires bootstrap files, <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/chez.scm">GNU GuixSD packaging</a> doesn't seems to have it figured out yet.</p>
  223. 			<h3 id="neko">NekoVM</h3>
  224. 			<p>Doesn't seems possible to build without <code>boot/*.n</code> files being present, which are NekoVM bytecode files.</p>
  226. 			<h3 id="nqp">Not Quite Perl (NQP)</h3>
  227. 			<p>
  228. 				Doesn't seems possible to build without <code>src/vm/moar/stage0/*.moarvm</code> files being present, which are MoarVM bytecode files.
  229. 				This means no Rakudo/Perl6.
  230. 			</p>
  232. 			<h3 id="gnulib">GNU gnulib</h3>
  233. 			<p><code>lib/javaversion.class</code>. Made <a href="https://hacktivis.me/tmp/0001-lib-javaversion.class-Remove-build-from-source.patch">[PATCH] lib/javaversion.class: Remove, build from source</a> to have it built from source.</p>
  235. 			<h3 id="gettext">GNU gettext</h3>
  236. 			<p>gnulib java blob; 3 Java class files in <code>gettext-tools/examples</code>; <code>gettext-tools/m4/csharpexec-test.exe</code> which doesn't have source code (C# is effectively proprietary anyway). Did <a href="https://github.com/gentoo/gentoo/commit/54b36e80f7c3910ae1557c2faafda3d6d62daf49">sys-devel/gettext: deblob</a> to fix it.</p>
  238. 			<h3 id="typescript">TypeScript</h3>
  239. 			<p>Compiler itself is written in TypeScript, no bootstrap path possible as the <a href="https://github.com/microsoft/TypeScript/commit/214df64e287804577afa1fea0184c18c40f7d1ca">commit introducing the compiler</a> is TypeScript code. Want TypeScript compiler? Get a blob from <code>npmjs.org</code>, like the <a href="https://github.com/microsoft/TypeScript/commit/99ec3a96880649eeaa08c3df30e3ae802048f4fe">Initial commit</a> tells you.</p>
  240. 			<p>
  241. 				Alternative might be <a href="https://github.com/swc-project/swc">swc</a> (<a href="#rust">Rust</a>). Note that <a href="https://deno.land/">Deno</a> (also <a href="#rust">Rust</a>) just <a href="https://github.com/denoland/deno/blob/main/tools/update_typescript.md">grabs pre-transpiled JS from Microsoft</a> and <a href="https://babeljs.io/">Babel</a> simply seems to depend on the <code>typescript</code> package.<br />
  242. 				And it should be noted that TypeScript seems to have no specification anymore. (Commit: <a href="https://github.com/microsoft/TypeScript/commit/91822db8e01e38e1f9d80142df67d3849851571d">Remove doc folder (old archived spec and assets), word2md script</a>)
  243. 			</p>
  244. 			<p>2025-03 Update: <a href="https://github.com/microsoft/typescript-go">TypeScript is porting it's compiler to Go</a>, for performance reasons but should also mean getting bootstrappability, hopefully full-source kind</p>
  246. 			<h3 id="dart">Dart</h3>
  247. 			<p>
  248. 				Yet another chicken-egg language without a single documented way to bootstrap it from source, I wish they would have learned from the other language from Google: Go.
  249. 			</p>
  251. 			<h3 id="rollup">rollup</h3>
  252. 			<p>
  253. 				<dl>
  254. 					<dt>chicken-egg</dt><dd>Uses rollup to build itself</dd>
  255. 					<dt>one-step circular dependency</dt><dd>rollup → acorn → rollup</dd>
  256. 					<dt>links to a two-step circular dependency</dt><dd>rollup → eslint → webpack → acorn → eslint</dd>
  257. 				</dl>
  258. 				I guess web development can also mean creating cyclic graphs of dependencies.<br />
  259. 				Note: acorn doesn't lists it's dependencies on npmjs because it publishes a pre-compiled version…
  260. 			</p>
  261. 		</main>
  262. <!--#include file="/templates/en/footer.shtml" -->
  263. 	</body>
  264. </html>