tag:blogger.com,1999:blog-64799322909496737452024-03-14T05:12:49.919-04:00Geek in a SuitA place for me to talk about thoughts relevant to software, consulting, etc.Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.comBlogger21125tag:blogger.com,1999:blog-6479932290949673745.post-32873454966989515952020-09-16T14:04:00.001-04:002020-09-16T14:14:28.913-04:00Jackson, YAML, and Kotlin data classes, and SnakeYaml formatting<p> There have been a few articles on how to use YAML parsing with kotlin data classes, including <a href="https://www.mkammerer.de/blog/snakeyaml-and-kotlin/" target="_blank">early frustrations with snakeyaml</a>, <a href="https://www.mkammerer.de/blog/kotlin-and-yaml-part-2/" target="_blank">how to do it</a> with <a href="https://github.com/FasterXML/jackson">Jackson</a>'s <a href="https://github.com/FasterXML/jackson-dataformats-text/tree/master/yaml">wrapping</a> of <a href="https://bitbucket.org/asomov/snakeyaml/src/master/">SnakeYaml</a>, and generally you can get a functional setup going. </p><p>Cool as far as it goes, but Jackson doesn't really expose much of SnakeYaml's <a href="https://www.javadoc.io/doc/org.yaml/snakeyaml/1.19/org/yaml/snakeyaml/DumperOptions.html" target="_blank">DumperOptions</a> (where all the fun config is). SnakeYaml has quite a few formatting options available to it which are essential to writing readable YAML, but these aren't exposed in the Jackson-supplied APIs. So, how to get this configuration exposed?</p><p>Unfortunately the answer is, while not hard, messy. Thankfully, Jackson's code is pretty open to extension. This is necessary, because to get your DumperOptions configuration in, you have to intercept where this option class is created, and you have to do that in an override of <a href="https://fasterxml.github.io/jackson-dataformats-text/javadoc/yaml/2.11/com/fasterxml/jackson/dataformat/yaml/YAMLGenerator.html" target="_blank">YAMLGenerator</a>. But... YAMLGenerator is constructed by <a href="https://fasterxml.github.io/jackson-dataformats-text/javadoc/yaml/2.11/com/fasterxml/jackson/dataformat/yaml/YAMLFactory.html">YAMLFactory</a>, so you also need to intercept THAT in a subclass which overrides the creation method, in order to plumb through your custom YAMLGenerator. Messy, but doable. </p><p>Here's what mine looked like:</p><pre style="background-color: #2b2b2b; color: #a9b7c6; font-family: menlo; font-size: 9pt;"><span style="color: #cc7832;">val </span><span style="color: #9876aa;">mapper</span>: YAMLMapper = YAMLMapper(MyYAMLFactory()).<span style="color: #ffc66d; font-style: italic;">apply </span><span style="font-weight: bold;">{<br /></span><span style="font-weight: bold;"> </span>registerModule(KotlinModule())<br /> setSerializationInclusion(<span style="color: #bbb529;">JsonInclude</span>.Include.<span style="color: #9876aa;">NON_EMPTY</span>)<br /><span style="font-weight: bold;">}<br /></span><br /><br /><span style="color: #cc7832;">class </span>MyYAMLGenerator(<br /> ctx: IOContext<span style="color: #cc7832;">,<br /></span><span style="color: #cc7832;"> </span>jsonFeatures: Int<span style="color: #cc7832;">,<br /></span><span style="color: #cc7832;"> </span>yamlFeatures: Int<span style="color: #cc7832;">,<br /></span><span style="color: #cc7832;"> </span>codec: ObjectCodec<span style="color: #cc7832;">,<br /></span><span style="color: #cc7832;"> </span>out: Writer<span style="color: #cc7832;">,<br /></span><span style="color: #cc7832;"> </span>version: DumperOptions.Version?<br />): YAMLGenerator(ctx<span style="color: #cc7832;">, </span>jsonFeatures<span style="color: #cc7832;">, </span>yamlFeatures<span style="color: #cc7832;">, </span>codec<span style="color: #cc7832;">, </span>out<span style="color: #cc7832;">, </span>version) {<br /> <span style="color: #cc7832;">override fun </span><span style="color: #ffc66d;">buildDumperOptions</span>(<br /> jsonFeatures: Int<span style="color: #cc7832;">,<br /></span><span style="color: #cc7832;"> </span>yamlFeatures: Int<span style="color: #cc7832;">,<br /></span><span style="color: #cc7832;"> </span>version: DumperOptions.Version?<br /> ): DumperOptions {<br /> <span style="color: #cc7832;">return super</span>.buildDumperOptions(jsonFeatures<span style="color: #cc7832;">, </span>yamlFeatures<span style="color: #cc7832;">, </span>version).<span style="color: #ffc66d; font-style: italic;">apply </span><span style="font-weight: bold;">{<br /></span><span style="font-weight: bold;"> </span><span style="color: #9876aa; font-style: italic;">defaultScalarStyle </span>= ScalarStyle.<span style="color: #9876aa;">LITERAL</span><span style="color: #cc7832;">;<br /></span><span style="color: #cc7832;"> </span><span style="color: #9876aa; font-style: italic;">defaultFlowStyle </span>= FlowStyle.<span style="color: #9876aa;">BLOCK<br /></span><span style="color: #9876aa;"> </span><span style="color: #9876aa; font-style: italic;">indicatorIndent </span>= <span style="color: #6897bb;">2<br /></span><span style="color: #6897bb;"> </span><span style="color: #9876aa; font-style: italic;">nonPrintableStyle </span>= <span style="color: #9876aa;">ESCAPE<br /></span><span style="color: #9876aa;"> </span><span style="color: #9876aa; font-style: italic;">indent </span>= <span style="color: #6897bb;">4<br /></span><span style="color: #6897bb;"> </span><span style="color: #9876aa; font-style: italic;">isPrettyFlow </span>= <span style="color: #cc7832;">true<br /></span><span style="color: #cc7832;"> </span><span style="color: #9876aa; font-style: italic;">width </span>= <span style="color: #6897bb;">100<br /></span><span style="color: #6897bb;"> </span><span style="color: #cc7832;">this</span>.<span style="color: #9876aa; font-style: italic;">version </span>= version<br /> <span style="font-weight: bold;">}<br /></span><span style="font-weight: bold;"> </span>}<br />}<br /><br /><span style="color: #cc7832;">class </span>MyYAMLFactory<span style="color: #72737a;">()</span>: YAMLFactory() {<br /> <span style="color: #bbb529;">@Throws</span>(IOException::<span style="color: #cc7832;">class</span>)<br /> <span style="color: #cc7832;">override fun </span><span style="color: #ffc66d;">_createGenerator</span>(out: Writer<span style="color: #cc7832;">, </span>ctxt: IOContext): YAMLGenerator {<br /> <span style="color: #cc7832;">val </span>feats = <span style="color: #9876aa;">_yamlGeneratorFeatures<br /></span><span style="color: #9876aa;"> </span><span style="color: #cc7832;">return </span>MyYAMLGenerator(ctxt<span style="color: #cc7832;">, </span><span style="color: #9876aa;">_generatorFeatures</span><span style="color: #cc7832;">, </span>feats<span style="color: #cc7832;">,</span><span style="color: #9876aa;">_objectCodec</span><span style="color: #cc7832;">, </span>out<span style="color: #cc7832;">, </span><span style="color: #9876aa;">_version</span>)<br /> }<br />}<br /></pre><p>This is pretty gross. It's overriding some internal methods in YAMLFactory and YAMLGenerator, but thankfully, it's only two classes, and not terribly deep into the mess. As a result, I managed to make use of the FlowStyle.BLOCK option, which fixes a <a href="https://stackoverflow.com/questions/52784112/jackson-yaml-serialization-object-arrays-format" target="_blank">known problem in Jackson's YAML handling</a>, where instead of this:</p><pre class="lang-java prettyprint prettyprinted" style="border-radius: 5px; border: 0px; box-sizing: inherit; color: var(--highlight-color); font-family: consolas, menlo, monaco, "lucida console", "liberation mono", "dejavu sans mono", "bitstream vera sans mono", "courier new", monospace, sans-serif; font-size: 13px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 1.6em; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><code style="background-attachment: unset; background-clip: unset; background-image: unset; background-origin: unset; background-position: unset; background-repeat: unset; background-size: unset; border-radius: 0px; border: 0px; box-sizing: inherit; color: var(--black-800); font-family: consolas, menlo, monaco, "lucida console", "liberation mono", "dejavu sans mono", "bitstream vera sans mono", "courier new", monospace, sans-serif; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">employees</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">-</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> name</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="typ" style="border: 0px; box-sizing: inherit; color: #2b91af; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">John</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
age</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="lit" color="var(--red-800)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">26</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">-</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> name</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="typ" style="border: 0px; box-sizing: inherit; color: #2b91af; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Sally</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
age</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="lit" color="var(--red-800)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">31</span></code></pre><p>you get this:</p><pre class="lang-java prettyprint prettyprinted" style="border-radius: 5px; border: 0px; box-sizing: inherit; color: var(--highlight-color); font-family: consolas, menlo, monaco, "lucida console", "liberation mono", "dejavu sans mono", "bitstream vera sans mono", "courier new", monospace, sans-serif; font-size: 13px; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 1.6em; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><code style="background-attachment: unset; background-clip: unset; background-image: unset; background-origin: unset; background-position: unset; background-repeat: unset; background-size: unset; border-radius: 0px; border: 0px; box-sizing: inherit; color: var(--black-800); font-family: consolas, menlo, monaco, "lucida console", "liberation mono", "dejavu sans mono", "bitstream vera sans mono", "courier new", monospace, sans-serif; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">employees</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">-</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
name</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="typ" style="border: 0px; box-sizing: inherit; color: #2b91af; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">John</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
age</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="lit" color="var(--red-800)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">26</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">-</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
name</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="typ" style="border: 0px; box-sizing: inherit; color: #2b91af; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Sally</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">
age</span><span class="pun" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">:</span><span class="pln" color="var(--black-750)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="lit" color="var(--red-800)" style="border: 0px; box-sizing: inherit; font-family: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">31</span></code></pre><p>There are other formatting niceties that you can do with SnakeYaml that aren't exposed by Jackson - no longer!</p><p>Anyway - this can always be factored into something improved. I'm honestly not sure how easy the Jackson project is to contribute to, but this could be exposed in public APIs without too much irritation. Regardless, I'm licensing the above code with MIT license (as permissive as I know how to make it) and also <a href="https://gist.github.com/cgruber/5c021f22bf9ed30bdcf1bf88baee2b7c" target="_blank">available at this github gist</a> so you can just use it if you feel like it.</p><p>P.S. I really wish <a href="https://github.com/square/moshi" target="_blank">Moshi</a> did YAML.</p>Christian Gruberhttp://www.blogger.com/profile/05296206780241940683noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-38427729037962932382019-02-21T16:53:00.000-05:002019-02-21T16:53:15.229-05:00Bazely thinking and the tale of the content-addressable cache...Bazel is awesome. It does a lot of things to ensure hermetic builds, etc. But part of thinking in these terms means that the bazel engineers think about certain problems from a different point of view. I discovered this in trying to debug a weird problem where my machine was working and a colleague's was failing both on a clean checkout.<br />
<br />
It all came down to the content addressable cache.<br />
<br />
So... what the heck am I talking about? Let me set the stage.<br />
<h3>
<br /></h3>
<h3>
Misleading cache hits</h3>
<div>
<br /></div>
I was working on a bazel conversion experiment (we're looking at migrating to bazel, but needed to try it out in a limited scope). It uses kotlin and the kotlin rules require downloading the kotlin compiler, located on github. The default in the bazel kotlin rules is 1.2.70 (at time of this writing), but I wanted to pull in a different version. All well and good. You set the version, you put in the sha256 of the file, and ... go. On my machine it worked flawlessly. On my colleague's machine, it dutifully downloaded the binary, and then threw a fit over a bad checksum. I repeat... we had exactly the same checkout. Same environment... we thought.<br />
<br />
I dug around and finally found out that (a) I had put in the wrong sha256 - the one from the default 1.2.70 version, and (b) it was satisfying my request, not from the network, but from a machine-wide "content-addressable" cache. On my colleague's machine, it had never downloaded any files before for bazel, so it tried to satisfy it from the network and choked when it fingerprinted the file. And (c) after digging I realized that it was supplying 1.2.70 when I asked for 1.2.71, because the content-addressable cache indexes by the hash, and the URL has no part in the cacheing. It literally only cared about the name when it wrote the file contents from the cache into my build working directory.<br />
<br />
Wait what? The URL played no part in the cache index?<br />
<br />
<h3>
Must be a bug</h3>
<div>
<br /></div>
I originally went to work up a repro-case, thinking that this is a clear bug. I asked for http://github.com/blah/blah/blah/blah-1.2.71.zip, and it gave me 1.2.70, because I had put the wrong sha256 hash in. It should have caught my error! Bad bazel. How dare it assume I knew what I was doing. That's not safe infrastructure. And then it hit me. Bazel and I were thinking of the world differently. The key was in the name "content addressable" (which I will call CA from now on, sorry Californians and Canadians). <br />
<br />
So, bazel was putting this in the CA cache because it was literally saying - the content is they key. Whatever the file name is, if the content hashes to <some number>, then any request for something with the same number must want the same content.<br />
<br />
I, on the other hand, was thinking in URL-centric terms. I wanted the file found at that location (I thought), and I supplied the hash to verify. These aren't incompatible world-views for the most part. Usually I actually do want the content, I just assumed it's going to be downloaded from that location. It's only around this error-handling question that they diverge, and when I consider Bazel's perspective on wanting to create fast, hermetic builds.<br />
<br />
Bazel simply assumes that you mean it when you put in the expected sha256. Bazel assumes you're not just naively cutting and pasting. It'll check the first time you go to download it, but in this case, I used a perfectly valid sha256 hash for which it had a valid file. And it served it. From the cache that is... addressed/keyed by the content of the file (or at least its hash).<br />
<h3>
<br /></h3>
<h3>
Is bazel right here?</h3>
<div>
<br /></div>
Yes and no. This is a tricky thing - Bazel is using sha256 hashes to make sure builds are repeatable and immutable (same inputs lead to the same outputs) and this is in service of both security and performance. Bazel thinks "same inputs, same output", and partly doesn't give a crap where the content came from. For most downloading rules you end up being able to give it multiple URLs, and it'll take whatever one it can use, as long as the content's hash matches. Even there, it's not seeing a canonical "location" as the key, but the content itself. It's largely only my prejudices that led me to assume otherwise.<br />
<br />
To what benefit? Well, assuming I don't make an error on my side, there are quite a few. For one, download poisoning is harder in a variety of ways out of scope here. Additionally, anything addressable by sha256 can now be downloaded once, and never downloaded again from the wire, even on a clean build (since it isn't "dirtied" because the hashes are the same). This can lead to a lot of benefits in continuous-integration machines where many projects downloading the same files over and over can avoid them. It also provides some solid ground for building distributed caches.<br />
<br />
I went to file the bug, but have decided that this, surprisingly, is a feature, not a bug.<br />
<h3>
<br /></h3>
<h3>
Thinking more bazely</h3>
<div>
<br /></div>
Now that I have adjusted my expectations, I can think about the content (or the hash) as the unit of account, and things like URLs as the fetch mechanism for satisfying the content if it isn't already supplied. This both should help me not make this particular mistake again, but also helps me understand a lot about the design decisions of the tooling. <br />
<br />
I realize that, in retrospect, this might seem obvious. It seems that way to me, too, now. Thinking about the web, privileging addresses not content is pretty easy. It's hard to quantify or express in words, but a lot of subtle things that bugged me about Bazel's (and Starlark's) design and idiosyncrasies have smoothed out in my brain because of this simple perspective adjustment. Hopefully it helps others reason more effectively about such things as well.Christian Gruberhttp://www.blogger.com/profile/05296206780241940683noreply@blogger.com2tag:blogger.com,1999:blog-6479932290949673745.post-58799344606250900642018-10-10T22:40:00.000-04:002018-10-10T22:40:16.974-04:00Suiting upI named this blog (and my tech-focused twitter account) GeekInASuit because I had spent a good chunk of my career in the financial industry doing technical architecture, and other technical consulting, and was the person who could talk to the nerds and the suits. I wore a suit, and so signaled in a way that finance folks would talk to me, but also had earrings, long-hair, and otherwise "signaled" that I wasn't <i>just</i> a suit. It was a great ride, and it was a lovely role, being the cultural translator, often gleaning important insights that helped clarify project details that could have been miscommunicated. I relished that part of my career.<br />
<br />
Then I took a job at Google, and everything changed. My role was decidedly technical, but all my customers were also technical. I lost the primary purpose of geekinasuit, and even more, Googler culture really venerates the geek, not so much the suit. To be honest, I kind of got shamed out of the suit. Lots of social pressure was applied, as well as a nearly endless supply of t-shirts - I swear, 20% of my compensation, by weight, was t-shirt. So I relented, and spent most of a decade wearing jeans and t-shirts, usually with nerdy slogans or fan-service. While in one sense, it didn't matter, I had come to like dressing up a bit. I enjoyed taking a bit of time for self-care and grooming beyond simple standard hygiene. So I was sadder than I realized, when I finally accepted it, and stopped upgrading my wardrobe as jackets and shoes and pants succumbed to wear and tear (and got a little tight, I admit). It was with sadness a couple of months ago that I realized that my last suit was actually no longer going to fit me. The lining was worn out, and the wedding I was preparing to fly to would require actually buying a new suit. I had let things go that far.<br />
<br />
That wedding coincided with my departure from Google/YouTube. I left for a variety of reasons, moral, financial, emotional - I left with some sadness and wistfulness, but also with a sense of maybe getting back to myself. I had gone down a deep hole in Google, lost a lot of professional contacts, reduced to only Google's technology stack. I mitigated it with doing a lot of my work in open-source, but it certainly wasn't a life I had led before, connecting with colleagues at conferences, serving customers more directly, and working with the technologies most of the industry uses. But also... I stopped appreciating having my identity swallowed by the behemoth that Google had become. Don't get me wrong - there are lots of good people and ideas and challenges at Google. I did work there that I'm very proud of (the Dagger and Truth frameworks, for example) but it also took me over, in many ways. So I left, to join Square, and help in their mission of economic empowerment (by helping them scale up their development).<br />
<br />
And I suited up. I decided to restore at least that, even if just as a symbol to myself, to be better, to push myself, to care for myself, as superficial as clothing and appearance are. So far, I've been suited-up 95% of the days I've been in the office, and it feels really good. It's a state-change in my brain, segmenting a work mode from other modes, and it oddly helps me stay focused (pretty necessary in the open-office wasteland that characterizes basically every tech company for some reason).<br />
<br />
I am, once again, a geek in a suit. And I love it.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWjIrG7t2ENZ9LKx0eR1UF6Q31yGG4oGMQlHqz-xd5dWufZBUHqlaggTtyLx3VveIYHVfqOsQsyqwDT5BwPd58x-MKbJgLAv-tE0SMjrpCiJ_S77JxOJYq_hfBtw-eddzETkmDfsoOt1s/s1600/suitup.gif" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="225" data-original-width="500" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWjIrG7t2ENZ9LKx0eR1UF6Q31yGG4oGMQlHqz-xd5dWufZBUHqlaggTtyLx3VveIYHVfqOsQsyqwDT5BwPd58x-MKbJgLAv-tE0SMjrpCiJ_S77JxOJYq_hfBtw-eddzETkmDfsoOt1s/s400/suitup.gif" width="400" /></a></div>
<br />Christian Gruberhttp://www.blogger.com/profile/05296206780241940683noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-74799003054541921292016-11-17T14:19:00.000-05:002016-11-17T14:19:39.701-05:00WTF does "vend" mean? A terminological journey with no clear ending.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://i.imgflip.com/1ecglg.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="123" src="https://i.imgflip.com/1ecglg.jpg" width="200" /></a></div>
So there I was, in the middle of a code review, adding a method which (in the language of dependency-injection) "provided" a value into a managed object graph with a different key, in advance of a migration. Doesn't matter why. But the word "provided" is so frequent that I went with a different word that (in my brain) seemed to mean the same thing: vend.<br />
<br />
Now, the root of this little language odyssey is simply that I hate repeating words unnecessarily. If you use dependency-injection, the word Provider is vastly overused thanks to <a href="http://github.com/google/guice" target="_blank">Guice</a>'s <span style="font-family: "courier new" , "courier" , monospace;">Provider<T></span> interface, the <a href="https://github.com/javax-inject/javax-inject" target="_blank">JSR-330</a> which standardized it, and it's baby brother <a href="http://github.com/google/dagger" target="_blank">Dagger</a> and other frameworks which adopted the standard terminology (Spring, J2EE, Tapestry, etc.)<br />
<br />
Since the API involved was a method annotated with <span style="font-family: "courier new" , "courier" , monospace;">@Provides</span> and the method was called <span style="font-family: "courier new" , "courier" , monospace;">provideBlah()</span> (real method name changed to protect the innocent), and I just wanted some variety in my life. So I described the change this way:<br />
<blockquote class="tr_bq">
Vends a [redacted] into the dagger graph, in advance of an API change where [redacted] will consume that in place of [redacted]. Part of a migration to make [redacted] require fewer assumptions (and fewer build deps) of its consumers.</blockquote>
Could have been "supplies" but I didn't want to imply the <span style="font-family: "courier new" , "courier" , monospace;">Supplier<T></span> interface, which is a thing. I went with "to vend".<br />
<br />
In that context, I got a drive-by comment. <br />
<br />
<div class="FooCss_style-row" style="background-color: #fcccd6; display: table-row; font-family: arial, sans-serif; font-size: 13px; position: relative; white-space: pre-wrap; z-index: 50;">
<div class="FooCss_style-header" style="display: table-cell; padding: 0px 7px 2px; white-space: nowrap;">
<span class="gwt-InlineLabel FooCss_style-collapser" style="cursor: pointer; display: inline; margin-left: 0.25em; vertical-align: top;">▾ </span><br />
<div class="FooCss_style-labels" style="cursor: pointer; display: inline-block; vertical-align: top;">
<div class="FooCss_style-label FooCss_style-author di-hover" style="cursor: default; display: inline;">
someuser</div>
<div class="gwt-Label FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
</div>
<div class="FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
3:24 PM, Nov 16</div>
</div>
</div>
<div class="FooCss_style-content" style="color: rgba(0, 0, 0, 0.870588); display: table-cell; max-width: 500px; padding: 0px 7px 2px; vertical-align: top; word-wrap: break-word;">
What does "Vend" mean in the context of this change?</div>
</div>
<br />
I was doing a cleanup using some of our awesome <a href="https://youtu.be/W71BTkUbdqE?t=13m48s" target="_blank">google-made bulk refactoring tools (notably Rosie)</a>, so this was one of those "dammit, why can't you just approve my change and let me get on with my life moments."<br />
<br />
At first, I just went ahead and answered:<br />
<br />
<div class="FooCss_style-row" style="background-color: #fcccd6; display: table-row; font-family: arial, sans-serif; font-size: 13px; position: relative; white-space: pre-wrap; z-index: 50;">
<div class="FooCss_style-header" style="display: table-cell; padding: 0px 7px 2px; white-space: nowrap;">
<span class="gwt-InlineLabel FooCss_style-collapser" style="cursor: pointer; display: inline; margin-left: 0.25em; vertical-align: top;">▾ </span><br />
<div class="FooCss_style-labels" style="cursor: pointer; display: inline-block; vertical-align: top;">
<div class="FooCss_style-label FooCss_style-author di-hover" style="cursor: default; display: inline;">
cgruber</div>
<div class="gwt-Label FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
</div>
<div class="FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
3:31 PM, Nov 16</div>
</div>
</div>
<div class="FooCss_style-content" style="color: rgba(0, 0, 0, 0.870588); display: table-cell; max-width: 500px; padding: 0px 7px 2px; vertical-align: top; word-wrap: break-word;">
<div>
<div class="quote-content" style="color: #888888;">
> What does "Vend" mean in the context of this change? </div>
</div>
<div>
Provide into the graph. </div>
</div>
</div>
<br />
Not to be disuaded, "someuser" pressed on:<br />
<br />
<div class="FooCss_style-row" style="background-color: #fcccd6; display: table-row; font-family: arial, sans-serif; font-size: 13px; position: relative; white-space: pre-wrap; z-index: 50;">
<div class="FooCss_style-header" style="display: table-cell; padding: 0px 7px 2px; white-space: nowrap;">
<span class="gwt-InlineLabel FooCss_style-collapser" style="cursor: pointer; display: inline; margin-left: 0.25em; vertical-align: top;">▾ </span><br />
<div class="FooCss_style-labels" style="cursor: pointer; display: inline-block; vertical-align: top;">
<div class="FooCss_style-label FooCss_style-author di-hover" style="cursor: default; display: inline;">
someuser</div>
<div class="gwt-Label FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
</div>
<div class="FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
3:56 PM, Nov 16</div>
</div>
</div>
<div class="FooCss_style-content" style="color: rgba(0, 0, 0, 0.870588); display: table-cell; max-width: 500px; padding: 0px 7px 2px; vertical-align: top; word-wrap: break-word;">
Normally, "vend" means to sell ...</div>
</div>
<br />
Ok... gonna try again to avoid the digression and <a href="https://xkcd.com/356/" target="_blank">nerd-sniping</a>...<br />
<br />
<div class="FooCss_style-row" style="-webkit-text-stroke-width: 0px; background-color: #fcccd6; color: black; display: table-row; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: normal; letter-spacing: normal; orphans: 2; position: relative; text-align: start; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px; z-index: 50;">
<div class="FooCss_style-header" style="display: table-cell; padding: 0px 7px 2px; white-space: nowrap;">
<span class="gwt-InlineLabel FooCss_style-collapser" style="color: black; cursor: pointer; display: inline; margin-left: 0.25em; vertical-align: top;">▾ </span><br />
<div class="FooCss_style-labels" style="cursor: pointer; display: inline-block; vertical-align: top;">
<div class="FooCss_style-label FooCss_style-author di-hover" style="cursor: default; display: inline;">
cgruber</div>
<div class="gwt-Label FooCss_style-subTitle" style="color: #888888; font-size: 10px; white-space: nowrap;">
</div>
<div class="FooCss_style-subTitle" style="color: #888888; font-size: 10px; white-space: nowrap;">
3:59 PM, Nov 16</div>
</div>
</div>
<div class="FooCss_style-content" style="color: rgba(0, 0, 0, 0.870588); display: table-cell; max-width: 500px; padding: 0px 7px 2px; vertical-align: top; word-wrap: break-word;">
<div>
Vend also implies supplying, and I was trying not to overload the term "provide" because in this context, "provide and bind" are both apt terms. Regardless, I've updated the description.<br />
And vend only means sell in a societal context of capitalist voluntary exchange. I can't imagine it would mean sell in the context of an API. :)</div>
</div>
</div>
<br />
This last paragraph was a total indulgence on my part, and the result of my having mainlined (liked, literally injected into my arm) econ textbooks and treatises for the last few years. And obviously my big mistake in the effort to avoid being nerd-sniped.<br />
<br />
Not to be so easily dismissed... "someuser" decided to call me out.<br />
<br />
<div class="FooCss_style-row" style="background-color: #fcccd6; display: table-row; font-family: arial, sans-serif; font-size: 13px; position: relative; white-space: pre-wrap; z-index: 50;">
<div class="FooCss_style-header" style="display: table-cell; padding: 0px 7px 2px; white-space: nowrap;">
<span class="gwt-InlineLabel FooCss_style-collapser" style="cursor: pointer; display: inline; margin-left: 0.25em; vertical-align: top;">▾ </span><br />
<div class="FooCss_style-labels" style="cursor: pointer; display: inline-block; vertical-align: top;">
<div class="FooCss_style-label FooCss_style-author di-hover" style="cursor: default; display: inline;">
someuser</div>
<div class="gwt-Label FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
</div>
<div class="FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
6:17 PM, Nov 16</div>
</div>
</div>
<div class="FooCss_style-content" style="color: rgba(0, 0, 0, 0.870588); display: table-cell; max-width: 500px; padding: 0px 7px 2px; vertical-align: top; word-wrap: break-word;">
<div>
<nit-picking-mode><br />
I agree that "vend" implies supplying something, but I have only seen it in the context of a sale. With all due respect, can you point to a definition of vend that means to supply or provide, that is not in a "societal context of capitalist voluntary exchange"? (I'm actually really curious, as in, I quite often read etymologies of words. :-) ) </div>
<div>
<div class="quote-content" style="color: #888888;">
> I can't imagine it would mean sell in the context of an API. </div>
</div>
<div>
That's why I was confused :-) <br />
</nit-picking-mode> <br />
Thanks for changing the description though :-) </div>
</div>
</div>
<br />
Oh, it's on like the break of dawn, now.<br />
<br />
I started looking. I couldn't find definitional resources (but I knew this was a particular inflection of use in the context of computer software and API design. <br />
<br />
I started with a web-search of the terms: "api which vends a type" just for starters. I was not disappointed. Nine relevant examples in the first three pages. <br />
<br />
Then I got philosophical, noticing that the code examples in which APIs which were described to "vend" things seemed to always be in Objective-C or Java sources. I started to think back, way into the early days of my career, steeped as they were in NeXTSTEP, and wondered whether there was a connection.<br />
<br />
Here is what I replied.<br />
<br />
<div class="FooCss_style-header" style="background-color: #fcccd6; display: table-cell; font-family: arial, sans-serif; font-size: 13px; padding: 0px 7px 2px; white-space: nowrap;">
<span class="gwt-InlineLabel FooCss_style-collapser" style="cursor: pointer; display: inline; margin-left: 0.25em; vertical-align: top;"><br class="Apple-interchange-newline" />▾ </span><br />
<div class="FooCss_style-labels" style="cursor: pointer; display: inline-block; vertical-align: top;">
<div class="FooCss_style-label FooCss_style-author di-hover" style="cursor: default; display: inline;">
cgruber</div>
<div class="gwt-Label FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
</div>
<div class="FooCss_style-subTitle" style="color: #888888; font-size: 10px;">
9:29 AM</div>
</div>
</div>
<div class="FooCss_style-content" style="background-color: #fcccd6; color: rgba(0, 0, 0, 0.870588); display: table-cell; font-family: arial, sans-serif; font-size: 13px; max-width: 500px; padding: 0px 7px 2px; vertical-align: top; white-space: pre-wrap; word-wrap: break-word;">
No, but I can find examples of its usage in tech, from which I apparently have picked it up over a couple of decades: <br />
Some of these you have to ctrl-f/cmd-f and search for "vend" as they're not in the description but in comments. Also, in some cases it is synonymous with "supplies" (as in via the return type) and in other cases with "offers" in the sense of exposing an API): <br />
<a href="https://www.google.com/url?sa=D&q=https%3A%2F%2Fgithub.com%2Fattic-labs%2Fnoms%2Fissues%2F2589" style="color: #3366cc;" target="_blank">https://github.com/attic-labs/noms/issues/2589</a><br />
<a href="https://www.google.com/url?sa=D&q=https%3A%2F%2Fgithub.com%2Frealm%2Frealm-cocoa%2Fissues%2F3981" style="color: #3366cc;" target="_blank">https://github.com/realm/realm-cocoa/issues/3981</a> <br />
<a href="https://www.google.com/url?sa=D&q=http%3A%2F%2Fstackoverflow.com%2Fquestions%2F37128296%2Frest-api-oauth2-type-authentication-using-aws-cognito%2F37141020" style="color: #3366cc;" target="_blank">http://stackoverflow.com/questions/37128296/rest-api-oauth2-type-authentication-using-aws-cognito/37141020</a><br />
<a href="https://www.google.com/url?sa=D&q=http%3A%2F%2Fnlp.stanford.edu%2Fnlp%2Fjavadoc%2Fjavanlp-3.5.0%2Fedu%2Fstanford%2Fnlp%2Fobjectbank%2FDelimitRegExIterator.html" style="color: #3366cc;" target="_blank">http://nlp.stanford.edu/nlp/javadoc/javanlp-3.5.0/edu/stanford/nlp/objectbank/DelimitRegExIterator.html</a> <br />
<a href="https://www.google.com/url?sa=D&q=https%3A%2F%2Fframework.zend.com%2Fapidoc%2F1.12%2Fpackages%2FZend_Pdf.Fonts.html" style="color: #3366cc;" target="_blank">https://framework.zend.com/apidoc/1.12/packages/Zend_Pdf.Fonts.html</a> <br />
<a href="https://www.google.com/url?sa=D&q=https%3A%2F%2Fdocs.oracle.com%2Fjavaee%2F7%2Fapi%2Fjavax%2Ffaces%2Frender%2Fpackage-summary.html" style="color: #3366cc;" target="_blank">https://docs.oracle.com/javaee/7/api/javax/faces/render/package-summary.html</a> <br />
<a href="https://www.google.com/url?sa=D&q=https%3A%2F%2Fvaadin.com%2Fapi%2F7.5.7%2Fcom%2Fgoogle%2Fweb%2Fbindery%2Frequestfactory%2Fshared%2FInstanceRequest.html" style="color: #3366cc;" target="_blank">https://vaadin.com/api/7.5.7/com/google/web/bindery/requestfactory/shared/InstanceRequest.html</a> <br />
<a href="https://www.google.com/url?sa=D&q=https%3A%2F%2Fjeremywsherman.com%2Fblog%2F2016%2F07%2F23%2Fwhy-im-meh-about-json-api%2F" style="color: #3366cc;" target="_blank">https://jeremywsherman.com/blog/2016/07/23/why-im-meh-about-json-api/</a> <a href="https://www.google.com/url?sa=D&q=http%3A%2F%2Fliftweb.net%2Fapi%2F25%2Fapi%2Fnet%2Fliftweb%2Fhttp%2FLiftRules.html" style="color: #3366cc;" target="_blank">http://liftweb.net/api/25/api/net/liftweb/http/LiftRules.html</a> <br />
(sourced from the first few pages of the google query "api which vends a type") <br />
I have a hypothesis: This language originated in the NeXTSTEP community (of which I was a part), and entered into the MacOS/iOS community lexicon from that source, and also into the Java community by way of a lot of NeXTSTEP folks joining Sun and related Java-oriented enterprises (at one point Javasoft was 25% populated by former Lighthouse Design people, of which I was one). So I suspect I picked it up early, but it is a very uncommon (as I find out in researching it) usage... but not purely in my head. :) <br />
Small addendum... a straw poll of my team which includes iOS developers as well as android developers suggests that it is vastly less common than I would have imagined from my own biases. That doesn't discount the above links but provides a bit of a ratio, a denominator for the numerator of anecdotes I cited above. Seems like a fringe usage, and sadly, provides no insight into from whence this minority usage actually derives. </div>
<br />
How common is this in use behind the walls of corporate secrecy? Doing an internal code-search I see a handful of examples with a cursory scan of initial results - all in API docs with this usage of "supply". It at least seems that I'm not entirely out of my mind, or at least others share my heterodox usage.<br />
<br />
So now I'm damned curious. How did I pick this up? I see these examples of the usage - is there a common source? Did we all pick it up from one place, or did we independently start using it the same way?<br />
<br />
If any of the three people left following this blog have a clue about this, I'd love to hear more insights. Christian Gruberhttp://www.blogger.com/profile/05296206780241940683noreply@blogger.com1tag:blogger.com,1999:blog-6479932290949673745.post-64694679935555216762016-06-01T13:23:00.000-04:002020-09-16T13:24:11.368-04:00The API apocalypse (APIocalypse?) is deferred, as the jury found that <a href="http://www.androidpolice.com/2016/05/26/jury-googles-use-of-oracles-java-apis-was-fair-use-oracle-loses-on-all-counts/">"Google's use of the APIs structure, sequence, and organization fell under fair use."</a><br />
<br />
Google had this to say about the matter.<br />
<br />
From here on in, I don't want to talk about the particulars of the case - I'm a Google employee, but my opinion is rooted in both copyright critique and my views as a software engineer. In fact, I used to work for Oracle, and probably would have resigned over this, to be honest, if I still worked there.<br />
<br />
I'm not happy about this. Don't get me wrong - it is a partial victory for sanity in software. But it leaves the travesty of applying copyright to APIs unanswered (though I'm not sure, since IANAL, that this case could have resolved anything about what I'm concerned about, since it's a lower-court ruling about specifics).<br />
<br />
The issue for me is this: a higher court found that APIs (application programmer interfaces - or in layman's terms, the specification of how to talk to a software library) are subject to copyright. Fair use means only it's a legitimate exception to what is otherwise copyright-able. So while I'm glad that this was a legit exemption, the whole underlying theory is problematic... and that has yet to be fixed.<br />
<br />
Having APIs (and their "structure and organization") be copyright-able in theory at all is insane. It's like saying "English grammar is subject to copyright, but if you want to talk to an English speaker, it's cool to use that grammar and lexicon - it's fair use."<br />
<br />
No... it's not <i>merely</i> fair use - it's the entire point of language. If you have to use different names, that's like saying "you can speak English, but you need to use different words when you speak". That means you're using a different language - specifically defeating the whole purpose of using...well... language.<br />
<br />
An API exists for the express purpose of providing a means by which one piece of code can speak to another piece of code. A public API (which one must rely on to write software on a language like Java, or using someone's supplied libraries) is a protocol - a language (or a specific subset of language, anyway - a local slang, if you will). Letting us copyright an API strains credulity to the breaking point.<br />
<br />
So I'm not convinced it's all over and our industry can breathe again, but at least we can catch a short-breath (until appeals happen, and until this fair-use exemption is used as precedent in other cases).Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-26879490522846323422016-03-09T13:42:00.000-05:002016-12-06T13:47:51.694-05:00Keep maven builds safe from "M.A.D. Gadget" vulnerabilityComing out of blogging retirement to point at a rather big issue, and to contribute to solving it a bit.
<p>
Per <a href="http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/">this blog article from Nov 2015</a> there is a rather large security vulnerability observed within the apache commons-collections library versions 3.0, 3.1, 3.2, 3.2.1, and 4.0. In the spirit of <a href="https://blog.sourceclear.com/commons-collections-deserialization-vulnerability-research-findings/">the fact that vulnerable classes are called "gadgets"</a>, a colleague of mine referred to this as the M.A.D. Gadgets bug. In essence, classes which reference Apache commons' vulnerable versions and perform serialization can effectively turn the entire JVM into a remotely exploitable exec() function (metaphorically speaking).
<p>
While people are busy swapping out vulnerable versions for newer ones, the way dependency graphs work in automated dependency management systems like maven, ivy, gradle, etc, is that a project might be obtaining vulnerable versions from a transitive dependency. That's bad bad bad. So, apart from updating deps, it's important to guard against a recurrence, and you can do that, at least in maven, via the maven-enforcer-plugin.
<p>
I threw together <a href="https://gist.github.com/cgruber/1ee25ba25f9e71da5cd0">this github gist</a> with an example configuration that can ban this dep from your deps graph, including (most importantly) inclusions via transitive dependencies that you didn't even know you had. Here is the gist's content (I'll try to keep both updated if I change them).
<p>
<pre style="font-size: 9px;"><code>
<project>
<!-- ... -->
<build>
<plugins>
<plugin>
<artifactId>maven-enforcer-plugin</artifactId>
<executions>
<execution>
<goals><goal>enforce</goal></goals>
<configuration>
<rules>
<bannedDependencies>
<excludes>
<exclude>commons-collections:commons-collections:[3.0,3.2.1]</exclude>
<exclude>commons-collections:commons-collections:4.0</exclude>
</excludes>
</bannedDependencies>
</rules>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
</code></pre>
<p>
So... fix your projects... but also make them better. Throw this into your project's parent pom, and then it will give you a build-breaking knowledge of whether you're vulnerable, and you can update your deps or use dependency exclusions to prune out any found occurrences you get from your upstream.
<p>
<em><sub>Edited to include all affected versions, and re-published from my original article in March, because blogger.com's url management is frustrating.</sub></em>Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-16608913921809553372013-08-20T12:13:00.001-04:002013-08-20T12:45:47.828-04:00Wow, I suck at blogging.<p>I just discovered, because of a <a href="https://code.google.com/p/google-guice/issues/detail?id=766">bug</a> filed on the <a href="https://code.google.com/p/google-guice">Guice</a> project, that my blog DNS settings were pointing at a domain parking page. Whoops! When I made the transition from Canadia to Amurika this year I totally neglected to fix up some domain settings. My bad.</p>
<p>But it sort of highlights that I don't have a good discipline around blogging. Examining this, I see about ten draft posts in my blogger account, that I never got around to posting, and which are now sort of obsolete and irrelevant. I'm not so narcissistic to think that everyone cares what I have to say but I provide zero value by never writing at all. :( So, sorry for that. I look with irony at my post of a few years ago stating that "I'm totally gonna blog again, now, promise!" Apparently not.</p>
<p>Life has been crazy - being at Google is a whirlwind. It's exciting, stressful, but also charming. You get very much "dug in" in certain ways, but most of those ways aren't awful - they just occupy your attention.</p>
<p>What IS wonderful is that I've been able to work on primarily open-sourced projects, being a part of the the java core-libraries team. This has meant working on the google core libraries offering <a href="https://code.google.com/p/guava-libraries">Guava</a>, dependency-injection frameworks such as <a href="https://code.google.com/p/google-guice">Guice</a>, and <a href="http://github.com/square/dagger">Dagger</a>, as well as a host of smaller projects such as <a href="http://github.com/google/auto">Auto</a> (let robots write it!) and contributing to my little testing/proposition framework, <a href="http://github.com/truth0/truth">Truth</a>. Seeing these things evolve, sometimes in response to each other, has been wonderful. And I get paid to do it! Why? Not because Google is purely altruistic, though Googlers seem to have a really strong bent towards contributing back. But these things really help Google develop world-class software at-scale.</p>
<p>I was in a little internal un-conference of core-librarians of the various supported languages, and my boss pointed out that the fact that we HAVE core libraries and tooling efforts like this is a major contributor to Google's competitiveness and capability. We fund people to figure out what patterns people use and what code developers re-write on every project and creating hardened, efficient, common implementations/APIs/frameworks out of them, where appropriate. We don't try to re-use everything, but we dig and see where we can "maximize the productivity of labour" (to borrow from the economists) of our colleagues by reducing their coding and testing burden to focus on the things that make their application unique from others. In short, we invest in future production, in future capacity for our developers, both in quality and velocity.</p>
<p>Often, we aren't writing tons of code to do it, but rather examining patterns and tweaking, deprecating certain approaches in favor of others, and using the rather stunning tooling we've evolved (<a href="http://google-engtools.blogspot.com/2011/10/build-in-cloud-distributing-build.html">blogged</a> <a href="http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html">about</a> <a href="http://google-engtools.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html">elsewhere</a>) and <a href="https://code.google.com/p/error-prone/">tools</a> we've open-sourced, to migrate our entire code-base from deprecated forms to new ones. But we also consider new approaches, libraries, and frameworks, both developed internally and externally. It's actually remarkable (to me) that a company this big can change direction so quickly, and adapt to new realities so quickly. The joke among my team is that we're starting to be less a core-libraries team, and more of a javac-enhancement team, since we are also doing a lot of building in static analysis and checks (thanks error-prone folks) into our tooling to prevent error at compile time as we are building new frameworks and tools.</p>
<p>While we've had a few false starts here and there, we are increasingly engaging in joint projects and accepting contributions into the codebase from external parties who benefit from the open-source work as well, which is gratifying. Nothing quite so happy as win-win exchanges.</p>
<p>All told, it's been a couple of years of full engagement, and not a lot of time to do tech blogging. But I'll give it another go, even if it's just to do updates like this from time to time. It's the best job I've had to date, and I am thrilled to be in contact with such high-quality professionals as I am on a daily basis.</p>Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-19239110889457389452011-07-14T10:58:00.006-04:002011-07-14T11:17:01.024-04:00Five years out, and still stable. (Or how I installed new git on old unix)<p>So, I have an OpenBSD 3.9 machine. It's circa 2006 since I last updated it. Lame, I know, but it's stable, doesn't really do much, has few problems and security holes, and runs nearly no services. But I store some files on it and have been using SVN to manage them. I wanted to start using Git, since I use it more consistently these days. I thought about Mercurial, but - meh - I've stopped fighting religious wars, even with myself.</p>
<p>Problem is - Git, even an old 1.4 version, doesn't exist in the ports tree of OpenBSD until 2007. So, what am I to do? I could update my copy of OpenBSD - an option. A good one that, in the long run, I really should do, even just for security fixes. I could bump the ports tree up a couple of versions until the git port exists, and then try to build it and hope the toolchain works. There are a lot of moving parts to the ports infrastructure, and they evolve between releases of OpenBSD. I played with that for a minute, and in the end, decided to go for broke, and do the naive thing that shouldn't work.</p>
<p>I un-tarred the latest Git 1.7 that's part of the latest OpenBSD release, into /usr/local, deleted the +DESC and +COMMENT files, and basically ran it to see what broke.</p>
<p>Well... it couldn't link to libc.so.58.0 - fair, since I only had libc.so.39.0. So I symbolically linked it. Did the same for libcrypto and libiconv. Seriously. Just symlinked them, hoping no symbols were used by git that had been added or changed in more recent versions.</p>
<p>Worked like a charm.</p>
<p>OpenBSD, and the various core libraries have been stable enough that half-again as many version bumps in libc haven't changed it enough that git needed it - likewise libcrypto. Kudos! I know I lucked out, but still - impressive.</p>Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com2tag:blogger.com,1999:blog-6479932290949673745.post-87962729673083644332010-07-21T14:46:00.001-04:002010-07-21T14:46:11.335-04:00Singleton and Request scope considered harmful? No! Scopelessness considered harmful.<p>I watched an e-mail thread discuss how Singleton and Request scope in Guice were harmful, because Singletons can only depend on Singletons (without magic) and that Request Scoped items can only depend on Request Scoped and Singletons... followed by the idea that because "we're stateless" with minimal global state, so we shouldn't need global things. And because it's cheap to build things so we can build them all the time.</p><p>I feel that this perspective comes from a lot of fighting with scopes and scoping problems and probably is an honest attempt at resolving them - if you're stateless, then scope IS somewhat irrelevant... but it also turns out that if you're not stateless (by accident) and you go scope-less, then you have state accidentally scattered all around your object graph without realizing you've copied it. Leave aside entirely that you're going to thrash garbage collection because your dependencies are stored in ivars which will go on the heap...</p><p>So what should happen here?</p><p>Other than Guice, all other dependency-injection containers of any substantial distribution use "Singleton" as the default scope, not "scopeless" (or what spring calls prototype scope). This is because Singleton is actually cleaner and easier to understand. There is one copy of this in the system... so it needs to either be stateless or carefully guard its state for multi-threaded access. But also the lifecycle is clearer - app starts up, component starts up, component shuts down, app shuts down. Scopeless (Guice "default" scope) actually have an arbitrary lifecycle based on who asks for them. </p><p>If you have Foo -> Bar (where -> means depends-on), and Foo is any real scope (singleton, session, request, per-thread, whatever), but Bar is scopeless (meaning a new one is created on every demand), then Bar's lifecycle is different if Foo depends on it than if Bash depends on it because it attaches to the lifecycle (lifetime... or scope, if you will <ahem>) of the dependent component.</p><p>This is freaky because it means it's sort of indeterminate until used (I call it the Heisenberg scope). And each time it's used it could be different.</p><p>Again, if it's stateless, no problem... but if it's stateless, Singleton is no problem... and cleaner... because you'll uncover scoping problems more quickly with more restrictive scope policies. But moving everything to no-scope means no clear lifecycle... or rather, whatever lifecycle and lifetime you happen to have in whatever calls it. </p><p>I think people look at scope as "magical" - especially in the Guice community. I don't see this kind of thrash in Picocontainer, Tapestry-IOC, or Spring user communities. And I think it's because "prototype" scope (scopeless) is seen as a quasi-factory behaviour, not a dependency/collaborator injection behaviour. The two are subtly different, and I think the distinction is lost in the Guice user community. I have ideas as to why this distinction arose between the Guice community's thinking and others', but I'll leave that for another post. </p><p>The point is, Scope implies a validity within a lifetime, and if something is stateless, there's no reason it shouldn't be the one and only copy with a lifetime of the entire application's lifetime. I've long posited that "games with scopes" is a dangerous thing in dependency injection, but this is solving the problem by dropping a 16 ton weight on your own head. It uses the most magical quasi-scope to create a fan of instances where a very small set of collaborators are required. </p><p>I'm getting close to believing that Fowler, Martin, and others were wrong, and that Dependency Injection (heck, occasionally O-O) are just too dangerous. Seriously. I can't imagine not using them, personally, but I find so many teams and projects where they just think about the problem so unhelpfully and then their clean-up efforts are worse than the mess they created. <sigh></p><p> </p>Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-4651761489360360312010-02-11T11:08:00.001-05:002010-02-11T11:08:21.523-05:00Shu-Ha-Ri not harmful... it's misunderstood and mis-applied.<p>Rachel Davies, for whom I have incredible respect, posted this post called "<a href="http://agilecoach.typepad.com/agile-coaching/2010/02/shuhari-considered-harmful.html">Shu-Ha-Ri considered harmful</a>". In it she points out that the basic notion of <a href="http://en.wikipedia.org/wiki/Shuhari">Shu-Ha-Ri</a> from <a href="http://en.wikipedia.org/wiki/Aikido">Aikido</a> of graduated styles of learning - the novice, the advanced student, the master should learn in different ways, the novice learning more by rote, and more importantly, from a single master, the advanced student trying the techniques in varying ways and comparing styles, then the master innovating new combinations and techniques. Cockburn, often cited in the Agile community somewhat gets this, but his <a href="http://alistair.cockburn.us/Shu+Ha+Ri">adaptation</a> varies from this concept just a bit.</p>
<p>Rachel has an important critique, which I accept. She points out that agile boot-camps and other styles of training for organizations and groups and teams in Agile practices often cite Shu-Ha-Ri, and require that teams do "just the scrum by the book" (or whichever method is to be used), at least initially. Then, as they master these techniques as they are put forth, they can adapt. She, however, sees a disrespect for the unique circumstance of the student in this. In her words, "<span style="font-family: arial, helvetica, hirakakupro-w3, osaka, 'ms pgothic', sans-serif; font-size: 13px; color: #333333;">I'm uncomfortable with approaches that force students to follow agile practices without questioning.<span style="color: #000000; font-family: Helvetica; font-size: 12px;">" I agree. But this is not what Shu-Ha-Ri implies. Shu should always include questioning... but the student should test the technique as presented, and part of the discovery is finding out its limits - how it works, and how it does not. But it's an experiment. It requires a control... and the control is the basic form, attempted as instructed, to get a baseline. Teaching any technique, physical or mental has a similarity in that respect. Is the metaphor limited - yes, and I argue that it is the agile-boot-camp folks who often mis-apply the martial arts concept.</span></span></p>
<p>I understand the concern she raises, especially the respect for context and unique character of the teams and the flexible nature of knowledge work... but this betrays a misunderstanding, or mis-application of Shu-Ha-Ri. Shu doesn't imply that the students are fungible. Technique is still taught in the context of the student (team). Sutherland has it wrong when he days "only when you have mastered the basic practices are you allowed to improvise." In Aikido, the students who are in a "Shu" mode are not improvising with "different" techniques, but they are applying them in different situations, and seeing how they fit. One adapts HOW to do the technique for a tall person, for a short person, a heavy person, and advanced Aikidoka, an unranked novice, etc. Likewise with Scrum, you apply the technique, but the coach helps the team use the technique in context. That's Shu. Ha then is where a team combines the techniques in unique ways. They may remove a practice, or replace it and see how that fits. Ri (mastery), they are inventing new techniques, or altering the basic forms in different ways. This is all quite reasonable, even in an agile coaching context.</p>
<p>In Aikido, especially, students practice the techniques in multiple contexts, so they can get a sense of the suppleness. Students are asked not to innovate initially, nor combine techniques before they have at least mastered the basic technique itself - so they're not thinking through each step of a move while they're doing it - they "get" it. They they can move closer to innovation.</p>
<p>Rachel's post, while understandably compassionate, confuses two separate things... models of instruction, training, and practice with notions of respect, oppression, and dominance. Telling a student to try the basic move and get it better before expanding isn't disrespectful, it's understanding the learning models of the student. In practice, it is quite possible for a student to <a href="http://en.wikipedia.org/wiki/Grok">grok</a> the technique more quickly, and if Sensei observes this, he will show the student something slightly more advanced, and have him practice this. Or, Sensei may see the student struggling to apply the move, and may change the context to let the student appreciate what's happening. The point of Shu-Ha-Ri is, as Rachel points out, to ensure that one doesn't miss the basics while playing with the innovative and the expansive. If it's being used to hold a student back, or somehow contains a disrespect for the student - that's a failing of the teacher (coach). Student-sensei relationships are adaptive to the needs of the student, else they become a mere posture of dominance and submission, without the deeper communication that's supposed to occur within a relationship of trust.</p>
<p>I feel Rachel is (unintentionally and understandably) mischaracterizing Aikido and Shu-Ha-Ri based on mis-perceptions prevalent in the community on how its concepts can be applied. Aikido is taught in many forms, but Agile is not looser, nor is Aikido tighter a discipline. Aikido is, in any good Dojo, taught with great sensitivity to the needs, the capacities and readiness of the student. It is taught to <i>groups</i>, <i>individually</i> - that is to say, it is demonstrated to the group, then practiced in pairs, with Sensei observing, correcting... coaching. If anything, rolling Agile out in a large organization, and taking large groups through paces in boot-camps which do one-size-fits-all is UNLIKE Aikido training.</p>
<p>Bootcamps fail to do Shu-Ha-Ri if they insist that all steps of Shu are learned at equal pace by all learners. This is not Shu-Ha-Ri being harmful - this is Shu-Ha-Ri being ignored.</p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com6tag:blogger.com,1999:blog-6479932290949673745.post-88466562403042550112009-08-26T15:52:00.000-04:002009-08-26T16:02:16.334-04:00I don't believe in "Business Value" (sort of)<p>Ok, let's define terms. I'm not against value-oriented thinking. in fact, I'm all about the customer. I think whatever can add value to the customer or the output the customer wants (presumably software, in my case), the better. But this term "Business Value" has crept in to the Agile community, as a bastardization of the Lean concept of "Customer Value-Added activities" and it worries me. I'm concerned with terms here, and the term is used two ways. I'm increasingly concerned about it's sloppy use and the impacts of such.</p>
<p>In Lean, specifically in <a href="http://en.wikipedia.org/wiki/Value_stream_mapping">Value Stream Mapping</a> there are two kinds of activities - those which add value to the customer, and those which do not. You analyze a process (say, software development from feature request to delivery into production) and you identify the time each activity takes, and whether that activity added or failed to add value to the customer. Writing code, writing tests, designing, working out clarifications of the requirements - these were customer-value-added activities. Waiting for a server to be deployed, waiting for three VPs to sign off on a design, etc. these were non-value-added activities. I like this kind of cycle-time measurement, because it forces the observer to be in the mind of the customer. If there's a business process which slows down the process, or, in lean terms, lengthens the cycle-time, it's an inefficiency and you try to find a way to remove it, circumvent it, etc.<br />
<img src="http://leansoftwareengineering.com/wp-content/uploads/2008/02/value_stream_map.gif" /></p>
<p>At a fairly large financial service client, I saw a rather crazy thing occur. The coaches and suits were doing a value stream mapping. There were business activities whose performance served a business need, but not the needs of the customer. This might be a strategic steering committee meeting which a Team Lead might be required to attend, or a process step that existed to satisfy a regulatory requirement imposed on the business. These weren't really "customer value added" activities, but the coaches and business were unwilling to see them as non-value-added activities, or "waste". This led to an interesting compromise - the concept of "business value added" activities.</p>
<p>Now, I don't mind if there's a piece of waste in your cycle, and you acknowledge it, but then decide, for strategic trade-off reasons, to keep it. That's a sane business decision, even if it rankles my personal aesthetic. It is the prerogative of a company's decision makers to judge the interests of their customer against the interests of the organization and of the shareholders and come to whatever balance they feel best serves the principles and values and mission of their company. But by calling it "business value added" what they did, in effect, was move it out of non-customer-value-added column, which effectively stopped people from considering whether or not to reduce or eliminate it. It became synonymous with "customer value".</p>
<p>On the other hand, often when people use the term "business value" in the Agile community, they mean it as the customer value of a product feature, and are using it to help focus people's prioritization of work/features on the backlog (worklist for the uninitiated). And I get it. In that context, you need to have product managers prioritizing based on value, not cost, for reasons that <a href="http://agiletoolkit.libsyn.com/index.php?post_id=400364">Arlo Belshee</a> can better explain. But the term business value gets mixed up in these various contexts, and I have heard consultants who, two years earlier would have called that committee review of design "waste" simply brush it off as "business value added" activity, and I think it's the sloppy language around BV and CV and NVA that is at the root of this phenomenon. In other words, hard nosed management consultants have stopped calling out the Emperor's nakedness. Since these external entities are the few empowered to call bullshit on a client, this means that less such is being called, to the detriment of our industry.</p>
<p>So yes, this is an argument about definitions and semantics. Yadda yadda yadda. But for my money, I want crisp meaning - especially if it allows the "cloak of truthiness" to descend and sow confusion about what's important. And if delighting your customers is critical to your long-term business survival, then anything not in that "value conversation" is waste (sorry, but it is - deal with it). You might have to live with some waste you can't get rid of yet, but you should never stop calling it like it is. Otherwise you begin to sink back into the very state from which you used value stream mapping to escape. And as far as I'm concerned, serving, nay, delighting the customer is the only way to be in business for the long haul.</p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com4tag:blogger.com,1999:blog-6479932290949673745.post-13996067382712103272009-06-20T18:41:00.000-04:002009-06-20T18:43:22.418-04:00Make Fake/Stub objects simpler with proxies.<p>I recently wrote a convenience component. It's a Java dynamic proxy creator with a default handler that simply delegates any methods called on it to a provided delegate, throwing an UnsupportedOperationException if such a method isn't implemented. Why? Because I was sick of writing huge Fake implementations of medium-to-huge interfaces when I needed a fake, not a mock, and I was frustrated with using EasyMock as a convenience to create Fakes and Stubs. This delegator allowed me to implement only the parts of the API provided by the Interface, and just not bother with the rest. This was useful for such things as factories or other "lookup" components which I was going to feed (in my tests) from some static hash-map.</p>
<p>This gets back to my new motto: I hate boilerplate. If I have to implement every interface method in a fake object, then I'm going to go crazy implementing public String getFoo() { return null; } all over the place. Especially since that method will never get called in my test, and in fact if it does, I want it to fail noisily. So I could write public String getFoo() { throw UnsupportedOperationException("Test should not call this method"; }. That's great, but if I have to do that for a component that's got thirty methods, my test classes are going to be littered with this. Instead, I can do:</p>
<pre>
public class MyTestClass {
private final hMap<Long> BAR_CACHE;
@BeforeClass
public void classInit() {
BAR_CACHE = new HashMap<Long>();
BAR_CACHE.put(1, new Bar(1, true));
BAR_CACHE.put(2, new Bar(2, false));
BAR_CACHE.put(3, new Bar(3, true));
BAR_CACHE.put(4, new Bar(4, true));
}
@Test
public void testFooValidityCount() {
BarFactory fooDep = Delegator
.createProxy(BarFactory.class,new FooDelegate());
Foo foo = new Foo(fooDep);
foo.processBars();
assertEquals(3, foo.getCountOfValidBars());
}
... more test methods that need to look stuff up...
public static class FooDelegate {
public void init() { /* ignore */ }
public Bar getBarWithId(int id) { BAR_CACHE.get(id); }
}
}
</pre>
<p>If we assume that FooFactory is an insanely long interface, then this allows me to do a very clean Fake implementation with only a few bits of implementation. Otherwise, a FakeFooFactory could be longer than all the test code in this testing class. The other thing I like about this, is that - especially if you're testing code that uses dependencies you're not intimately familiar with, nor have access to the source, you can quickly implement an empty delegate and let the exceptions in the test guide you towards the leanest implementation. You'll get an exception any time your delegate is missing a method used by your System Under Test on your dependency. Handy.</p>
<p>Essentially, the Delegator is simply a pre-fab proxy InvocationHandler, with some reflection jazz in the invoke() method to lookup methods on the delegate, or throw... but it saves a bunch of extra boilerplate. I've had fake classes so large THEY needed tests. This is a much nicer approach for non-conversational dependencies (where mocks would be better). I tried to do this with EasyMock, but the implementations were really awkward due (ironically) to the fluent interface style. This is just a rare case where it's cleaner to just have a minimal implementation.</p>
<p>Unfortunately, the code I wrote is closed-source for my employer, but it's simple enough that everything I've done above should let you implement your own.</p>
<p>... and remember, kids... Mocks, Stubs, and Fakes are all Test Doubles, but they're all not the same thing.</p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com2tag:blogger.com,1999:blog-6479932290949673745.post-51921838826035650902009-04-12T00:38:00.000-04:002009-04-12T00:40:21.401-04:00Re-thinking Object-Relational Mapping in a Distributed Key-Store world<p><a href="http://java.sun.com/jdo/">JDO</a>, <a href="http://en.wikipedia.org/wiki/Java_Persistence_API">JPA</a>, <a href="http://www.hibernate.org/">Hibernate</a>, <a href="http://www.oracle.com/technology/products/ias/toplink/index.html">Toplink</a>, <a href="http://cayenne.apache.org/">Cayenne</a>, <a href="http://en.wikipedia.org/wiki/Enterprise_Objects_Framework">Enterprise Object Framework</a>, etc., are workable object to datastore mapping mechanisms. However, you still have to optimize your approach to the underlying data system, which in most cases historically have been RDBMS. Hence these systems, their habits and idioms are all quite strongly tied to <a href="http://en.wikipedia.org/wiki/Object-relational_mapping">Object-Relational Mapping</a>. Since <a href="http://www.google.com/">Google's</a> <a href="http://code.google.com/appengine/">App-Engine</a> released a <a href="http://code.google.com/appengine/docs/java/overview.html">Java app hosting service</a>, with a datastore wrapped in JDO or JPA via <a href="http://www.datanucleus.org/">DataNucleus</a> this week, people have been playing, and the difficulties of these very habits have become clearer. It's easy, when using JDO or JPA with <a href="http://en.wikipedia.org/wiki/BigTable">BigTable</a>, to design as if we're mapping to an RDBMS, which a distributed <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">column-oriented DBMS</a> like BigTable is not. There are some good articles on the net about how to think about this sort of data store:</p>
<p><a href="http://www.mibgames.co.uk/2008/04/15/google-appengine-bigtable-and-why-rdbms-mentality-is-harmful/">http://www.mibgames.co.uk/2008/04/15/google-appengine-bigtable-and-why-rdbms-mentality-is-harmful/</a></p>
<p><a href="http://torrez.us/archives/2005/10/24/407/">http://torrez.us/archives/2005/10/24/407/</a></p>
<p><a href="http://highscalability.com/how-i-learned-stop-worrying-and-love-using-lot-disk-space-scale">http://highscalability.com/how-i-learned-stop-worrying-and-love-using-lot-disk-space-scale</a></p>
<p>I'm struggling with this myself, being a long-time (longer than most) user of Object-Relational mapping frameworks. For example, one cannot do things with BigTable like join across relationships to pre-fetch child object data - a common optimization. Keystores are bad at joins, because they're sparse and inconsistently shaped. The contents of each "row" may not have the same "columns" as each other, so building indexes to join against is difficult. We actually need to re-think normalization, because it's not the same kind of store at all.</p>
<p>Interestingly, OO and RDBMS actually DIDN'T have an impedance mis-match in one key area in that <a href="http://en.wikipedia.org/wiki/Entity-relationship_model">Entity-Relationship models</a> bore a striking structural resemblance to <a href="http://en.wikipedia.org/wiki/Class_diagram">Object-Composition or Class diagrams,</a> and the two could be mapped. Both RDBMS schemata and Object Models were supposed to be somewhat <a href="http://en.wikipedia.org/wiki/Database_normalization">normalized</a>, except where they were explicitly <a href="http://en.wikipedia.org/wiki/Denormalization">de-normalized</a> for performance reasons. With distributed key-stores, we're potentially better off storing duplicate objects, or possibly building massive numbers of crazy indices. At this point, I don't know what the right answer is, habit wise, and I work for the <a href="http://www.google.com/">darned company that made this thing</a>. It's a different paradigm, and optimizing for performance on a sparce key-store is a different skill and knack. And since we're mapping objects to it, we'll have to get that knack then work out what mapping habits might be appropriate to that different store.</p>
<p>Eventually we will work out the paradigms, with BigTable and its open-source cousins (<a href="http://hadoop.apache.org/hbase/">HBase</a> and <a href="http://hypertable.org/">HyperTable</a>) creating more parallelism and horizontal scaling on a "cloud-y web" as I've seen it called. The forums - especially the <a href="https://groups.google.com/group/google-appengine-java">google-appserver-java</a> and other forums where objects and keystores have to meet - they will grow the skills and habits within the community. But we have to re-think it, lest we just turn around and say "that's crap" and throw the baby (performance/scaling) out with the bathwater (no joins/set-theory). </p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com5tag:blogger.com,1999:blog-6479932290949673745.post-56859136089308907522009-03-28T16:45:00.000-04:002009-03-28T16:48:59.259-04:00Hiatus... should be over<p>So apologies that I haven't posted anything - I never intended to be away from the blog for this long, so soon after starting it. I have recently been in transition, and am now working for Google as an internal software development coach. Having disclosed that, I should mention that this blog isn't Google-specific, and mostly the things I'm speaking about here are things I have seen elsewhere, fodder from discussions with other Agile Community members and practitioners, and may include aspects from my current job, but not specifically, and usually blended in with the former. In other words, I won't be leaking Google's secrets on this page. ;)</p>
<p>Having said that, it's so far quite a fun place to work, with a wide variety of development cultures. There's a great spirit of exploration and experimentation in my new firm, which allows different teams to try different things. I leave it as an exercise for the reader to see how this can be challenging, organizationally, but it certainly allows G to innovate - as is nicely shown in the marketplace.</p>
<p>Anyway, I have four or so articles partially written from before that I hope to publish in the next couple of weeks. It's been an adjustment (haven't been an actual employee for quite some time now), but I'm starting to get a bit of a stable schedule so I can pay attention to this sort of thing (blogging).</p>
<p>Cheers,</p>
<p>Christian - the Geek in a Suit</p>
<p>Oh, P.S. I wore a suit to my first meeting with the team I'll be working with, and got soundly ribbed for it. lol. It's ok... I'm a big boy. I can take the heat. -cg.<br /></p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com1tag:blogger.com,1999:blog-6479932290949673745.post-83046538428924510922009-01-23T09:06:00.000-05:002009-01-23T09:11:04.919-05:00Are mocks just stubs by another name, or something more?<p>[An older article that I published internally to a client community, reprinted with changes to remove client references]</p>
<p>An opinion that I've run into among some of my clients is that mock objects are just what we've always called stubs, and that it's an old approach. It's actually a quite common perspective - one I have held myself for part of my career. In fact early mock object approaches were very much like "fake implementations", but modern mocks are different. While they can both be considered "test doubles" or "test stand-ins", stubbed out interfaces or provide expected data for the system under test, mock objects provide behavioural expectation. These terminologies can be confusing, but we can sort that out.</p>
<p>jMock and EasyMock are two examples of mock object frameworks which allow for the specification of behaviour. jMock uses a domain specific language (DSL) such that you code the expectations in a fairly verbal syntax (often called a <a href="http://en.wikipedia.org/wiki/Fluent_interface">fluent interface</a>). Something along the lines of</p>
<blockquote>
<p>myMock.expects(methodCall(SOME_METHOD)<br />
.which().returns(A_VALUE))<br />
.then().expects(methodCall(OTHER_METHOD))</p>
</blockquote>
<p>... which should be vaguely like english to the initiated. EasyMock, on the other hand, uses a "proxy/record/replay" approach instead, which some find easier. The point is that they both define a set of expected interactions, rather than a first this state, then the next state only.</p>
<p>Martin Fowler, around the middle of his article <a href="http://martinfowler.com/articles/mocksArentStubs.html" title="Mocks Aren't Stubs">"Mocks Aren't Stubs"</a>, after describing fakes vs. mocks approaches in more detail. He starts to use a clarifying terminology which I like:</p>
<blockquote>
<p>"In the two styles of testing I've shown above, the first case uses a real warehouse object and the second case uses a mock warehouse, which of course isn't a real warehouse object. Using mocks is one way to not use a real warehouse in the test, but there are other forms of unreal objects used in testing like this.</p>
<p>"The vocabulary for talking about this soon gets messy - all sorts of words are used: stub, mock, fake, dummy. For this article I'm going to follow the vocabulary of Gerard Meszaros's book. It's not what everyone uses, but I think it's a good vocabulary and since it's my essay I get to pick which words to use.</p>
<p>"Meszaros uses the term Test Double as the generic term for any kind of pretend object used in place of a real object for testing purposes. The name comes from the notion of a Stunt Double in movies. (One of his aims was to avoid using any name that was already widely used.) Meszaros then defined four particular kinds of double:</p>
<ul>
<li>Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.</li>
<li>Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).</li>
<li>Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.</li>
<li>Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.</li>
</ul>
<p>"Of these kinds of doubles, only mocks insist upon behavior verification. The other doubles can, and usually do, use state verification. Mocks actually do behave like other doubles during the exercise phase, as they need to make the SUT believe it's talking with its real collaborators - but mocks differ in the setup and the verification phases."</p>
</blockquote>
<p>This use of behavioural expectations for "Test Double" objects is quite handy, especially in systems where you have components which make use of heavy service objects or (gasp) singletons which may do more than a simple "call-and-answer" method-call on the object. Deep call chains may require stronger behavioural testing. Being able to provide an object to the system under test that expects to be called a certain way and in a certain order can create a much more precise test, and reduce the amount of code you have to write in a "fake" object. Otherwise, to fully and precisely test, one ends up with severely large number of Test Dummies and ballooning "Fake Objects", which themselves could have error and often have to be tested. Having a system like jMock or EasyMock can reduce the size of your testing code, thus removing some code that could easily become out-of-sync with the system under test, and either introduce false-positive errors or become insufficient and therefore meaningless. So less code, less maintenance, less syncing problems, and the tests are able to be more precise at the same time.</p>
<p>Another approach is to radically re-factor your code into smaller components, each of which is substantially simpler to test. If a component does one thing and does it well, and components interact and collaborate, then you often can use simpler mock behaviours, or simple fakes without as much effort on the testing. Great resources for testable code are available <a href="http://misko.hevery.com/category/testability/">here</a> .</p>
<p>Mocks aren't always easily intuitive for someone used to building fakes (it wasn't for me, and it still occasionally trips me up), but once you're comfortable with the approach, it can be much crisper. This is especially true as people try to get better coverage in isolated unit tests, or who are trying to test-drive their software. The linked Fowler article is a good one, and certainly worth reading for those trying to figure out how to more meaningfully test components without having to start up external servers or simulators.</p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-64542284992263327032009-01-02T19:11:00.002-05:002009-01-02T19:13:38.875-05:00Why I hate the java.util collection library.<p><rant></p>
<p>It's very simple. While it was a vast improvement over Vector and Dictionary of their day, the Collections library as of Java 1.2 did what all new Java APIs from Sun seem to do... require lots of code to use simply and allow the user to do invalid things.</p>
<p>I won't spend a lot of time on the former, since it could be the topic of a whole other blog post, and I should preface all of this by saying that Java is my most proficient language. So this is not an anti-Java bigot speaking... just a frustrated user who wishes Sun and the community wouldn't keep heaping bad APIs on top of bad APIs.</p>
<p>Sorry... back to the point.</p>
<p><strong>Immutability - it's all backwards... what's with that?</strong><br /></p>
<p>The big issue I have with the Collections API are about allowing the user to do wrong things. This amounts to Sun having inverted Mutability vs. Immutability in the class heirarchy. Immutability, in any language that wants to guarantee semantics, ease concurrency and resource contention, and otherwise clean things up should have immutability as a default. Something set shouldn't be volatile or mutable unless specified as such, and the semantics should enforce this. But with the collections API, we have immutability as an optional characteristic of Collections. To use a simplistic example, you can do this:</p>
<pre>
Collection c = new ArrayList();
c.add(foo);
c.add(bar);
</pre>
<p>Collection has an add() method. This means that, by definition, Collection is mutable. "But wait!" you cry, you can obtain an immutable version of this collection by calling Collections.immutableCollection(c);. Sure. At which point you have something that conforms to the contract of Collection, but which may throw exceptions if part of that contract is relied upon. In other words, you should not have access to methods that allow you to break the contract. To allow this is to have written a bad contract with ambiguity. Now, to properly guard against the possibility of stray immutable collections, you may be forced to check immutability before executing the contract (the add() method) or you may have to guard against the exceptions with try-catch logic and exception handling. You can see some of this in the concurrent library's implementations of concurrent collections. It's not bad, but could be simpler with an immutable collection interface.</p>
<p>Additionally, consider how hard it is to create anonymous one-off implementations of Collection. You have to implement not only size, contains(), iterator(), but all the storage logic. If you're wrapping an existing data structure and merely wanted a read-only view on the structure, you are forced to implement all that extra API in your code purely to satisfy the optional contract provided in the Collection definition.</p>
<p>The point here is that an immutable Collection is a sub-set of the functionality of a MutableCollection. That should be obvious, but apparently wasn't to Sun. Consider had Sun used the model used in NeXTSTEP (and now Apple's Cocoa) APIs. Collection would have been defined as (simplified):</p>
<pre>
public interface Collection<T> extends Iterable<T> {
public Iterator<T> getIterator();
public int getSize();
public boolean isEmpty();
public boolean contains(T object);
public boolean containsAll(Collection<T> object);
public T[] toArray(T[]);
}
</pre>
<p>and</p>
<pre>
public interface MutableCollection<T> extends Collection<T> {
public boolean add(T object);
public boolean addAll(Collection<T> objects);
public boolean remove(T object);
public boolean removeAll(Collection<T> objects);
public boolean retainAll(Collection<T> objects);
public void clear();
}
</pre>
<p>This would, ultimately, mean that a Collection instance, typed as a Collection would not have any mutable methods available to invoke, let alone that would need guarding against stray immutable invocations. There would then be two strategy for guaranteeing immutability. One... cast the stupid thing as Collection, and onothing that has access to the cast can get at the MutableCollections methods (except by explicit reflection). Alternately, add a "getImmutableCopy()" method to the MutableCollection interface that creates a shallow copy that is NOT an implementor of MutableCollection... merely of Collection. Then you have a safe "snapshot" of the mutable object that can be freely passed around without worry that something else will modify it.<br /></p>
<p>Ok, why is this such a big deal? It's about having code that means what it says. If I have to guess about the run-time state, or more concrete type of an instance to know if it's OK to invoke one of its methods or not, then I'm working with implicit contract, and that's murky territory. Java, by making an ImmutableCollection a special implementation of Collection, has inverted the hierarchy of contract, and exposed methods that are not truly available for all implementors. Optional interfaces are fine, but you don't expose the optional interface above where it's true. It's a basic piece of encapsulation that the Java folks just seemed to forget.</p>
<p>Now, this is a decade too late, this little rant. Truth is, I made it when I worked at Sun, but was quite the junior contracting peon, and had no real voice. Now, I have a blog, and am free to whine and be annoyed in public. :) But I hope to make a more general point here about contract. Optional contracts (APIs) need to be handled very carefully, and in a way such that an unfamiliar programmer can understand what you meant from how the contract reads. Look at your interfaces from the perspective that it should not offer what it (potentially) cannot satisfy. Polymorphism doesn't require "kitchen-sinkism" in an API. Just careful, thought-out layers.</p>
<p>The issues of testability also arise here, insofar as a class that has to implement optional APIs must, therefore, have more tests to satisfy what are, in essence, boilerplate code. If I made a quick-and-dirty implementation of Collection as a wrapper around an already existing, immutable data structure, then I have to implement all of those methods and test them, when in fact, half of them will throw an exception. This means (to me) that they probably shouldn't even exist. Code with a lot of boilerplate code (lots of extra getters and setters, or over-wrought interfaces) tend to be hard to test, and one of my big annoyances in life (these days) is testing boilerplate code. Ick.</p>
<p></rant></p>Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com2tag:blogger.com,1999:blog-6479932290949673745.post-41193743726257693072008-12-11T17:50:00.000-05:002008-12-11T19:57:06.961-05:00Testability - re-discovering what we learned and forgot about software
development.<p>(or, why agile approaches require good old-fashioned O-O)</p>
<h3>What are we all talking about? (the intro)</h3>
<p>Testability comes out of an attempt to understand how agile processes and practices change how we write software. Misko Hevery has written some rather wonderful stuff on his blog, and starts to get into issues around singletons, dependencies, and other software bits that get in the way of testability, and starts to look at testability as an attribute. (Full plug at the end of this post) But in particular, he also starts looking at what design and process changes can we start to use to make code more testable. And while we're at it, what is the point? Is testability the point? It's important, especially as a way to remove barriers from working in an agile environment, if that's what we've chosen. There are reasons related to quality as well. But I think there are some deeper implications, which Misko and others have implied and, on occasion, called out. It's that we've forgotten the point of Object-Orientation and what it was trying to achieve in the 80's and 90's, and are re-discovering it.</p>
<h3>What have we forgotten? (the reminiscence)</h3>
<p>But what is the essence of what Misko is saying? Martin Fowler (coiner of the Inversion of Control term) and others have written wonderful articles on "Law of Demeter" and other principles. In general, they are all looking at how we grow software, and between all the thinkers and talkers and doers, it seems to me we're re-discovering some key concepts that we all learned in college but forgot in the field. The key points are:</p>
<ol>
<li>Manage complexity by separating concerns and de-coupling code.</li>
<li>Map your solution to the business problem.</li>
<li>Write code that is not brittle with respect to change.</li>
<li>Use tools that empower your goals, don't change your goals to fit the limits of the tools.</li>
</ol>
<p>In other words, what we all learned when we were taught textbook Object-Oriented Analysis and Design and Programming. Now, O-O has had its various incarnations, and I would contend that all the architectural threads of AOP, Dependency Injection, as well as a more conservative take on O-O have all stemmed from these key principles which were at the core of the Smalltalk revolution and early attempts to get rapid development cycles, and what is often now called "agile" development.</p>
<h3>How did we forget all this? (the rant)</h3>
<p>So why did we forget all this? Five reasons, I suspect:</p>
<h4>Selling Object-Orientation</h4>
<p>We did a crappy job of selling O-O. You might not think so, since O-O is so prevalent (or at least O-O languages are). However, we didn't sell the 4 notions I mentioned above, we sold business benefits that were, in essence, lies. They didn't need to be, but usually were. These are things like "O-O will make you go faster because of reuse," or "O-O will help you reduce costs because of reuse," and on and on. These can be true, but are usually the result of a longer evolution of your software in an O-O context, and the costs of realizing the benefits that we sold often would be too high for businesses to stomach. In fact, O-O won, in my view, because managing complexity became fundamentally necessary when software scale became huge.</p>
<h4>Cheap, fast computers</h4>
<p>Fast computers have allowed us to do so many bad things. Room to move and space to breath unfortunately gave us less necessity for discipline. We removed the impetus to be efficient and crisp and to think through the implications of our decisions in software. However, we're catching up with hardware in terms of real limits. Moores law may still apply, but we are starting to have limits in memory. A client of mine observed that Google is an interesting example. He works for an embedded software firm, and noted that they have probably similar scaling issues as an embedded (say, phones or similar devices) device company, because sheer volume of traffic forces Google against real limits, much the way resource constraints on a telephone forces those companies against their constraints. However most of us live in a client-server mid-level-traffic dream of cheap hardware so that we can always "throw more hardware at it".</p>
<h4>Java</h4>
<p>Java is an O-O language, and really was the spear-head that won the wars between O-O and structured programming in the '90s. However, bloated processes and territorialism have kept Java from fixing some of its early issues that prevented it from solving problems such as I mention above in efficient ways. The simple example is reflection. If a langauge requires that I create tons of boilerplate code (try-catch, look-up this, materialize that) to find out if an object implements a method, and then to invoke it, it needs to provide a way for me to eliminate the boilerplate. If it can't do it in a clean way, it should at least provide convenience APIs for me to do the most common operations without all that boilerplate. Sadly, the core libraries of Java were bloated even in the beginning, because of the tension between the Smalltalk, Objective-C people on one hand, the C++ people on the others, and Sun not caring really, because they were a hardware company. So because Java won the O-O war (don't argue, I'm generalizing) its flaws became endemic to our adoption of O-O practices. I'm going to mention that J2EE bears about half of Java's responsibility, but I'll leave that for another flame. Nevertheless, the design and coding and idiomatic culture that spawned from these toolsets have informed our approaches to O-O principles for over a decade.</p>
<h4>The .com bubble</h4>
<p>The dot-com bubble compounded our Java woes by introducing 6-month diploma programmers into the wild - nay, into senior development positions, and elevated UI scripting a-la JSP and ASP, which allowed for the enmeshment of concerns beyond anything we'd seen for a while in computing. All notions of Model-View-Control separation (or Presentation, Abstraction, Control) were jettisoned while millions of lines of .jsp and .asp (and ultimately .php) script were foisted onto production servers, there to be maintained for decades (I weep for our children). While this was invisible in early small internet sites, the Dot-Com bubble which careened the internet into a primary vehicle for business, entertainment, culture, and these days even politics caused a growth in number, interaction, and complexity of these sites that has caused unmitigated hell for those who found their "whipped-up" scripted sites turn into high-traffic internet hubs. Much of this code has been re-written out of necessity, and yet it caused the travesty that is Model-1 MVC and other attempts to back-into good O-O practice from a messy start. These partial solutions were propagated as good practice (which, by comparison with the norm, they were) and a generation of students learned how to do O-O from Struts and other toolsets. Ignored in that process were wonderful tools like WebObjects or Tapestry which actually did a fair job of doing O-O AND doing the web, but I'll leave that point here.</p>
<h4>Design Patterns</h4>
<p>A small corollary to the dot-com bubble is that combining Java, and Patterns concepts from the Gang of Four, these new developers managed to create a code-by-numbers style of design, where you don't describe architecture with patterns, you design with patterns up-front. This has led to some of the worst architecture I've ever seen. Paint-by-numbers has never resulted in a Picasso or Monet, and rarely results in anything anyone would want to see except the artist's mother. Design Patterns and pattern-languages aren't bad - far from it. However, they are a language to discuss architecture, they are not an instruction manual. They should be symptoms of a good software design, not an ingredient.</p>
<h4>Really big, bloated projects</h4>
<p>Lastly, really really big projects have taken all of the above and raised the stakes. We are now finding that the limits of software aren't the Hardware (thank you Moore), but rather the people. A whole generation of us attempted to solve this by increasing the process weight around the development effort. This satisfied some contractual issues with scale, but in general failed to attend to the issues raised in <a href="http://en.wikipedia.org/wiki/The_Mythical_Man-Month">The Mythical Man-Month</a>, despite 40 years of its having been published.</p>
<p>A side-effect of really big projects is that when you have that much money on the table, risk-mitigate goes into high-gear, and <a href="http://www.schneier.com/blog/archives/2008/10/does_risk_manag.html">people are bad at risk analysis</a> and planning. We tend to manage risk by telling ourselves stories. We invent narratives that help us manage our fears, but don't actually manage risk. So we make very large plans. Idealistic (even if they're pessimistic) portrayals of how the project shall be. Then, because we want to "lock down" our risk, we solicit every possible feature that could potentially be in, including, but not limited to, the kitchen sink, to make sure we haven't forgotten it. It goes in the plan, but by this point we have <a href="http://agilesoftwaredevelopment.com/2006/12/kano-model-of-customer-satisfaction">twice the features any user will ever ever use</a> and 80% of the features provide, maybe, 20% of the value. So we actually increase risk to the project's success while we are trying to minimize and control it. This kitchen-sinkism leads to greater and larger projects, but then large projects bring prestige as well, so there are several motivation vectors for large-projects. Most of them aren't good.</p>
<h3>Enter Agile (the path to the solution)</h3>
<p>Agile software started to address the human problem of software, and I won't go into it much here, as it's well covered elsewhere. However, one can summarize most agile methods by saying that the basics are</p>
<ol>
<li>Iterate in small cycles</li>
<li>Get frequent feedback (ideally by having teams and customers co-located)</li>
<li>Deliver after each iteration (where possible)</li>
<li>Only work on the most important thing at a time.</li>
<li>Build quality in.<br /></li>
<li>Don't work in "phases" (design, define, code, test)</li>
</ol>
<p>This is a quick-n-dirty, so no arguments here. It's just an overview. But from these changes there are tons of obstacles, issues, and implications. They are, indeed, too numerous to go into. But a light non-exhaustive summary might include:</p>
<ul>
<li>You can't go fast unless you build quality in</li>
<li>You can't build quality in unless you can test quickly</li>
<li>You can't test quickly if you can't build quickly</li>
<li>You can't test quickly if you aren't separating your types of testing</li>
<li>You can't test quickly if your tests are manual</li>
<li>You can't automate your tests if your code is hard to test (requires lots of setup-teardown for each test)</li>
<li>You can't make your code more amenable to testing if it's not modular</li>
<li>You can't ship frequently if you can't verify quickly before shipping</li>
<li>You can't build quality in if you ship crap</li>
<li>You can't get feedback if you can't ship to customers</li>
<li>etc.</li>
</ul>
<p>Lots of "can't" phrases there, but note that they're conditionals. Agile methods don't actually fix these problems, they expose them, and help you do root-cause analysis to solve them. For example, look at some of those chains there.</p>
<p>If I, for example, take my "<a href="http://www.laputan.org/mud/">Big Ball of Mud</a>" software system and re-tool it to de-couple it's logical systems and components into discrete components in its native language (say, Java), then I suddenly can test it more helpfully, because I can test one component without interference from another. Because of this, my burden of infrastructure to get the same test value goes down. Because of this my speed of test execution improves. This causes me to be able to test more frequently (possibly eventually after each check-in). This causes me to be able to make quick changes without as much fear, because I have a fast way of checking for regressions. This allows me to then be less fearful of making changes, such as cleaning up my code-base... Oh wow - there's a circle.</p>
<p>In fact, it is a positive feedback loop. Starting to make this change enables me to more easily make the change in the future. But once I'm moving along in this way, I start to be able to ship more frequently, because my fast verification reduces the cost of shipping. This means I could ship after three iterations, instead of twelve... or eventually every iteration. It means I can make smaller iterations, because the cost of my end-of-iteration process is going down... There are several feedback loops in process during any transition to a more agile way of doing things, as the agile approach finds more and more procedural obstacles in the organization.</p>
<p>But... and here's the big but... if you start to do an agile process implementation and don't start changing how you think about software, software delivery, how you write it, and how it's designed, you're going to run up against an internal, self-inflicted limitation. You can't move fast unless you're organized to accommodate moving fast. Your code base is part of your environment in this context. So starting to help developers subtly move in this direction, and increase the pace at which they transition the existing code-base into a more suitable shape for working efficiently is critical. This, as it turns out, involves our dear old O-O.</p>
<h3>What are testability, O-O, and other best practices today? (the recipies)</h3>
<p>There's a wealth of info out there. These don't just include software approaches but also team practices. <a href="http://martinfowler.com/">Martin Fowler</a>, <a href="http://misko.hevery.com/about/">Misko Hevery</a>, <a href="http://en.wikipedia.org/wiki/Kent_Beck">Kent Beck</a>, <a href="http://www.objectmentor.com/omTeam/martin_r.html">(Uncle) Bob C. Martin</a>, <a href="http://www.linkedin.com/pub/0/13b/0ab">Arlo Belshee</a>, and a host of others I couldn't name in this space provide lots of good text on these. These include <a href="http://martinfowler.com/articles/injection.html">dependency-injection</a>, <a href="http://portal.acm.org/citation.cfm?id=1122100">continuous code review (pair programming)</a>, <a href="http://www.truthtable.com/RadicalCol-ocation.html">team co-location</a>, <a href="http://en.wikipedia.org/wiki/Separation_of_concerns">separation of concerns</a>. On the latter point, <a href="http://en.wikipedia.org/wiki/Aspect-oriented_programming">Aspect Oriented Programming</a> is a nice approach which I see as another flavour of O-O, conceptually, in that it attempts to get at some of the same key problems. It is often mixed either with O-O, or with Inversion of Control containers. Fearless <a href="http://www.refactoring.com/">refactoring</a>, <a href="http://martinfowler.com/articles/continuousIntegration.html">continuous integration</a>, build and test automation (I'm a big fan of Maven, btw, since, for all its problems, it makes dependency explicit). <a href="http://en.wikipedia.org/wiki/Test-driven_development">Test Driven Development</a> (and it's cousin <a href="http://www.extremeprogramming.org/rules/testfirst.html">test-first development</a>). Also, the use of <a href="http://www.martinfowler.com/bliki/DomainSpecificLanguage.html">Domain Specific Languages</a> has become quite helpful in both mapping the business problem to technology, but also eliminating quality problems by defining the language of the solution differently. And of course, wrapping this development in a management method that helps feed the process and consume its results - such as <a href="http://en.wikipedia.org/wiki/Scrum_(development)">Scrum</a>, or the management elements of <a href="http://en.wikipedia.org/wiki/Extreme_Programming">Extreme Programming</a>.</p>
<p>These are a sampling of practices that affect how you organize, re-think, design, create, and evolve your software. They rely on the basic principles and premises of agile, but require, in implementation, the core elements that O-O was trying to solve. How can we manage complexity, address the business problem, write healthy code, and be served, not mastered, by our tools.</p>
<h3>Prologue (the plugs)</h3>
<p>I'm a big Misko Hevery fan these days (I can feel him cringing at that appellation). There's a lot I've had to say to my clients on the subject of testable code, designing for testability, and tools and technologies, but Misko seems to have wonderfully summed up much of my discussion on the topic on his <a href="http://misko.hevery.com/">"Testability Explorer" blog</a>. He explains issues like the problem with Gang-of-Four Singletons, why Dependency Injection encourages not only testable code, but it does so by separating concerns (wiring, initialization, and business logic), and all sorts of good stuff like that. It helps to read from the earlier materials first, because Misko does build on earlier postings so later ones may assume you have read the earlier ones and that you are following along. Notwithstanding, his posts are cogent, clear, and insightful and have helped to crystallize certain understandings that I've been formulating over my years of software development into much more precise notions, and he's helped me learn how to articulate and explain such topics.</p>
<p>Misko has also recently published a <a href="http://misko.hevery.com/code-reviewers-guide/">code-reviewer's guide to testable code</a>. Sorely needed in my view. I also want to make a quick shout-out to his <a href="http://code.google.com/p/testability-explorer/">testability explorer tool</a>, which is fabulous, and I'm working on a <a href="http://maven.apache.org/">Maven 2 plugin</a> to integrate it into <a href="http://maven.apache.org/guides/mini/guide-site.html">site reports</a>.<br /></p>
<p>Also, I've built a Dependency Injection container (<a href="http://code.google.com/p/israfil-micro/">google-code project here</a> and <a href="http://www.israfil.net/projects/micro/israfil-micro-container/index.html">docs here</a>) suitable for use on Java2 Micro Edition CLDC 1.1 platform, because I had to prove to a client that you could do this in a reflection-free embedded environment. It's BSD licensed, so feel free to use it if you want.</p>
<p>Lastly, Mishkin Berteig and co. have a decent blog called <a href="http://www.agileadvice.com/">Agile Advice</a> (on which I occasionally also blog) which nicely examines the various process-related, cultural, organizational, and relational issues that working in this way brings up. My posts tend towards the technical on that blog, but occasionally otherwise as well.</p><br />
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com3tag:blogger.com,1999:blog-6479932290949673745.post-10177875093971036742008-12-08T13:21:00.000-05:002008-12-08T19:27:14.067-05:00Agile Engineering Practices course in Ottawa, December 18-19<p>To my stalwart readers: I'm putting on my first public version of an Agile Engineering Practices training that I've presented with various clients over the last couple of years.</p>
<p>The training will cover a few areas, including:</p>
<ul>
<li>Workflow of an Agile Software Development Cycle</li>
<li>Impact of Agile on Conception, Design, Construction, and Verification</li>
<li>Conceptual consequences of Agile methods
<ul>
<li>Software as an emergent property</li>
<li>Modularity, Dependency, and Component-Orientation</li>
<li>Incremental design</li>
</ul>
</li>
<li>Infrastructure needs to support Agile methods
<ul>
<li>Build systems</li>
<li>Version Control and SCM</li>
<li>Continuous Integration</li>
</ul>
</li>
<li>Development Practices and Approaches of Agile methods
<ul>
<li>Testing, Test-First, and Test Driven Development</li>
<li>Writing testable code</li>
<li>Collaborative software development</li>
<li>Pair Programming</li>
</ul>
</li>
</ul>
<p>There is a small overlap with the Scrum training, mostly for context, but a good chunk will focus on the notion of testable, flexible code, and how infrastructure and an agile approach changes your design concepts and techniques. Also, what architectural features will help your code evolve in a healthy way, rather than spin out of control, or paint you into a corner.</p>
<p>Anyway, if you're in Ottawa, or know anyone there who could benefit from such training, let them know. It'll be a small course (no more than 11 students). The cost is initially discounted from its ultimate price of $1,200.00. A steal at $750.00 (CAD of course). You can look at the course <a href="http://www.berteigconsulting.com/AgileEngineeringPracticesTrainingDecember2008OttawaOntarioCanada">here</a> or can <a href="http://www.berteigconsulting.com/node/add/registration">register here</a>.</p>
<p>I'm planning to arrange for more of these in other cities in the new year, including Toronto, Somewhere in the 519 area code - probably Kitchener/Waterloo, as well, possibly, as out west in Saskatoon or Calgary or Vancouver. If you are interested in these cities, let me know, and I'll keep you in the loop.</p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com0tag:blogger.com,1999:blog-6479932290949673745.post-57159492743253525632008-11-17T10:56:00.000-05:002008-11-17T11:00:11.223-05:00Saying you're agile, doesn't make you agile.<p>So a lot of posts have come up in response to <a href="http://jamesshore.com/Blog/The-Decline-and-Fall-of-Agile.html">James Shore's article on Agile's decline and fall</a>, and the implications of using "scrum alone, and misapplied." While I agree with James about the bulk of his content, there's a much simpler issue at work, and it's an issue that's been hanging around since the earliest days. It's about branding.</p>
<p><strong>Agile is a brand</strong></p>
<p>I can call myself anything. I can call my self "Christian" (that's my name). I can call myself Canadian. I can call myself a consultant. These are descriptive, verifiable, and helpful designations in understanding what I'm about. I can also call myself "agile." This, depending on how the label is understood, can be helpful or not. Do I mean that I'm physically agile? Do I mean that I'm flexible in my thinking? Do I mean that I observe a particular method of software development? Does that mean that I'm iterating? The term is, at that level, meant to be evocative, and therefore is a very easy to apply, but slightly meaningless brand.</p>
<p>Scrum is a stronger brand, because it has some licensing, and there's a "by the book" approach. Extreme Programming is also a stronger brand. You can measure the organization's application of practices and determine if they're really doing XP or Scrum (to a larger extent). In fact Agile (as a brand) has always suffered from this weakness, in that it can be applied so generically as to be unhelpful as a description. Because of this, Agile has failed a lot in its history, because anyone can do a little piece of agile, call themselves "Agile", and fail. "Agile" is then a nice scapegoat for the failure. Waterfall is in the similar unenviable position, but it was received wisdom in the software industry for so long no one really questioned that it was one of the culprits until alternatives were seriously proposed.</p>
<p><strong>So is "Agile" failing?</strong></p>
<p>At one point I worked with a client that implemented Scrum. Only they didn't. They let teams determine their own processes in many ways, except that they all iterated. But the infrastructure of Scrum™ wasn't there. But they did not deliver working software every iteration. They did not have a clearly defined product owner. They had ambiguity about what "the team" was, in fact. No scrum-master-alike role. Little criteria for acceptance was provided to the teams. Few teams were allowed to prioritize defects to the top above new functionality. Basically I'm listing off a set of Scrum assumptions/directives that were not followed. Yet this organization called what it was doing "Agile." This company did not deliver software for several years, and is now not "doing agile" as an organization. (Some of the teams are actually moving forward with stronger agile processes, but the outer shell of the organization is moving towards stronger "contract-oriented" delivery in large phases.)</p>
<p>So did Agile fail? Did they fail Agile? Most Scrum proponents would probably suggest (and we did) that they weren't even doing agile in the first place. In fact, they were (on some teams) starting to do some of the engineering practices. The teams were learning, but the organization wasn't. They held retrospectives and didn't apply the learning. They weren't fearless in their willingness to change and adapt to what they were discovering about their process. So in effect, they started to do Scrum-but, ended up doing 1/4 XP with iteration, and management did not accept what was obvious to the rank and file. I do not consider this an "agile failure" or a "failure of agile" because, frankly, Agile wasn't even given a fair shake.</p>
<p>I contend, based on my experience there, that had they implemented Scrum "by the book", it may well not have "saved" them. Their engineers, however, are extremely competent (when permitted) and some of their best immediately began to implement agile-supportive engineering practices. And as consultants we helped them with infrastructure issues like moving towards a more coherent infrastructure around continuous integration, separation of types of testing, etc. I think they could have succeeded, because the ingredients were there. Scrum was exposing problems, insofar as they were using its elements. But without applying the learning, Scrum is as useless as any other learning system. As <a href="http://www.agileadvice.com/2008/11/16/scrumxplean/the-decline-and-fall-of-agile-and-how-scrum-makes-it-hurt-more/">Mishkin Berteig opines frequently in his courses</a>, "Scrum is very hard."</p>
<p>Incidentally, In case you know my clients, don't be too hard on these guys. Management tried very hard, was undermined, interfered, and meant well. For the most part, agile implementations I've observed have been partial implementations largely doomed to produce less value less quickly than was otherwise possible.</p>
<p><strong>Can "Agile" fail if you are using something else.</strong></p>
<p>What James talks about is not "agile" or "scrum" failing, but a wolf in sheep's clothing. I'm not copping out. As I said, most of the implementations I've seen have been half-baked partial agile implementations, which, in retrospective analysis done for my clients, have failed in the obvious faults left where missing practices were ignored or seen as too risky or costly. These methods are systems which work together for reasons. Lean analysis can help, but it still is a learning-system closely aligned to scrum in mental-state (if not in brand), and it will merely uncover things that you have to fix.</p>
<p>If I call myself a Buddhist, but become a materialist freak... did Buddhism fail, or am I truly worthy of the label? If I label myself a Christian, but then fail to love my neighbour, is Christianity failing, or am I failing Christianity? If I call myself a Muslim, then violate the strictures of the Qur'an (by, say, killing innocents), is Islam violent, or am I doing an injustice to the name of the religion? I mean no disservice to any religion when I call them a "brand", but like any other label, you can think of them that way. If a brand is easy to apply, then it's easier to distort and pervert from its original intent.</p>
<p>If I take a bottle of Coca-cola, replace the contents with prune-juice and Perrier, then put it in a blind taste test with Pepsi, which subsequently wins... did Pepsi beat Coca-cola in a blind taste test? No. I subverted the brand, by applying it to something else entirely.</p>
<p>If I've made my point clear, the problem should be obvious. Agile is a weak brand, which can be misapplied. Therefore, in a sense, given that Agile was only a loose community of like-minded methods and system practitioners, maybe it's OK that Agile declines, as a brand. The problem is that such a decline will discourage people from applying the management processes and engineering practices that can encourage organizational learning and product quality.</p>
<p>If that happens, then the sky has fallen, because we weren't making high-quality software before Agile, and I would despair if we just gave up.</p>
<p><strong>Not a fad for me</strong><br /></p>
<p>I'll keep teaching and coaching "agile" teams, encouraging them to implement the full range of agile practices (above and below the engineering line). Not because I'm a believer, but because I find these useful practices which, when applied in concert, powerfully enable teams to deliver high-quality software. And because I enjoy working this way. And because I deliver faster when I do them. That's the bottom line. Agile might be a fad, but I don't really care. I wasn't using these techniques just because they had become popular. Many of them I had used before I had ever heard of the terms Scrum, XP, or Agile. It's a bit like Object-Orientation, which I've been using since the NeXTSTEP days in the early 90's. At that point, it wasn't clear that O-O had "won" over structured programming, and it was not really a "fad" until the advent of Java. That didn't stop me from using a helpful paradigm that improved my quality, integrity, and productivity. And it shouldn't stop anyone else. CIO.com will announce the new fad soon. Use it if its useful. Use Agile methods if you find them useful. Was O-O hard when I first was trying to do it? Sure! There are books on its pitfalls and how to screw it up. (And in fact I find little of it out there, despite the prevalence of O-O supportive languages.) But I still advocate for its use, and use it myself.</p>
<p>The quality, integrity, and delivery of your software is in your hands. Do what you gotta do.</p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com2tag:blogger.com,1999:blog-6479932290949673745.post-42180223398028310632008-11-04T18:01:00.001-05:002008-11-04T18:15:00.854-05:00code coverage != test driven development<p>I was vaguely annoyed to see <a href="http://www.kevinwilliampang.com/post/Is-Code-Coverage-Really-All-That-Useful.aspx">this blog article</a> featured in JavaLobby's recent mailout. Not because Kevin Pang doesn't make some good points about the limits of code coverage, but because his title is needlessly controversial. And, because JavaLobby is engaging in some agile-baiting by publishing it without some editorial restraint.</p>
<p>In asking the question, "Is code coverage all that useful," he asserts at the beginning of his article that <a href="http://en.wikipedia.org/wiki/Test-driven_development">Test Driven Development</a> (TDD) proponents "often tend to push code coverage as a useful metric for gauging how well tested an application is." This statement is true, but the remainder of the blog post takes apart code coverage as a valid "one true metric," a claim that TDD proponents don't make, except in Kevin's interpretation.</p>
<p>He further asserts that "100% code coverage has long been the ultimate goal of testing fanatics." This isn't true. High code coverage is a desired attribute of a well tested system, but the goal is to have a fully and sufficiently tested system. Code coverage is indicative, but not proof, of a well-tested system. How do I mean that? Any system whose authors have taken the time to sufficiently test it such that it gets > 95% code coverage is likely (in my experience) thinking through how to test their system in order to fully express its happy paths, edge cases, etc. However, the code coverage here is a symptom, not a cause, of a well-tested system. And the metric can be gamed. Actually, when imposed as a management quality criterion, it usually is gamed. Good metrics should confirm a result obtained by other means, or provide leading indicators. Few numeric measurements are subtle enough to really drive system development.</p>
<p>Having said that, I have used code-coverage in this way, but in context, as I'll mention later in this post.</p>
<p>Kevin provides example code similar to the following:</p>
<pre>
String foo(boolean condition) {
if (condition)
return "true";
else
return "false";
}
</pre>
<p>... and talks about how if the unit tests are only testing the true path, then this is only working on 50% coverage. Good so far. But then he goes on to express that "code coverage only tells us what was executed by our unit tests, not what executed correctly." He is carefully telling us that a unit test executing a line doesn't guarantee that the line is working as intended. Um... that's obvious. And if the tests didn't pass correctly, then the line should not be considered covered. It seems there are some unclear assumptions on how testing needs to work, so let me get some assertions out of the way...</p>
<blockquote>
<ol>
<li>Code coverage is only meaningful in the context of well-written tests. It doesn't save you from crappy tests.</li>
<li>Code coverage should only be measured on a line/branch if the covering tests are passing.</li>
<li>Code coverage suggests insufficiency, but doesn't guarantee sufficiency.</li>
<li>Test-driven code will likely have the symptom of nearly perfect coverage.</li>
<li>Test-driven code will be sufficiently tested, because the author wrote all the tests that form, in full, the requirements/spec of that code.</li>
<li>Perfectly covered code will not necessarily be sufficiently tested.</li>
</ol>
</blockquote>
<p>What I'm driving at is that Kevin is arguing against something entirely different than that which TDD proponents argue. He's arguing against a common misunderstanding of how TDD works. On point 1 he and I are in agreement. Many of his commentators mention #3 (and he states it in various ways himself). His description of what code coverage doesn't give you is absurd when you take #2 into account (we assume that a line of covered code is only covered if the covering test is passing). But most importantly - "TDD proponents" would, in my experience, find this whole line of explanation rather irrelevant, as it is an argument against code-coverage as a single metric for code quality, and they would attempt to achieve code quality through thoroughness of testing by driving the development through tests. TDD is a design methodology, not a testing methodology. You just get tests as side-effect artifacts of the approach. Useful in their own right? Sure, but it's only sort of the point. It isn't just writing the tests-first.</p>
<p>In other words - TDD implies high or perfect coverage. But the inverse is not necessarily true.</p>
<p>How do you achieve thoroughness by driving your development with tests? You imagine the functionality you need next (your next increment of useful change), and you write or modify your tests to "require" the new piece of functionality. They you write it, then you go green. Code coverage doesn't enter into it, because you should have near perfect coverage at all times by implication, because every new piece of functionality you develop is preceded by tests which test its main paths and error states, upper and lower bounds, etc. Code coverage in this model is a great way to notice that you screwed up and missed something, but nothing else.</p>
<p>So, is code-coverage useful? Heck yeah! I've used coverage to discover lots of waste in my system. I've removed whole sets of APIs that were "just in case I need them" APIs, because they become rote (lots of accessors/mutators that are not called in normal operations). Is code coverage the only way I would find them? No. If I'm dealing with a system that wasn't driven with tests, or was poorly tested in general, I may use coverage as a quick health meter, but probably not. Going from zero to 90% on legacy code is likely to be less valuable than just re-writing whole subsystems using TDD... and often more expensive.</p>
<p>Regardless, while Kevin is formally asking "is code coverage useful?" he's really asking (rhetorically) is it reasonable to worship code coverage as the primary metric. But if no one's asserting the positive, why is he questioning it? He may be dealing with a lot of people with misunderstandings of how TDD works. He could be dealing with metrics bigots. He could be dealing with management-imposed-metrics initiatives which often fail. It might be a pet peeve or he's annoyed with TDD and this is a great way to do some agile-baiting of his own. I don't know him, so I can't say. His comments seem reasonable, so I assume no ill intent. But the answer to his rhetorical question is "yes, but in context." Not surprising, since most rhetorically asked questions are answerable in this fashion. Hopefully it's a bit clearer where it's useful (and where/how) it's not.</p>Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com1tag:blogger.com,1999:blog-6479932290949673745.post-866002577575561882008-10-21T13:19:00.000-04:002008-10-21T13:24:17.579-04:00Hi, my name is Christian, and I'm a geek in a suit.<p>I've been consulting in the high-tech field for over a decade, and programming for substantially longer than that. In fact, my earliest programs were BASIC programs on my C64, though my first true love was an Amiga 1000 owned by my ultra-cool uncle. But these days I wear a suit. I wear a suit because I consult with "geeks" and "suits" and all comers in between. Most of my clients have been in the Financial sector, so dressing up provides credibility with the customer, and my thinking has moved drastically towards a customer-focus over my decade of consulting.</p>
<p>I've started this blog, because I consider things from a few perspectives. As a technologist, I think about software feasibility, artistry, craft, and shippable result. As a geek I think about cool, new, innovative tech. As someone who has to act on behalf of the customer in many situations, I think about value and fitness-to-purpose and total cost of ownership. As a project manager I have to think about predictability, output, project throughput, cost, and so on. As a consultant, I think of cross-cultural conflict resolution, needs analysis, and communications (geeks and suits speak vastly different languages, leaving aside ethno-linguistic or gender communication issues). So my observations hit a lot of these areas. They often will stray into the Agile and Lean categories, though often I end up working in non-agile, non-lean environments, so they're not exclusive to these communities of interest.</p>
<p>I already blog on <a href="http://www.agileadvice.com/author/christian-gruber/" title="AgileAdvice.com">AgileAdvice.com</a>, but really this is intended as a vehicle for personal thinking that's probably going to stray a bit from that blog's editorial perspective, though there will be some overlap. I may cross-post some. Regardless, I wanted a way to start talking about things important to me, professionally, from a variety of perspectives, whether or not I have an audience. Hopefully, someone will find this useful.</p>
<p>Cheers,</p>
<p>Christian Gruber</p>
Anonymoushttp://www.blogger.com/profile/11149956661907218421noreply@blogger.com3