Pacers are human beings like every other runner. They can make mistakes, get injured, get tired, and so many other unforeseen situations. When we decide to follow a pacer we have to accept that anything can happen in such a long course, which might result in glory or frustration. It is about accepting a variable we can’t control. I had a personal experience with a pacer who was going faster than the average pace for the goal time. Most of my fuel was burned in the first half of the course, leaving me exhausted in the second half.
Let’s use LibRunner to investigate the case. If you are not familiar with LibRunner yet, read [an introduction we published previously] to learn how to run the following code. As published in Strava, the 4:10h pacer of the Calgary Marathon 2023 was running too fast for more than 30 km. But first, what is the average pace we would expect from him to finish the marathon in 4:10 hours?
use librunner::running::{Race, MetricRace, ImperialRace, Running, MetricRunning, ImperialRunning};
use librunner::utils::converter;
use librunner::utils::formatter;
fn main() {
const MARATHON_DISTANCE: u64 = 42195; // meters
let duration = converter::to_duration(4, 10, 0);
let marathon: MetricRace = Race::new(MARATHON_DISTANCE);
let running: MetricRunning = Running::new(duration);
println!("The pace to run a marathon in {}h is {}/km",
formatter::format_duration(running.duration()),
formatter::format_duration(running.average_pace(&marathon)));
}
It prints: “The pace to run a marathon in 04:10:00h is 05:55/km”. Looking at his splits, he was mostly below this pace and only slowed down in the last 8 km to the finish line. So, what was his average pace before he slowed down? Adding to the previous code:
let mut splits: Vec<Duration> = Vec::new();
splits.push(converter::to_duration(0, 5, 53)); // 1km
splits.push(converter::to_duration(0, 5, 38)); // 2km
splits.push(converter::to_duration(0, 5, 44)); // 3km
splits.push(converter::to_duration(0, 5, 37)); // 4km
splits.push(converter::to_duration(0, 5, 29)); // 5km
splits.push(converter::to_duration(0, 5, 43)); // 6km
splits.push(converter::to_duration(0, 5, 33)); // 7km
splits.push(converter::to_duration(0, 5, 46)); // 8km
splits.push(converter::to_duration(0, 5, 30)); // 9km
splits.push(converter::to_duration(0, 5, 33)); // 10km
splits.push(converter::to_duration(0, 5, 27)); // 11km
splits.push(converter::to_duration(0, 5, 33)); // 12km
splits.push(converter::to_duration(0, 5, 37)); // 13km
splits.push(converter::to_duration(0, 5, 25)); // 14km
splits.push(converter::to_duration(0, 5, 42)); // 15km
splits.push(converter::to_duration(0, 5, 54)); // 16km
splits.push(converter::to_duration(0, 5, 42)); // 17km
splits.push(converter::to_duration(0, 5, 41)); // 18km
splits.push(converter::to_duration(0, 5, 50)); // 19km
splits.push(converter::to_duration(0, 5, 51)); // 20km
splits.push(converter::to_duration(0, 5, 43)); // 21km
splits.push(converter::to_duration(0, 5, 39)); // 22km
splits.push(converter::to_duration(0, 5, 43)); // 23km
splits.push(converter::to_duration(0, 5, 37)); // 24km
splits.push(converter::to_duration(0, 5, 42)); // 25km
splits.push(converter::to_duration(0, 5, 43)); // 26km
splits.push(converter::to_duration(0, 5, 42)); // 27km
splits.push(converter::to_duration(0, 5, 41)); // 28km
splits.push(converter::to_duration(0, 5, 36)); // 29km
splits.push(converter::to_duration(0, 5, 37)); // 30km
splits.push(converter::to_duration(0, 5, 34)); // 31km
splits.push(converter::to_duration(0, 5, 40)); // 32km
splits.push(converter::to_duration(0, 5, 47)); // 33km
splits.push(converter::to_duration(0, 5, 41)); // 34km
splits.push(converter::to_duration(0, 5, 51)); // 35km
let race: MetricRace = Race::new_from_splits(&splits);
let race_running: MetricRunning = Running::new_from_splits(&splits);
println!("The average pace of these {} splits is {}/km",
race.num_splits(),
formatter::format_duration(race.average_pace()));
}
It prints: “The average pace of these 35 splits is 5:39/km”. This is 16 seconds faster splits! What would be the finishing time if the pacer decided to maintain that pace until the end of the race? Adding to the previous code:
let faster_marathon: MetricRace =
Race::new_from_pace(MARATHON_DISTANCE,
race.average_pace());
println!("If he continued with the pace of {}/km to the finish line, \
he would finish the marathon in {} hours.",
formatter::format_duration(faster_marathon.average_pace()),
formatter::format_duration(faster_marathon.duration()));
}
It prints: “If he continued with the pace of 05:39/km to the finish line, he would finish the marathon in 03:59:06 hours”. This is almost 11 minutes faster! When runners are moving faster than what they trained for, they will hit the wall between 30 to 35 km, which is when their glycogen stores deplete. Therefore, any runner following that pacer would likely hit the wall and finish behind their initial goal.
Effectively balancing the energy consumption throughout the race, considering the elevation profile, and managing hydration and nutrition, definitely makes the marathon a strategic distance. We can have our strategy when we learn how our bodies perform under pressure or we can follow somebody else’s strategy, taking the risk of going off rails.
It is not about trusting a marathon pacer. It is about controlling as many variables as possible and being able to reason during the race whether it is a good idea to follow a pacer or not. For more tips on how to have more control over your pace, read this article published in Canada Running Magazine.
]]>LibRunner is an open-source library published on GitHub and distributed by creates.io. We decided to write it in Rust for the following reasons:
Performance: Rust is fast, compared to C, and often replaces C/C++ in production code due to its memory safety guarantees. The domain of sports often requires real-time features, so Rust allows calculations to be as fast as they can be, without the overhead of garbage collection.
Popular: Rust has been elected by an annual Stack Overflow survey as the most loved language for 7 consecutive years. It has a large and strong community of users and contributors as well as a great number of sponsors, from small start-ups to global players, investing in the language and its ecosystem. It implements the state-of-the-art on developer experience, making developers happy about the process of building high-performance software.
Reusable: Publishing LibRunner on creates.io is so straightforward that we did it as soon as its first feature was available. Today, any Rust project can add it as a dependency and use it for running-related applications. In the browser space, LibRunner can be used by WebAssembly applications. In addition to that, LibRunner can also be used by other programming languages through bridges. We intend to support other languages in the future in a on demand basis, ensuring the same underline logic across platforms and technologies.
High Level, yet native, writing Rust code feels like writing in a high-level language such as Kotlin, Scala, and C#. Yet, the result is native, platform-specific, and self-contained software.
LibRunner can help you calculate the average speed and pace to complete a distance within a duration, calculate the distance of a race from duration and pace, calculate the duration and distance of the race from a set of splits, calculate the pace of positive and negative splits of a race and so many other features that we are constantly adding to it as we learn more about the field. We are also open to suggestions, demands, and contributions.
In the LibRunner documentation, we have examples of code for every feature. These examples are guaranteed to work because the testing tool runs them against the current version. You can copy and paste them into your code and make the necessary changes to the problem you are solving.
Let’s go through these quick steps to get started with LibRunner. We start a Rust project from zero to unlock the way to hero:
visit https://rustup.rs and install rustup, an installer for the programming language Rust. Once installed, update and check the toolchain:
$ rustup update
$ rustc --version
$ cargo --version
create your new running application:
$ cargo new runningapp
a folder called runningapp
is created. Go into it and run the project:
$ cd runningapp
$ cargo run
it prints “Hello World”, meaning you have a working code to start from. Open the project in your favourite code editor and make two changes:
4.1. add LibRunner to the project’s dependencies:
$ cargo add librunner
It adds a new dependency to your Cargo.toml
file:
[dependencies]
librunner = "0.6.0"
4.2. replace the content of the file src/main.rs
with the code below:
use std::time::Duration;
use librunner::running::{Race, MetricRace, ImperialRace, Running, MetricRunning, ImperialRunning};
use librunner::utils::converter;
use librunner::utils::formatter;
fn main() {
let duration = converter::to_duration(4, 0, 0); // 04:00:00
let m_marathon: MetricRace = Race::new(42195);
let m_running: MetricRunning = Running::new(duration);
println!("The pace to run {}km in {}h is approximately {}/km at {:.2}km/h",
converter::to_km(m_marathon.distance),
formatter::format_duration(m_running.duration()),
formatter::format_duration(m_running.average_pace(&m_marathon)),
converter::to_km_h(m_running.speed(&m_marathon)));
let i_marathon: ImperialRace = Race::new(46112);
let i_running: ImperialRunning = Running::new(duration);
println!("The pace to run {} miles in {}h is approximately {}/mile at {:.2}mph",
converter::to_mile(i_marathon.distance),
formatter::format_duration(i_running.duration()),
formatter::format_duration(i_running.average_pace(&i_marathon)),
converter::to_mph(i_running.speed(&i_marathon)));
}
then run the project again:
$ cargo run
which generates the following output:
The pace to run 42.195km in 04:00:00h is approximately 05:41/km at 10.55km/h
The pace to run 26.2 miles in 04:00:00h is approximately 09:09/mile at 6.55mph
That’s it! You are now using LibRunner in no time. Keep an eye on this website to learn more about future updates and all things geeks love about running.
]]>8 years ago I failed to write a book. I wrote it until chapter 3 but I couldn’t stand all the criticism coming from the editor and the reviewers. As the deadline to deliver chapter 4 was approaching, I was still overwhelmed by all the work to catch up with their feedback. So, I quit, but I learned something very important from that experience: the scrutiny over the writing of books makes them very good references.
It’s true that the time required to write and publish a book is incompatible with the rapid pace Information Technology evolves, making it quickly obsolete. However, Clojure is well known for its long term stability and backward compatibility. Clojure books have a very slow obsolescence and they are worth buying.
Most Clojure programmers I know, including myself, love to own Clojure books, but beginners may find interesting to have access to some books for free, before starting a new book collection. The Toronto Public Library is there to help.
I don’t know about other public libraries out there, but the Toronto Public Library is a model to follow. There is a central reference library where books are for your-eyes-only and dozen other branches all over the city, each one adapted to the needs of the neighbourhood. They offer not only books, but also e-books, audio-books, magazines, seminars, courses, all sort of multimedia material, image/video editing and 3D printing. When you subscribe, you gain access to all these services, most of them for free, with a library card that you can also use online to borrow books delivered to a branch near you.
This is a list of Clojure books you can borrow and keep for 21 days and renew them two times for the same length of time, making a total of 63 days!
It is also possible to put your hands on books about other LISP languages:
Even when some books aren’t physically available, they can be accessed online thanks to a partnership with Safari Books Online. I would highlight:
Clojure is probably the most popular programming language among the functional ones. It is good to see that it is also accessible to everyone living in Toronto. When you borrow one of these books to learn it you’re going to fill a spark that will change the way you think about programming forever. It will be the beginning of your own Clojure book collection.
]]>A year ago I wrote an article about my Knowledge Portfolio in 2022. Indeed, knowledge is an investment, but I changed my mind about diversifying it. At the time, I described the domains, the programming languages, and the projects I planned to spend time with. To this day, some of them flourished and others failed. Since time is a valuable asset, spending it with a limited set of subjects will lead to mastery sooner. I’m trying to downsize this year, so let’s talk about what happened in 2022 and what to expect in 2023.
First, let’s look at the projects. Digger is a tool that I actively use at work to document everything I discovered in relational databases. It evolves very slowly, with only 46 commits in 2022, mostly dependency upgrades, issues with the embedded H2 database, security advisories, and the gradual migration from Java to Kotlin. No new features were added in 2022, but I’m planning to finish the migration to Kotlin and implement the model synchronization in 2023. Today, Digger only stores the elements that are documented. The rest of the model is dynamically loaded throw an active database connection. What we want to do is to store the entire model and keep it in sync with detected database changes, making an active connection optional or on-demand.
Minimily is another project that didn’t evolve at all. I didn’t even used it in 2022, which was a bad idea. It helps managing my finances and home inventory, which are really necessary in times of crisis. As a new year’s resolution, I decided to put it up and running and start using it from January 1st. To my surprise, it is working just fine, but I confess that I need a little recap of Clojure to make sense of the code again. In 2023, I’m going to evolve it according to my needs and implement the concept of “Family” as a group of users with access to the same accounts and assets.
CSVSource was the only project I started in 2022 and implemented all features as planned. It works as a command-line tool, but also as a library. Apart from solving a data engineering problem, this project also helped me to learn Rust, a complex language that delivers the satisfaction of a memory-safe app without the burden of garbage collection. In 2023, I intend to document it better, increase test coverage, and support NoSQL databases as well.
Controlato didn’t leave the conceptual phase in 2022. It’s an idea that I like very much, but I’m just not motivated anymore to move it forward with Python. Maybe, if I change the stack in 2023 I can finally get it done, but I have no decision made yet. It will just wait for the right time.
Two projects were completely abandoned: SampleReads and SpitFHIR. The first was planned to be developed in Elixir, but I only found time to learn one new programming language, which was Rust. The project was abandoned after the initial bootstrap and will be deleted in the coming days. SpitFHIR was also abandoned after the initial bootstrap because its goal was too ambitious. The FHIR standard is very large and complex, leaving no room to low budget implementations. The project will also be deleted in the coming days. Future projects will likely be very small, focused on automation opportunities.
From the list of knowledge domains, only home automation and utilities remain active. Healthcare and sports are now deprecated. From the list of programming languages, only Clojure, Go, Kotlin, and Rust will have a chance in 2023. Rust is probably the one that will get most of my attention.
Finally, I’m going to migrate this blog from Jekyll to Hugo. I’m tired of dealing with Ruby issues, a technology that I don’t even use myself. Hugo is extremely fast and has a nicer template engine that allows me to easily share the same custom template among my blogs. The only caveat is that Github automatically builds any Jekyll website without extra configuration, while Hugo requires a Github action. As long as it does its job, I’m ok with that.
]]>I was a Ph.D candidate in 2009, in the Electrical Engineering department, at Université catholique de Louvain. A coleague of mine, Russian, told me about this conference in Saint Petersburg and asked me if I wanted to submit a paper. I agreed, we wrote it together and it got accepted. Since she was in Moscow at the time, I thought she would present the paper, but our lab had a policy that they could only pay for trips from Belgium to Russia, not from Russia to Russia. So, I had to go.
The first step was to get a Visa. To my surprise, I needed an invitation from someone in Russia to be able to apply. Fortunately, the conference itself was the invitee. I went to the Russian consulate in Brussels, with all required documents, and my feeling was like getting into a secret service office to be interrogated. When it was my turn, the two-story security guy at the entrance pointed at me and commanded to hurry up inside. Short line, straight people, the process was actually fast. Perhaps because of my Brazilian passport, a friendly country that welcomes Russians for decades. In other words, a weak country afraid of any armed conflict. Some may say that if a world power invades Brazil, they would accept the new regime right away without fight, and wait for the carnival to reach peace.
It was a cheap flight, with connection in Frankfurt, Germany. I arrived in Saint Petersburg late afternoon, unaware of a big celebration that was taking place in the city: The Scarlet Sails during the White Nights Festival. The taxi driver had a hard time to drop me at the hotel because the streets were full of people and traffic jams, but nothing that distracted my attention from spectacular fireworks in the horizon. I finally arrived at the hotel and was eager to walk around, see the festivities, but for my astonishment, the receptionist confiscated my passport for inspection. They gave me a paper and asked me to pass by in the morning to recover it. I mean, really?! How can you possibly suspect that a Brazilian individual could be of any harm to Russia?! Let me tell you something about Brazilian espionage: the chef of our secret service released in his social networks a document showing he tested negative for COVID-19. Printed in his document was the equivalent of his social security number. The next day, hundreds of loans were taken on his behalf, he was registered as supporter of a soccer team, and even registered in a opposition party. As you can see, if I was a secret agent it would be clearly stamped on my face. Anyway, I was scared, but not in panic yet. Watching the fireworks through the window was somehow relaxing.
The next morning was a beautiful sunny day, the beginning of the conference. With my passport in hand, I walked all the way to the venue, enjoying a very charming city, full of people, parks, monuments, clean, and enlightened by the sun reflected on serene water channels. The conference was in the vicinities of the great Hermitage Museum, comparable to the Louvre in Paris. After finishing my presentation, I had the chance to visit the museum with fellow researchers and also went in a boat tour through the channels of the city. Everything was magnificent, showing all the progress Russia has achieved over the years, with the introduction of capitalism, after the fall of communism. Today, all that prosperity is in check, shrunken by the ego of a single insane person: Vladimir Putin.
In January 31, 1990, McDonald’s opened its first restaurant in the old Soviet Union. That day, the line of people stretched the length of five soccer fields. More than 30,000 ordinary Soviet citizens waited six hours or more. When they tried their first hamburger and fries, it really was a life-changing moment for them. Today, a huge number of companies are leaving Russia, including McDonald’s, some in protest to the aggression to Ukraine, others disrupted by broken supply chains. Russians learned to do many things from the occident, but its isolation will freeze its innovation. Thousands of highly skilled people already left the country and it will take a while for them to return - if ever - with this hostile government in power. The recovery of Russia to pre-conflict levels is certainly two generations apart.
That conference went well, I presented the paper, made good friends, but I’m afraid this war will put us apart for decades to come. I wish I could visit Saint Petersburg again, but I don’t think it will be possible any time soon due their isolation. Also, I never had the chance to visit Ukraine, but if I had I would be deeply sorry to know that all the places I visited are now destroyed by unprepared or ruthless soldiers. I hope Ukrainians get their territory back, join the European Union, get reconstructed and enjoy prosperity in the long term.
]]>I’m over a year working in the healthcare industry. Before that, I was involved in education and finance. These are industries with significant impact on people’s lives, but the impact of healthcare is far more important because we may be poor or poorly educated, but being poorly healthy is a life threatening situation. A bug in an education system is upsetting, but it can be fixed without major impact. A bug in a financial system is troublesome to many people but they all have a chance to recover with the help of government and insurance companies. In a healthcare system, a bug can kill people. Loosing a simple allergy reaction record can represent severe complications during a surgery, for example.
There is no shortage of complex problems to solve in healthcare and they are pushing me to become a better engineer. While working at PointClickCare, I was exposed to a popular standard called HL7® FHIR®, which has been used not only to integrate heterogeneous systems, but also to enable a marketplace of applications that exponentially expands the offering of services for patients and care givers.
FHIR® (Fast Healthcare Interoperability Resources) is for healthcare what Swift is for finance. It is not simple. It takes some time to digest but it seems to have just enough complexity to cover the needs of healthcare without limiting systems and users. What I don’t get is the meaning of the word “Fast” in the acronym. I’m pretty sure it’s not about runtime performance. It depends on the library and the design of the system. I think it’s probably about productivity because there is a mature ecosystem out there to build solutions within hours, under the most optimistic circumstances of course.
FHIR® - pronounced like “fire” - structures healthcare data into resources, which also match the concept of resource in REST (RepResourceresentational State Transfer), making it an excellent standard to build healthcare APIs. Resources are composed of strongly typed elements. From that, we get that resources also follow the object-oriented paradigm in the way Java and C# implement it. They use abstractions similar to classes, attributes, methods, and inheritance. Resources are serialized in multiple formats such as JSON, XML, and RDF. The most powerful characteristic of the standard is extensibility. New resources can be created - by extending the Basic resource -, existing resources and elements can be extended, new elements can be added, all that while still compliant with the standard and without breaking interoperability. Here is how a Patient resource looks like in JSON format:
But the Patient resource does not contain anything about the health of a patient. For that, we need other resources like Condition and Observation. The Condition below indicates that the Patient above has a mild form of asthma. Notice the reference to the Patient in the attribute “subject”:
Many other resources are available to exchange healthcare data, but we don’t have to build anything from scratch to deal with those resources. There are many FHIR® implementations out there and one will probably match your stack, saving a ton amount of work. The most commonly used are Hapi FHIR (Java), Firely .NET (C#), and fhir.js (JavaScript). They are high level languages that match all abstractions required by the standard. Here are some advantages of using these libraries:
A single client implementation can connect and exchange data with multiple compliant servers.
We don’t have to build an in-house FHIR® client because one probably exists for our stack, giving us data ready to be processed.
If more data needs to be exchanged out of what FHIR® already offers, then the extension framework can be used to serve the data without breaking compatibility with existing clients.
Some active members of the community make FHIR® servers openly available for learning purpose. Hapi FHIR is one of those. Visit http://hapi.fhir.org/ to explore generated fake resources. We can even call the endpoints, as documented in the Swagger page:
$ curl --location --request GET 'http://hapi.fhir.org/baseR4/Patient/1963546'
This call returns the following response:
In addition to building APIs, FHIR® can be used to define messages to post in messaging systems such as RabbitMQ and Kafka. It can also be used to create documents that represent all records of a patient or an entire organization, with the intent of archiving or transferring large volumes of data. I can go on and on, listing everything that can be done with FHIR®, but this is a blog post, not a book. But, that’s probably a subject that I’m going to explore further, to help with its dissemination and perhaps to inspire other industries to take similar initiatives.
* HL7 and FHIR are the registered trademarks of Health Level Seven International and their use does not constitute endorsement by HL7.
]]>The City of Waterloo, located in Ontario - Canada, has an Open Data Portal that publishes raw data about infrastructure, services, environment, transportation, etc. Residents can use the data to oversee public investments and services, identify gaps, discover development opportunities, and even create new business. We figured out another use for the Portal: test CSVSource. It publishes a variety of CSV files. Among them, we found a an inventory of every single tree planted on the streets of Waterloo. Isn’t it cool?!
CSVSource is one of the repositories in my portfolio. I first introduced it a month ago. Its goal is to convert a CSV file to a SQL file with insert statements, simplifying the data ingestion in relational databases. To have fun building CSVSource, we looked for an interesting dataset in the Open Data Portal and put it in the folder /examples. We are glad to inform that we’ve got the minimal Rust code in place to convert those CSV files to SQL. CSVSource implements convention over configuration, with the following default behaviors:
the name of the CSV file is used as the name of the table in the insert statements.
the first line is skipped because it contains the headers that describe the columns.
the headers in the first line are used as columns of the table.
the column separator is comma.
each line in the CSV turns into an insert statement.
if the value contains at least one alphanumeric character then it is quoted, but if the value contains a valid number then it is not quoted.
If you have these basic requirements then CSVSource is ready for you. Otherwise, wait for the availability of arguments that will customize these conventions. For the moment, simply type:
$ csvsource --csv waterloo_tree_inventory.csv
It converts this CSV:
X,Y,OBJECTID,TREEID,CIVIC_NO,STREET,LOCATION,SPECIES_NAME,SPECIES_LATIN,SPECIES_CODE,LANDUSE,ROADSEGMENTID,PARK,WARD,PLANTED_BY,MONTH_PLANTED,YEAR_PLANTED,STOCK_TYPE,STOCK_SIZE,STATUS,STATUS_DATE,CREATE_BY,CREATE_DATE,CREATE_YEAR,CREATE_MONTH,UPDATE_BY,UPDATE_DATE,SOURCE,SOURCE_DATE,OWNERSHIP,CATEGORY,ROOT_PATHWAYS,SOIL_VOLUME_M3,INITIAL_ACCEPTANCE_DATE,FINAL_ACCEPTANCE_DATE,OVERHEAD_HYDRO,DEVELOPMENT_AGE,PLANNING_COMMUNITY,SUB_WATERSHED,MAP_DBH_CM,GIS_NOTES,HEIGHT_ESTM_LIDAR_2014_M,HEIGHT_ESTM_LIDAR_2019_M,TAG1,GLOBALID,INSPECTED_YEAR
-80.5095189090518,43.412175164406,1,10007057,27,ACTIVA AVE,BOULEVARD,Autumn Brilliance Serviceberry,Amelanchier x grandiflora 'Autumn Brilliance',AMGRAB,ROW,40056,,5,CITY CONTRACTOR,NOVEMBER,2014,BALL AND BURLAP,50 mm,REMOVED,2020/07/30 12:53:44+00,,2015/12/23 10:34:03+00,2015,December,GIS_DATA,2018/03/05 17:14:35+00,Tree Inventory,,CITY,Small Tree,N,,,,None,1997,LAURENTIAN WEST,BORDEN CREEK,7,ADAM BUITENDYK,0,10,,0a2719cc-94b2-42c7-80b7-5dc0c5fe0f24,
-80.4806046253164,43.4464481686235,2,153401,10,CAMERON ST N,LAWN,Norway Maple,Acer platanoides,ACPL,ROW,11547,,10,UNKNOWN,UNKNOWN,0,UNKNOWN,UNKNOWN,ACTIVE,2017/01/31 15:14:06+00,Mark Grondin,2009/10/17 00:00:00+00,2009,October,Esri_Anonymous,2017/01/31 20:14:06+00,Tree Inventory,2009/10/17 00:00:00+00,CITY,Maple_Norway,N,,,,Three phase,1908,KING EAST,UPPER SCHNEIDER CREEK,55,Field Inspection,13,9,,f9ccd885-1a91-497c-b1df-4818419373ac,
to these SQL insert statements:
insert into small_waterloo_tree_inventory
(X, Y, OBJECTID, TREEID, CIVIC_NO, STREET, LOCATION, SPECIES_NAME, SPECIES_LATIN, SPECIES_CODE, LANDUSE, ROADSEGMENTID, PARK, WARD, PLANTED_BY, MONTH_PLANTED, YEAR_PLANTED, STOCK_TYPE, STOCK_SIZE, STATUS, STATUS_DATE, CREATE_BY, CREATE_DATE, CREATE_YEAR, CREATE_MONTH, UPDATE_BY, UPDATE_DATE, SOURCE, SOURCE_DATE, OWNERSHIP, CATEGORY, ROOT_PATHWAYS, SOIL_VOLUME_M3, INITIAL_ACCEPTANCE_DATE, FINAL_ACCEPTANCE_DATE, OVERHEAD_HYDRO, DEVELOPMENT_AGE, PLANNING_COMMUNITY, SUB_WATERSHED, MAP_DBH_CM, GIS_NOTES, HEIGHT_ESTM_LIDAR_2014_M, HEIGHT_ESTM_LIDAR_2019_M, TAG1, GLOBALID, INSPECTED_YEAR)
values
(-80.5095189090518, 43.412175164406, 1, 10007057, 27, 'ACTIVA AVE', 'BOULEVARD', 'Autumn Brilliance Serviceberry', 'Amelanchier x grandiflora ''Autumn Brilliance''', 'AMGRAB', 'ROW', 40056, NULL, 5, 'CITY CONTRACTOR', 'NOVEMBER', 2014, 'BALL AND BURLAP', '50 mm', 'REMOVED', '2020/07/30 12:53:44+00', NULL, '2015/12/23 10:34:03+00', 2015, 'December', 'GIS_DATA', '2018/03/05 17:14:35+00', 'Tree Inventory', NULL, 'CITY', 'Small Tree', 'N', NULL, NULL, NULL, 'None', 1997, 'LAURENTIAN WEST', 'BORDEN CREEK', 7, 'ADAM BUITENDYK', 0, 10, NULL, '0a2719cc-94b2-42c7-80b7-5dc0c5fe0f24', NULL);
insert into small_waterloo_tree_inventory
(X, Y, OBJECTID, TREEID, CIVIC_NO, STREET, LOCATION, SPECIES_NAME, SPECIES_LATIN, SPECIES_CODE, LANDUSE, ROADSEGMENTID, PARK, WARD, PLANTED_BY, MONTH_PLANTED, YEAR_PLANTED, STOCK_TYPE, STOCK_SIZE, STATUS, STATUS_DATE, CREATE_BY, CREATE_DATE, CREATE_YEAR, CREATE_MONTH, UPDATE_BY, UPDATE_DATE, SOURCE, SOURCE_DATE, OWNERSHIP, CATEGORY, ROOT_PATHWAYS, SOIL_VOLUME_M3, INITIAL_ACCEPTANCE_DATE, FINAL_ACCEPTANCE_DATE, OVERHEAD_HYDRO, DEVELOPMENT_AGE, PLANNING_COMMUNITY, SUB_WATERSHED, MAP_DBH_CM, GIS_NOTES, HEIGHT_ESTM_LIDAR_2014_M, HEIGHT_ESTM_LIDAR_2019_M, TAG1, GLOBALID, INSPECTED_YEAR)
values
(-80.4806046253164, 43.4464481686235, 2, 153401, 10, 'CAMERON ST N', 'LAWN', 'Norway Maple', 'Acer platanoides', 'ACPL', 'ROW', 11547, NULL, 10, 'UNKNOWN', 'UNKNOWN', 0, 'UNKNOWN', 'UNKNOWN', 'ACTIVE', '2017/01/31 15:14:06+00', 'Mark Grondin', '2009/10/17 00:00:00+00', 2009, 'October', 'Esri_Anonymous', '2017/01/31 20:14:06+00', 'Tree Inventory', '2009/10/17 00:00:00+00', 'CITY', 'Maple_Norway', 'N', NULL, NULL, NULL, 'Three phase', 1908, 'KING EAST', 'UPPER SCHNEIDER CREEK', 55, 'Field Inspection', 13, 9, NULL, 'f9ccd885-1a91-497c-b1df-4818419373ac', NULL);
To use CSVSource, please clone the repository locally and compile it from source. You will need to install Rust first, and then run the following commands:
$ git clone https://github.com/htmfilho/csvsource.git
$ cd csvsource
$ git fetch origin 0.1.0
$ git checkout tags/0.1.0 -b 0.1.0
$ cargo build --release
$ cargo install --path .
I’m still learning how to cross-compile CSVSource to multiple operating systems. Until then, we need to compile from source, but the usage is the same:
$ csvsource --csv waterloo_tree_inventory.csv
The output is the file waterloo_tree_inventory.sql
in the same folder, with all insert statements.
In future releases, you will be able to:
change column separator to tab.
attach a prefix and a suffix content from other files.
set a table name different from the file name.
set the column names and their types different from the headers.
create insert statements that insert multiple records.
wrap multiple insert statements within a transaction scope.
But we are not limited to these. Let us know if you have any special needs by creating an issue in our repository.
Rust is definitely complicated. It took me a month to write the equivalent code that I wrote in 3 days in Go. The code might be memory safe, but I wouldn’t use the adjective “correct” like many Bloggers and Youtubers out there. A runtime panic exception is an evidence that correctness depends on the programmer, not the language. I still believe that Go is better than Rust, but it is delightful to see a Rust application running, using very minimal resources.
]]>I didn’t even go too far building my portfolio and I’m already rethinking my repositories. After some meditation, I didn’t feel compelled enough to pursue one of the domains and I noticed that two repositories were covering the same feature. Maybe it’s better to change my mind now, when things are still unclear than later, when I may end up abandoning repositories for lack of enthusiasm. So, what is about to change?
The first change is giving up the domain of sports. I remain excited about running and I’m booked to a marathon at the end of May, but I don’t think I need more than what Strava and Garmin Connect already offer me. They are integrated with easy-to-carry devices, collecting real-time data about my heart beats, location, pace, elevation, etc. These apps also support multiple sports and are relatively cheap. It would take a long time to reach the same level of maturity and by then, they would be far more advanced. In summary, an unfruitful effort.
The domain of sports was covered by Pycific, but this repository needs to be repurposed now. I’m going to explore something that is also part of my daily routine: reading. I currently use Goodreads, a social network of writers and readers. This app hasn’t evolved for a long time and has limited features. The only one I care about is keeping track of the books I read. And even that doesn’t fulfill my needs. So I decided to approach this problem with SampleReads: an app that keeps track of the book I read entirely or - and here is its differential - partially. I can register I read from page 47 to 81, make notes, write reviews, all counting to my reading stats. I also want to add books that are not in public databases, like my car’s driver manual, or a magazine or an obscure e-book.
SampleReads is going to be simple. So, I would rather approach a simple problem with a technical stack that I’m not familiar with. Since I don’t know enough of Elixir/Phoenix, I’m going to use it instead of Python/Django, the stack previously used in Pycific. However, I already use Elixir/Phoenix in Controlato. Well, I guess I have to migrate Controlato to Python/Django then. It actually makes sense because Controlato is a complex application and it would be harder to complete it in Elixir/Phoenix. Also, Controlato deals with data analysis, which is better supported by Python.
The third and last change, for the time being, is the discontinuation of Liftbox, a simple tool originally planned to replace my Dropbox subscription. I realized, when I was writing my previous post, that Minimily already implements this feature with AWS. All I have to do is to make it cloud agnostic. After discontinuing Liftbox, CSVSource is now the only repository in Rust.
By this time, all changes were implemented in the repositories and all broken links in previous posts were changed to point to this post as a form of clarification.
]]>RMM Level 2 starts today with the bootstrap of the repositories. Creating the minimal code necessary to run the apps. We’re not fulfilling any requirements yet or discussing design decisions, just initializing the projects using the tools in the ecosystem of each programming language, getting pure and simple skeletons. I’m curious to know which technology offers the best experience!
Programming languages suffer strong influence from their creators. Some creators are open to other people’s ideas - as we can see in Python, Java and Rust - incorporating them on demand. Other creators are more conservative, evolving their languages but limiting other people’s influence. These creators are known as opinionated. Users must agree with their opinions to feel comfortable with their programming experience. This is the case of Go, Clojure, and Elixir. The same phenomenon happens with frameworks as well. Developers are pushed in a lower or higher level of compliance to a pre-defined design.
Actually, the more opinionated a language is the easier it is to learn. It also has less issues with backwards compatibility, since it tends to have just enough features. Less opinionated languages like Python, Java, and Rust are popular but they are pretty hard to master due to their abundance of features. Too many features lead to breaking changes, which happened from Python 2 to Python 3 and from Java 8 to Java 11. Rust is still new but some evidences show that they make syntax changes without breaking compatibility, which means they are increasing the compiler complexity to keep this guarantee.
In the framework space, the opinionated ones are easier to learn but harder to maintain. Entire systems were ruined by frameworks that were discontinued - like JBoss Seam - or redesigned - like Apache Struts 1 to Struts 2 - or had backwards incompatibilities - like the struggle of Github to migrate from Rails 3.2 to 5.2. Instead of opting for a framework, one could opt to compose libraries. This approach takes longer, is more complex, but the issues are limited to individual libraries, which can be upgraded or even replaced without major issues. A large scale library issue recently happened with Log4j security vulnerabilities. Some people opted to upgrading it to newer versions. Others decided to replace it with Logback. No changes are required in the code if the library is properly integrated, which is the case when the project uses SLF4j, an interface for log libraries. In summary, opinionated languages are easier to master, but opinionated frameworks may be challenging.
We are going to navigate through tools and frameworks in the context of the repositories. We start with Digger and Minimily, since they were already bootstrapped, but we’re revisiting them without pushing any code. Then we bootstrap Controlato, Pycific, SpitFHIR, CSVSource, and Liftbox, pushing the resulting code.
Digger is written in Java, with the help of frameworks like Spring Boot and Hibernate, and connectivity with H2, PostgreSQL, and Microsoft SQL Server databases. We bootstrapped it in August 2019, using Spring Initializr, which is still in use nowadays. It consists on visiting that website, fill in a form with information about the project, list all dependencies, and generate a zip file with a Spring Boot application ready to run.
After unzipping the file locally, we can run mvn spring-boot:run
and visit http://localhost:8080
to see it serving. Java is not opinionated at all, but Spring is just a little bit. The zip contains all it takes to have dependency injection, convention over configuration, dependencies, but developers have autonomy to make all other design decisions.
Minimily is written in Clojure with the help of libraries, not frameworks. In this case, the language is opinionated but the choices and combination of libraries is totally up to the developer, who is the opinionated one. We bootstrapped Minimily in November 2016, using Leningen. There are more options out there but Leiningen remains the most popular, just like its old friend Maven. Here is a basic example:
$ lein new minimily
It generates just the minimal necessary to write a library or a little tool. We can follow templates to do more elaborated apps using Pedestal:
$ lein new pedestal-service minimily
Some templates support wiring dependencies at the command line, like Luminus:
$ lein new luminus minimily +h2 +immutant
Looks like the more options we add to lein new
the more opinionated the project becomes. But at least we come to decide which opinion we agree with. If we run the latest command and then:
$ cd minimily
$ lein run
we get the following page at http://localhost:3000:
Luminus generates just enough content to start with, competing here with Phoenix and Django.
Controlato is going to be developed in Elixir, using the Phoenix Framework. Both are heavily opinionated, which means they believe they are offering the state of the art in terms of design. Phoenix generates the directory structure and all the files we need for our application. To install it, run the following commands:
$ mix local.hex
$ mix archive.install hex phx_new
Then, generate the project:
$ mix phx.new controlato
* creating ...
Fetch and install dependencies [Yn] Y
The output of this command also explains the next steps:
We are almost there! The following steps are missing:
$ cd controlato
Then configure your database in config/dev.exs and run:
$ mix ecto.create
Start your Phoenix app with:
$ mix phx.server
There is a caveat here. The command mix ecto-create
only works if the password of the user postgres
is postgres
. We’re simply changing the password to postgres
to move forward. Here is what we got in the browser after completing the steps:
The Controlato repository now contains hundreds of files in different formats and languages, doing different things. If we agree with them or not, it doesn’t matter. We just have a lot of stuff to digest.
SpitFHIR is going to be written in Go, which is the most opinionated language in my portfolio. They hardly accept new language features from the community with the intent to keep is as simple as possible. On the other hand, to do what we want to do, we will need all design knowledge we have accumulated over the years because there is no opinionated framework to generate tons of files for us. Here, we have to carefully pick the libraries and make them work together. All we can do for now is to clone the repository, initialize it as a Go module, and create a main.go file, which will be the entry point of the app.
$ git clone git@github.com:htmfilho/spitfhir.git
$ cd spitfhir
$ go mod init
$ mkdir -p cmd/spitfhir
$ touch cmd/spitfhir/main.go
We opened the main.go
file and added the following Go code:
package main
import "fmt"
func main() {
fmt.Println("SpitFHIR")
}
Then we built it and ran:
$ go build cmd/spitfhir/main.go
$ ./main
SpitFHIR
It will take time until we reach the equivalent of Phoenix, but at least it will contain only what is needed and we will fully understand it.
We will write Pycific in Python and Django. Python is not opinionated, but Django is as much as Phoenix. It makes a lot of decisions for us. The advantage of Django over the competition is a full featured admin application that gives full control over the database to the administrator. It definitely saves a lot of time. Here are the commands we used to bootstrap the repository:
$ git clone git@github.com:htmfilho/pycific.git
$ cd pycific
$ python3 -m venv venv
$ source venv/bin/activate
$ python -m pip install Django
$ pip freeze > requirements.txt
$ django-admin startproject pycific .
$ python manage.py migrate
$ python manage.py runserver
The process was straightforward and more refined than Phoenix. Once completed, we can see the result at http://127.0.0.1:8000:
CSVSource and Liftbox are bootstraped the same way because they are both Rust standalone applications. They are simple projects, so, no big bootstrap needs. Cargo do the entire job for us:
$ git clone git@github.com:htmfilho/csvsource.git
$ cd csvsource
$ cargo init
$ cargo run
Hello, world!
At least, cargo init
generates the first hello world code, something that go mod init
doesn’t.
On the bright side, Clojure and Leiningen offered the best bootstrap experience. We can range from a very basic library to a complex web app, using the same tooling and just changing the arguments. That’s very elegant. Python and Django offered the best framework experience. Rust and Cargo were just fine. On the dark side, Java and Spring depend on a website to do a decent job. Go required some background to write the first code, since it didn’t generate any. Elixir and Phoenix were expecting the database to be configured according to their own rules.
I don’t like when I don’t understand what is behind the scene. At this point, I don’t know what Phoenix and Django are doing to serve those pages. It doesn’t matter how nice they were trying to save my time, but I still have to understand the whole mechanism, otherwise I won’t be able to solve problems, do upgrades, and investigate root causes. I like the fact that Spring doesn’t generate too much stuff. I still need to write the controller, the components, the services, the repository, etc, but I feel some level of discomfort about what all those annotations are hiding from me in the code. What I really like - and this is personal - is how Clojure, Go, and Rust let me think about the design without telling me what to do. I understand that I will need a lot of training before I come even close to what Django and Phoenix already put on my plate. But at least I’m forced to do it. What is my motivation to do the same for Django and Phoenix? None. I feel like jumping into the requirements already.
NOTE: I recently changed the repositories to refine my portfolio and documented the changes in another blog post.
]]>After proposing the Repository Maturity Model (RMM), I wanted to put it in practice in my portfolio. The first step is to implement Level 1, which consists on writing the content to inform visitors about the repository and guide contributors through the process of fixing bugs and enhancing features. It doesn’t necessarily need to have working code, but anyone should be able to understand what the repository is about and how to contribute to it. I’m reporting here what we have done to accomplish that in all repositories.
Documentation is often neglected at the initial stages of a project. It might be due to the fact that “documenting” is less fun than “coding”. I agree with that, but in my experience, leaving the whole documentation to be written at the end is more painful than gradually writing it as the project evolves. Writing also leads to thinking and thinking is what we need to create great products. Amazon, Google, Twitter, Spotify and others have built a culture of documentation, and those who follow lean practices like to test ideas before implementing them.
Documenting since the beginning makes the content lighter, cleaner, better. It passes through several revisions each time someone contributes with more content. So a repository reaches Level 1 when it contains enough content not only for users and contributors, but also to give them a chance to document as they use and build the product. After getting some inspiration from opensource.guide, we came up with the following artifacts for Level 1:
Github somehow helps us to implement Level 1. When creating a repository, it suggests creating a readme file. It is then generated in the root of the repository. Github also has a checklist to make the repository welcoming to contributors, as we can see in the figure below.
You can find this checklist at “Insights > Community”. It automatically checks for the presence of files in predefined locations, completing the checklist as the following files are created:
.github/ISSUE_TEMPLATE/bug_report.md
(issue templates).gitbub/ISSUE_TEMPLATE/feature_request.md
(issue templates).github/PULL_REQUEST_TEMPLATE/pull_request_template.md
(pull request template)CODE_OF_CONDUCT.md
(code of conduct)CONTRIBUTING.md
(contribution guidelines)LICENSE
(license)README.md
(readme file)We are left with short description, user guide website, and tags. The short description is the product summarized in a sentence, in the About section of the repository. It also includes a paragraph at the beginning of the README file, as highlighted in read in the figure below.
The figure also highlights the tags, which are commonly used keywords across Github. They help to make the project discoverable by Github’s search engine. We can also see the link to the user guide website above the tags. For the website, Github helps with Github Pages, whose configuration you can find at “Settings > Pages”. Any HTML content in the folder /docs
is served as a static website. All we have to do is to generate that HTML content and publish there. We’re using Asciidoctor for that, which processes the Asciidoc format and generates a good looking HTML page, like the one below.
In the last 5 days, we went through all repositories, produced all the documentation required by RMM Level 1 and now we have all of them ready to receive code. The following table compiles everything that was produced.
There is a reasoning behind the open source licensing of these repositories. I’m using GPL-3.0 when the product targets end-users. We hope people concentrate their efforts around these GPL repositories, but if they decide to go in another direction, at least they have to keep the source open elsewhere. I’m using Apache 2.0 for libraries, giving the necessary freedom developers need to fulfill the requirements of their solutions. I’m using MIT for tools that target developers and non-technical people. Here is the distribution:
We reached a point where all the repositories are minimally organized to welcome contributions. This is the result of implementing the RMM Level 1. At this point, we’re ready to pursuit Level 2, which is delivering a minimal viable product. It means a lot of code, in different languages, solving different problems. You’re welcome to participate by writing code, documentation, tests, etc.
NOTE: I recently changed the repositories to refine my portfolio and documented the changes in another blog post.
]]>