The second version of the Scrollable Tables project cleans up deficiencies from the first version, adds accessibility support, and is generally all-round awesome.
A project I’m working on involves running Oracle queries on a PeopleSoft system and getting the results as CSV files. I use those query results, plus information from other places (the New York State Department of Education’s website and Degreeworks information for academic programs) to derive a Postgres database that is the backend for a “Transfer App” (https://transfer-app.qc.cuny.edu) for the City University of New York.
In working with the PeopleSoft query results, I needed to explore the results before deciding how to combine them in the derived database, and Excel, Google Sheets, and Numbers all are too limited for working with some of the results, which are millions of lines long. So I wrote a Python script to turn CSV files into database tables.
It’s a convenient utility, so I put it up on GitHub for sharing: https://github.com/cvickery/csv2db
Code and documentation are publicly available at: https://github.com/cvickery/scrollable_tables
Mark Guzdial is a provocateur. In today’s blog post, the title says it all: “teachers as provocateurs … all learning starts from a problem.” An example he gives is teaching students about loops because it will save them from having to cut and paste (or re-type!) code in the future. Instead, he suggests, let them do it the hard way until it becomes a problem for them, at which point the natural process of disturbing one’s equilibrium because it doesn’t work anymore will lead to learning about loops in a natural way.
I like this perspective a lot, but have found it difficult to use in practice. Part of the issue is my own unwillingness to give students the time and leeway to develop their own perception of the problem with writing repetitive code. To me, once a student types dozens of lines where a two-line loop would suffice should be enough to motivate them to learn a better way. But, in fact, not everyone has the same low tolerance for repetitive labor as I do. And I don’t have the patience to let nature take its course. There is only so much time available in the semester!
The other issue, of course, is that “beginning” coders come to the course with different backgrounds. Some already know about looping from outside experience, while others have never thought of the computer’s ability to automate mundane tasks, and accept typing nearly the same thing over and over as simply consistent with their world view of how things get done in the world. How to reconcile these different backgrounds?
Linking, loading, header files, and libraries—what are they all about? This post is a description of how these concepts interrelate for C/C++ code. Since the Arduino language is really C++, this material applies in that context as well. In fact, this post aims to explain these concepts specifically as they relate to Arduino code development.
Wait, how did function get in here? Functions are blocks of code that solve one of the earliest problems of software development: how to re-use code without having to copy and paste it into each project that could use it. There are three interrelated things to do with a function: declare it, define it, and reference it.
- The declaration gives the “signature“ of the function: its name, return type, and the number of parameters to pass to it, along with the types of those parameters. A function definition may serve double-duty as the function’s declaration, but the situation that is important for linking and loading is when the declaration is put in a separate file, which is added to the compilation code by including it at the top of the code file (at the “head of the code file”) that uses it. These are called “header files” and by convention their names end in
- The definition gives the signature of the function but also includes, inside curly braces following the signature, the
statements that make up the body of the function—the statements that are executed when the function is referenced.
- References are the pieces of code that “invoke” or “call” a function—that cause the statements in the function body to be executed. When the compiler encounters a function reference, the reference must agree with a known function signature in order to be compiled.
The above description of things related to functions deals with source code: what the programmer has to deal with when preparing a project to be compiled. Now we jump to what has to happen to load compiled code into memory for execution.
The machine language code generated by the compiler has to be loaded into the computer’s memory from one or more disk files. All the compiled code for the program has to be loaded into memory: the code for the functions you defined (setup(), loop(), and possibly others); the code for the functions your code references; some special code that provides the “runtime environment” for your code. (For Arduino, this is a function named main() that calls your setup() function once, and then endlessly calls your loop() function over and over again.) Each function definition gets assigned its own location in memory, and each function call has to be filled in with the address that identifies the corresponding definition’s location in memory.
For this discussion, a library is a file that contains a set of function definitions that are somehow related, such as to control NeoPixels on an Adafruit board. In Arduino, these libraries are compiled as part of the same process that compiles your
.ino file(s) because the machine language to be generated depends on what Arduino board you are using. For C/C++ applications running on a regular computer, these library files are typically precompiled (to save time building an application) because the processor is known ahead of time.
Linking is the process of deciding what code files to pass on to the loading process. The machine code for your “user defined” definitions of setup() and loop() will come from a file produced by the compiler when it translated your
.ino file(s) into machine language. The Arduino runtime code will come from a file that is available inside the package of files that come with the Arduino application. In addition there may be library files that contain definitions for functions that a project uses beyond those that are part of the runtime or user-defined functions.
When the compiler processes your code (and header files), it leaves information in the machine language file that tells where the function references are located within the file and what the functions are that they refer to. It’s the linker’s job to look at this information and then find the other files that contains the needed function definitions, and to link the files together so that the machine language function references connect to the corresponding machine language function definitions.
The linker leaves to the loader the task of filling in the actual memory locations used by functions.
We end with a slightly arcane and somewhat pedantic statement of the role of header files in an application. People often say that header files tell what libraries to use. That’s incomplete, but not incorrect. People also say that header files are the libraries that a project uses, and that is (pedantically) incorrect.
A more accurate statement would be that a header file provides the code for function signatures that allow a compiler to generate code for function references in your code. The machine instructions for those function references then have to be “fixed up” (a technical term) by the linker and loader for the program to be able to run.
Header files contain other information besides function signatures. They may also define named constants and class information that are used for working with a library.
Whatever the current sales promotion is that Tesla is having (free supercharging; vehicle upgrade, whatever) my referral code will get it for you.
I use lftp in a launchctl script to transfer and delete files from an Axway/Tumbleweed secure server, but after upgrading to macos Sierra, the connection failed (and looped forever retrying) with a brief flash of a “DH GEX group out of range” message.
Searching for the message reveals that it’s an ssh issue (DH is diffie-hellman): Apple apparently changed ssh to use a shorter keys by default in order to encourage use of TLS. At least I think that’s what happened. Could be that the secure server changed its key negotiation requirements the same day I upgraded to Sierra. The explanation of the error message (and solution to the problem) showed up in a Linux server forum.
First I mistakenly tried to rebuild lftp, which I could not do because ./configure died saying it couldn’t find the readline headers even though they were there (brew link –force readline). The second dead end was to try to change the fish:connect-program setting for lftp, but that had no effect. Finally, I scrolled through the lftp man page far enough to realize that the proper configuration setting is sftp:connect-program.
So I created ~/.lftprc and put this line in it:
set sftp:connect-program "ssh -a -x -o KexAlgorithms=diffie-hellman-group14-sha1"
I upgraded my office workstation from El Capitan to Sierra last week after waiting a while to make sure there were no problems with the new version of the operating system: my office workstation is used for some campus-wide services, and I go into panic mode when upgrades break things. So after a decent interval it seemed safe to let the upgrade go forward.
It’s a little embarrassing in hindsight how long it took me to recover from the upgrade, but there were no indications whatsoever what had gone wrong: just a cryptic HTTP 500 error code for part of one of the two virtual hosts on the system. Lots of time looking at mail configuration; PHP include_path, etc. Finally: “pg_connect() not found.” I wasn’t getting my “unable to connect to database” error because … postgres wasn’t available at all. The connection couldn’t fail because the code to make the connection couldn’t even be executed.
I’m not the only one to have the problem, of course, and I finally was able to google the correct problem. The solution was simple, just add these two lines to php.ini:
Since I don’t use the PDO interface to Postgres, I probably need only the second line, but I didn’t do the experiment to make sure.
So now I’m back to the old problem: when Apple updates the OS, the path to pgsql.so will undoubtedly change without documentation (that I know of) to alert me to the new path, just as the demotion of Postgres being included in the PHP installation by default was discontinued without documentation (that I know of).
Anyone buying a new Tesla can get free supercharging and a $1000 credit towards a new Model S or Model X by using this referral code. Tesla updates the terms of the referral offers quite regularly, but the referral code itself is the same for all changes in the terms. As of this update (April 2018), the first person to use my code gets a $500 credit towards service or accessories, and the first five people get unlimited free supercharging.
And, yes, I think my Model S is terrific!
Google Forms is terrific for gathering information, but it sucks as a workflow tool. Today I received a Word document that instructed people to fill it out, print it, collect three signatures on the paper copy, scan that, and return it by email. No way to track progress; no record of what’s happening, but partial and unsynchronized records in both paper and email trails.
With Google Forms, we can be sure the entire chain from initiator to final approval is handled by authenticated individuals who are signed into the organization’s Google account. And we can collect form responses in a spreadsheet.
But we can’t populate the form with information from a database that provides, for example, the identities of the people who have to approve or review the user’s request. No drop-down lists generated from a table or no-sql document list somewhere.
And there will have to be code to trigger notifications to alert reviewers that they need to look at / approve a request.
For now it looks like we have to use Google Sheets for our “CMS” and HtmlService to generate the form(s) and GoogleScript code to interact with the users.
I’ve done most of this before for our “Assessment Document Repository.” It’s a bit slow, but at least the capability is there.
My first TODO is to update a Google Sheet when someone clicks a link that says “approve” in an email message.