fouronnes3a day ago
Very cool article!
I also implemented a spreadsheet last year [0] in pure TypeScript, with the fun twist that formulas also update backwards. While the backwards root finding algorithm was challenging, I also found it incredibly humbling to discover how much complexity there is in the UX of the simple spreadsheet interface. Handling selection states, reactive updates, detecting cycles of dependency and gracefully recovering from them is a massive state machine programming challenge! Very fun project with a lot of depth!
I myself didn't hand roll my own parser but used Ohm-js [1] which I highly recommend if you want to parse a custom language in Javascript or TypeScript.
> One way of doing this is to keep track of all dependencies between the cells and trigger updates when necessary. Maintaining a dependency graph would give us the most efficient updates, but it’s often an overkill for a spreadsheet.
On that subject, figuring out the efficient way to do it is also a large engineering challenge, and is definitely not overkill but absolutely required for a modern spreadsheet implementation. There is a good description of how Excel does it in this famous paper "Build systems a la carte" paper, which interestingly takes on a spreadsheet as a build system [2].
[0] https://victorpoughon.github.io/bidicalc/
[2] https://www.microsoft.com/en-us/research/wp-content/uploads/...
wslha day ago
The idea of backward updating is fascinating but is not generally feasible or computable. What kind of problems can you solve backwardly?
fouronnes3a day ago
> not generally feasible or computable
You'd be surprised. It really depends on how you define the problem and what your goal is. My goal with bidicalc what to find ONE solution. This makes the problem somewhat possible since when there are an infinity of solution, the goal is just to converge to one. For example solving 100 = X + Y with both X and Y unknown sounds impossible in general, but finding one solution is not so difficult. The idea is that any further constraint that would help choose between the many solutions should be expressed by the user in the spreadsheet itself, rather than hardcoded in the backwards solver.
> What kind of problems can you solve backwardly?
This is the weakness of the project honestly! I made it because I was obsessed with the idea and wanted it to exist, not because I was driven by any use case. You can load some premade examples in the app, but I haven't found any killer use case for it yet. I'm just glad it exists now. You can enter any arbitrary DAG of formulas, update any value, input or output, and everything will update upstream and downstream from your edit and remain valid. That's just extremely satisfying to me.
AlotOfReadinga day ago
Have you looked into prolog/datalog? You're dancing around many of the same ideas, including backwards execution, constraint programming, stratification, and finding possible values. Here's a relevant example of someone solving a problem like this in prolog:
https://mike.zwobble.org/2013/11/fun-with-prolog-write-an-al...
oritrona day ago
> I haven't found any killer use case for it yet
You might dig into an operations research textbook, there are a number of problems solved with linear programming techniques which might make sense for your interface... In fact might be more intuitive for people that way and with commercial potential.
somata day ago
I am not sure if I know what I am talking about or if it counts in this scenario but constraint solvers come to mind. I am mainly familiar with them in a CAD context so I am struggling to think of a use for them in a spreadsheet context. But I think being able to say given these endpoints find me some values that fit could be a very valuable tool.
But like I said I am not sure that I know what I am talking about and I may be confusing backwards calculation with algebraic engines. I would love for algebra solvers to be a first class object in more languages.
WillAdamsa day ago
I implemented bi-directional solving in a very simple "Proportion Bar" app --- sort of --- one side would calculate at the specified scaling factor (so 100% could do unit conversions), the other would calculate the scaling factor necessary to make the two sides agree.
btillya day ago
While the general problem is not always tractable, some of the special cases are pretty important.
Take, for example, backprop in machine learning. The model operates forwards. Then you solve backwards to figure out how to update the terms.
skaushik92a day ago
Speaking from experience, I find budgeting spreadsheets to be a great usecase for this.
devnotes77a day ago
[dead]
ivanpribeca day ago
Reminds me of spreadsheet-fortran (https://github.com/lwsinclair/spreadsheet-fortran), a project creating a VisiCalc lookalike in FORTRAN 66, which even runs on a PDP-11.
afandiana day ago
Quote:
#define MAXIN 128 // max cell input length
enum { EMPTY, NUM, LABEL, FORMULA }; // cell types
struct cell {
int type;
float val;
char text[MAXIN]; // raw user input
};
#define NCOL 26 // max number of columns (A..Z)
#define NROW 50 // max number of rows
struct grid {
struct cell cells[NCOL][NROW];
};
vidarha day ago
According to Bob Frankston, Bricklin's co-founder[1]:
> The basic approach was to allocate memory into fixed chunks so that we wouldn't have a problem with the kind of breakage that occurs with irregular allocation. Deallocating a cell freed up 100% of its storage. Thus a given spreadsheet would take up the same amount of space no matter how it was created. I presumed that the spreadsheet would normally be compact and in the upper left (low number rows and cells) so used a vector of rows vectors. The chunks were also called cells so I had to be careful about terminology to avoid confusion. Internally the term "cell" always meant storage cell. These cells were allocated from one direction and the vectors from the other. When they collided the program reorganized the storage. It had to do this in place since there was no room left at that point -- after all that's why we had to do the reorganization.
> The actual representation was variable length with each element prefixed by a varying length type indicator. In order to avoid having most code parse the formula the last by was marked $ff (or 0xff in today's representation). It turned out that valid cell references at the edges of the sheet looked like this and created some interesting bugs.
It leaves out a lot of details - if you're skimping enough you could allocate variable length row vectors, but it seems they wanted to avoid variable length allocations, in which case you could start with a 255 byte array pointing to which subsequent equal-sized chunk represents each in-use row. You'd need at most 126 bytes per row in actual use to point into the chunks representing the cell contents. But this is just guesses.
[1] https://www.landley.net/history/mirror/apple2/implementingvi... and https://news.ycombinator.com/item?id=34303825
ksra day ago
Dan Bricklin wrote a JavaScript spreadsheet engine around 2008 https://github.com/DanBricklin/socialcalc. I extended it and adapted it to Drupal CMS around that time https://www.drupal.org/project/sheetnode.
staplung19 hours ago
Cool article but I think the write-up no longer matches the actual code. Snippets in the article use `*p->p` a lot. The *p is a parser struct defined above as
struct parser {
const char* s;
int pos;
struct grid* g;
};
struct parser {
const char *s, *p;
struct grid* g;
};
airstrikea day ago
> Maintaining a dependency graph would give us the most efficient updates, but it’s often an overkill for a spreadsheet.
It's not overkill at all. In fact, it's absolutely necessary for all but the simplest toy examples.
OliverMa day ago
Isn’t the existence & success of visicalc a direct counter to this?
gregatesa day ago
> Since the formulas did depend on each other the order of (re)calculation made a difference. The first idea was to follow the dependency chains but this would have involved keeping pointers and that would take up memory. We realized that normal spreadsheets were simple and could be calculated in either row or column order and errors would usually become obvious right away. Later spreadsheets touted "natural order" as a major feature but for the Apple ][ I think we made the right tradeoff.
It would seem that the creators of VisiCalc regarded this is a choice that made sense in the context of the limitations of the Apple ][, but agree that a dependency graph would have been better.
https://www.landley.net/history/mirror/apple2/implementingvi...
Edit: It's also interesting that the tradeoff here is put in terms of correctness, not performance as in the posted article. And that makes sense: Consider a spreadsheet with =B2 in A1 and =B1 in B2. Now change the value of B1. If you recalc the sheet in row-column OR column-row order, B2 will update to match B1, but A1 will now be incorrect! You need to evaluate twice to fully resolve the dependency graph.
SoftTalkera day ago
Even LaTeX just brute-forces dependencies such as building a table of contents, index, and footnote references by running it a few times until everything stabilizes.
bombcara day ago
It is possible (though very rare) to get a situation in LaTeX where it keeps oscillating between two possible “solutions” - usually forcing a hbox width will stabilize it.
Or do what everyone does and reword something ;)
gregatesa day ago
VisiCalc didn't do this, though. It just recalculated once, and if there were errors you had to notice them and manually trigger another recalc.
fsckboy20 hours ago
but wasn't it documented to do it in some sort of "down and to the right" order, and if you wrote your formulas "up and to the left" everything would be hunky dory?
tables generally have row and column sums, subtotals, and averages down and to the right.
airstrikea day ago
Is anyone using visicalc today? I'm not sure how its past success, however fantastic, can be translated into "a dependency graph is often an overkill for a spreadsheet"
OliverMa day ago
The clause "it's absolutely necessary for all but the simplest toy examples" is what I was disagreeing with. But I wouldn't be surprised to hear that visicalc adopted one as soon as it was technically feasible in later versions.
airstrikea day ago
visicalc is not the benchmark you think it is. it's decades old. this day and age, dependency graphs for any real world use case will definitely need a dependency graph. it helps no one to suggest otherwise, and actually makes light of a specific engineering task that will for a fact be required of anyone looking to build a spreadsheet engine into a product
OliverMa day ago
I'm not suggesting otherwise. I'm saying that your "toy example" comment is very dismissive of something that was an extraordinary accomplishment of its day. They invented spreadsheets without it. Dependency graphs are excellent and widely useful things we should all be happy to adapt and reach for, far beyond spreadsheets. We should be grateful that they're available to all of us to build into software products so readily. I've used them repeatedly and I'm sure I will many times in the future.
What I'm trying to communicate is this: this product _invented_ spreadsheets, but you dismiss the implementation with a sneer.
airstrikea day ago
I didn't dismiss Visicalc at all. In fact, I even said it had fantastic success.
I dismissed the article's claim that maintaining a dependency graph is overkill for a spreadsheet. That's a false statement. It might have been true at the time, but it's not true today. The phrase as written in TFA is poor form and misleading to beginners.
izacusa day ago
You were arrogantly yammering about software design in an article comments for a product that literally proves you wrong.
Take that as a lesson in humility for the future.
airstrike21 hours ago
Few things are more arrogant than telling someone you're teaching them a lesson...
This is a product from 1979! It does not prove me wrong at all. Trade offs are different today. Decisions that applied then don't apply now. The quoted text from TFA is wrong and simplistic.
izacus14 hours ago
Give it up man.
airstrike11 hours ago
No, thanks.
vor_a day ago
That's cringy.
[deleted]a day agocollapsed
SoftTalkera day ago
A still-very-common use case for spreadsheets is just to manage lists of things. For these, there are no formulas or dependencies at all. Another is simple totals of columns of numbers.
There are many common spreadsheet use cases that don't involve complicated dependency trees.
zsergea day ago
It's a common CPU vs RAM decision to make. Dependency graph consumes memory, while recalculating everything for a number of iterations could happen on stack one formula at a time in a loop. On 6502 it mattered. On modern CPUs, even with RAM crisis I'm sure for 99.9% of spreadsheets any options is good enough. Say, you have 10K rows and 100 columns - it's 1M calculations to make.
airstrikea day ago
Keeping a dependency tree is not complicated
SoftTalkera day ago
It's more complicated than not keeping one, at least.
airstrikea day ago
But not keeping one has a cost too. Which cost is higher? Generally, I argue, not tracking dependencies is the higher cost for any real spreadsheet in production use cases.
fsckboy20 hours ago
visicalc was rewriten in the 80's for computers newer than the apple ][, so are you asking specifically about the original implementation or later implementations? it makes a difference and mostly disobviates the question.
izacus14 hours ago
It absolutely is, it's bizarre seeing HNers crapping over approaches used by one of the most successful products in history (Visicalc was literally a system seller) as a "toy" approach.
airstrike11 hours ago
In 1979!
bonsai_spoola day ago
Are there good command-line interfaces for spreadsheets? I don't do anything super financially-important and I'd prefer to stay in the terminal for quick editing of things, especially if I can have Vi keybindings.
ziea day ago
There is SC and now sc-im: https://github.com/andmarti1424/sc-im
You can also literally run Lotus 123 if you want. Someone has binaries to make it work on linux. or under dosemu
freedombena day ago
Neat, thank you! sc-im looks amazing, and it's even in the Fedora repos (though the repo version doesn't support xlsx, so I'll compile myself and try it out)
Edit: Quite painless! Opened some test xlsx files without issue. Did get a stack trace on a very complicated one, so when I have time I'll try and dig in deeper. Added a doc to the wiki in case it's helpful to other: https://github.com/andmarti1424/sc-im/wiki/Building-sc%E2%80...
bombcara day ago
zie14 hours ago
Awesome sauce! I was too lazy to track this down, thanks for sharing! (for readers not following, this is Lotus 123 on Linux)
hunter4309a day ago
Visidata[0] is a killer data swiss army knife. It's even inspired off Visicalc
somata day ago
It's weird but visidata is my favorite spreadsheet.
"But... visidata is not a spreadsheet"
I know, that's what makes it so weird.
On contemplation, I think I grew dissatisfied with the normal spreadsheet data model, I wanted something bettered structured than the "it's a big bag of cells" that spreadsheets present, I wanted row security. The best I found was the relational database. I currently use a local postgres db for most things I would have used a spreadsheet for. The interfaces sort of suck in comparison but at least I have sane data structures.
bombcara day ago
Microsoft access was always a moderately mediocre database but it was also always an amazing spreadsheet for so many use cases.
rauli_a day ago
I actually created one for some time ago. It's nothing special but it has Vi keybindings.
vsliraa day ago
This is brilliant! Thank you for creating it
somata day ago
I want to mention teapot. First an apology, it's not actually a good match for for question, sure, it's a curses spreadsheet, but it was made by someone who thought about the fundamentals of the problem a little too much. So it is probably a little too weird for someone who just wants to spreadsheet as Dan Bricklin intended.
https://www.syntax-k.de/projekte/teapot/
In short cell address are normalized @(1,2,3) instead of A1 or r1c1. real references so address rewriting hacks($A$1) are not needed. formula references so you can use a single master formula, and clocked expressions which allow circular dependencies/simulation.
Probably a little too different for casual use but worth taking a look at, if nothing else to challenge your ideas of what a spreadsheet has to be.
While looking up the website I found a rewrite in rust, which is cool I guess, someone is keeping the dream alive, I will leave a link to that as well.
chungya day ago
Emacs with org-mode and evil-mode seems to be up your alley.
ziea day ago
and Emacs SES, built-in spreadsheets outside of Org-mode: https://www.gnu.org/software/emacs/manual/html_mono/ses.html...
fiddlerwoaroofa day ago
This might be programmer-brain, but I find sqlite is pretty nice for things people would use a spreadsheet for. It’s a little bit higher friction, but when I started designing a Improv-like terminal spreadsheet a while ago, I eventually realized I was just reinventing databases.
freedombena day ago
Oh man, a TUI spreadsheet application that can edit ODF or XLSX format would be absolutely killer. Would love to hear if anyone knows of such a tool
paddy_ma day ago
I would think visidata could.
zsergea day ago
A slightly larger implementation at the end of the post does that to some extent - https://github.com/zserge/kalk (CSV import export, Excel-like "locking" of rows/columns like $A$1). If there's a need for such a project - I'm happy to add ODF or XLSX, more compatibility with Excel formulas etc. I'm not sure about Vi keybindings, I personally find spreadsheets easier to use in a non-modal manner.
nhatchera day ago
It's a bit low on my priority list, but I'm working on that!
airstrikea day ago
Pretty sure I can build one based on code I already have. If others are interested in this, please let me know and I'll bang it out in the next couple of weeks.
f1shya day ago
I would be interested! Clixel
phonona day ago
segmondya day ago
vibe one. ;-)
0x20cowboya day ago
sc has been around for quite a while: https://github.com/robrohan/sc there are several versions floating around.
breadsniffera day ago
Anyone know what kind of departments/parts of business were the first adopters of visicalc?
SoftTalkera day ago
All kinds of operational departments. I'm sure it was used for accounting, payroll and commissions, inventory tracking, I know that teachers used it for gradebooks as I helped set them up when I was in high school (early 1980s).
Pretty much anything that you used to do on paper with a columnar notebook or worksheet and a calculator, or anything that could be represented in tabular form could probably be implemented in VisiCalc, Lotus 123, and others. Spreadsheets are probably the most successful software application that was ever invented. Certainly one of the most.
WillAdamsa day ago
One of my most vivid memories from childhood was being in a computer store which sold Apple ][s when a gentleman drove up in an (awesome) black Trans Am and declared to the salesperson, "I want a Visicalc" --- after explaining that it was a computer application and that the potential customer didn't have an Apple, the salesperson proceeded to put together pretty much my dream machine (at the time), an Apple ][ w/ dual-disk drives and 80 col. card and green display and 132 col. dot matrix printer, and of course, a copy of Visicalc.
After paying by writing out a check, I helped load everything into his car and he drove off into the sunset --- I was then allowed to choose a reformatted disk from the box as a reward and chose _The Softporn Adventure_ (which I then stupidly removed the label from, but it wasn't something I wanted to explain to my parents...).
tonyedgecombea day ago
I would guess anybody doing bookkeeping or accounting.
Back then it was common for people to buy a whole system for their requirements. Hardware and software.
bombcara day ago
Software was often more expensive than the computer - and sometimes by far!
Managers loved spreadsheets more than the accountants I feel, being able to fiddle numbers and see what changed was, well, a game changer.
TMWNNa day ago
Accountants, and individuals within all kinds of businesses (what we today would call shadow IT). Imagine something like this:
* Person who deals with numbers all day goes to a computer store to browse.
* He sees VisiCalc, and immediately understands what it can do. It *blows his mind*.
* He wants to buy it right away. Pays for $2000 Apple II computer with disk drives to run $100 software; price is no object.
* Shows friends and colleagues.
* They rush to computer store. Repeat.
mlhpdxa day ago
This was one of the projects students did when I helped teach APCS to high schoolers as a TEALS volunteer (FracCalc).
Some of the implementations went way overboard and it was so much fun to watch and to play a part.
Even as a “seasoned” developer I learned some tidbits talking through the ways to do (and not do) certain parts. When to store input raw vs processed, etc.
airstrikea day ago
Reminds me of https://tomasp.net/blog/2018/write-your-own-excel/
4leafclovera day ago
Very nice read!
Though I think the definition of the parser struct should be
struct parser {
const char* s;
const char* p;
struct grid* g;
};
pstuarta day ago
Other open source command line spreadsheets:
https://github.com/drclcomputers/GoSheet
https://github.com/xi/spreadsheet/
https://github.com/andmarti1424/sc-im
https://github.com/saulpw/visidata
https://github.com/bgreenwell/xleak
https://github.com/SamuelSchlesinger/tshts
https://github.com/CodeOne45/vex-tuisomat19 hours ago
If we are doing a survey, you missed one of the most interesting ones
manoDeva day ago
VisiCalc has, undoubtedly, the highest impact-to-complexity ratio in the history of software so far.
rnxrxa day ago
This a great article - both interesting and potentially really useful to folks teaching- or learning- programming.
khazhouxa day ago
I’m genuinely worried that we’re the last generation who will study and appreciate this craft. Because now a kid learning to program will just say “Write me a terminal spreadsheet app in plain C.”
jdswaina day ago
Which is somewhat akin to downloading one today. If, however, that same kid started small, with a data model, then added calculation, and UI and stepped through everything designing, reviewing, and testing as they went, they would learn a lot, and at a faster pace than if they wrote it character by character.
II2IIa day ago
The thing is, any generation can say something similar. Just look at the article: it manages to produce and describe the creation of a simple spreadsheet, yet the code and accompanying description would only fill a small pamphlet.
There are various reasons for that, and those reasons extend beyond leaving out vital functionality. While C is archaic by our standards, and existed at the time VisiCalc was developed, it was programmed in assembly language. It pretty much had to be, simply to hold the program and a reasonable amount of data in memory. That, in turn, meant understanding the machine: what the processor was capable of, the particular computer's memory map, how to interface with the various peripherals. You sure weren't going to be reaching for a library like curses. While it, like C, existed by the time of VisiCalc's release, it was the domain of minicomputers.
I mean, can the current generation truly understand the craft when the hard work is being done my compilers and libraries?
dualblocksgamea day ago
[dead]
diablevva day ago
[dead]
tracker1a day ago
Kinda cool to see... TBH, I'd be more inclined to reach for Rust and Ratatui myslf over C + ncurses. I know this would likely be a much larger executable though.
With MS Edit resurrected similarly, I wonder how hard it would be to get a flushed out text based spreadsheet closer in function to MS Excel or Lotus 123 versions for DOS, but cross platform. Maybe even able to load/save a few different formats from CSV/TSV to XLSX (without OLE/COM embeds).