Degrade before failing
Telemetry is secondary workload. Collection becomes less detailed before the host becomes less reliable.
Insight-Technology Oy
Small by choice. Technical by default.
An independent systems engineering company based in Helsinki. Most of us build software, operate infrastructure, review protocols, or answer technical support requests.
2011established in Helsinki
30–40people, depending on contractors
~4/5directly in technical work
02:00when architecture becomes support
01 / ORIGIN
The collector survived.
Insight-Technology was founded by engineers from distributed monitoring, industrial systems, and telecom infrastructure. The original internal project investigated failures in a geographically distributed compute environment.
The collector survived. The application it was monitoring did not.
This was considered promising.[03:14:07] target application unresponsive
[03:14:07] collector state ........ healthy
[03:14:08] journal flush .......... complete
[03:14:08] evidence preserved ..... yes
observer> perhaps this is a product02 / ENGINEERING PRINCIPLES
Five rules we can operate.
There is no separate innovation organization. If something becomes a product, the people building it normally have to explain its behavior to customers.
Bound everything
Queues, buffers, retries, and logs all have selected maximum sizes.
Offline is a normal network state
Disconnected systems are not edge cases. They simply do not have a route at the moment.
Compatibility is part of the product
Firmware inside machinery will not receive a compiler upgrade because we asked nicely.
Debuggability beats cleverness
A boring distributed system can be operated for ten years.
03 / SELECTIVE HISTORY
Fifteen years of mostly intentional behavior.
Insight-Technology founded
Two rooms, five desks, and an alarming quantity of second-hand networking equipment.
First commercial deployment
Approximately 1,800 engineering workstations. The customer requests a dashboard. Engineering provides a CSV export. Negotiations continue.
FleetWatch 1.0
Linux and Windows support. Version 1.1 mostly fixes assumptions Version 1.0 made about Windows.
Relay architecture
Introduced after “occasionally connected” is found to include computers aboard ships.
ChipWatch
A hardware-test logging library becomes a product. Its stern comment about calling malloc() remains.
FleetWatch 5
Independent streams and resumable sessions, plus a compatibility layer that support repeatedly prevents engineering from removing.
ChipWatch schema compiler
Compile-time schema generation and version-aware host decoding arrive.
Still here
The existing mission statement is determined to compile successfully and remains in production.
04 / HELSINKI OFFICE
One hardware lab. One small machine room.
Also several whiteboards, a library containing an unreasonable number of protocol specifications, and an espresso machine whose network interface has been permanently disabled.
Hot-desking has been attempted twice. Both attempts ended when engineers quietly moved all the keyboards back.
- hel-beacon-01
- production relay
- hel-beacon-03
- desktop named incorrectly; too dangerous to rename
- build-moose
- retired
- build-moose-final
- not final
- build-moose-final2
- current