View profile

Solving the 'exit' problem of Pino transports and many more Adventures in Nodeland - Issue #11

Matteo Collina
Matteo Collina
Hi Everyone! Another week has passed and it came with so many important Open Source contributions in the communities I participate in: Pino, Fastify and Node.js Core. Check them out!

I already wrote extensively of the new transport system in [email protected] that we are developing at it will allow you to create in-process transport similar to Winston or Bunyan, with no overhead on the main thread, as they will run inside a worker thread.
One of the biggest challenges of this approach is handling the shutdown of the process correctly. Two cases cause Node.js to exit: it can exit naturally if there are no more asynchronous things to do or manually by calling `process.exit(code)`. For the first case, we are calling unref() on the Worker thread to let the process exit even if the transport thread is running. In the second case, we thought of adding a handler to the `‘exit’` event to gracefully shut down the worker and ensure that all log messages are delivered independently of the status of the main thread. However, this approach had a significant obstacle to overcome: the `‘exit’` handler must be synchronous.
To achieve this, I updated the thread-stream `.end()` method to close the stream synchronously - this relies on work my friend mafintosh did a long time ago on Node.js stream: the autoDestroy mode ( Thanks to autoDestroy, we can guarantee that all streams emit a `‘close’` event when the underlining resource is fully closed. This setting became `true` by default in Node v12 onwards, while in readable-stream, it is still false by default.
Add close option by mcollina · Pull Request #6 · pinojs/pino-abstract-transport · GitHub
Finally, I developed a demo for my upcoming talk at JSNation, where I showed how to use ElasticSearch with the new transport system. I had to make minimal changes to the transport:
This would not have been possible without the help of David Mark Clements, who helped me out when I was stuck!
My colleague Simone Busoli fixed fastify-multipart on Node.js v16. While the change itself looks trivial - one line removed - it was actually really hard to understand why the plugin stopped working on the latest and greatest of Node.js, and it might be possibly relevant to you.
ci: add Node 16 by simoneb · Pull Request #235 · fastify/fastify-multipart · GitHub
fastify-static got a significant update as well with the support of pre-compressed assets! Thanks to this change, you will now be able to avoid compressing assets on the fly, saving precious CPU cycles.
The long-time Fastify collaborator Manuel Spigolon has been working relentlessly to improve the data validation/serialization support inside Fastify to make it more flexible and maintainable at the same time. The introduction of the ajv-compiler dependency allows us to remove the direct dependency to Ajv, making it fully swappable. Check out the PR, the change is coming to a Fastify release this week.
feat: compilers factory (transient ajv dependency) by Eomm · Pull Request #2862 · fastify/fastify · GitHub
I’m so grateful for Stephen Belanger’s work to improve the performance of async_hooks in Node.js. Thanks to this change, the overhead in handling promises was significantly reduced from 82% to “just” 35%. This is part of the plan that we drafted at Diagnostics Summit back in
async_hooks: use new v8::Context PromiseHook API by Qard · Pull Request #36394 · nodejs/node · GitHub
Like all new things, we spotted a bug in this new optimization, as it breaks when used inside Jest: In fact, this is problematic even outside of Jest:
This newsletter has reached 800 subscribers! I’m really happy you all find this interesting, keep the comments coming as they are deeply motivating.
I would also like to thank Glen Keane for proofreading my rumblings!
See you all next week!
Did you enjoy this issue? Yes No
Matteo Collina
Matteo Collina @matteocollina

I write about my journey as a core contributor of Node.js, as an author and a maintainer of many modules - including Fastify and Pino. In addition, I speak at conferences, and I will add links to all my talks in case you missed one.

If you don't want these updates anymore, please unsubscribe here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Created with Revue by Twitter.