Code behaving differently on serverless environment
TLDR: Discovered that Cloudflare Workers were producing a different output than my local environment when running python code. The issue was due to limited buffer space for stdout/stderr on Cloudflare Workers. Reducing the number of debug stdout or stderr resolved the issue.
I've always relied on print statements over logging for debugging since my C days and never ran into any issues until recently. This website is built in python using Flask and Frozen-Flask. I use Jinja to generate HTML files from markdown files. For parsing Markdown files, I use a combination of built-in and custom Markdown extensions so I can auto-generate table of contents and have custom shortcodes for image collages, captions, etc. Markdown extensions rely on writing regular expressions to identify blocks for processing and each markdown extension also has a priority level which determines the order in which they are processed. To debug the markdown's parsing, I added a whole lot of debug stdout to help me debug the markdown extensions.
Since the code was never meant to run on a production environment I assumed that the print statements would be harmless. The Frozen-Flask tool generates all the static files for deployment on Cloudflare CDN and the Flask server never executes on the Cloudflare Workers. Then more recently I added additional functionality for my travel pages to create image thumbnails at various resolutions. This worked as expected on my local environment but when I deployed it to Cloudflare, exactly half of my images would get optimized.
The next obvious step for debugging was to make sure all the packages versions and the python version was the same on both environments. After verifying identical versions on both environments, I added more debug print statements for each step of the image processing pipeline. By this point, my log files were 16,000 lines (~2MB) for a single deployment. Going through the log files I saw the error BlockingIOError: [Errno 11] write could not complete without blocking when processing every other image. A quick research revealed that this was due to limited buffer space for stdout/stderr on Cloudflare Workers.
To confirm that this issue was caused by stdout/stderr, I added a sys.stdout = open(os.devnull, 'w') on top of my main python script to suppress all stdout which immediately resolved the issue.
I've now switched to using the python logging module for debugging where I have a more fine-grained control over the level of verbosity and can direct the output to a file without resorting to hacky solutions. Now I've identified at least one distinction between logging and print statements.