I have a zest for 3d printing, which is why I’ve created a new 3d Printer called the Lemontron.

I’ve been passionately working on this project with a single goal in mind: to create a 3D printer that is as versatile as it is portable. Lemontron is designed to fit neatly into a standard filament box, making it the perfect tool for anyone who values efficiency without compromising on quality.

Whether you’re a seasoned 3D printing enthusiast or just getting started, Lemontron offers something new. It’s easy to assemble, disassemble, and store, without sacrificing the performance you expect from a full-sized 3D printer. With Lemontron, you can take your zest for printing anywhere.

Check out the Lemontron at lemontron.com.

Install Tailscale

curl -fsSL https://tailscale.com/install.sh | sh

Create non-root user & enable SSH

Create ubuntu user:

adduser ubuntu
usermod -aG sudo ubuntu

Most hosts allow keys to be uploaded at time of vps creation. Copy the keys to the new user.

rsync --archive --chown=ubuntu:ubuntu ~/.ssh /home/ubuntu

Login as ubuntu

ssh ubuntu@example

Quiet the login message

touch .hushlogin

For Meteor Apps

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt-get install -y nodejs jq build-essential
sudo npm install -g pm2
pm2 startup

Install docker

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce
sudo usermod -aG docker $USER

Log out and back in!

There are actually so many reasons. First, Docker’s 2375 is bound to localhost, so you might have looked into creating a proxyport with netsh, but read on. On Windows, Docker is actually running in a VM, so there’s a firewall there. There’s also a Windows firewall, so how can we connect to the Docker VM remotely?

Overcoming the complexities of network traversal

Tailscale can be used to get around the issue while keeping everything secure. You are of course always free to install Tailscale directly on your computer. However, installing the Tailscale Docker Extension will allow you to connect to the underlying Docker VM. Crucially, Tailscale provides distinct IP addresses for these distinct machines. This allows you to easily overcome the complexities of network traversal, as you can simply use an IP for the Docker VM.

No need to mess with Firewall, netsh portproxy, or any of that nonsense.

How to directly connect to the Docker Machine by IP

1. Install the Tailscale Docker extension
2. Enable the setting “Expose daemon on tcp://localhost:2375 without TLS” in Docker Desktop
3. Copy the IP address from the newly added Docker Machine (This is the Virtual Machine Docker is running under, not the Windows host)
4. Add "hosts": ["tcp://0.0.0.0:2375"] to the Docker Machine configuration in Settings.
5. Test the connection by running docker -H tcp://[HOST_IP]:2375 ps

We can plainly see how solid of a solution this is. Not only is it insanely simple and easy to use, you are now able to remotely access the docker daemon even from outside of your LAN. Incredible!

Life360, a company that claims its product is a “family safety service” has been selling location data on its 31 million customers, including kids, since 2016.

PathSense defends itself by citing the app “improved driver safety and saved numerous lives” but was it even technically necessary to save all that data to the cloud, or could it have been possible to respect user privacy without losing its major selling points?

That’s exactly the challenge I faced when I began work on a location-based app.

I have had the pleasure of working on a new dwell detection algorithm. What started as a “scholarly” kernel-based approach, making attempts at polling updates at regular intervals, quickly revealed, through testing, iOS would begin to throttle GPS updates to reduce the amount of time spent background processing. Working within the limitations of iOS, we discovered how to play nice, and add some top shelf sophistication.

With the help of my crack team of pedestrian and vehicular co-founders, we created an algorithm that performed…unreasonably well! We reached a hit rate above 90% by using the least amount of updates possible. Counterintuitively, the GPS remains OFF when in motion, and turns on High Accuracy during periods of stillness. Our battery usage is below 4% per day.

Basically, we draw a circle around you and move it with you, but when you walk or are still, we allow the circle to trail behind, sticking to the ground rather than following you. If you ever escape a circle, it becomes a “leave” event. After backtracking, we can determine the dwell’s starting event, being the first location update of the cluster of updates existing inside the circle.

Many dwells are created from only two points, the activity recognition system triggering an initial GPS poll for the starting point, and then a second poll when the user has moved a significant distance based off of low power draw signals from other sensors that the OS uses. I think this is a metric of how efficient a dwell detection algorithm is because it demonstrates how little the algorithm has to fight against the data it receives.

I really enjoyed the process of utterly completely understanding everything involved in dwell detection. I specifically loved how once we landed on the right algorithm, the code for dwell detection was reduced from 200 lines of useless interpolation, averaging, heuristics and edge case handling, to only the essential 20 lines. Everything just fell together all at once near the end. I think that satisfaction is a moment in my life I will always remember.

Also, during the entire time, it was never a question of if we needed to send the location updates to a server, that part was nonsense and should have never come up.

Yet another impossible to remember thing!

sudo lsof -i:22

Need to generate a PDF using the Arial font? Ubuntu doesn’t bundle proprietary software to avoid licensing issues, however you can easily install the Microsoft suite of fonts with one simple command.

sudo apt install ttf-mscorefonts-installer
sudo fc-cache -f -v

That’s it!

By running this one simple command you will install all these fonts on Ubuntu! Most importantly you will install Arial, the most common Helvetica substitute.

1. Andale Mono
2. Arial Black
3. Arial (Bold, Italic, Bold Italic)
4. Comic Sans MS (Bold)
5. Courier New (Bold, Italic, Bold Italic)
6. Georgia (Bold, Italic, Bold Italic)
7. Impact
8. Times New Roman (Bold, Italic, Bold Italic)
9. Trebuchet (Bold, Italic, Bold Italic)
10. Verdana (Bold, Italic, Bold Italic)
11. Webdings

When developing a React Native app with a meteor backend, you may be tempted to make use of the meteor client found on npm. However, that is not the best way!

How not to implement a Meteor backend in React Native

When we search for a package that can integrate a Meteor backend into a React Native project, we are presented with several options, but no official package. These packages all seem to work at first, but once you dive deeper you discover differences from the official client, including bugs. Crucially, none of the packages follow Meteor DDP to spec, or differ from the official implementation. The way quiessence (one of meteor’s internal mechanisms) events fire, and out-of-order steps in the method call pipeline can cause bugs that are usually race conditions. In contrast, the meteor official code is frequently updated, inherently supports the latest features, and is battle tested and is in widespread use.

The solution

When building a Meteor app for deployment, Meteor generates a js bundle. Using this knowledge, we can make use of the official, battle-tested client, and in doing so, make React Native just a little tiny bit less shitty.

import { Random } from 'meteor/random';

Meteor.call('Test', Random.id(), (err, res) => {
	console.log(err, res);
});

Importing meteor within a React Native project seems like madness!

Table of contents

• Setup and build the meteor client
• Polyfills
• Get imports to work
• useTracker
• Meteor updates

Prior work

This blog post is based off of the work of Urigo and his meteor-client-bundler and has been made more efficient by better polyfills and more readily modified scripts.

Quick overview of the folder structure

You can see below that there is a meteor-client folder, inside which is a folder for a stub of a meteor project, called app. The file create-bundle.js is the primary script which will run the meteor bundler and copy the output to a place within your react-native project directory. There is also post-bundle.js which is an asset used by create-bundle.js. Inside your utils folder in your project you will be copying in the polyfill.js file which will seamlessly translate web API’s to React Native API’s.

📂 .meteor-client
   📁 app
   📄 create-bundle.js
   📄 post-bundle.js
📂 util
   📂 meteor
      📄 polyfill.js
      📄 meteor-client.js

Setup and build meteor client

Let’s get started! Create a folder to house the bundler and make a blank meteor app inside.

mkdir .meteor-client
cd .meteor-client
meteor create app

The bundler’s purpose is to build the meteor client to a temporary folder and copy the client to a specific directory in your project, in this case utils/meteor/meteor-client.js. It also installs some boilerplate code to the top and bottom of the bundle. Contents of create-bundle.js:

 const execa = require('execa');
 const fs = require('fs');
 const path = require('path');
 
 const tempDir = '/tmp/meteor-client';
 const cwd = process.cwd();
 const destinationFile = path.resolve(cwd, 'utils/meteor/meteor-client.js');
 
 execa.sync('meteor', ['build', '--debug', '--directory', tempDir], {cwd: path.resolve(cwd, '.meteor-client/app'), stdio: 'inherit'});
 
 // Create top of file
 const runtimeConfig = '__meteor_runtime_config__ = Object.assign({meteorEnv: {}}, window.__meteor_runtime_config__);\n\n';
 fs.unlinkSync(destinationFile);
 fs.writeFileSync(destinationFile, runtimeConfig);
 
 // Copy modules to file
 const buildPath = path.resolve(tempDir, 'bundle/programs/web.browser');
 const program = require(path.resolve(buildPath, 'program.json'));
 program.manifest.filter(pack => {
     return (pack.where === 'client' &&
         pack.type === 'js' &&
         path.dirname(pack.path) === 'packages') ||
         pack.path.indexOf('global-imports') > -1;
 }).forEach(pack => {
     const packFile = path.resolve(buildPath, pack.path);
     const packContent = fs.readFileSync(packFile).toString() + '\n\n';
     fs.appendFileSync(destinationFile, packContent);
 });
 
 // Append bottom of file
 const postBundleFile = path.resolve(__dirname, 'post-bundle.js');
 const postBundleContent = fs.readFileSync(postBundleFile).toString();
 fs.appendFileSync(destinationFile, postBundleContent);

Contents of post-bundle.js

 // Disable hot-code-push
 if (Package.reload) {
     Package.reload.Reload._onMigrate(function () {
         return [false];
     });
 }

Install the devDependencies:

npm install --dev execa

In your package.json add a new script:

"scripts": {
    ...
	"meteor-client-bundle": "node .meteor-client/create-bundle.js"
}

And run it…

npm run meteor-client-bundle

In your root component, import the polyfill and the meteor client:

import './utils/meteor/polyfills';
import './utils/meteor/meteor-client';

import React from 'react';
import { registerRootComponent } from 'expo';
import App from './containers/App';

registerRootComponent(App);

Polyfills

This generated bundle is not much use yet, it was intended to be run within a browser environment. So we must polyfill all those browser apis. This polyfill depends on react-native-mmkv-storage.

npm install react-native-mmkv-storage

Contents of /util/meteor/polyfill.js:

// Good stuff that is required to run meteor client
// https://blog.meteor.com/leverage-the-power-of-meteor-with-any-client-side-framework-bfb909141008

import { Dimensions, Linking, Vibration, Alert, AppState } from 'react-native';
import NetInfo from '@react-native-community/netinfo';
import MMKVStorage from 'react-native-mmkv-storage';
import { baseUrl } from '../../constants/baseUrl';

export const polyfillMmkv = new MMKVStorage.Loader().withInstanceID('localStorage').initialize();

global.__meteor_runtime_config__ = {'DDP_DEFAULT_CONNECTION_URL': baseUrl};

let currentUrl = '';
let readyState = 'complete';

const netInfo = {
	changeListeners: [],
	addEventListener(eventName, callback) {
		if (eventName === 'change') this.changeListeners.push(callback);
	},
	removeEventListener(eventName, callback) {
		if (eventName === 'change') this.changeListeners.splice(this.changeListeners.indexOf(callback), 1);
	},
};

// Forward react native NetInfo events to polyfill
NetInfo.addEventListener(info => {
	// console.log('update', info);
	Object.assign(netInfo, info);
	netInfo.changeListeners.forEach(callback => callback(info));
});

// Sync currentUrl with new url
Linking.addEventListener('url', event => {
	if (event.url) currentUrl = event.url;
});

function parseUrl(url) {
	const urlArr = url.split('#');
	return {
		get href() {
			return urlArr[0];
		},
		set href(url) {
			return Linking.openURL(url);
		},
		get pathname() {
			return this.href;
		},
		get hash() {
			return urlArr[1] ? `#${urlArr[1]}` : "";
		},
		set hash(hash) {
			return Linking.openURL(this.href + '#' + hash);
		},
		replace(url) {
			return Linking.openURL(url);
		},
		toString() {
			return url;
		},
	};
}

export const localStorage = {
	setItem: (id, val) => {
		polyfillMmkv.setString(id, String(val));
	},
	getItem: id => {
		return polyfillMmkv.getString(id);
	},
	removeItem: id => {
		polyfillMmkv.removeItem(id);
	},
	clear: () => {
		polyfillMmkv.clearStore();
	},
};

Object.defineProperties(window, {
	localStorage: {
		get: () => localStorage,
	},
	sessionStorage: {
		get: () => localStorage,
	},
	document: {
		get() {
			return {
				readyState,
				addEventListener(eventName, callback) {
					if (eventName === 'deviceready' || eventName === 'DOMContentLoaded') {
						callback();
					} else if (eventName === 'visibilitychange') {
						AppState.addEventListener('change', e => {
							callback({'visibilitychange': e.match(/inactive|background/) ? 'hidden' : 'visible'});
						});
					}
				},
				getElementsByTagName() {
					return {
						item() {
							return {
								appendChild() {
								},
							};
						},
					};
				},
				createElement() {
					return {
						setAttribute() {
						},
						get pathname() {
							return '';
						},
					};
				},
				get hidden() {
					return AppState.currentState.match(/inactive|background/);
				},
				get visibilityState() {
					return this.hidden ? 'hidden' : 'visible';
				},
			};
		},
	},
	self: {
		configurable: true,
		value: global,
	},
	addEventListener: {
		configurable: true,
		value(eventName, callback) {
			switch (eventName) {
				case 'load':
					break;
				case 'online':
					netInfo.changeListeners.push(callback);
					break;
				case 'hashchange':
					let oldHash = location.hash;
					Linking.addEventListener('url', event => {
						const {hash} = parseUrl(event.url);
						if (hash !== oldHash) callback(event);
						oldHash = hash;
					});
					break;
			}
		},
	},
	attachEvent: {
		configurable: true,
		value(eventNameWithOn, callback) {
			const eventName = eventNameWithOn.replace('on', '');
			return this.addEventListener(eventName, callback);
		},
	},
	onload: {
		set(callback) {
			return this.addEventListener('load', callback);
		},
	},
	onhashchange: {
		set(callback) {
			return this.addEventListener('hashchange', callback);
		},
	},
	outerWidth: {
		configurable: true,
		get() {
			return Dimensions.get('window').width;
		},
	},
	outerHeight: {
		configurable: true,
		get() {
			return Dimensions.get('window').height;
		},
	},
	alert: {
		get() {
			return Alert.alert.bind(Alert);
		},
	},
});

const geolocation = navigator.geolocation;

Object.defineProperty(window, 'navigator', {
	get() {
		return {
			get onLine() {
				return !!netInfo.type;
			},
			get vibrate() {
				return Vibration.vibrate.bind(Vibration);
			},
			get geolocation() {
				return geolocation;
			},
			get connection() {
				return netInfo;
			},
		};
	},
});

Object.defineProperty(window, 'location', {
	configurable: true,
	get() {
		return parseUrl(currentUrl);
	},
	set(url) {
		return Linking.openURL(url);
	},
});


let newWindowMock = {closed: false};

Object.defineProperty(window, 'open', {
	configurable: true,
	value: url => {
		newWindowMock.closed = false;
		Linking.openURL(url);
		return newWindowMock;
	},
});

Get imports to work

At this point you can access the meteor packages but unless you are a maniac, this is really unsatisfying.

To be able to import meteor packages, you can implement an adapter in babel. Using AST explorer I developed a “import from meteor package” transpiler. By including the transpiler in babel.config.js babel (which is built in) will immediately make the new syntax work.

const adaptMeteor = (babel) => {
	const {types: t} = babel;
	return {
		visitor: {
			ImportDeclaration(path) {
				const modulePath = path.node.source.value;
				if (!modulePath.startsWith('meteor/')) return;
				const moduleName = modulePath.slice(7);
				const newNode = t.variableDeclaration('const', [
					t.variableDeclarator(
						t.objectPattern(path.node.specifiers.map(r => {
							return t.objectProperty(t.identifier(r.imported.name), t.identifier(r.local.name), false, true);
						})),
						t.identifier(`Package['${moduleName}']`),
					),
				]);
				path.replaceWith(newNode);
			},
		},
	};
};

module.exports = function(api) {
	api.cache(true);
	return {
		presets: ['babel-preset-expo'],
		plugins: [adaptMeteor],
	};
};

useTracker

Here’s a useTracker function you can drop into your project!

import { Meteor } from 'meteor/meteor';
import { Tracker } from 'meteor/tracker';
import { useReducer, useEffect, useRef, useMemo } from 'react';

// Used to create a forceUpdate from useReducer. Forces update by
// incrementing a number whenever the dispatch method is invoked.
const fur = (x) => x + 1;
const useForceUpdate = () => useReducer(fur, 0)[1];
const useTrackerNoDeps = (reactiveFn, skipUpdate = null) => {
	const {current: refs} = useRef({
		isMounted: false,
		trackerData: null,
	});
	const forceUpdate = useForceUpdate();
	// Without deps, always dispose and recreate the computation with every render.
	if (refs.computation) {
		refs.computation.stop();
		// @ts-ignore This makes TS think ref.computation is "never" set
		delete refs.computation;
	}
	// Use Tracker.nonreactive in case we are inside a Tracker Computation.
	// This can happen if someone calls `ReactDOM.render` inside a Computation.
	// In that case, we want to opt out of the normal behavior of nested
	// Computations, where if the outer one is invalidated or stopped,
	// it stops the inner one.
	Tracker.nonreactive(() => Tracker.autorun((c) => {
		refs.computation = c;
		const data = reactiveFn(c);
		if (c.firstRun) {
			// Always run the reactiveFn on firstRun
			refs.trackerData = data;
		} else if (!skipUpdate || !skipUpdate(refs.trackerData, data)) {
			// For any reactive change, forceUpdate and let the next render rebuild the computation.
			forceUpdate();
		}
	}));
	// To clean up side effects in render, stop the computation immediately
	if (!refs.isMounted) {
		Meteor.defer(() => {
			if (!refs.isMounted && refs.computation) {
				refs.computation.stop();
				delete refs.computation;
			}
		});
	}
	useEffect(() => {
		// Let subsequent renders know we are mounted (render is committed).
		refs.isMounted = true;
		// In some cases, the useEffect hook will run before Meteor.defer, such as
		// when React.lazy is used. In those cases, we might as well leave the
		// computation alone!
		if (!refs.computation) {
			// Render is committed, but we no longer have a computation. Invoke
			// forceUpdate and let the next render recreate the computation.
			if (!skipUpdate) {
				forceUpdate();
			} else {
				Tracker.nonreactive(() => Tracker.autorun((c) => {
					const data = reactiveFn(c);
					refs.computation = c;
					if (!skipUpdate(refs.trackerData, data)) {
						// For any reactive change, forceUpdate and let the next render rebuild the computation.
						forceUpdate();
					}
				}));
			}
		}
		// stop the computation on unmount
		return () => {
			var _a;
			(_a = refs.computation) === null || _a === void 0 ? void 0 : _a.stop();
			delete refs.computation;
			refs.isMounted = false;
		};
	}, []);
	return refs.trackerData;
};
const useTrackerWithDeps = (reactiveFn, deps, skipUpdate = null) => {
	const forceUpdate = useForceUpdate();
	const {current: refs} = useRef({reactiveFn});
	// keep reactiveFn ref fresh
	refs.reactiveFn = reactiveFn;
	useMemo(() => {
		// To jive with the lifecycle interplay between Tracker/Subscribe, run the
		// reactive function in a computation, then stop it, to force flush cycle.
		const comp = Tracker.nonreactive(() => Tracker.autorun((c) => {
			const data = refs.reactiveFn();
			if (c.firstRun) {
				refs.data = data;
			} else if (!skipUpdate || !skipUpdate(refs.data, data)) {
				refs.data = data;
				forceUpdate();
			}
		}));
		// In some cases, the useEffect hook will run before Meteor.defer, such as
		// when React.lazy is used. This will allow it to be stopped earlier in
		// useEffect if needed.
		refs.comp = comp;
		// To avoid creating side effects in render, stop the computation immediately
		Meteor.defer(() => {
			if (!refs.isMounted && refs.comp) {
				refs.comp.stop();
				delete refs.comp;
			}
		});
	}, deps);
	useEffect(() => {
		// Let subsequent renders know we are mounted (render is committed).
		refs.isMounted = true;
		if (!refs.comp) {
			refs.comp = Tracker.nonreactive(() => Tracker.autorun((c) => {
				const data = refs.reactiveFn(c);
				if (!skipUpdate || !skipUpdate(refs.data, data)) {
					refs.data = data;
					forceUpdate();
				}
			}));
		}
		return () => {
			refs.comp.stop();
			delete refs.comp;
			refs.isMounted = false;
		};
	}, deps);
	return refs.data;
};

export function useTracker(reactiveFn, deps = null, skipUpdate = null) {
	if (deps === null || deps === undefined || !Array.isArray(deps)) {
		if (typeof deps === 'function') {
			skipUpdate = deps;
		}
		return useTrackerNoDeps(reactiveFn, skipUpdate);
	} else {
		return useTrackerWithDeps(reactiveFn, deps, skipUpdate);
	}
}

Meteor updates

Meteor updates are a simple matter of accepting the update and re-running the bundler:

cd .meteor-client/app
meteor update
cd ../../
npm run meteor-client-bundle

I’m always facing the problem of calculating the distance between two points on the surface of the Earth. Most of my readers will already be aware of the Haversine formula, so let’s dive in and see how many earth shattering speed enhancements we can make.

Good old haversine

Haversine is a pretty solid way to calculate distances. Although it assumes the earth is a sphere, it sacrifices very little and reaches a good level of speed. Despite it being the more performant among the top distance algorithms, the haversine still involves 7 computationally expensive trigonometric functions (4 sines, 2 cosines, 1 arc tangent, 2 powers of two, and 2 square roots) and looks like this:

const R = 6378.137;
const toRad = n => n * Math.PI / 180;

export const getDistance = (a, b) => {
    const dLat = toRad(b.lat - a.lat);
    const dLon = toRad(b.lng - a.lng);
    const aLat = toRad(a.lat);
    const bLat = toRad(b.lat);
    const f = Math.pow(Math.sin(dLat / 2), 2) + (Math.pow(Math.sin(dLon / 2), 2) * Math.cos(aLat) * Math.cos(bLat));
    const c = 2 * Math.atan2(Math.sqrt(f), Math.sqrt(1 - f));
    return R * c;
};

I benchmarked the haversine taking 1.413s to process 5 million coordinate pairs.

Optimizing for small distances: the fast haversine

My own use case allowed me to take some additional shortcuts. I got rid of the Sin calculations. On top of that, the Cos of two close latitudes is approximately the Cos of its average. That allowed me to remove one Cos and assume some error. Finally, the two Pow were inlined, so no need to call an external generic power function if only to multiply a number by itself. The final algorithm, with some additional inlining to optimize even more, looks like this:

const R = 6378.137;
const PI_360 = Math.PI / 360;

export const getDistance = (a, b) => {
    const cLat = Math.cos((a.lat + b.lat) * PI_360);
    const dLat = (b.lat - a.lat) * PI_360;
    const dLon = (b.lng - a.lng) * PI_360;
    const f = dLat * dLat + cLat * cLat * dLon * dLon;
    const c = 2 * Math.atan2(Math.sqrt(f), Math.sqrt(1 - f));   
    return R * c;
};

Benchmarking 5 million coordinate pairs took 303.719ms. We are doing well with impressive gains. This one took only 21% of the time compared to baseline.

Reaching the end-game, the sloppy haversine

After finding the fast haversine formula, I could not stand still. I knew there was one more way to push the formula even further by revisiting our good friend trigonometry. By now we know that haversine exists to accommodate the earth’s spherical math. However when we are looking close to the earth’s surface it no longer appears as a sphere and instead looks flat. So if we can find a factor to multiply by the longitude delta, and a factor to multiply by the latitude delta, we can use trigonometry as if we are dealing with a flat plane. This is what is known as a projection which is when you flatten the latitude and longitude into x and y coordinates that represent kilometers. That code is this magnificent sloppy boy:

const kRad = Math.PI / 180;
export const getDistance = (a, b) => {
    const kx = Math.cos(a.lat * kRad) * 111.321;
    const dx = (a.lng - b.lng) * kx;
    const dy = (a.lat - b.lat) * 111.139;
    return Math.sqrt((dx * dx) + (dy * dy));
};

This new algorithm takes a mere square root and cosine call and completes 5 million coordinate pairs in a break neck speed of 215.79ms which is only 15% of the baseline!

Going even further for a specific use case

Usually we want to calculate the distance of your current location to many other locations, so your current location will be the same for each calculation. In the sloppy haversine, the variable kx, which is the factor to multiply the x coordinate by, will not change unless the first location changes. We can make a closure that reuses the cosine, so that each individual comparison uses only one square root call:

const kRad = Math.PI / 180;
export const ruler = (a) => {
    const kx = Math.cos(a.lat * kRad) * 111.321;
    return (b) => {
        const dx = (a.lng - b.lng) * kx;
        const dy = (a.lat - b.lat) * 111.139;
        return Math.sqrt((dx * dx) + (dy * dy));
    } 
};

Removing the square root

Hilariously, if you are only comparing distances and not displaying them, there is no need for the square root. Simply comparing the result of (dx * dx) + (dy * dy) will do. And with the last math call gone, we can plainly see here that with just a little work, we can avoid doing any work, which is kind of my life philosophy.

So, we can conclude that haversine is very generalized, and you should choose algorithms knowing how they work, not just blindly importing them!

For DigitalOcean’s 2021 hackathon I set out to create a dashboard for my Emissions station in Duluth, GA.

You can check progress on the publicly accessible website queen.jamesloper.com

I am aiming to build the following features…

1. Accounts with ability to link devices
2. Webcam view of the interior of the station from Wyze cam
3. Screen cast of the emissions computer screen (for diagnostics)
4. Call logs for the business phone line (provided by Twilio)
5. To determine the exact time the employee opened and closed for the day, I am making an integration with myQ to see when the garage door is open or closed
6. To check profit, I am making a scraper for the government provided dashboard website

Development & DigitalOcean

The mongo database will be hosted on DigitalOcean managed MongoDB, and the app will be served by a DigitalOcean droplet. I will run a copy of the development environment on my laptop and then deploy to “production”, but to ease the transition, I will connect my development environment straight to the MongoDB since the beginning. This way I don’t have to maintain two databases.

How will I determine the profit?

Each car that buys an emissions test uses up a “ticket” provided by the government. This is recurring cost for the emissions business, and since each test is a fixed cost, profit can be calculated from just the number of used tickets. If I can scrape the government website, I can calculate profit to a high degree of accuracy.

How will the webcams/streams work?

Depending on the protocol, it can be easy or hard. For the screen cast, it’s easy. I bought a capture card with pass-thru to grab the HDMI signal running to the emissions station screen. A cheap computer running mjpg-streamer exposes the feed as a URL. To embed that in a webpage, all I have to do is put an <img> tag with the url set as the src. Any URL pointing to a MJPEG stream can also be embedded into a web page this way.

I also want to get some networked webcams, so I chose Wyze Cam V2. It is capable of serving an RTSP stream, but I had to convert that to something web browsers can view. For this I made an RTSP to MJPEG proxy server. I hand off most of the work to FFMPEG, streaming the output from stdout to the client, however care must be taken to build a buffer and flush it upon each frame.

How are the call logs set up?

When adding a Twilio account, I fetch the historical logs and massage the data into my own format before inserting it into the Mongo DB. From that point, a webhook is enabled so that whenever Twilio gets a phone call, they will make an HTTP call to my REST endpoint. This does mean that while developing the app, I have to make my computer publicly accessible, for example by using ngrok for this.

How will the garage door sensor be set up?

I integrated myQ and will install a smart garage door sensor and will query it every 5 minutes to see if it is up or down.

While hosted mongodb solutions exist, they aren’t for everyone. Scaling, security, and reliability is just one tutorial away. In this fairly deep tutorial we cover everything from the initial installation to optional steps such as SSL and a secondary replica. This guide is targeted for Ubuntu 20.

Quick links

1. Getting started
2. Enable automatic startup mongod server on server boot
3. Create the admin user
4. Enable SSL
5. Adding a secondary
6. Enable swap
7. Simple MongoDB backups with mongodump
8. Diagnostics cheat sheet

Getting started

In this guide, the primary will be db.example.com. Start by logging in and performing the apt ritual. You should already set up a user, in this case my user is ubuntu.

ssh ubuntu@db.example.com
sudo apt-get install gnupg curl
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list
sudo apt update && sudo apt upgrade
sudo apt-get install -y mongodb-org

Optional: Create a data directory. You’ll want to do this so that the database doesn’t reside on the boot drive. This is a good practice! XFS is the filesystem of choice for mongodb data directories.

sudo mkdir /mnt/db/mongodb
sudo chown -R mongodb:mongodb /mnt/db/mongodb

Create a barebones configuration just to get it functioning. Edit /etc/mongod.conf.

storage:
  dbPath: /mnt/db/mongodb
  
net:
  bindIp: 0.0.0.0
  
replication:
  replSetName: rs0
  oplogSizeMB: 100

• dbPath should be set to the path you made. Or you can just remove the whole storage section if you are going to use the default.
• 0.0.0.0 makes your server accessible via the web.

Restart mongod:

sudo service mongod restart

Enable automatic startup mongod server on server boot

Register it with systemctl to start when your server boots

sudo systemctl enable mongod

At this point you can connect to the db, but it’s also wide open for attackers to take over! Next up we will deal with security.

Create the admin user

In this section, you will configure your database to only accept username & password authentication. While we are in here we might as well take care of initializing the replica set as well. Using your existing SSH session, connect to your database and create an admin user:

mongosh
use local
rs.initiate()
use admin

db.createUser({
  user: 'admin',
  pwd: 'password',
  roles: [{role: 'root', db: 'admin'}]
})

Edit /etc/mongod.conf again! Specify a security policy. This one is absolutely required if your server is bound to 0.0.0.0! Once done, you must authenticate every time (even through localhost!)

security:
  authorization: "enabled"
  keyFile: /data/key

Create key file

This step is kind of a song and dance but you just have to do it to appease the mongo gods.

sudo touch /data/key
sudo chmod 400 /data/key
sudo nano /data/key

And just put some random string in this file!

Restart mongod:

sudo systemctl restart mongod
sudo systemctl status mongod

Try it out! You want to make sure you can log in.

mongo mongodb://admin:password@db.example.com

Enable web server

LetsEncrypt requires your server to respond to a challenge in order to fulfill a certificate request, so we’ll need a tiny web server to answer their challenge:

sudo add-apt-repository ppa:nginx/stable
sudo apt-get update
sudo apt install nginx

At this point, navigating to db.example.com will result in the demo page for Nginx. Nothing further is really required to set up Nginx, and quite frankly no other software should even be running on your production database server.

Install certbot & certificate

sudo apt update
sudo apt install certbot python3-certbot-nginx
certbot --nginx -d db.example.com

At this point, you should be able to visit the NGINX page again, but this time it will be served over HTTPS! Next, we need to set up automatic renewal and application of the cert to MongoDB. Set up a bash script that will concatenate your NGINX certificates together to create your MongoDB server certificate:

cd ~
nano renew-mongo-cert.sh

#!/bin/bash
DOMAIN=db.example.com
newestFull=$(ls -v /etc/letsencrypt/archive/$DOMAIN/fullchain*.pem | tail -n 1)
newestPriv=$(ls -v /etc/letsencrypt/archive/$DOMAIN/privkey*.pem | tail -n 1)
cat $newestFull $newestPriv > /etc/ssl/mongo.pem
service mongod restart

chmod +x renew-mongo-cert.sh

Enable SSL in the mongo server configuration in /etc/mongod.conf by adding this under the net declaration:

  tls:
    mode: requireTLS
    certificateKeyFile: /etc/ssl/mongo.pem

Run the script for the first time:

./renew-mongo-cert.sh

Set the file permissions for the outputted pem file. This is only needed one time.

chmod 600 /etc/ssl/mongo.pem
chown mongodb:mongodb /etc/ssl/mongo.pem

Add the script as a cron job with crontab -e

0 0 1 * * /root/renew-mongo-cert.sh

Reboot, and now you can connect through SSL. Note that the URL now contains the ssl param:

mongo mongodb://admin:password@db.example.com/?ssl=true

Adding a secondary

Follow the same process above to get mongo installed and configured, switching out db.example.com with your secondary’s hostname.

All nodes in a cluster must share the same secret key– that’s how they authenticate with each other. We’ll make that file next. Do this step on both servers!

echo "your-secret-key" > /data/key

Edit /etc/mongod.conf

> security:
>  keyFile: /data/key

Then restart the mongo server again

service mongod restart

Next, log back in to the mongo shell.

mongo mongodb://admin:password@db.example.com/?ssl=true

The following mongo shell script is to have a secondary that your BI tools could beat up without affecting your primary. That’s why hidden is set to 1 and not 0. This ensures that the node can not become the primary in the event of a primary malfunction.

rs.add({host: "secondary.example.com", priority: 0, votes: 0, hidden: 1})
rs.slaveOk()

If you want to provide more hosts for your app to connect to, you instead want to make them visible. The minimum number of nodes in a cluster jumps from 1 straight to 3, otherwise a cluster of 2 can not perform a vote to decide which node will become primary.

Enable swap

In the case of large load, having a few gigs of swap space can help your database not crash. Unfortunately turning swap on can mask critical memory issues which would be better solved by adding more ram. It’s always up to you to determine if you want to use swap or not. If you want to go down that route, here is how.

Tip: You can verify at any time that your swap is working by running sudo swapon --show

Run the following commands:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Now you have swap space, but it won’t persist after reboot. Fix that by editing your fstab file with sudo nano /etc/fstab and paste the following /swapfile swap swap defaults 0 0

Simple MongoDB backups with mongodump

This is probably a horrible solution for a database over 1 gig, but I just got it functional so I will share it in the hopes that someone can benefit from it.

mkdir backups
mkdir backups/hourly
cd backups
nano backup.sh

Create a bash script like this:

#!/bin/bash
mongodump --uri="mongodb://user:pass@localhost:27017/test" --out="dump"
zip -r "hourly/$(date +"%Y-%m-%d %I:%M%p").zip" dump
find hourly/* -mtime +1 -exec rm {} \;

This script creates a new backup and zips it, and deposits it in the hourly folder, then deletes any hourly backups that are older than a day. You can make this run hourly with crontab -e

Diagnostics cheat sheet

• Clear logs with echo "" > /var/log/mongodb/mongod.log
• Get logs with tail /var/log/mongodb/mongod.log
• Edit config at nano /etc/mongod.conf
• Key file should be 600 permissions
• mongodb user should own /var/log/mongodb
• Unlink socket with rm /tmp/mongodb-27017.sock
• Start mongod service with service mongod start