New Horizons

This is a second post in a 2 post series of my experience building inventory websites over the last few years. The first post talks about building inventory websites using MEAN stack and superfluousness of using MEAN for smaller inventory footprint / user base. If you haven't read the first post, you could read that here

In this post, I'd like to talk about a simpler version of inventory website (with just enough functionality) that you could build right now. I'll add my code at the end too.

The framework in question is called 'streamlit'. Streamlit is a python based framework to showcase your data science work developed with data scientists in mind. It natively runs in a web browser backed by a python script. And it is interactive - you can create and respond to user actions. If you know what Jupyter notebooks are (if you dont, they are a browser based execution environment backed by the terminal , again great for data science experimentation ), streamlit has what Jupyter notebook has , plus controls like text input, buttons, sliders etc.

Before we talk about why streamlit is the right framework, let me cover the bare bones of an inventory website in terms of database, web server and web page.

The fundamental building block of any inventory application is a data base containing all the information. This is the heart of the entire system. Think of this as CENTRAL STATE. Every user action either reads or updates this one central state. The user actions one could perform are usually called CRUD ( Create - Read - Update - Delete ). All the inventory actions could be composed using CRUD operations.

The second building block of an inventory application is a web server. Think of this as the brain of the system. It accepts requests from the user and sends responses. It is also connected to the database and does CRUD operations on behalf of the user. A web site is the interface for the user to get current inventory, update inventory , create new inventory items etc. When you enter an address in the browser bar, that's the first request made to the web server. It responds with a website. Now once the browser has the website, you can interact with your inventory through it. When you click a button, push a slider, select an option etc, your website sends a request to the web server on your behalf. The server has some pre defined rules for each request. Some requests make the server request inventory from the database and send it to the webpage.

So in summary , you need a database, a web server and a webpage. In the MEAN world, you use a MongoDB database, Express.js server , Angular 2 framework for building a website and all of this built the nodejs environment. If the above jargon doesnot make sense to you, that's alright. Replace everything by javascript and more javascript to give it some shape.

Now let's talk about streamit. First things first, it's based on python. Oh wait, you did go through the initiation into python in the previous post.

Moving on. I've give you a very simple recipe of an inventory website. All in a less than 100 lines of code [obviously, you can't launch into space with this. Oh wait, can you ? ]

Install python and streamlit on your PC ( or server ), you can find installation instructions here. Take all your inventory in excel sheets. You can even convert it into .csv files, but if you don't, that's fine too. Write a .py file with the interaction you need. Look at the code for more details. Open a terminal and run 'streamlit run <filename>.py' and there you have it.

An inventory website being server on you LAN network. Just give everyone your IP address and port and they could use it too. No more writing code for the web server, html, css, javascript, angular boiler plate etc. One file, some python code and your inventory excel sheet or csv. This setup solves a simple problem I call visibility. Every one know the current state of inventory. All you have do is keep the excel sheet up to date and your users would see the latest inventory on the website automatically.

There are some problems it does not solve. For example, there's no real authentication. So don't use for external facing websites, obviously. Apart from that, you could create an excel sheet of user name and passwords and link it to your streamlit app to authenticate people.

Without further ado, you can find sample code here.

Call or text me if you need any help.

This is a blog about the evolution of my thought process about building websites to maintain and disseminate information on inventory over the past three years. It has been a long journey from tinkering with databases and javascript and finally discovering the zen of csv files and python.

How does one manage inventory? There is no denying that everyone has this problem here. Even after commissioning special storage cabinets and labelled shelves , boxes, pouches and keeping spreadsheets on them, we were at a loss.

In spite of all that work of organising, inventory still has been a game of darts. We still had issues like sending the wrong parts to field. Field personnel and us fumbling on the part names, matcodes, manufacturing part numbers, purchase orders. Then there were other problems - like differences in packaging, similar names with different uses and the list goes on.

And we were back at how does one manage inventory. If you have noticed, questions like these have a way of starting slowly, seeming like a wrestle, followed by periods of being swept under the rug and one day, ending in the monster suddenly re-surfacing in intimidating forms.

Hyperbole aside , we noticed that there were several facets to the same problem. Let me see, if I could list all of them.

1. Not everyone uses a certain spare and hence not every one knows every thing. This is a Scope Problem.

2. Not everyone uses the same terminology. This is a Language Problem.

3. A person who knows and uses a spare part, might not necessarily be the best at describing it . This is a Language Problem too.

So what do you do when you decide to deal with the problem ? Like any good accountant - you look for the ground truth and build from there. It started with a simple and unoriginal idea. Descriptions are causing all the confusion, what if we could see a picture. May be, magically , this inventory problem would be solved.

So, for the next few weeks, our technicians religiously catalogued all our inventory all the while taking pictures of everything. For a while, the store looked like a forensics lab.

Initially, we used a laptop with a spreadsheet for data collection and a mobile camera for pictures.

Later, we graduated to using a simple google form. [ I assure you we gave due regard to the confidentiality & Integrity of the information ]

Now that we had all the information, our next step was to look for ways to make it available to everyone in the department. Our requirements were simple

1. Everyone in the department can search for a spare part and find a picture, current inventory , location and any other meta data.
2. The store keeper can add new spares and update inventory.
3. If you are not a store keeper, you cannot make any changes to the inventory.

Right around this time, I had finished reading 'Eloquent Javascript' and a few books and tutorials on the MEAN stack including 'Web Application Development using MEAN stack'.

So charged with the problem and a means to solve it, I did the following which I present to you as Paradigm #1.

Paradigm #1 :

Build a full featured web site for tracking inventory. One where you could add, remove, update inventory. One with a login window, running on our local network.

So that's what we built.

1. A website built using HTML, CSS, Javascript using the Angular 2 framework. I relied heavily on Bootstrap for styling. The website had three pages. Add, Update, Search. The search page would let you type in a keyword and it returned matches for matcode, part number, description, meta tags etc.
2. All the data was stored in a No SQL database called MongoDB. Schema was really simple, one to two nests.
3. Express.js Server. All the javascript code was heavily borrowed. We had created simple REST APIs and middleware for the Create-Read-Update-Delete loop. We tried integrating passport.js for authentication but eventually gave up. It had some 'session' issue that we could not solve.
4. Node.js — Well although I mentioned this in the end, this was the basis for everything. My familiarity of working with Node.js in one of my previous projects really helped. [More on that in a different post.]

It look us about 3 week - one to two hours on each day , of tinkering got it running. Mostly everyone was happy. Except the guy who staked his life on the efficacy of excel sheets. This was uncharted territory for him and mostly he was worried that he would get a few weird columns in the excel sheet. We used mongoexport to export the database into excel sheet.

Well, looking back, the 'photo' idea seemed useful. As always, the person who appreciated it the most was the one who faced the most problems on a minute by minute basis, THE STOREKEEPER.

In its entirety, these are problems paradigm 1 solved , aka PROs :

• Availability
• Easy Inventory management
• Visibility

CONs of Paradigm 1 :

• Javascript Web App Development, especially Angular 2 is overkill for this application for a small user base.
• Each feature like search, pagination, mongodb integration took us time.
• Huge codebase. We literally spent many afternoons trying to solve a tiny bug.
• You cannot make batch updates to inventory. I mean you could if we wrote another page for that. Think of a basic Javascript TODO with a submit button.
• You need two different applications — a client app and a server app.

After this adventure, I also built a single terminal biometric attendance system using javascript.

Fast forward to March 2020, I began to take obsessive interest in data science. I entered the world of rectangular data aka dataframes. I was rapidly learning data manipulations and visualisations using Python packages.

Let me take a moment and acknowledge the delight of python.

For people coming from C, Java, Javascript, they are taken aback by the simplicity of syntax. Python one-liner codes are so popular, and there's so many of them, people have written books on them. Here's one. I can't resist but show you the python code for swapping two variables. ⇒ a,b = b,a. I could tell about list and dictionary comprehensions too , but you'll figure it. They are simple and elegant. If there's one thing I'd want you to take away from this, it would be : LEARN PYTHON.

In fact, let me introduce you to the guiding principles of Python language.

• Beautiful is better than ugly.
• Explicit is better than implicit.
• Simple is better than complex.
• Complex is better than complicated.
• Flat is better than nested.
• Sparse is better than dense.
• Readability counts.
• Special cases aren't special enough to break the rules.
• Although practicality beats purity.
• Errors should never pass silently.
• Unless explicitly silenced.
• In the face of ambiguity, refuse the temptation to guess.
• There should be one—and preferably only one—obvious way to do it.
• Although that way may not be obvious at first unless you're Dutch.
• Now is better than never.
• Although never is often better than right now.
• If the implementation is hard to explain, it's a bad idea.
• If the implementation is easy to explain, it may be a good idea.
• Namespaces are one honking great idea—let's do more of those!

A popular library for data manipulation in python is Pandas. You can import data from csv, xlsx, images, sql db, api, pdf into pandas and then filter, search and visualise. For example, if you have list of your inventory, it is really easy to do statistical analyses. If you want to know what you can do with pandas ? Look here

Getting back to our inventory problem, our application started misbehaving. Unresponsive database requests. Turns out a lot of packages we used to build our inventory app have been deprecated. We were living under a rock.

Now with these new found super powers, I started to rethink the inventory app problem. We already had the data. Our problem was to somehow use python and pandas and built a new website so that everyone got access.

Enter 'streamlit'. I think it was truly serendipity that brought us together. One lazy Sunday, I read a tweet about how they needed an equivalent of streamlit for Julia. Julia a high performance and scalable data analysis language. It claims to combine the speed and scalability of C with 'Python'y syntax. This tweet stoked my interest in streamlit and I spent the next few hours being blown away by the simplicity of it. With streamlit, you could build 'data' apps i.e websites backed by any of your favourite data containers like csv, excel, sql etc. One of its many great features is , it runs on python. Data scientists have been using streamlit to build and show case their data experiments. If you want to see what is possible with streamlit, go here. [have patience, it takes a while to load]

This is already too long for a single read, let me cover my streamlit experience in the next post. Stay tuned.