throwawaystress

How are you organizing your data for your RAG applications? I've searched all over and have found tons of tutorials about how the tech stack works, but very little about how the data is actually stored. I don't want to just create an application that can give an answer, I want something I can use to evaluate my progress as I improve my prompts and retrievals.

This is the kind of stuff that I think needs to be stored: - Prompt templates (i.e., versioning my prompts) - Final inputs to and outputs from the LLM provider (and associated metadata) - Chunks of all my documents to be used in RAG - The chunks that were retrieved for a given prompt, so that I can evaluate the performance of the retrieval step - Conversations (or chains?) for when there might be multiple requests sent to an LLM for a given "question" - Experiments. This is for the purposes of evaluation. It would associate an experiment ID with a series of inputs/outputs for an evaluation set of questions.

I can't be the first person to hit this issue. I started off with a simple SQLite database with a handful of tables, and now that I'm going to be incorporating RAG into the application (and probably agentic stuff soon), I really want to leverage someone else's learning so I don't rediscover all the same mistakes.

I recently finished a masters in statistics and took multiple machine learning courses in the CS department along the way. I also minored in CS in college and have been "software-engineering-adjacent" for several years, having worked on stored procedures for the backend of a CRM for a couple years, and for a consulting firm doing data analysis in R and Python.

Despite all that, I never hear back from places I applied to. I'd really like to work at a FAANG (especially Microsoft), and it feels like I'll never get there. I thought that by going to grad school I'd kind of jump-start my career in that direction, and instead I'm doing work in SAS for an insurance company. Meanwhile, I'm watching friends who went to a 3 month bootcamp get jobs as data engineers (which is what I'd like to do), and folks I trained at previous jobs go on to become directors of analytics.

What am I doing wrong? I kind of think I need to go back to the drawing board, as I wonder if I'm just not an attractive candidate.

How important is it for data scientists to have clean, modular, reusable code? Here’s my problem: while working on a project, I’ll start off in Jupyter notebooks, toying around with the data, doing some EDA, etc. Eventually I’ll pull out some of that code into functions in a Python file, and call those functions from the Jupyter. Neat.

The problem is, as I get more and more functions, I want to organize them more, make them more generalizable and consistent, etc. I’ll also get carried away with organizing files and source control, cleaning up my notes, and making documentation to explain what models/data/source files/results exist, what they mean, etc.

And then I realize I’ve been spending less and less time getting results, and more on this “overhead”. I struggle to balance the desire the rush ahead and get results with the compulsion to make the code “beautiful” and to have the project in the cleanest possible state. I’ve seen plenty of other projects with terrible organization, no documentation, and confusing, poorly formatted code. But if I’m not producing value, my neatness doesn’t matter.

All in all, I’m feeling pretty unproductive because of these habits. Any advice?

I’ve used experience both Python and R for years, in multiple jobs, and still, whenever I need to make a visualization that’s anything more complicated than a single-panel bar/scatter/etc plot, it takes tons of trial-and-error to get what I want.

This is in contrast to my experience coding in general, where I build things out, have some tests, and use the little bits of libraries that I need. Visualizations seem like a whole other beast, and are incredibly frustrating given how simple it seems to want to just “move this title over there!”, or “swap the order of these bars on all the facets!”.

On a related note, is data analysis in Python just generally supposed to be more of a hassle than in R? I like Python for general coding, but sometimes pandas feels like pulling teeth.

I'm looking for advice, commiseration, and perspective, if you have it.

I'm in my early 30s and an aspiring data scientist. I minored in CS in college, and was able to pick up a lot at jobs in my early 20s, including Python, R, and Oracle. I'm back in grad school now for a masters in statistics, and am hoping to transition away from Oracle programming (my last job) to data science.

I'm feeling overwhelmed by the sheer number of technologies out there that I "should" learn: Python and R (and their libraries), Spark, Hadoop, TensorFlow, and pytorch. You also need to know how to productionalize things: Docker, AWS, distributed databases, the latest web stuff (sockets?), etc. I could spend all the time in the world learning the new tech, and by the time I'm done, it'll be outdated by the new stuff. It's exhausting, and I don't know if I have the drive (or mental health) to commit to a career where I could be replaced when I'm 50 by a young lion fresh out of college who knows the latest stuff and is willing to work for pennies on the dollar.

When I was 21 I felt confident in my abilities; 10 years later, I don't at all. I feel like an idiot for how short a distance I've gone in my career. At this point in my career, I've never made over $75k and I'm jealous of my peers who are making close to $200k, and of kids out of college making over $100k. I think I know less than the undergrads at my university, and I think I give up on things more quickly now than I used to. At my first job, when my peers started to move on to grad school or higher roles other companies, I admired what they were doing, but I was afraid of the unknown at the time, in a toxic relationship, and had some decision paralysis about whether this was the right career fro me. Now I feel light years behind them. They're all in "senior this" and "senior that" roles, getting very highly paid, and I'm nowhere.

I'm not sure where to go from here. 30s are old in tech, and I don't think I can catch up.