Yesterday I spent 4 hours regexing out HTML data from not so well maintained data sources and screamed at my computer, because I was not able to find patterns. I know, please don't ask. This is not high level programming. We're talking .replace and .match, with the occaisonal if/else over hundreds of sets. My colleague calmed me down and took the time to help me. Bless him. This colleague has multiple projects within the company, because, to me, he is a modern web dev: Building scrapers from scratch with Node, uses multiple frameworks, you get the jist.
My path in this company should have been: Learning the code he writes, so I can help him maintaining the software and be on standby, if something breaks. This won't be happening in the forseeable future.
My problem: I do not understand his code. Or anyone elses. While I basically use if/else, he uses ternary operators. I write functions by hand, the uses arrow functions and symbols I can hardly understand. Multiple generations of Javascript foundation changes and yearly new crazes are lying between us. We do not have the time to code review his projects. Also, he should not be my teacher. But: I want to be able to help him and not be a burden.
I tried $learnToCode websites, YouTube tutorials, bought "Master Classes" from the usual suspects. But I'm literally drowning in the market of javascript. There is no clean line learning "This is the 2000s" to "Welcome to the year 2022" javascript.
For example, I do not get Typescript; I do not understand, how big projects can use Node, with it's thousands of dependencies, while nobody cares, that 20 to 100 of them are deprecated, or in short: I do not understand, how this whole eco system works. But I want to. Do you have any suggestions on how to unclutter this mess, which absolutely overwhelms me?
If you care to learn modern JavaScript, which you should, check out Eloquent JavaScript [0]. It’s a free web-based ebook providing an introduction to modern JavaScript.
Also, stop with the negative narrative. You are overwhelmed, not helpless. Instead of telling yourself you don’t understand code and giving up, just Google said code. It’s easier than ever. In a few weeks, you won’t need to reach for Google. If you encounter a dependency you don’t understand, try to find some documentation.
Accept that there are black boxes that “just work” despite being deprecated or lacking documentation.
Your job is not to learn the history of JavaScript and how it evolved from your understanding. Your job is to get up to speed with modern JavaScript enough to make functional programs to solve the business’ needs.
This is pretty integral to my job and I’ve been in the industry for 11 years or so. You can’t know everything right away and it’s overwhelming to try to attempt to do so. Learning certain parts has its time. For now just focus on what is in front of you.
“Method X produces Y and that’s what I need” next step.
The #1 thing you can do for yourself first though is accepting that what you learned 20 years ago probably means dick. JS has radically changed and that's OK. Go into this book with that mindset and you will come out a champ.
https://news.ycombinator.com/item?id=31660869
Ask HN: Best alternative jobs for “outdated” skills with small websites/apps?
> you should be applying at small to medium-sized companies that are not tech companies. There's a ton of work in these companies and hiring is more focused on solving business problems
It's OK. Just tackle it one step at a time. Forget Typescript for now. Don't use third party libraries until you need them. Arrow functions are optional. Ternaries are a form of jargon/an optional shortcut.
If the other developer's code is too hard to read, that's okay, set it aside for now and you can come back to it when you have a little more experience.
Start with the problem you're actually trying to solve (which isn't terribly clear, sorry). You have XML and HTML you're trying to parse? Which is it? (XML will be much easier than HTML, because it's much stricter. HTML is notoriously loose, which is why there is only like three real browsers in the world.) What are you trying to do with the parsed input?
Regex probably isn't the best tool to use here. There are specialized parsers for these data formats that we can help you use, if we better understood your needs.
Ask. Ask a lot. Find a mentor if you can. Ask on Stack Overflow if you can't. It's totally okay. Modern JS is a new and exploding field and we're all pretty ignorant about it. Fortunately, 80% of it is relatively easy to learn, and the remaining 20% you can usually ignore or work around. Just take it one step at a time... it's not you. It really just is a chaotic ecosystem.
About my regex problem: This is a structual mess. JSON/XML with HTML code in the data fields. We process them and send them to multiple job boards. Our clients mainly use HRM software or some CMS, some of which are only able to spit out whatever HTML is displayed on their career sites. This code often does not even have classes or IDs. Most of the times we are dangling together whatever is between two headlines, praying those won't change. But they do, because the recruiters put fields, where they not belong. I call myself code cleaner, not web dev nowadays. We are not able to use APIs, because the receiving job boards either don't offer one, the client doesn't, or it's just not worth it financially.
I will take a step back and reevaluate my situation.
I’d suggest rather than attempting to do big regexes that capture a bunch of stuff in one call, break it down to a bunch of smaller, more targeted calls - one call to capture the text of the whole record, another with 3 variants to get the title, another with 2 variants to pick up a tag line, etc.
Most web devs don't need to do that, and that you're willing to tackle it at all just shows how willing to learn you are, despite the frustration.
If you hate this situation, it's totally understandable lol. That kind of work has all the tedium of dealing with someone else's arcane data format, and none of the joy of seeing your creativity come to life. Some people love that sort of work, and specialize in it, becoming backend people or DB engineers or data scientists or the such, but it's not usually what web devs are known for (who tend to focus instead on UIs and some level of design and interactive stateful apps). Nothing wrong if ETL just isn't your cup of tea. I'd go crazy if I had to do that often, too.
Anyhow, if I'm understanding you right, you have HTML embedded in either JSON and/or XML. Do you know what "escaping" is in the text embedding sense? Like if you have quotes inside quotes, or tag brackets inside tags, how to separate each layer of embedding? If your JSON and XML files are cleanly escaped, you should be able to (as a first step) just iterate through the files and get the HTML parts out (without regex).
Like if the HTML is just a data string inside JSON, you can transform the JSON into an array of HTML strings using array.map() or object.values.map().
In the XML, if the HTML is stored in CDATA fields, you can access it using an "XPath" selector... you know how CSS has selectors that let you say headings should be styled one way, paragraphs another? XML has its own selector language that lets you directly target a certain node inside the document, without using regex, by specifying the hierarchical path that takes you there (like a CDATA inside a description inside a job inside a company, or whatever). Although there is a learning curve to XPath, it is much more suited to the task than regex, because the regex can't easily account for the complexity within XML (especially when there's nested layers).
It would help if you can post some example snippets, but that might be better suited for Stack than HN (though feel free to link to it here).
Once you have the HTML out, then you can run it through a sanitizer -- that's an optional step, but would let you strip out unnecessary divs, old font tags, whatever, keeping old basic formatting (headers, paragraphs, links, bold, etc.) which should be much cleaner to hand off to your clients. That would be much easier to embed on someone else's site vs a scraped page with all the HTML mess from someone else's framework.
I know there is a lot of complexity in each of those steps, but there are great tools and documentation for each step of the way. That's just to get you started.
At the end of the day what you're doing isn't really a Javascript issue at all, it's just a different kind of work that Javascript happens to be able to handle if you really need it to (but so can Python or Java or specialized command line tools like jq). It's a different body of work, which is why your casual web dev skills aren't providing easy answers. It's OK! You can learn it once and make it work (and then decide never to do that again, like I did lol). Or switch tracks, totally up to you :)
But feel free to ask here or on Stack if you have followups!
Go straight to the last book in the series, called "ES6 & Beyond" [1] which specifically covers the most important features of JavaScript that have appeared since you last worked with it.
For hands on practice with newer JS features, the best resource is Execute Program's "Modern JavaScript" [2] course. Once you work through that, move on to their TypeScript Basics course [3]. Both are amazing. Execute Program is the best tool for learning a programming language that I've ever used.
[0]: https://github.com/getify/You-Dont-Know-JS/blob/1st-ed/READM...
[1]: https://github.com/getify/You-Dont-Know-JS/blob/1st-ed/es6%2...
[2]: https://www.executeprogram.com/courses/modern-javascript
[3]: https://www.executeprogram.com/courses/typescript-basics
Having a wiki is a good tip, I already have started to maintain a small database with snippets I use often. But it's just that. My next focus will be having a project which does not have to many features. Baby steps it is.
Here's what I suggest:
1) Get the language basics really well.Like really well.
2) Move on to more complex concepts,again don't skim read, but do again and again and again.
3) learning from code written by a more experienced colleague may not be the best option. He may have a certain style, approach and etc. If you can't understand the code , you won't learn much from it.
4) Get yourself all ' you don't know JS' series and devour it. The books aren't easy read if you really try to understand the concepts but way better than some random guy on YouTube covering the subject in 2 min video.
5) I've recently did a private job with Regex,which I barely know. It was painful.. but it gets easier. Again, start with basics and build up, instead of trying to run complex queries and not fully understanding them.
6) Be methodical. Don't jump from one tutorial to another,from one book to another. Read one book,do all the exercises, only then move to something else.
P.S: Regexing HTML is by far the worst thing a dev could get to do, that's why most would advise against it. Don't be hard on yourself.
Good luck!
For the syntax thing you might wanna pick out one particularily "bad" one and have your collegue explain it to you. Takes him maybe 15 mins and will safe you a good chunk of time. Where I work asking is expected. I ask our Network/Linix guru for things all the time. He asks me as well when I use something he does not know. Pretending to know everything won't get anybody ahead.
As for the regex hell: I always found tools that visualize the matches in realtime useful (like this: https://projects.verou.me/regexplained/). Just make sure the flavour of regex matches yours.
Edit: I suspect your collegue relies heavily on ES6 syntax (as I would). A overview of that can be found here: https://www.w3schools.com/Js/js_es6.asp
Make sure to give yourself a little time to play, e.g. try to replace something where you would use a for loop with an Array.forEach loop or something like that.
The debugger is your friend. Figure out how to get it attached anywhere if you haven't already.
That means running node with --inspect-brk=<port number> that also means figuring out where spawns are.
This exercise will greatly aid you in overcoming any surprising control flow going on by allowing you to single step the program. JS is one of those languages I just have to cling to my debugger in.
Two: node has two "ClassLoaders"(CommonJS/ESM). They don't mention squat about the differences between them until the middle of the nodejs documentation.
Start learning dependencies one by one. There's really no other way around it.
Avoid using regex for XML/HTML. Try something more akin to SAX parsers. Seriously. There:s way too much pain in doing regex XML processing. You could literally rewrite the damn thing in java faster than getting a shoddy, flaky JS monstrosity reliable.
God help you if they're doing weird Async/Event nonsense.
Keep cracking at it, but be aware, I think JS has broken my brain, because I just don't even like programming anymore since I started dealing with the language and all the necessary tranpilation/toolchains.
I don't know if I can offer any suggestions beyond just Googling syntax you don't understand and experimenting with it until you feel you understand it. From there it just a matter of time before you can glance your eye over the code and understand what it's doing without needing to decipher every line. I just want to say you're not alone. I think most frontend devs have gone through something similar at some point over the last decade - I know I have.
As for deprecated dependencies, that's a huge problem. Senior devs generally do care and will be selective around dependencies. One of the problems is that the ecosystem is evolving so quickly. Libraries come into and out of fashion almost over night so frequent maintenance is needed to keep dependencies up to date. My guess is that either devs are not putting enough thought into what dependencies they're using, or your not currently assigning enough time as a team to review & maintain your existing dependencies - this is definitely something worth raising with your team imo.
The last thing I'll say is that the use of ternary operators are something I'm very critical of and it's interesting that you mentioned them in your post. A lot of devs seem to have this mindset that if you can convert 5 lines of code to 1 or 2 that's always a good thing, but compressing 5 lines of code to 2 often comes at the cost of clarity and readability. It sounds like you might need to think about coding standards with your team. Is part of the problem here that the team is writing confusing / messy code, and it's that coupled with your lack of knowledge of modern JS that's resulting in the stress you're feeling here?
I would look for the old-ish articles and tutorials that introduce es6, for example: https://babeljs.io/docs/en/learn/ idk if you need to go earlier, but surely there are tons of articles on each update to the language.
I think it's good to realize most of it is not new, it's largely the same stuff, just slightly different and more convenient.
I believe that's your main problem. You are seeing a foreign language kinda close to what you know and expect to understand it right away (while it's the same "Javascript", using different writing ways is more akin to another dialect). It takes a bit of time and practice to familiarize yourself with it. Try to decompose it until you become fluent. When you learned to read, you would read a word one letter at a time. Do the same: read the code one operator at a time, until you understand what's going on and how they work together.
For example, a ternary is just a compressed if/else with a different syntax. An arrow function is just an inline function with a different syntax. etc...
> I do not get Typescript
Typescript is Javascript with type safety. In Javascript, if you pass a number to something that expects a string, it will convert without any complaint. Typescript will complain that it's the wrong type. You can go much further than this with complex types that have to be passed correctly.
> thousands of dependencies, while nobody cares, that 20 to 100 of them are deprecated
Some people care, but for most dev, it's a black box that just works. It's frankly not something to worry about when starting.
> I do not understand, how this whole eco system works
If I had to sum it up, it would be building on the shoulders of giants, over and over again, at a break-neck speed, valuing improvements over backward-compatibility. I'm personally not a fan of it, but that's how it is.
Nowadays the Internet is literally drowned in resources and you may quickly feel overwhelmed by everything. The trick here is to choose only a handful and stick with them for the time being.
Some of my personal favorites for starting out are https://www.freecodecamp.org/ to get your feet wet and up to speed with JS. You will build projects and explore some of the fancier syntax you saw.
You may also want to start building a strong CS foundation at some point. This is going to be really beneficial as you move forward. Arguably one the best (and completely free) resources for starting out is CS50 - https://cs50.harvard.edu/x/2022/.
Also don't follow courses blindly, take notes and try to implement things yourself. Moreover, start tinkering with some projects as soon as possible.
Good luck.
MDN is a very reliable source of reference for all of JavaScript.
Have you tried using an XML parser instead?
- https://stackoverflow.com/a/1732454
With that aside, for parsing HTML with node, use cheerio. It has a jQuery-style API and IIRC is more lightweight than jsdom (which is what I'd suggest if cheerio isn't enough). If you're parsing HTML in a browser, you don't need any libraries, use a DOMParser. Arrow functions are just normal functions but with `this` automatically bound to the `this` of the scope they are in. MDN is your friend.
I can’t even parse that enough to decide if it should get an up or down vote and I haven’t even tried regex.
EDIT: I see
Regular expressions are the worst.
Also if someone scraped web pages and then stored the HTML, that seems like a bad idea. You would want to actually extract the data at that point.
Also regular expressions are the worst way to extract data from HTML.
I think the real problem is the business model and technical approach. It almost sounds like you are scraping a massive number of websites using regexes, which is guaranteed to break somewhere on an almost daily basis since developers change the HTML and regexes are very brittle.
You are probably taking advantage of a lot of other people's work, possibly quite a bit of it copyrighted and certainly most of it NOT curated with the idea of some other company profiting off of it.
So you deserve your pain.
But don't blame JavaScript or lack of familiarity with new versions when it's all of your other life choices that are the real problem.
And not if/else type of, but rather oop, software design and organization of code kind of things.
What will help is to master a single decent beginners course that focuses on applying fundamentals to building practical projects. And it’s better to be python or ruby based course (to minimize friction and infra pain).
Don’t jump on anything else until you finished and have some coherent picture.
Only then you should proceed with learning other concepts bit by bit.
Js world looks complicated for beginners because of a diversity of tools and techniques like bundling, transpiling etc.
It is not easy to comprehend and it is not necessary, you just focus on your specific goals and moving from a goal to another goal you gradually build holistic picture.
You don’t just “get” Typescript etc. Im still learning typescript after 2 years professionally working with it.
Are you suggesting a different technology stack for the fundamentals? Others seem to go the same way, but they are suggesting it within the ecosystem of JS, just not with the overhead of the new technologies.
It's not that important in general. But I assumed that if you're already overwhelmed by JS infra, you might get a smoother initial experience with less messy yet still interpreted dynamic language like ruby.
The ternary operator is born out of the need for an expression form of if-else. Some languages have if-else already implemented as an expression so they don't need a ternary operator.
Typescript exists to provide a static-typing discipline to javascript, if you don't care about this obviously you won't see the importance of typescript.
From day 0, a javascript project starts as a goal driven project of finishing things. Write hacky code for now we will figure them down the road. From that point on the hacks keep piling on top of hacks.
Another issue is "syntax overload". Javascript allows for doing one thing ten different way. So keep doing things your own way as long as you don't have to review other people's code. The new things JS brings out doesn't bring significant syntactical improvement and yet adds to the syntax overload syndrome.
XPath/XSLT3/Xquery are full-fledged functional programming languages that specialize in querying and transforming tree data structures. They make it so easy to work with XML and JSON. You don’t have to hurt yourself using the wrong tools!
You are very lucky nobody at your company caught on yet. My suggestion is hire me as a subcontractor - I will do your job for half your wage, you will have plenty of spare time to do whatever.
knowing the internal modules of nodejs such as fs, events, process, streams
knowing utility modules like lodash, asyncjs, rxjs
these could help.
but the fact is if your workmate is writing unreadable and unmaintainable code, hes the problem, not you.
you need linters, code conducts, test sequences, dependency upgrade schedules, node runtime update schedules, things like that.
i once left a high paying job because a colleague there is code golfing on our code. the team suffered more when i left lol.
in life choose your battles, dont stick with a shitty one. when applying for work, it is not just you applying for their company, it is you screening if their tech stack and coding practices is the right fit for you. screen hard.
> but the fact is if your workmate is writing unreadable and unmaintainable code, hes the problem, not you.
That's a stretch, especially since OP also says:
> Or anyone elses.
Idiomatic new js/typescript is a completely different beast from old javascript. For all intents and purposes, it's a different language entirely. No one would be expected to pick it up in a minute. Sounds to me like this guy has decades old knowledge and doesn't know where to learn the new stuff, since the target is moving from one year to the next.
You yourself are suggesting today, that he learns libraries that are no longer necessary, like lodash.
A bit like learning Golang when you know C99.
For example, there is an excellent introductory FP course on Coursera by Martin Odersky ("Functional Programming Principles in Scala"). It happens to be based on the Scala language syntax, but most of what you learn there will be directly applicable in other FP languages too. For me, taking this course was a game changer ~10 years ago. Highly recommended.
Once you understand the basics, picking up stuff like "arrow functions" (lambdas) will be just a matter of adjusting to the JS syntax.
Javascript is a mess, especially at smaller shops which may face pressures to ship projects to a client that work, regardless of the state of the code.
Breathe. You're overwhelmed and that's ok. Just take it day at a time, line at a time and don't be afraid to have to google the same thing over and over again. It eventually sticks.
Don't be afraid to ask for help after you've exhausted yourself trying to solve something. Lots of other great advice in this post too. You'll get there.
What you are describing is commonly known as ES6 or ES2015 syntax. The language got some features at that time that solve gotchas related to using JS and arrow functions are a great example. These features make JavaScript easier to work with if you use them, and learning the most important ones doesn't take more than a day. https://www.boardinfinity.com/blog/top-10-features-of-es6/ After 20165, they only released like 2 features per year, and most of those haven't had the same adoption.
Arrow functions are a great example, as they don't have their own `this` and it makes the `this` keyword work how you think it should.
> ternary operators vs if/else
Sometimes new features make sense in a certain context. If/else makes sense if your code has actual branching, but if the assignment of a single variable is all you need to branch for, ternaries are great.
``` // classic JS let variable;
if (someCondition) { variable = 'foo'; } else { variable = 'bar'; }
// with ternary
const variable = someCondition ? 'foo' : 'bar'; ```
> We do not have the time to code review his projects.
That's crazy.
Ternaries are also a great example of a feature that can be abused. As conditions get more complex, ternaries become more opaque and are a worse choice. More concise code isn't necessarily better. This can be true for a lot of JS. Fancier code isn't better if the rest of the team can't understand it, and if you don't have a review system, you can't blame him for writing in his own accent and you not being able to understand. Developers on a team need oversight. I catch bugs my teammates write, and they do for me too. If you can't understand it when the PR is made, you're not going to understand it later.
So my answer to you is to try to start doing code review. Reading code is important, and a PR should be focused enough that you can track what a feature or bugfix is actually doing. If you can't, you should be speaking up.
> Also, he should not be my teacher.
Why not? What dev wants their teammates to struggle? I love helping more jr developers, and I've never had someone on any team bristle when I asked them to help me.
