Which is better for Freelance Data Engineers Scala or Python?

Picking up freelance gigs can be a challenge especially when just starting out. So which language is better for getting freelance gigs Scala or Python? In today’s episode Big Data Big Questions I answer a question about freelance options for Data Engineers. Watch the video below to find out about Scala and Python Freelance options.

Transcript – Python vs. Scala Freelance Data Engineers Video

Hi. I’m Thomas Henson with thomashenson.com. Today is another episode of Big Data Big Questions. Today’s question focuses around Python versus Scala for freelancers. Today, we’re going to tackle that question. Find out more right after this.

Today’s question comes in from our Big Data Big Questions series. If you have any questions, go to thomashenson.com Big Data Big Questions. Submit your questions. That’s where today’s is coming from.

Today, our question’s coming in from Gill Taso [Phonetic]. I hope I’m saying your name right. I hope I’m not mispronouncing it. Thanks for following, and thanks for asking a question, Gill.

Today, Gill’s question is, “I saw your video on learning how to become a data engineer, and it was very interesting.” Thank you, Gill. “I subscribe to your Twitter and your YouTube channel.” Well, thanks again. That’s awesome. That’s how you can make sure that you never miss an episode. Currently Gill is a junior data engineer who works mainly with technologies like Impala and Hive, with a little bit of Spark SQL.

Said, “I would like to improve my skills in Spark. I have intermediate skills in Python, but as I review the industry, it seems it’s better to learn Scala, which is very tough, at least for me. Do you think it’s better for me to sharpen my skills in Python or transition to Scala now? Also, I’m looking for ways to be a freelance data engineer in the short-term, maybe two to three years, because I’m looking more for more flexibility. Do you have any advice to give me for training, skills, and technology?”

Thanks for the question, Gill. That’s actually a two-part question, I feel like, so I feel like there’s a couple different ways that we can look at it. First, the question is, for a junior data engineer that’s working a really good intermediate skills in Python, but should you transition more to Scala? The second part of the question is, what should you do for freelancing?

There’s two different ways to look at it. If we look at it in your current career, where the industry’s going, is Python a viable option, or should you start to do Scala? Or, where’s that market going to be at for freelancing? I think those are two different things. What we see for freelancing and then what we see for industry and in your career progression, and especially if you’re in the Spark community. It seems like you’ve got, hinted around that, you’ve been working with Spark for a little while, and starting to get into more of the Spark, and that’s why you’re starting to look at Scala.

We’ll break it down real quick. First, my thoughts on Python versus Scala, just in the industry, I think that, with you saying that you have intermediate Python skills, if you’re looking at Spark, and that’s what you’re really wanting to specialize a little bit more in, I don’t think it would hurt you. I think it would be beneficial for you to start going ahead and looking into Scala, and maybe doing some research, there.

I definitely wouldn’t push back, and say, “I don’t need those Python skills anymore.” Both of them are interpreters. You can write both of those inside of Spark, so you’re covered there. There a little more functionality, it seems like, and more documentation around Scala. Part of it is because it’s expanding out that Java, so it keeps a lot of people from having to write those Java jobs versus just using the Scala interpreter ripple or just using the functions there.

From where I see it, and where you want to go, and especially if, you being a junior, saying that you’re a junior, data engineer, if you want to progress, I don’t think there’s a problem with you learning Scala. I just wouldn’t say that you really need to jump to it now.

It’s hard for us to, as technologists, as engineers, as developers, we love playing with new toys. Any time there’s a new language or a new toy, a new kind of technology, whether it be some kind of project with dockers containers, or Mesos, or Yarn, anytime there’s something new, we always look at that, like, “Oh, man!” That’s a shiny new object we want to play with.

What should we do from our career perspective? I don’t think you have to go all in on Scala. I think that, even with Python, and where we see Python now, and just look at the different open source projects and everything that’s going on with Python. I think you’re set there. One recommendation that I was given early in my career, that I want to pass along to you, is being a junior engineer, saying you have intermediate Python skills, what I would start to do is, I would start to maybe start looking at specialization.

Look at a couple different open source projects, maybe get involved, that revolve around Python. That can help you build your foundation, your portfolio, and yourself as a career. You still have options.

You’ve heard me talk about it before in the data engineer video. Take 30 minutes. Take 30 minutes a day and maybe two to three times a week, mix in some Scala research in there. That way, you’re learning it, and you’re starting to get it, but you’re also still contributing to your Python skills, which, like I said, you’re at the intermediate phase. There’s a way for you to specialize, because Python’s a pretty big language, and so there are different specializations that you can do. I would look at that.

The second part of the question that’s a little bit different, and I’m going to have somewhat of a different opinion here, is you’re asking about options for freelancing. Which one should you focus on if you want to freelance?

The way that I would look at that is, there are three important things that I look at for freelancing. If you’re freelancing, you’re focused on your brand. That’s how you’re going to get freelancing jobs. That’s how you’re going to be able to market yourself as a freelancer and continue to get that revenue there.

That starts with specialization. Like I was talking about a little bit with Python and starting to pick a specialization, I would really focus on it if that’s what I wanted to do. If I wanted to get into freelancing, wanted to be a freelance data engineer, you really want to start focusing and start specializing.

That’s not to say that you’re not going to pick up other skills and continue to learn those, but I think specialization, especially when you’re new to freelancing, is going to be very, very important, even more so important than just in your current career, where you’re working, maybe, at your company. Specialization is important there, but not as important as a freelancer. So, the second thing that’s really big with freelancing is the market. What is the total market need? Just looking based off that, if you were looking and asking that question, “Hey, I want to be a freelancer,” Python or Scala? There’s going to be a lot more jobs that are available right now, a lot more gigs to be able to get, that are going to revolve around Python, and I think that trend’s going to continue. I don’t think that Scala’s going to overtake that market.

Even segmented in the big data area, too, so even if we got outside of what’s going on with the big data community, I think those jobs are still going to continue to be maybe two to three times more for Python developers.

You’re looking at the brand. You’re looking at the market need, but then you’re also, the third key for freelancing is your reputation. Being new, you’re not going to have a lot of reputation, so what are some things that you can do to be able to get your name out there, and also get involved in some jobs?

It’s going to go back to the open source. I would look at some open source projects, maybe around the areas that you want to specialize in, and just get involved. Getting involved in those is sometimes just as easy as joining the user discussion list, and maybe answering some questions there. Getting on the developer list, and just maybe reading the emails, and see how it’s going.

I’m telling you, just getting on those email lists and seeing what’s going on, and as you start to contribute on there, that’s a great way for you to be able to, one, have code that you can point back to, or that you’ve been a part of the community. Two, that helps your reputation, so it’s going to be easier for you to find jobs. There’s a lot of times that you’ll see jobs, or requests for podcast interviews, or just a lot of different things, all come from that open source email list. I would get involved there.

Also, too, another thing that you do, just being new to the area of freelancing, is maybe you look for one of the gig, one of the freelancing places online, and start doing some of those jobs. A lot of times, those jobs, because you’re just starting out, they’re going to be very competitive, and you’re going to maybe be doing it for 10 or 20 bucks, or 30 bucks an hour, for a project, but that helps you build that portfolio. I wouldn’t focus too much on being a part of that, and saying that’s going to be your revenue for freelancing, but I think it’s a great way for you to get started, and start to see how it works for freelancing, and if it’s something that you’re interested in, too.

It’ll help you build that reputation, but I think the open source, long term, is going to really build that reputation for a lot longer than even the gig sites, and some of the other things. Three things for freelancing. Definitely, for that question around Python versus Scala, I would continue to focus on Python for your freelancing gigs, because I think that’s going to be your fastest way to be able to build that specialization. I think the market need’s there, and then also being able to brand yourself as well. That’s today’s episode of Big Data Big Questions.

Make sure you subscribe, so that you never miss an episode. Also, go, if you have any questions for Big Data Big Questions, check it out, thomashenson.com, I’ve got a place out there for you to just fill out. I can answer any questions you have.

Thanks again. See you!