Major Hadoop Release! Hadoop 3.0 is has dropped! There is a lot of excitement in the Hadoop community for a 3.0 release. Now is the time to find out what's new in Hadoop 3.0 so you can plan for an upgrade to your existing Hadoop clusters. In this … [Continue reading]
Is there a learning Roadmap for Data Engineers? Data Engineers are highly sought after field for Developers and Administrators. One factor driving developers into that space is the average salary of 100K - 150Kwhich is well above average for IT … [Continue reading]
Have you ever been running a script in from the HDFS command line gotten this error? Or running one of your favorite HDFS or Hadoop fs commands... Maybe you were trying to remember the HDFS URL and couldn't figure it out? Well it happens … [Continue reading]
What are the options for using Python in Hadoop? Python developers are looking to transition their Python Skills in the Hadoop Ecosystem. In a recent episode of Big Data Big Questions I answered question about using Python on Hadoop. Let's take a … [Continue reading]
How involved should Data Engineers be in learning Machine Learning Algorithms? For the past few years Data Scientist are one of the hottest jobs in IT. A huge part of what Data Scientist do is selecting Machine Learning Algorithms for projects … [Continue reading]
Which is better for Freelance Data Engineers Scala or Python? Picking up freelance gigs can be a challenge especially when just starting out. So which language is better for getting freelance gigs Scala or Python? In today's episode Big Data Big … [Continue reading]
Why read a fighter pilot book? Ever heard of the OODA loop? It's the basis for agile development. Observe, Orient , Decide, and Act (OODA) is the feedback loop coined by John Boyd. The point of the loop is to go through these steps repeatedly … [Continue reading]
How do you keep up with all the news going on in the Big Data community? Announcing the Big Data Beard Podcast, a Podcast devoted to Big Data news, architecture, and the software powering the big data ecosystem. Watch the video below to learn how I … [Continue reading]
How do you manage OneFS snapshots from the CLI? It's easy to use the isi snapshot snapshots commands. We have worked through setting up Isilon's OneFS Snapshots from the WebCLI in multiple Isilon Quick Tips. Let's turn our focus now to setting … [Continue reading]
Let's say you have two sets of structured or unstructured data. How to combine two sets of data (relations) in Pig Latin? Look at the example below. If you wanted to combine the cereal and price data sets what would you use? Pig Latin offers Joins … [Continue reading]
Want to know one of the hardest part for me installing Hadoop with Ambari? Setting up Passwordless ssh for all nodes so that Ambari Agent could do the install. Looking back it might be a trivial thing to get right, but at that time my Linux skills … [Continue reading]
“Is it possible to build a prediction model based on real-time processing data frameworks such as the Kappa Architecture?” Yes we can build models based on the real-time processing and in fact there are some you use every day.... In today's … [Continue reading]