Here are several examples with explanations and sample datasets for practicing SerDe. This post contains a compilation of all of the articles previously published on my site.
Hive SerDe – RegEx – Example1
SerDe: is short-form for Serializer/Deserializer. A SerDe allows Hive to read in data from a table, and write it back out to HDFS in any custom format. The SerDe interface allows you to instruct Hive as to how a record should be processed. Anyone can write their own SerDe for…
Hive SerDe – RegEx – Example2
In this post, we will be learning different patterns in input regular expressions itself. With a slight change in delimitation, the below dataset is similar to the previous one.Data200 6459CREATE TABLE sampleTab1(col1 string, col2 string)ROW FORMAT SERDE ‘org.apache.hadoop.hive.serde2.RegexSerDe’WITH SERDEPROPERTIES(“input.regex”=”([^ ]*)\\s([^ ]*)”,”output.format.string”=”%1$s %2$s”)STORED AS TEXTFILE;LOAD DATA LOCAL INPATH ‘Desktop/SpaceDelimter.txt’ INTO TABLE…
Hive SerDe – RegEx – Example3
Regular expressions are difficult to understand, but not in contrast to the data they contain. You will begin to love it once you understand the structure of the regular expressions. Often we use regular expressions for the complex data which is not in a pattern that can be handled by…
Hive SerDe – RegEx – Example4
“Build step by step. Push yourself, but not too hard. Learn, keep it fun.”It is definitely harder to read and comprehend 100 characters in a regular expression syntax. But if there’s a proper learning pattern, it’s not impossible to master. Below dataset is an example that we are increasing the…
Hive SerDe – RegEx – Example5
This is a continuity part of the series “Hive SerDe Regular Expressions”. As we discussed earlier, the Hadoop Hive regular expression functions define precise patterns of characters in the given string and are useful for extracting strings from the data and validating current data, e.g. validating the year, verifying the…
Hive SerDe – RegEx – Example6
This is a continuity part of the series “Hive SerDe Regular Expressions”. Most probably, the series will end before it hits 10. RegexSerDe uses regular expression (regex) for serialization/deserialization. You can deserialize data using regex and extract groups as columns. You can also serialize a row object using a string…
Hive SerDe – RegEx – Example7
The dataset below is one of the most popular and active dataset on the Internet. Regular expression has been written for this log by a few developers. However, our goal is to learn the regex patterns and see how many methods we can use to translate this log to a…
Hive SerDe – RegEx – Example8
Similar to the previous article, the dataset below is also common and a good example to learn regular expressions.Dataset (for copying purpose):82.133.98.11 – – [15/May/2019:07:47:12 -0900] “GET /org.apache.com/bin/param/Vars/ReadmeFirst?view1=1.5?view2=1.4 HTTP/1.1” 299 393982.133.98.11 – – [15/May/2019:07:55:17 -0900] “GET /org.apache.com/bin/view/Page/org.apache.comGroups?view=1.2 HTTP/1.1” 299 494982.133.98.11 – – [15/May/2019:08:11:19 -0900] “GET /org.apache.com/bin/param/Page/ConfigurationVariables HTTP/1.1” 299 5678982.133.98.11 -…
Hive SerDe – RegEx – Example9
The Relational Database Architecture is the simplest model, since it does not need any complex structuring and does not entail complicated architectural processes. The simplicity of SQL, where even a beginner can learn to run basic queries in a short time, is a major part of the reason for the…
Hive SerDe – Cannot validate SerDe Error
A return code or an error code is an alphanumeric or numbered code that is used to evaluate the essence of the error and why it occurred. It is difficult to define the problem in most programming languages or applications, even though there is some explanation of the error. It…
Big Data and SQL 
One comment