In this blog post, the first of a series, we’ll present a selection of non-technical but common issues that data professionals face in organizations. We will offer a diagnosis of “why” such problems arise and present some potential solutions.

We aim to contribute to the online data discourse by broadening the discussion to non-technical but important challenges to getting data work done, and to normalize the idea that, despite the decade-long hype around data, the day-to-day work is filled with common interpersonal and organizational challenges.

This blog post is based on a talk given by myself and Cesar Legendre at the 2024 meeting of the Royal Statistical Society of Belgium in Brussels.

Communicating About Data

In this first blog post, we will focus on problems that can occur when communicating to organizational stakeholders (colleagues, bosses, top management) about data topics.

Below, we present a series of short case studies illustrating a problem or challenge we have seen first-hand. We describe what happened, the reaction from organizational stakeholders or clients, a diagnosis of the underlying issue, and offer potential solutions for solving the problem or avoiding it in the first place.

Case 1: The Graph Was Too Complicated

What Happened

• In a meeting among the data science team and top management, a data scientist showed a graph that was too complicated. We have all seen this type of graph - there are too many data points represented, the axes are unlabeled or have unclear labels, there is no takeaway message.

The Reaction

• This did not go over well. You could feel the energy leaving the room, the executives start to lose interest and disconnect. Non-data stakeholders started checking their phones… If the goal was to communicate the takeaways from a data analysis to decision makers, it failed spectacularly.

The Diagnosis

• There was no match between the goals of the stakeholders (e.g. the management members in the room, who were the invitees to the discussion) and the data scientists presenting to them. The role of the executive in these types of discussions is to take in the essence of the situation at a high level, and make a decision, provide guidance or direction to the project. An inscrutable graph with no clear conclusion does not allow the executive to accomplish any of these tasks, and so they might rightly think that their time is being wasted in such a discussion.

The Solution

• The solution here can be very simple: make simple, easy-to-understand graphs with clear takeaways (bonus points for including the conclusion in writing on the slide itself). This is harder than it sounds for many data professionals, because we are trained to see beauty in the whole story, and we often want to present the nuances that exist, particularly if they posed challenges to us in the analysis or cleaning of the data.

• However, especially in interaction with management stakeholders, you don’t need to show everything! We have found that a good strategy is to focus on the message or conclusion you want to communicate, and to show the data that supports it. This isn’t peer-review – you have explicitly been hired by the organization to analyze and synthesize, and present the conclusions that you feel are justified. In a healthy organization (more on that below), you are empowered as an expert to make these choices.

Case 2: The Graph Was Too Simple

If complicated graphs can be the source of communication issues, then simple graphs should be the solution, right? Unfortunately, this is not always the case.

What Happened

• In a meeting among the data science team and top management, a data scientist presented a graph that showed a simple comparison with a clear managerial / decision implication.

The Reaction

• Rather than being rewarded for showing a clear and compelling data visualization that had decision implications, the executives’ response was: “This isn’t good enough.” One executive in particular insisted that they needed drill downs, for example per region, per store within region, product category within store, etc.

• The marching orders from this meeting were to produce the additional visualizations and send them to management. However, there was no plan for following up on this additional work or any commitment for any resulting actions. The data team left the meeting with the feeling that this was just the beginning of a long and complicated cycle in which an endless series of graphs would be made, but that nothing would ever be done with them.

The Diagnosis

Why does this happen? There are at least 2 diagnoses in our opinion.

• The first explanation assumes good intentions on the part of the management stakeholder. The simple truth is that not everyone is equally comfortable with data or with making data-driven decisions. Among some stakeholders there is a feeling that, if they had all of the information, they would be able to completely understand the situation and decide with confidence. A related belief is that more information is better (this is why organizations build an often overwhelming number of dashboards, showing splits of the data according to a never-ending series of categories). Furthermore, some decision makers want to make many local decisions – e.g. to manage each country, region, store, or individual product in its own way, though this can quickly become unmanageable.

• The second explanation does not assume good intentions on the part of the organizational stakeholder. To executives who are used to simply going with their intuition or gut feeling, being challenged to integrate data into the decision-making process can be a somewhat threatening proposition. One strategy to simply ignore data completely is to leave the analysis unfinished by sending the data team down a never-ending series of rabbit holes while continuing to make decisions as usual.

The Solution

The solution here depends on the diagnosis of the problem.

• If one assumes that the issue is a desire for mastery and a discomfort with numbers, a helpful strategy is to re-center the discussion on the high-level topic at hand. In the context of the graph shown above, one approach would be to say something along the lines of: “What are we trying to do here? We conducted an experiment to see whether using different types of promotions would impact our overall sales. We saw that customers who received Promotion A were 1.5 times more likely to buy the product than customers who received Promotion B. Our testing sample was representative of our customer base, and so the analysis suggests that we expand the use of Promotion A to all of our customers.”

• If one assumes that the issue is a desire to ignore data completely and continue on with business as usual, there’s not much that you as a data professional can do to change the managerial culture and address the underlying issue. A single occurrence of such behavior could just be a fluke, but if you keep finding yourself in this position, we would encourage you to reflect on whether you are in the right place in your organization, or in the right organization at all. If all you do is chase rabbits, you will wind up exhausted, never finish anything, and your work will have no impact.

Case 3: When A Graph Leads to Stakeholder Tunnel Vision

What Happened

• In a meeting among the data science team and management, a data scientist presented a simple graph showing some patterns in the data. (The graph here is a correlation matrix of the mtcars dataset, included for illustrative purposes).

The Reaction

• For reasons that were unclear to the data team, a manager exhibited an odd focus on a single data point and asked a great many questions about it, derailing the meeting and the higher-level discussion in the process.

• An example of what this looked like, applied to the correlational graph above:

  • “Ah so the bubble with Weight and Weight is a big blue circle. Well, this Weight score is very important because this is one of our key metrics. So why is the dot so blue? And why is it so big? And what does it mean that the Weight and Weight has one of the biggest and bluest bubbles, compared to the other ones?” (Note to the readers: the big blue circle indicates the correlation of the Weight variable with itself; the correlation is by definition 1, and explains why the bubble is big (it’s the largest possible correlation) and blue (the correlation is positive). There is nothing substantively interesting about this result.)

The Diagnosis

• As in the case of the simple graph above, the problem might stem from discomfort on the part of a stakeholder who is not at ease with understanding data and graphs. Although this feels unnatural to many data professionals, the simple truth is that data makes many people uncomfortable. It is possible that this extreme focus on a single detail in a larger analysis reflects a desire for mastery over a topic that feels scary and uncontrollable.

• Another potential explanation concerns the performative function of meetings in the modern workplace. In an organizational setting, meetings are not simply a neutral environment where information is given and received. Especially in larger companies, and in meetings with large numbers of participants, simply making a point or digging into a detail is a strategy some people use to draw attention to themselves or show that they are making a contribution to the discussion. (Even if the contribution is negligible or counter-productive!)

The Solution

• As described above, the recommended solution here is to take a step back and recenter the discussion. In the context of our correlation analysis above, we might step up and say – “What are we trying to do here? We are showing a correlation matrix of the most important variables, and we see that the correlations in this plot show that larger and more powerful cars get fewer miles per gallon (are less fuel-efficient). In our next slide, we show the results of a regression analysis that identifies the key drivers of our outcome metric.” And then move on to the next slide in order to continue with the story you are giving in your presentation.

Case 4: When the Data Reveal an Uncomfortable Truth

What Happened

• In a meeting with top management, a data scientist showed a simple chart of a metric by location, and made a simple observation about the pattern shown by the data – “ah, our metric is lower in Paris and Berlin vs. London.” This was a relatively simple observation, very clear from the graph, and the data scientist thought that this was some basic information that management should be aware of.

The Reaction

• The reaction on the part of the senior stakeholder, however, was explosive. There was a fair amount of yelling and shouting, not particularly related to the topic at hand. It was as if a toddler in a suit was having a meltdown. The meeting ended quickly, and the executive made it clear he did not want the project to continue or to see the data scientists again.

The Diagnosis

• The executive was aware of the situation that the data visualization revealed. However, the facts that the data made clear were problematic for the executive’s scope of responsibility. Truly addressing the issue would have been difficult if not impossible given the organizational structure and geographic footprint. Rather than acknowledge the problem and work to solve it, the most expedient solution was to “shoot the messenger.”

The Solution

• There’s not a lot that a data scientist, or even a data scientist leader, can do in these situations. Nevertheless, shouting in meetings is completely unacceptable. Our recommendation is to simply look for another job. There is enough work right now for data profiles, and you owe it to yourself to seek out a more professional organization.

Summary and Conclusion

The goal of this blog post was to describe some common non-technical problems that are encountered by data professionals while working in organizational settings. We focused here on communicating about data, and some of the many things that can go wrong when we talk about data or present the results of a data project to colleagues.

Communication in any organization can be difficult, and communication about data particularly so. Data topics can be complex, and such discussions can make people uncomfortable, or reveal facts that some stakeholders would prefer remain hidden. In many organizations, employees do not have much training or experience in data, and this often understandably leads to confusion as data projects start to be rolled out.

Our goal here was not simply to make a list of potential difficult situations, but to give you some things to think about and some solutions to try when you run into similar problems. In our view, it’s worth talking about what can and does go wrong in data projects, because we believe this work is important. It is our hope that, the more we as a field talk about problems like these, the better awareness is (among data practitioners and their stakeholders!) and hopefully the better the whole field works. There is a tremendous opportunity to do better, so let’s take it!

Coming Up Next

Our next blog post will focus on interpersonal issues that can pose problems in data projects. Stay tuned!

Post-Script: R Code Used to Produce the Graphs Shown Above

  

# Load necessary libraries
library(ggplot2)
library(dplyr)
library(ggpubr)
library(corrplot)


# Set seed for reproducibility
set.seed(43)

################################
# The graph was too complicated
################################


# Generate a random permutation of 8 predefined group means
# These represent different average values for each group
group_means <- sample(c(10, 15, 20, 20.01, 30, 35, 40, 40.2))

# Display the randomly sampled group means
group_means

# Create a data frame with two columns: Group and Value
# Group: factor with 8 levels, each repeated 30 times (240 observations in total)
# Value: for each group mean, generate 30 random values from a normal distribution
# with the specified mean from the group_means vector and a standard deviation of 5
data_too_complicated <- data.frame(
  Group = factor(rep(1:8, each = 30)),
  Value = unlist(lapply(group_means, function(mean) rnorm(30, mean, 5)))
)


# Initialize a ggplot object using the dataset 'data_too_complicated'
ggplot(data_too_complicated, aes(x = Group, y = Value, fill = Group)) +
 # Add a boxplot layer to show the distribution of values for each group
  geom_boxplot()  +
  # Overlay jittered points to display individual observations
  # width = 0.2 controls horizontal spread; alpha = 0.5 makes points semi-transparent
  geom_jitter(width = 0.2, alpha = 0.5)  +
  # Add statistical comparison between all pairs of groups using t-tests
  # comparisons = all pairwise combinations of Group levels
  # label = "p.signif" shows significance stars; hide.ns = TRUE hides non-significant results
  stat_compare_means(comparisons = combn(levels(data_too_complicated$Group), 2, simplify = FALSE),
                     method = "t.test", label = "p.signif", hide.ns = TRUE) +
  # Apply a minimal theme for a clean look
  theme_minimal() +
  # Add plot title and axis labels; customize legend title for fill
  labs(title = "Estimated Profit Per Client Group",
       x = "Client Group",
       y = "Value",
       fill = 'Client Group') +
  # Format y-axis: scientific notation for labels and 10 evenly spaced breaks
  scale_y_continuous(labels = scales::scientific,
                     breaks = scales::pretty_breaks(n = 10)) +
  # increase y-axis text size and add a black border around the plot
  theme(axis.text.y = element_text(size = 12),
        plot.background = element_rect(colour = "black", fill = NA, size = 1))

###########################
# The graph was too simple
###########################


# simulate the promotion data
data_too_simple <- data.frame(
  # Define the 'Promotion' column as a factor with two levels: Promotion A and Promotion B
  Promotion = factor(c("Promotion A", "Promotion B")),
  # Define the 'Value' column: calculated values for each promotion
  # Promotion A: 23 * 1.5 * 2.1; Promotion B: 23 * 2.1
  Value = c(72.5, 48),
  # Define the 'SE' (Standard Error) column for each promotion
  # Promotion A: 2.5; Promotion B: 2
  SE = c(2.5, 2)
)


# plot the simulated data
ggplot(data_too_simple, aes(x = Promotion, y = Value, fill = Promotion)) +
  # Add bar chart layer with actual values (stat = "identity")
  # Bars are dodged for side-by-side comparison and width set to 0.7
  geom_bar(stat = "identity", position = position_dodge(), width = 0.7) +
  # Add error bars to represent standard errors
  # ymin and ymax define lower and upper bounds; width controls bar cap size
  # Position dodged to align with bars
  geom_errorbar(aes(ymin = Value - SE, ymax = Value + SE), width = 0.2, position = position_dodge(0.7)) +
  # Add text labels showing rounded Value with a percentage sign
  # vjust adjusts vertical position above bars; size sets font size
  geom_text(aes(label = paste0(round(Value), "%")), vjust = -1.5, size = 5) +
  # Manually set fill colors for each promotion for a visually appealing palette
  scale_fill_manual(values = c("Promotion A" = "#FF5733", "Promotion B" = "#33C3FF")) +
  # Apply a minimal theme for a clean and modern look
  theme_minimal() +
  # Add plot title, axis labels, and subtitle explaining error bars
  labs(title = "Comparison of Promotion A and Promotion B",
       x = "Promotion",
       y = "Customer Purchase (%)",
       subtitle = "Error bars represent standard errors") +
  # Configure y-axis: set limits from 0 to 100 and breaks every 10 units
  scale_y_continuous(limits = c(0, 100), breaks = seq(0, 100, by = 10)) +
  # Customize theme: axis text and titles size, bold plot title, and black border around plot
  theme(axis.text = element_text(size = 12),
        axis.title = element_text(size = 14),
        plot.title = element_text(size = 16, face = "bold"),
        plot.background = element_rect(colour = "black", fill = NA, size = 1))

##########################################
# Graph leads to stakeholder tunnel vision
##########################################


# Load the mtcars dataset
data(mtcars) 
# make a subselection of the columns
mtcars <- mtcars %>% 
  select(wt, hp, cyl, disp, qsec, mpg, drat)

# Calculate the correlation matrix
cor_matrix <- cor(mtcars)

# Rename the columns of the correlation matrix to proper English names
colnames(cor_matrix) <- c("Weight", "Horsepower", "Cylinders", "Displacement", 
                          "1/4 Mile Time", "Miles per Gallon", "Rear Axle Ratio")
rownames(cor_matrix) <- colnames(cor_matrix)

# Create a correlation plot using circles to represent correlation strength 
corrplot(cor_matrix, method = "circle", type = "upper",
         # Define color palette: gradient from red (negative) to white (neutral) to blue (positive)
         # Generate 200 color steps for smooth transitions
         col = colorRampPalette(c("red", "white", "blue"))(200),
         # Set text label size for variable names and color to black
         tl.cex = 0.8, tl.col = "black",
         # Set color legend size and position on the right
         cl.cex = 0.8, cl.pos = "r",
         # Add a title to the plot
         title = 'Correlation of Important Variables',
         # Adjust plot margins: bottom, left, top, right
         mar = c(1, 1, 2, 1))
# put a box around the edge of the plot
box(which = "figure", col = "black", lwd = 3)


##########################################
# When Data Reveal an Uncomfortable Truth
##########################################

# Simulated data for 7 cities
data_uncomfortable_truth <- data.frame(
  City = c("London", "Paris", "Berlin", "Madrid", "Rome", "Amsterdam", "Vienna"),
  Business_Metric = c(85, 78, 65, 60, 55, 50, 45)
)

# Order the cities by Business Metric from largest to smallest
data_uncomfortable_truth <- data_uncomfortable_truth %>%
  arrange(desc(Business_Metric))


# Create the bar chart
data_uncomfortable_truth %>%
  # set up plot aesthetics
  ggplot(aes(x = reorder(City, -Business_Metric), 
             y = Business_Metric, 
             fill = City)) +
  # specify barplot and width of bars
  geom_bar(stat = "identity", width = 0.7) +
  # minimal theme
  theme_minimal() +
  # make axis labels and title
  labs(title = "Business Metric by City",
       x = "City",
       y = "Business Metric") +
  # format text and background of the plot
  theme(axis.text.x = element_text(angle = 45, hjust = 1),
        plot.title = element_text(size = 16, face = "bold"),
        legend.position = "none",
        plot.background = element_rect(colour = "black", fill = NA, size = 1))

The main conclusions of our analysis are:

• Employers are increasingly looking for “hybrid” profiles who can take on tasks that are related to multiple job titles (e.g. both data scientist and data engineer)
• The hot job title of 2025 is “data analyst”, though employers are also looking for engineering talent across different sub-disciplines (e.g. cloud, develops and data engineers)
• Data jobs are located in many different data-rich sectors, including IT, finance, healthcare, consulting and manufacturing
• Five years after the COVID-19 pandemic began, employers are expecting data employees to return to the office en masse
• Finally, across the data job profiles, the top requested tools are related to coding, cloud services, and data analysis, while the top requested skills focus on problem solving, soft skills and project management

For all of the details and more, read on below!

Data

¹ Sector, way of working, tools and skills were extracted based on open-coding of the job description text by LLMs (Gemini from Google).

Results

More “Hybrid” Job Titles

One of the first things that popped up to us in this analysis was that fully 11% of job titles encompassed multiple roles. Some examples (reproduced exactly as listed in the job ads):

• ML Engineer / Data Scientist
• BI Business Analyst / Data Engineer (Business-Intelligence-Consultant)
• AI Engineer / Data Analyst
• IT Business Analyst / IT Consultant (Data Engineer)
• SAP Team Leader SAP BI / Data Science (Data Engineer)

This trend is a reversal from what we saw in previous years, where a single job title (e.g. data engineer) would split into separate sub-roles (e.g. devops engineer and cloud engineer) as time went on. In 2025, companies are looking for data profiles who can handle a larger technical scope.

In order to better understand which roles were being sought after in a single hire, we conducted a cluster analysis of the co-occurrence of job titles in our dataset:

Figure 1: Clustering of Roles Based on Co-Occurrence in Job Ads

This analysis shows that the job titles fall into three primary groups:

• The first (on the left-hand side) is business analyst, which appears in a cluster by itself. This makes sense, in that business analyst is a broad and relatively generalist job title, covering dashboarding as well as other types of general business analysis.
• We next have a cluster consisting of IT engineering profiles: devops and cloud engineer. These profiles both work to deploy data solutions and move data around within an organization.
• Finally, we have the third group (on the right-hand side) whose members all work with data in some capacity. The data engineer job title is more on the storage and transfer side, and it appears in its own sub-cluster. The remaining titles (in orange in the graph above) relate to analysis or model-building in some capacity.

Most Popular Data Job Titles in 2025

The plot below shows the most frequently occurring job-titles in our jobs dataset:

Figure 2: Most Popular Job Titles

We see three main takeaways from this analysis

• The hottest data job title in 2025 in Europe is business analyst; these profiles work on dashboards, business intelligence, and other general business topics linked with data. It makes sense that such a generalist title would be most frequent in the data.
• Second, the engineering titles (cloud, devops & data engineers) are very sought after right now. These three titles are more-or-less equally mentioned, and together account for 53% of all job titles in our data.
• Data scientist is much less popular than in previous years. Specifically, in our 2021 analysis, there were more openings for data scientists than for data engineers. In our 2023 analysis, there were around 2 data engineer job ads for every data scientist ad. In 2025, there are 3 data engineer job ads ford every data scientist ad, indicating the continued relative decline of data science in comparison to data engineering.

Sectors

Our analysis indicates that companies in data-rich sectors are the most interested in hiring data profiles. Indeed, in these domains, the potential added value from data and data analysis is substantial:

Figure 3: Number of Data Jobs Per Company Sector

• IT and cybersecurity rank number one. Looking more closely at these job descriptions, we find jobs that sit in the IT silos of organizations. These employees are expected to work almost exclusively with IT tools, databases, SQL, cloud technologies.
• The second most popular sector is financial services. Here, we see many banks and insurance companies represented, and the jobs are a mix between more generalist profiles (e.g. data scientist / engineer) and specialist profiles such as financial analyst.
• Healthcare & life sciences comes next. In this group of organizations we find medtech companies working with data-intensive machines like EEG tools, along with hospitals, national cancer research centers, and SAAS companies targeting the healthcare industry.
• There is healthy demand, as in previous years, in the consulting sectors. These job ads are all focused on finding people to go perform data work at various (nearly always unnamed) clients.
• Finally, we see a fair mount of jobs in the manufacturing sector. A great many of these positions focus on working with data from industrial processes in factories, whether in storing, analyzing and visualizing such data or on optimizing industrial processes based on data captured in factories.

It is safe to say that, in 2025, a great many sectors are looking for data profiles, and that people with data skills are sought after in many different types of organizations!

Way of Working

One of the many ways that the COVID 19 pandemic changed our lives was in the shift to remote and hybrid ways of working.

In 2025, however, it seems pretty clear that companies in Europe are expecting data workers to return to the office. This is a big change from a similar analysis in 2023. In 2023, there were more hybrid jobs advertised than there were onsite jobs. In the current data the trend has reversed itself, fairly heavily, in the opposite direction.

Figure 4: Number of Data Jobs Per “Way of Working”

Private vs. Public Organizations

The graph below shows the split in job advertisements between private and public organizations.

It is clear that the vast majority of data jobs in our analysis are in the private sector. However, there are a non-trivial amount of jobs in the public sector as well. In our data, public entities looking for data profiles include public transport companies (e.g. national rail and regional transportation networks), national research centers, finance ministries and tax services, along with national, regional and city governments.

Figure 5: Number of Data Jobs in the Private vs. Public Sectors

Top Tools & Skills

Now let’s dig in a bit more to the substance of the job advertisements, in particular into what qualities prospective employers are looking for. For this analysis, we relied on LLMs to examine the text of each job description and to extract the tools and skills that were expected of job applicants.

Tools

The graph below shows the top requested tools. The following themes strike us as interesting:

• Coding (Python, SQL, etc.) and development (e.g. VSCode) tools are the number one requested toolset in this analysis. Despite the fact that LLMs are making some coding tasks easier, data professionals are still expected to know how to code!
• In contrast to previous years, cloud tooling and topics (e.g. containerization, Azure, Cloud computing, AWS, GCP, etc.) occurs in many places on our list. This increased focus is new, and indicates that the cloud is increasingly where companies work with data and deploy data solutions.
• Tools related to data analysis also make the list, whether on the BI side, or tools used for more complicated analysis like data science, statistics, and modeling.

Figure 6: Top Tools for Data Jobs

Skills

The graph below shows the top-requested skills in our job ad data. The following points stick out to us:

• The number one skill in this analysis is problem solving and analytical thinking. At a high-level, working with data entails a never-ending series of problem solving and analytical thinking exercises. It makes sense that these skills take the top spot!
• The second and third most-requested skills here are soft skills – namely collaboration and teamwork and communication and interpersonal skills. This represents a big change from previous years, where technical skills were ranked much more highly than soft skills. There seems to be an increasing realization that, in the data professions, being able to communicate to others what you’ve done and why you’ve done it is critical.
• Finally, project management seems to be a sought-after skill – having the ability to “get things done” is important, not always easy in an organizational setting, and not necessarily intuitive for many technical profiles.

Figure 7: Top Skills for Data Jobs

One final thing we’ll take a look at in this blog post is to examine whether some of the job titles place more or less accent on the different skills in our analysis

Clustering Job Titles & Skills

In order to understand how the job titles (e.g. data analyst, scientist, data analyst, etc.) differ by the requested skills, we made the below clustermap. In this map, lighter colors indicate higher prevalence of a given skill for a given job title compared to the others, while darker colors indicate a lower prevalence of a given tool for a given job title compared to the others.

The graph below gives an overview of the differences in employer expectations in regards to skills for the different data job titles. There are too many data points to go over them all in detail, but the following results struck us as the most interesting:

• Business analysts need the most communication and interpersonal skills, likely because they often work more closely with the business and therefore have a greater need to explain their work to non-experts.
• AI engineers are expected to be constantly learning, no doubt because this sub-domain moves so rapidly that knowledge quickly becomes out-of-date.
• DevOps need to be good at operations and support to others – their job is to keep processes running, and when the processes fail to get them back up again.
• Financial analysts need specific domain knowledge more than other data profiles. Indeed, the financial data space is one where domain knowledge is critical, given the regulation surrounding specific types of modelling applications (e.g. credit risk scoring) in the banking sector.
• Finally, data analysts are asked to have a problem-solving and analytical thinking mindset, compared to the other roles.

Figure 8: Clustering of Skills and Roles

A Data Career in Europe in 2025

So, what does this analysis suggest about how to think about a data career in 2025 in Europe? In our thinking, there are two primary types of work that data professionals could consider pursuing:

Focus on Moving the Data:

• Job titles include data and cloud engineering, along with devops
• These titles dominate the data jobs ads in our analysis
• This expertise is needed because A) data need to be in good shape to be used effectively, B) the task is enormous (often due to past under-investment) and C) few companies are where they need to be to take advantage of the data they have.

Focus on Understanding the Data:

• Job titles include business analyst / data analyst / data scientist
• There is strong demand for these job titles in our analysis
• This expertise is needed because A) understanding data is pre-requisite to act on it B) the ability to analyze, visualize and communicate clearly about data is a rare and valuable talent and C) other engineering and IT profiles are less good at this (in our experience)

Summary and Conclusion

Our main takeaways from our analysis of 8,086 European data job ads are as follows:

• Employers are increasingly looking for “hybrid” profiles who can take on a broader set of responsibilities (e.g. handle both data engineering and data science work)
• The bulk of the data job titles can be clustered into two main groups with different emphases:
  • Data analysis in some shape or form (notably data analyst, along with data scientist, statistician, etc.)
  • Moving data from one place to another (e.g. engineering in some capacity, notably cloud, develops and data engineers)
• Data jobs are to be found in many different sectors, and present in both private and public organizations. In short, data opportunities exist in a great many places!
• Across the data profiles, employers are looking for candidates who are familiar with tools related to coding, cloud services, and data analysis
• Across the data profiles, employers seek candidates who have problem-solving and project management skills, and who balance their technical know-how with soft skills

Coming Up Next

How has the perception and practice of the field of data science evolved over the past decade? An early and influential article in the Harvard Business Review from 2012 made a great deal of promises and predictions about what the field of data science would deliver. The next blog post will discuss how those predictions have held up over the past 13 years. Stay tuned!

In the previous post, we used Principal Components Analysis and data visualization techniques to understand the first 2 principal components evident from the survey responses at the municipality level. The results of these analyses showed that the first two principal components concerned feelings about one’s place of residence and transport and mobility, respectively. An analysis of the principal components according to province revealed, for example, that residents in West Flanders felt best about where they lived and that residents in Antwerp Province were most likely to use environmentally sustainable transport.

In this post, we will focus on the third and fourth principal components from the PCA analysis, doing a deep dive into the revealed themes, and map the towns and provinces with the highest and lowest scores on these dimensions.

Read on to learn more!

The Gemeente-Stadsmonitor Survey

The Gemeente-Stadsmonitor (Municipality and City Monitor) is conducted every three years by the Agency of the Interior and Statistics Flanders, and the most recent survey wave was conducted in 2023. For the 2023 survey, a representative sample of residents between 17 and 85 years old was sent the survey, and in in total 389,714 people filled it out. According to the website, all 300 Flemish municipalities should be in the data, but in fact only 299 are.¹

The survey contains questions on 11 broad topics, as designated by the creators of the survey:

• Armoede (Poverty)
• Cultuur en vrije tijd (Culture and leisure)
• Demografie (Demography)
• Klimaat, milieu en natuur (Climate, environment and nature)
• Lokaal bestuur (Local government)
• Mobiliteit (Mobility)
• Onderwijs en vorming (Education and training)
• Samenleven (Living together)
• Werk (Work)
• Wonen en woonomgeving (Living and living environment)
• Zorg en gezondheid (Care and health)

For more information about the survey, you can check out this website (in Dutch). The 2023 survey form containing all of the question text and answer options can be found here.

Language Use in This Post

The Gemeente-Stadsmonitor survey is conducted in Dutch; all survey questions are written in this language. I’m writing this post in English in order for it to be more widely-accessible.

The question descriptions in the charts below will be displayed with the Dutch language descriptions. I’ll provide English language translations (created with machine-translation via Co-Pilot) throughout the text and tables. It should be possible to follow everything described in this post, even if you don’t know any Dutch!

Finally, though the full name of the survey is the “Gemeente-Stadsmonitor”, in the text below I will refer to the survey as the “Stadsmonitor” for simplicity. This is also the term that is used in the Flemish media when talking about the survey and its results.

The Data

The data, at the gemeente / municipality level, are freely available to the public. You can download subsets of the data, or have all of it in a 100+ tab Excel file.

I downloaded the Excel file with all of the data, and spent a significant amount of effort preparing it for analysis. The data preparation code is available on Github here. The code for the analysis presented below is available here.

The present analysis considers only data from the most recent survey wave, conducted in 2023. The data for the analyses below are taken from the answer options that indicate agreement with the question topic. For example, the underlying data for the question “Zich thuis voelen bij mensen in de buurt” (“Feeling at home with people in the neighborhood”) are the percentage of respondents per municipality who agree with this statement.

The dataset contains 299 rows (1 per gemeente/municipality), with the answers to each question contained in 200 columns.

Analysis Part 1: Using PCA to Uncover Latent Themes in the Stadsmonitor Data

The goal of the first set of analyses is to understand the latent themes of the survey items. Each survey question is written to assess residents’ thoughts or feelings about a specific topic (the 11 subjects outlined above, e.g. Mobiliteit / Mobility). However, it is often the case that survey items have a higher-level grouping that is evidenced by the iter-relationship of responses to the questions.

One common data analytic technique that is often used to bring clarity to this underlying structure is called PCA (Principal Components Analysis). Principal Components Analysis is a technique that tries to reduce a set of variables into a smaller dimensional space. In the current case, we have variables describing the gemeente / municipality-level responses to the 200 questions in the Stadsmonitor. PCA allows us to find a smaller number of independent components that describe the variation in the responses to these questions. Within this reduced-dimensional space, we are better able understand the relationships among the questions, municipalities and regions.²

In this blog post, I will use the term topic to describe the designation given by the survey authors, and principal components to describe the data-driven groupings of survey questions from our statistical analysis.

Principal Components 3 and 4

In this post, we will focus on the third and fourth dimensions uncovered by our PCA analysis. PCA analysis returns a list of underlying themes (called Principal Components or PCs in statistical terms), ranked in terms of their importance in explaining the variation in the responses to the survey questions. Each question gets a score (called a loading in statistical terms) for each Principal Component. The loadings range in between -1 and +1, and the larger a question’s loading on a principal component (either in a positive or negative direction), the more the question is reflective of the theme represented by that component.³

Note that, below, we will begin our discussion with principal component 3. For a detailed description of the first two dimensions from the principal components analysis, please see the first blog post in this series.

Principal Component 3

The table below shows the questions with the highest scores (loadings) - both positive and negative - on the third principal component. Examining these questions will give us an idea of the subject matters of the third principal component.

As can be seen in the table below, the third principal component encompasses questions that focus on a variety of different social and societal themes.

On the positive end of this principal component, we find items that focus on environmentally conscious behavior (e.g. vegetarian eating, buying organic products, limiting plastic use, etc.) and positive attitudes towards diversity (e.g. believing that having residents from different backgrounds enriches life in the town/municipality). On the negative end of this principal component, we find questions related to negative attitudes towards diversity (e.g. having unpleasant neighbors of foreign backgrounds, feeling that municipality residents come from too many different origins, etc.) and being satisfied with town services (e.g. satisfaction with facilities for elderly people, childare, etc.). Interestingly, the negative side of PC3 also encompasses items about subjective poverty and payment difficulties (suggesting that municipalities that score low on this dimension are less wealthy) and working in one’s own municipality (suggesting that more residents in municipalities that score low on this dimension work where they live).

Principal Component 4

The questions with the highest scores (both positive and negative) on the fourth principal component are shown in the table below.

As can be seen from an examination of the items in the table, this principal component deals with two different topics at its positive and negative poles.

On the positive end of this principal component, we find items that focus on satisfaction with one’s local government (e.g. trust in government, satisfaction with communication towards and consultation of residents, satisfaction with contact with the municipality, etc.). On the negative end of this principal component, we find questions related to bike usage (owning and using bicycles for for short distances, leisure travel, etc.) and sustainable living (e.g. having homes which are energy efficient and well-insulated & commitment to climate-friendly investments; note that bike use could also be seen as a facet of sustainable living).

Plotting the Stadsmonitor Questions According to Their Scores on Principal Components 3 & 4

We can plot the individual survey items in the tables above in the two-dimensional space defined by the scores of each question on the third and fourth principal components, mapping out where the questions fall with regards to one another within this space. The information shown in the plot below is the same as that contained in the above tables, but the visualization provides another way of understanding the relationships among the questions and the themes of the third and fourth principal components. The items are colored by the question topic, as determined by the agency that administered the survey.

Note that we are showing just a subset of the questions in this plot; there are 200 questions, and plotting them all yields in an enormous jumble that is difficult to read and to interpret.

The main concerns of principal component 3 (displayed on the horizontal or x-axis) come through clearly in the plot. On the positive (right-hand) side of principal component 3, we see questions in yellow describing environmentally conscious behavior (e.g. “Vegetarisch eten” / “Vegetarian eating”) and questions in red describing positive attitudes towards diversity (e.g. “Verschillende herkomst verrijking” / “Different origins bring enrichment”). On the negative (left-hand) side, we see items in light purple and green describing satisfaction with town services (e.g. “Tevredenheid over kinderopvang” / “Satisfaction with childcare”), items in red describing negative attitudes towards diversity (e.g. “Onprettig buren andere herkomst” / “Unpleasant neighbors with foreign backgrounds”) and items in black representing subjective poverty and payment difficulties (e.g. “Subjectieve armoede” / “Subjective poverty”).

The main concerns of the fourth principal component (displayed on the vertical or y-axis) are also evident in the plot. At the positive (upper) side of component 4, we see the items about satisfaction with local government in light blue (e.g. “Vertrouwen in gemeentebestuur” / “Trust in municipal government”). Conversely, at the negative pole (bottom end) of this dimension, we see items in red-violet and yellow representing bike use (e.g. “Verplaatsingen vrije tijd - Fiets algemeen” / “Leisure travel - Bike in general”) and items in dark blue representing sustainable living (e.g. “Duurzaamheid woning - energiezuinig” / “Sustainability house - energy efficient”).

Analysis Part 2: Plotting the Gemeenten / Municipalities and Provinces According to Their Scores on Principal Components 3 & 4

We can use the results of our PCA analysis to plot each gemeente / municipality in the two-dimensional space defined by the third and fourth principal components, coloring the points according to the Flemish province in which they are located. We can also compute the average of each province along the principal components; these province averages are indicated by the larger dots on the plot below.

There is a clear ordering of the provinces according to the third principal component (displayed on the horizontal axis). The positive end of this component concerns environmentally conscious behaviors & positive attitudes towards diversity. Gemeenten / municipalities in Flemish Brabant have, on average, the highest scores on this dimension, indicating that residents in this province are more likely to report performing environmentally conscious behaviors and having positive attitudes towards diversity. The provinces of East Flanders and Antwerp fall somewhere in the middle, while Limburg falls more to the lower side of principal component 3. The negative end of principal component 3 concerns negative attitudes towards diversity and satisfaction with town services; the province with the lowest average score is West Flanders. The average low score on principal component three indicates that residents in West Flanders report on average more negative attitudes towards diversity and are also more satisfied with town services than their fellow citizens in the other Flemish provinces.

There is much less variation by province along the fourth principal component (displayed on the vertical axis). The one data point that sticks out to me is the low average score for Antwerp province on principal component 4. This indicates that residents in Antwerp province, on average, report greater bike use and more environmentally sustainable behaviors than residents in the other Flemish provinces.

Focusing on the Gemeenten / Municipalities at the Extremes of Principal Components 3 & 4

In the plot above, I’ve focused on the center of the coordinates defined by the principal components three and four. However, there are gemeenten / municipalities that fall at both positive and negative extremes along both of these two dimensions. The plots below display the gemeenten / municipalities with the most extreme scores at the positive and negative ends of each principal component.

Highest Scores On PC 3 - Environmentally Conscious Behavior & Positive Attitudes Towards Diversity

The plot below shows the gemeenten / municipalities with the highest scores on principal component 3. These are the places where survey respondents say that they engage in environmentally conscious behavior and have positive attitudes towards diversity.

Lowest Scores On PC 3 - Negative Attitudes Towards Diversity & Satisfaction With Town Services

The plot below shows the gemeenten / municipalities with the lowest scores on theme / principal component 3. These are the places where survey respondents report negative attitudes towards diversity, satisfaction with town services, and subjective poverty.

Highest Scores On PC 4 - Satisfaction With Local Government

The plot below shows the gemeenten / municipalities with the highest scores on principal component 4. These are the places where survey respondents report the greatest satisfaction with their local government.

Lowest Scores On PC 4 - Bike Usage and Sustainable Living

The plot below shows the gemeenten / municipalities with the lowest scores on principal component 4. These are the places where survey respondents report the highest degrees of bike usage and sustainable living.

Summary and Conclusion

In this blog post, we used principal components analysis to analyze the summary data per gemeente / municipality from the 2023 Stadsmonitor survey. This analysis followed up on the first post in this series; we focused here on the third and fourth principal components of our PCA analysis.

The third principal component concerned environmentally conscious behavior and positive attitudes towards diversity at the positive end of the dimension and negative attitudes towards diversity and satisfaction with town services on the negative end. The fourth principal component concerned satisfaction with one’s local government at the positive end and bike usage and sustainable living at the negative end.

We then plotted the gemeenten / municipalities and provinces in the two-dimensional space defined by principal components 3 and 4. There was a clear ordering of the provinces according to the third principal component. Respondents Flemish Brabant are more likely to report performing environmentally conscious behaviors and having positive attitudes towards diversity, while respondents in West Flemish municipalities feel, on average, greater negative attitudes towards diversity and greater satisfaction with town services.

There was less variation by province according to the fourth principal component; the one clear difference was that gemeenten / municipalities in Antwerp province were much more likely to report bicycle ownership and use and were more likely to engage in environmentally sustainable behaviors. Finally, we made plots of the gemeenten / municipalities that scored the highest on the positive and negative poles of the third and fourth principal components.

This is the final post examining the themes revealed by our principal component analysis of the 2023 Stadsmonitor survey. In contrast to the first post, the separation of themes between the principal components was less clear here. For example, both principal components 3 and 4 contained items related to environmentally-friendly behaviors (e.g. environmentally conscious behavior like fair trade for PC 3 and sustainable living practices such as energy efficient housing for PC 4) and attitudes towards local government (e.g. satisfaction with services for PC 3 and satisfaction with communication, consultation and contact for PC 4). While there is a logic underlying this separation, the differences are somewhat subtle. This, combined with the fact that each subsequent principal component explains less variation in the underlying survey responses, suggests that we have reached the limit of what we can learn from our PCA analysis of the Stadsmonitor survey data.

Coming Up Next

There are two posts coming in the hopefully not-too-distant future. The first will be a technical deep-dive into the PCA analyses presented in this post and the previous one, and we will describe (with code) the details of the analyses presented above. The second is based on a talk that I gave at the Royal Statistical Society of Belgium with Cesar Legendre of Prophecy Labs. In this post, we will describe lessons learned from our professional experiences in doing data science in an organizational context.

Stay tuned!

Flanders, the Northern, Dutch-speaking part of Belgium, conducts a regular survey of the people who live here. The survey is called De Gemeente-Stadsmonitor (The Municipality and City Monitor), and covers a great many topics, from large societal issues to housing to mobility and climate.

This post describes an analysis of the summary data (aggregated per municipality) from the most-recent survey wave, giving a data-driven view of the main underlying themes of the survey. We will analyze these themes in some detail, and make a map of Flemish municipalities (gemeenten in Dutch) and provinces (West Flanders, East Flanders, Flemish Brabant, Antwerp, and Limburg) according to their relationship to the themes and to one another.

In which municipalities and regions do residents feel good about where they live, and in contrast, where do residents say they feel unsafe? In which municipalities and regions do residents use environmentally sustainable transport and where do they seem dependent on the car? Read on to find out!

The Gemeente-Stadsmonitor Survey

The Gemeente-Stadsmonitor (Municipality and City Monitor) is conducted every three years by the Agency of the Interior and Statistics Flanders, and the most recent survey wave was conducted in 2023. For the 2023 survey, a representative sample of residents between 17 and 85 years old was sent the survey, and in in total 389,714 people filled it out. According to the website, all 300 Flemish municipalities should be in the data, but in fact only 299 are.¹

The survey contains questions on 11 broad topics, as designated by the creators of the survey:

• Armoede (Poverty)
• Cultuur en vrije tijd (Culture and leisure)
• Demografie (Demography)
• Klimaat, milieu en natuur (Climate, environment and nature)
• Lokaal bestuur (Local government)
• Mobiliteit (Mobility)
• Onderwijs en vorming (Education and training)
• Samenleven (Living together)
• Werk (Work)
• Wonen en woonomgeving (Living and living environment)
• Zorg en gezondheid (Care and health)

For more information about the survey, you can check out this website (in Dutch). The 2023 survey form containing all of the question text and answer options can be found here

Language Use in This Post

The Gemeente-Stadsmonitor survey is conducted in Dutch; all survey questions are written in this language. I’m writing this post in English in order for it to be more widely-accessible.

The question descriptions in the charts below will be displayed with the Dutch language descriptions. I’ll provide English language translations (created with machine-translation via Co-Pilot) throughout the text and tables. It should be possible to follow everything described in this post, even if you don’t know any Dutch!

Finally, though the full name of the survey is the “Gemeente-Stadsmonitor”, in the text below I will refer to the survey as the “Stadsmonitor” for simplicity. This is also the term that is used in the Flemish media when talking about the survey and its results.

The Data

The data, at the gemeente / municipality level, are freely available to the public. You can download subsets of the data, or have all of it in a 100+ tab Excel file.

I downloaded the Excel file with all of the data, and spent a significant amount of effort preparing it for analysis. All of the data preparation and analysis code is available on Github here.

The present analysis considers only data from the most recent survey wave, conducted in 2023. The data for the analyses below are taken from the answer options that indicate agreement with the question topic. For example, the underlying data for the question “Zich thuis voelen bij mensen in de buurt” (“Feeling at home with people in the neighborhood”) are the percentage of respondents per municipality who agree with this statement.

The dataset contains 299 rows (1 per gemeente/municipality), with the answers to each question contained in 200 columns.

Analysis Part 1: Using PCA to Uncover Latent Themes in the Stadsmonitor Data

The goal of the first set of analyses is to understand the latent themes of the survey items. Each survey question is written to assess residents’ thoughts or feelings about a specific topic (the 11 subjects outlined above, e.g. Mobiliteit / Mobility). However, it is often the case that survey items have a higher-level grouping that is evidenced by the iter-relationship of responses to the questions.

One common data analytic technique that is often used to bring clarity to this underlying structure is called PCA (Principal Components Analysis). Principal Components Analysis is a technique that tries to reduce a set of variables into a smaller dimensional space. In the current case, we have variables describing the gemeente / municipality-level responses to the 200 questions in the Stadsmonitor. PCA allows us to find a smaller number of independent components that describe the variation in the responses to these questions. Within this reduced-dimensional space, we are better able understand the relationships among the questions, municipalities and regions.²

In this blog post, I will use the term topic to describe the designation given by the survey authors, and theme to describe the data-driven groupings of survey questions from our statistical analysis (e.g. the principal components).

Revealed Themes (Principal Components)

In this post, we will focus on the top two themes uncovered by the PCA analysis. Our analysis gives us a list of underlying themes (called Principal Components or PCs in statistical terms), ranked in terms of their importance in explaining the variation in the responses to the survey questions. Each question gets a score (called a loading in statistical terms) for each Principal Component. The loadings range in between -1 and +1, and the larger a question’s loading on a principal component (either in a positive or negative direction), the more the question is reflective of the theme represented by that component.³

Theme 1 (PC 1)

The table below shows the questions with the highest scores (loadings) - both positive and negative - on the first principal component. Examining these questions will give us an idea of the subject matter of the first theme.

As is clear in the table, the first theme is heavily concerned with feelings about one’s place of residence, in the neighborhood or gemeente / municipality.

On the positive end of this principal component, we find items that focus on feeling good about where one lives. The questions deal with feeling at home, comfortable, safe, having good contacts with neighbors, etc. On the negative end of this principal component, we find questions related to feeling insecure or unsafe where one lives. The questions with the largest negative scores concern nuisances in the neighborhood, conflict, attitudes towards diversity, feeling unsafe, and interestingly, trust in the federal government.

A Word on Interpretation

PCA analysis is based upon the underlying correlations among the responses to the survey items at the gemeente / municipality level. We can therefore use the analysis to understand how responses to the survey questions are related to one another.

Firstly, items that have higher scores on PC 1 are positively correlated with one another, and items that have lower scores on PC 1 are also positively correlated with one another. For example, gemeenten / municipalities that have higher average scores on the item “Feeling at home with people in the neighborhood” (PC 1 loading of .88) also have higher average scores on the item “Satisfaction with contact in the neighborhood” (PC 1 loading of .87). And at the other end of PC 1, gemeenten / municipalities with higher average scores on the item “Feeling of insecurity in the neighborhood” (PC 1 loading of -.73) also have higher average scores on the item “Chat with people of non-Belgian origin” (PC 1 loading of -.72).

Furthermore, items with positive loadings on a given principal component are negatively correlated with items with negative loadings on that component. For example, gemeenten / municipalities with higher average scores on the item “Feeling at home with people in the neighborhood” (PC1 loading of .85) have lower average scores on the item “Feeling of insecurity in the municipality” (PC1 loading of -.74).

Note that these relationships are correlations, and not causal relationships. The table of loadings below shows that gemeenten / municipalities where residents speak more with people of non-Belgian origin (loading of -.72) also feel less safe in their neighborhood (loading of -.73). However, it is not the case that people feel less safe because they speak more frequently with individuals of non-Belgian origin.

Theme 2 / PC 2

The questions with the highest scores (both positive and negative) on the second underlying theme are shown in the table below. This theme is heavily focused on transport and mobility, important topics in a country where the near-constant traffic jams have enormous negative impacts on the economy and commuter well-being.

On the positive dimension of the principal component, the questions focus on environmentally sustainable transportation. Many of the top items concern bicycle use - whether respondents frequently use their bikes, the state of the bike infrastructure, and whether they feel safe while cycling. Mixed in with these items are questions related to walking and being physically active. At the negative end of the second theme, we find mostly items about frequent car usage.

Plotting the Stadsmonitor Questions According to Their Scores on Themes 1 & 2

We can plot the individual survey items in the tables above in the two-dimensional space defined by the scores of each question on the first two principal components, mapping out where the questions fall with regards to one another within this space. The information shown in the plot below is the same as that contained in the above tables, but the visualization provides another way of understanding the relationships among the questions and the themes of the first two principal components. The items are colored by the question topic, as determined by the agency that administered the survey.

Note that we are showing just a subset of the questions in this plot; there are 200 questions, and plotting them all yields in an enormous jumble that is difficult to read and to interpret.

The main concerns of the first theme (displayed on the horizontal or x-axis) - of feelings about one’s place of residence - are very clear. On the positive (right-hand) side, we see the questions describing positive feelings about where one lives (e.g. “Zich thuis voelen in de buurt” / “Feeling at home in the neighborhood”). On the negative (left-hand) side, we see the items about feeling insecure or unsafe in the place where one lives (e.g. “Onveiligheidsgevoel in de buurt” / “Feeling of insecurity in the neighborhood”).

The main concerns of the second theme (displayed on the vertical or y-axis) - all about transport and mobility - are also quite clear. At the positive (upper) side, we see the items about using bikes or other environmentally sustainable transportation modes, whereas on the negative (bottom) side, we see items about using the car.

Notice that the question topics (assigned by the survey writers) are grouped into higher level themes by the PCA analysis. For example, the first theme / principal component - about feeling good or bad where one lives - contains survey items from the Wonen en woonomgeving (Living and living environment), Samenleven (Living together), and Cultuur en vrije tijd (Culture & leisure) topics.

Analysis Part 2: Plotting the Gemeenten / Municipalities and Provinces According to Their Scores on Themes 1 & 2

We can use the results of our PCA analysis to plot each gemeente / municipality in the two-dimensional space defined by the first two principal components, coloring the points by the color of the Flemish province in which they are located. We can also compute the average of each province along the principal components; these province averages are indicated by the larger dots on the plot below.

There is a clear ordering of the provinces according to the first theme / principal component (displayed on the horizontal axis), which concerns feelings about one’s place of residence. Gemeenten / municipalities in West Flanders have, on average, the highest scores on this dimension, indicating that residents in this province feel the most positive about where they live. West Flanders is followed closely by Limburg, while Antwerp and East Flanders fall somewhere in the middle. Flemish Brabant has the lowest average scores on the first theme / principal component, indicating that gemeenten / municipalities in this province are the least positive about their place of residence, and that on average feelings of insecurity are higher there.

There is much less variation by province along the second theme / principal component (displayed on the vertical axis). On this transport and mobility dimension, Antwerp scores the highest, indicating that gemeenten / municipalities in this province, on average, indicate greater use of environmentally sustainable transportation. West Flanders falls in the middle, while the remaining provinces are located on the frequent car usage side of this dimension.

Focusing on the Gemeenten / Municipalities at the Extremes of Themes 1 & 2

In the plot above, I’ve focused on the center of the coordinates defined by the first two principal components. However, there are gemeenten / municipalities that fall at both positive and negative extremes along both of these two dimensions. The plots below display the gemeenten / municipalities with the most extreme scores at the positive and negative ends of each theme.

Highest Scores On Theme / PC 1 - Feelings About One’s Place of Residence

The plot below shows the gemeenten / municipalities with the highest scores on theme / principal component 1. These are the places where survey respondents feel the most positive about where they live.

Lowest Scores On Theme / PC 1 - Feelings About One’s Place of Residence

The plot below shows the gemeenten / municipalities with the lowest scores on theme / principal component 1. These are the places where survey respondents report the greatest feelings of insecurity where they live.

Highest Scores On Theme / PC 2 - Transport & Mobility

The plot below shows the gemeenten / municipalities with the highest scores on theme / principal component 2. These are the places where survey respondents report the greatest use of environmentally sustainable transportation; people here regularly walk or use the bike in their daily lives.

Lowest Scores On Theme / PC 2 - Transport & Mobility

The plot below shows the gemeenten / municipalities with the lowest scores on theme / principal component 2. These are the places where survey respondents report the highest degrees of frequent car usage; people here travel more by car in their daily lives.

Summary and Conclusion

In this blog post, we used principal components analysis to analyze the summary data per gemeente / municipality from the 2023 Stadsmonitor survey. This analysis allowed us to uncover the main themes that underlie the survey responses. We focused here on the first two themes or principal components.

The first theme concerned feelings about one’s place of residence. At the positive end of this dimension, we found questions related to feeling good about where one lives. At the negative end of this dimension, we found questions related to feeling insecure or unsafe where one lives.

The second theme concerned transport and mobility. At the positive end of this dimension, we found questions related to the usage of environmentally sustainable transportation, while at the negative end of this dimension, we found questions related to frequent car usage.

We then plotted the gemeenten / municipalities and provinces in the two-dimensional space defined by the first two themes / principal components. There was a clear ordering of the provinces according to the first dimension. Respondents in West Flemish municipalities feel, on average, the best about their places of residence, while residents in Flemish Brabant report the greatest feelings of insecurity about where they live. There was less variation by province according to the second dimension; the one clear difference was that gemeenten / municipalities in Antwerp were much more likely to report greater use of environmentally sustainable transportation such as biking or walking, compared to the other provinces. Finally, we made plots of the gemeenten / municipalities that scored the highest on the positive and negative dimensions of the two dimensions, highlighting the places that feel the best vs. worst about where they live, and the places that make the greatest use of environmentally sustainable transport options vs. the places where residents are most likely to make use of the car in their daily lives.

Coming Up Next

The next two posts will also focus on the data from the 2023 Stadsmonitor survey. We will further explore the dimensions of the PCA analysis described above, and understand how Flemish gementeen and provinces differ according to the third theme / principal component.⁴ The final post in this series will be a technical one, and will present (with code) the details of the analyses described above.

Stay tuned!

Shrimp farmers often have laboratory tests conducted of the water in their shrimp ponds. The data they receive provide vital information that they can use to adjust their farming practics, with the goal of keeping their animals healthy and attaining the largest possible harvest.

How reliable are the results of such lab tests? Using data we collected from 60 different laboratory tests of water from the same pond, we use the coefficient of variation to understand the extent to which tests of the same parameters from the same pond water vary across tests and across laboratories.

You can check out the full post here!

Exasperated Data Scientists Applying for Jobs. Credit: Jesse Blum via Midjourney

Introduction

Has “data science” peaked? It’s an intriguing question, with strong arguments for and against this provocative proposition. On the one hand, deep learning has recently captivated popular attention, with amazing innovations in text (e.g. large language models such as Chat-GPT) and image generation (e.g. services such as DALL-E and Midjourney, the rise of deepfakes in the political sphere, etc.) promising to disrupt many professions from screenwriting to advertising to customer service. On the other hand, many organizations struggle in the data/AI space, facing problems ranging from basic (e.g. collection and storage of basic transactional data) to sophisticated (e.g. struggles with migration to the cloud, deployment of data solutions into existing business or client-facing applications), not to mention the ever-present talent problem (with the relative lack of qualified professionals and a workforce that is not in all cases ready for the digital disruption that data has the potential to deliver). It appears as though progress in developing and adopting data solutions is bifurcated - a relative few companies are adopting advanced analytics solutions to great success, while many are taking their first steps and struggling to fully adopt the promise of AI.

In this landscape, what does the job market for data professionals look like in 2023? Our analysis of European data science job ads from 2022 suggested that European companies looking for data talent placed a great deal of importance on technological skills, to the detriment of softer skills such as empathy and communication, which are important in a larger organizational context.

This blog post presents an update of our 2022 analysis of European data job advertisements, with an examination of a larger portfolio of roles, in order to understand the rapidly-evolving labor market and data role landscape. In contrast to last year’s analysis, we focus here exclusively on the Belgian job market, as this is the country where both authors live and work.

The results revealed a great deal about the data job market in 2023 in Belgium. Specifically, business analyst, devops engineer, and data engineers were the most in-demand job titles, recruitment agencies and consultancies were the main posters of data job advertisements, and about half of the posted jobs were in smaller-sized companies. Three years after the start of the COVID pandemic, Belgian employers expect data employees to work from the office at least some of the time. As was the case last year, the most in-demand data tools are Python and SQL, and each role within the data space is associated with a distinct collection of tools that sets each role apart from the others. In contrast to last year’s analysis, employers are placing increasing importance on soft skills such as communication and collaboration. Finally, our data suggest that employers frequently re-post the same job description multiple times (presumably to remain at the top of the “new jobs” list), and use emojis to appeal to a younger, tech-savvy demographic.

For all of the details, read on below!

Data

Our primary data source consists of 25,965 unique job descriptions scraped from a major online job board. Specifically, we scraped all jobs that were returned with the query “data scientist” from January to April 2023. For each job description, we extracted the following pieces of information: the job title, the job description text, the posting company name, company size, company sector and the way of working (e.g. office, remote, or hybrid).

What Roles are Most In Demand?

We first examined which roles were most sought-after. The most-frequent job titles for the scraped job advertisements (searching explicitly for “data scientist”) was software developer, followed by project manager. This is quite interesting, and perhaps captures important areas of overlap: software tools such as Python, CI/CD, are common between the data scientist and software developer roles, while the agile methodology is common between the data scientist and project manager roles.

Figure 1: Most Frequent Job Titles Returned with the Query “Data Scientist”

Data scientist roles, meanwhile, were only the 8th most-common job title for searches for data scientist jobs!

A Focus on Data Roles

For the remainder of this blog post, we will focus exclusively on data roles (N = 4280), specifically on the following job titles:

• business analyst
• devops engineer
• data engineer
• data analyst
• cloud engineer
• data scientist
• statistician
• machine learning engineer

Figure 2: Number of Data Jobs Per Role

Who Is Looking for Data Talent?

Company Sector

In Belgium, by far the largest sector recruiting for data profiles is the staffing and recruitment sector. Anyone who has spent any time in the data space has surely dealt with recruiters! In Belgium, it appears that many companies feel unable to recruit data profiles on their own (or they’re simply looking for freelancers), and so many jobs are advertised and filled by recruitment agencies.

Figure 3: Number of Data Jobs Per Sector

The other major sector represented in the above graph is the services / consulting sector (e.g. financial services, IT services, etc.). In Belgium, in 2023, this is where many of the jobs are. Some of these jobs are of course with the big consulting companies, but others are in small tech/data consultancies, often but not always specialized in a specific sector (e.g. finance).

Company Size

As was the case last year, most data jobs are at smaller organizations - in between 1-500 employees. Many of these companies are smaller startups or consultancies with a digital / tech flavor baked in from the very beginning.

Figure 4: Number of Data Jobs Per Company Size

Three Years Since COVID Began - Are Data Profiles Expected Back in the Office?

The COVID pandemic radically changed the way that many people in the data space work. While the pre-COVID norm in Belgium placed a great deal of importance on being physically present at the office, when the pandemic began, many companies adopted remote working practices overnight.

Three years after the pandemic began, what are the expectations in terms of ways of working for data profiles? According to our data, the most frequent way-of-working is hybrid, followed closely by on-site roles, with remote appearing least frequently.

Figure 5: Number of Data Jobs Per “Way of Working”

It therefore appears that companies in Belgium are reluctant to fully embrace remote working for data roles in 2023. It’s clear that, for many jobs, we’re not going back to the pre-pandemic world just yet, as evidenced by the large number of hybrid roles. But given the relatively small share of remote jobs, it is clear that Belgian organizations expect data employees to be on-site at least some of the time.

What Are Employers Looking For?

Unlike last year’s analysis, the current data do not contain employer-provided keywords of the desired skills for each job description. Therefore, we used openAI) models to analyze each job description and return the tools and skills mentioned in the job description texts.

To accomplish this task, we relied on state-of-the-art language models like GPT-Turbo-3.5 (May 2023 version) and GPT4, accessed via the official APIs. Coupled with prompt engineering techniques, these models enabled efficient and accurate extraction of the desired information from the job description texts.

The LLMs (Large Language Models) were able to extract the tools (python, kubernetes, etc.) from the texts relatively easily, as data tools are typically referred to by their unique names. However, the extraction of skills presents a bit more of a challenge. The language used to describe the same skill can vary between sectors or even individual job listings. For instance, the skill of ‘time management’ could be described in several ways, including ‘time organization’ and ‘efficiency in scheduling’. The LLM skill-extraction analysis therefore gave us a long list of skills that contained many terms that embodied the same concept, but described in different words.

To make this list more manageable, we turned to sentence embeddings, using the text-embedding-ada-002 model. By grouping together skills with a high level of similarity (i.e. cosine similarity of 0.95 or above), we were able to aggregate the long list of separate skills (e.g. ‘time organization’ and ‘efficiency in scheduling’) into a fewer number of higher-level categories (e.g. ‘time management’) that captured the essence of the underlying skill.

Tools

Despite the fact that the data science toolkit is constantly evolving, the top two tools are the same as last year, the programming languages Python and SQL. These are both undeniably important for most data science jobs, and are joined on the list by other programming languages like c# and JavaScript.

Figure 6: Top 25 Tools for Data Jobs

We see many tools related to the cloud, such as AWS, Azure, GCP, Kubernetes, Terraform, and Ansible. Increasingly, it appears, data science work is taking place in cloud-based systems, making experience with cloud platforms a valuable asset.

Unsurprisingly, given the link between data science and software, we see a number of software development tools in the list, such as git, CI/CD, bash, and Jenkins.

Data science is a large universe, and one aspect involves data visualization. This aspect is represented in the list with the most popular BI tools, Power BI and Tableau.

Finally, we see a number of tools based on the previously en vogue buzzword “big data”: Scala, Spark, Databricks, and Kafka.

Skills

When examining the most frequently-mentioned skills in the data job advertisements, we see a number of different themes emerging.

Figure 7: Top 25 Skills for Data Jobs

In contrast to last year, we see a number of “soft” skills such as communication, collaboration, and stakeholder management. In our view, these skills are fundamental to getting data projects done, and we are encouraged that the job advertisements increasingly mention this critical skill set.

We also see a number of data-related skills, such as analysis, visualization, ML engineering, and statistics. These skills get at the analysis, interpretation, and modeling of data, which are all core tasks in the data science role.

Interestingly, and importantly given that data science is a role that exists to serve overall business needs, we see a number of what we term “general business skills” such as problem solving, project management, and business analysis.

Finally, we see a number of skills related to software development such as devops, automation, security, and CI/CD.

Clustering Roles by Tools

In order to understand how the roles (e.g. data scientist, data analyst, etc.) differ by the requested tools, we made the below clustermap. We first calculated the percentage of job ads per role that contained each tool, filtering on tools that appeared in more than 50 job ads. These percentages were converted to z-scores per tool, such that higher numbers indicate that a given tool was mentioned more often for a given role compared to the others. This final matrix was then passed to the cluster map algorithm, which performs a simultaneous clustering of both the job roles and of the mentioned tools.

In this map, lighter colors indicate higher prevalence of a given tool for a given role compared to the others, while darker colors indicate a lower prevalence of a given tool for a given role compared to the others.

This analysis gives a good overview of the differences in employer expectations in regards to tools for the different data roles.

Figure 8: Clustering of Tools and Roles

Cloud engineers are expected to know about cloud platforms (e.g. Azure, Google Cloud Platform and AWS), devops engineers are expected to know about deploying data solutions (e.g. Chef, Puppet) and infra as code (e.g. Ansible), while data engineers are expected to know about tools that are required for building data pipelines (e.g. ETL, Spark, Kafka, etc.). ML engineers are expected to know about deep learning (e.g. Tensorflow) and big data (e.g. Scala, Spark, Hadoop), while statisticians are expected to know SAS (but not Python, according to the graph!), and data scientists are expected to understand machine learning and statistics with Python. Finally, business analysts are expected to master tools to design data flows (e.g. UML) and dashboards (e.g. Power BI), while data analysts are expected to know tools that allow them to build dashboards (e.g. Tableau, Power BI).

One of the new up-and-coming data roles is the devops engineer. According to this analysis, they manage cloud infrastructure (e.g. OpenShift, Ansible), take care of deployment, industrialization of a data solution and integration into business systems (e.g. Chef, Puppet). This job title appears to reflect a splitting of the “data engineer” profile, and is a sign of the market evolving. Employers seem to be realizing that one person can’t do everything that’s needed in order to develop and deploy data solutions! As a consequence, data engineers, in this analysis, seem to be expected just to work on the data pipelines, leaving behind the cloud and devops aspects that were in their scope last year.

What to Watch Out For?

The data and AI space is very hot right now, promising new technologies and possibilities that many employers do not fully understand. As such, it is filled with creative, ambitious people trying to do something new (bring business value through data), but also with opportunists trying to make a quick buck (both employers and job candidates can be guilty of this!).

In analyzing the 2023 data job descriptions, we noticed a couple of things that prospective job applicants should be aware of when navigating the data job market.

Duplicate Job Ads

One tendency is for employers to post the same job ad multiple times, often weekly, presumably to ensure that the job description is returned at the top of the list. We found that 25% of the jobs in our dataset were duplicated, and that this pattern occurred most frequently among jobs posted by recruitment agencies.

Trying to Impress the Cool Kids

Inspired by the iconic “How Do You Do, Fellow Kids?” meme, Steve Buscemi in the role of an old-style recruiter trying to convince young data scientists that a particular job description is ‘cool’ enough.

When looking to attract young talent, job posters often employ a variety of creative marketing techniques. These include using social media, sharing engaging videos, spreading relatable memes, and posting motivational quotes. However, the conventional format of job descriptions may limit their creative arsenal of techniques.

As a result, integrating non-verbal communication tools like emojis into these descriptions can be a strategic move to bring them to life, making them more dynamic and visually engaging. A money bag emoji 💰, for instance, could emphasize the salary aspect, while a globe emoji 🌎 might indicate a remote working opportunity. Emojis can also subtly hint at a company’s culture (a consistent use of emojis might suggest a more casual, friendly, or creative work environment).

Figure 9: Count of Data Job Descriptions With Different Number of Emojis

Our analysis reveals that 5.3% (1,376) of job advertisements contain emojis, with 465 postings including only one emoji, and 16 descriptions containing 17 or more emojis! The registered symbol ® (both emoji and unicode form) is the most frequently used, followed by the check mark ✅ and the rocket 🚀.

Figure 10: Most Frequent Emojis in Data Job Descriptions

The registered symbol ® might be used to emphasize the authenticity or official status of the company, reflecting a sense of professionalism and legitimacy; the check mark ✅, often associated with completion or validation, could signify the fulfillment of certain criteria, thereby reassuring potential candidates that the position meets specific standards or qualifications; and the rocket 🚀, symbolizing growth, innovation, or a dynamic approach, could convey the company’s forward-thinking mindset, aspiration for growth, or enthusiasm for new ideas, making the job opportunity seem more exciting and progressive.

The Future of Data Analysis - Ask a Model!

The path to a successful data job can often feel like navigating a labyrinth. With nearly 26,000 data job descriptions posted in the first 4 months of this year (only in Belgium!), aspiring data scientists find themselves overwhelmed by an ocean of information.

AI Assistants Trying to Navigate Thousands of Data Job Descriptions. Credit: Jesse Blum via Midjourney

As a young data scientist, you do not have to fear, as the future of AI is here to change the game! Imagine having a personal AI assistant that accompanies you on your job search, simplifying the process, and illuminating the way to your ideal data role. The solution lies in an AI assistant powered by a RAG (Retrieval Augmented Generation) model, combining the best of both worlds: the search capabilities of a search engine and the interactive experience of a chatbot. Here’s how this approach works:

1. The Question: Picture yourself as a curious job seeker, eager to explore the realm of data opportunities. Armed with your burning questions, you simply ask away.
2. Question Transformation: The AI assistant springs into action, converting your questions into a document embedding (using OpenAI API)
3. Semantic Similarity Search: with your question’s document embedding, the AI assistant conducts a semantic similarity search, sifting through a vast pool of job posts to find the N most relevant matches.
4. Prompt Engineering: with the most pertinent job listings in hand, it’s time for some prompt engineering. By summoning the power of advanced generative models like GPT-4 or GPT-3, the solution uses a compelling and insightful prompt.
5. Text Generation: As your questions reach the generative model or LLM (Large Language Model), the solution brings back comprehensive answers,
6. Answer Building: with the text generation at hand, this is further enriched with relevant links and other artifacts.

The following video shows our AI assistant in action:

Summary and Conclusion

In summary, in this blog post, we describe the results of a data analysis of 25,965 job ads for the Belgian market returned for the query “data scientist” from a major online job board in the first four months of 2023.

We found that the most-frequently-returned job ads for the data scientist were not data science jobs! The most frequent job title by far was “software developer”, followed a distant second by “project manager”.

When focusing on data roles specifically, we found that the top three most common job titles were business analyst, devops engineer, and data engineer. The vast majority of the data job ads were posted by recruiting companies and IT consultancies. In 2023 in Belgium, the route to a data job passes much more through external companies than through direct hiring for internal roles (what consultancies typically call the “client-side”). There were many more roles at smaller companies than there were at larger organizations, and data professionals in the Belgian market are expected to be present at the office, either occasionally (through a hybrid work arrangement), or full-time.

Using insights provided by a chat GPT analysis of the job description text, we were able to examine which software tools and skills employers were looking for data roles. As in last year’s analysis, Python and SQL were the most-frequently mentioned tools. In contrast to last year, soft skills such as communication and stakeholder management appeared frequently in our analysis. In our view, this is encouraging, and suggests that employers are recognizing that a mix of technical and people skills are necessary to deliver successful data projects.

A simultaneous clustering of the software tools and data roles revealed how technologies and tasks that were previously the purview of data engineers (e.g. managing cloud infrastructure and deployment), are being split among newer up-and-coming roles (e.g. the cloud part usurped by the cloud engineers and the deployment part usurped by the devops engineers). This example serves as a nice illustration of the fast-pace of simultaneous evolution of technology and data job titles that characterizes the data space in 2023.

Finally, we uncovered a number of types of behaviors from prospective employers that job applicants should be aware of. Firstly, we saw that recruitment companies in particular often post the same job multiple times, likely to surge to the top of returned results on the job platform. We also found that some employers made somewhat gratuitous use of emoji’s in their job ads, presumably to appear “cool” to prospective applicants.

We hope you’ve found this analysis interesting and perhaps even useful. The data job market in Belgium in 2023 is vibrant and large, and our analysis should be encouraging to prospective data workers - there’s lots of opportunities out there - we hope you find a job that you love!

Data Scientists Find Professional Satisfaction. Credit: Jesse Blum via Midjourney

You can find the data and code used for this analysis on Github here.

The Data

The data come from two different sources. The sampling frame is the Billboard year end top 100 song charts for two different genres: country and R&B/hip-hop from the years 1990 until 2021. Note that while the second genre encompasses both R&B and hip-hop, from the period 1990 onward, the bulk of the songs listed lean more towards hip-hop than R&B.

I scraped most of the Billboard data in July 2021 and used the excellent Python package LyricsGenius to extract the song lyric data from the Genius website. Hat tip to Mark MacArdle’s script on Github that made it really straightforward to get these data!¹

In total, the raw dataset contains lyrics for 2754 R&B/Hip-Hop songs and 2444 Country songs that appeared in the Top 100 year-end Billboard song rankings. Some songs are repeated in the raw dataset, because a given song can be popular across multiple years. After removing duplicate songs, we are left with 4620 songs for the current analysis: 2423 R&B/hip-hop and 2197 country songs.

The head of our dataset, called clean_df, looks like this:

The column lyrics_clean contains the song lyrics, from which I’ve removed carriage returns and additional text that is not part of the lyrics (e.g. [Verse 1], etc.). The column lyrics_scrubbed contains the same text as lyrics_clean, but with stopwords removed and all letters set to lower case.

Empath Python Library

The Empath approach to text analysis is an interesting (but at this point somewhat old-fashioned) way of analyzing text. Empath uses a dictionary approach, constructing a priori categories representing different subjects (e.g. positive emotion, health), and determining keywords that represent the different categories (e.g. “joy” is a word in the “positive emotion” category). A given text is evaluated against each of the categories in the dictionary, and a final score per category is determined per text. This final score can simply be the count of the number of words in a category present in the text; a more common approach is to divide the number of matching words per category by the total number of words in the text. This “normalizes” or adjusts for the fact that longer texts will likely have more matches for each category simply because they contain a greater number of words.

The following table, taken from the paper describing the development of the Empath library, gives examples of 8 of the linguistic categories and some of the words for each:

Empath Category Examples

There are many advantages to using the dictionary approach - it is transparent, fast to execute, relatively light on computational resources, and has been validated by human raters. The Empath program, which has 194 linguistic categories, was built using a combination of top-down (e.g. inspiration for categories from ConceptNet) and bottom-up (e.g. dictionary words created by a data-driven approach using a vector space model and word embeddings to find words that are representative of each category) approaches. The Empath dictionaries were further validated by crowdsourced workers on Mechanical Turk, adding a final layer of human evaluation to the dictionary construction process. For a detailed overview, check out this paper which describes the creation and validation of the Empath library.

The downside of the dictionary approach is that it cannot deal with the specific context in which each word is used, meaning that it does not pick up on sarcasm and cannot disambiguate uses of a word that can mean different things depending on the context. For example, the word “hit” can refer to an act of physical violence or a popular song; dictionary-based approaches would not be able to tell the difference between these uses.

(Note that we’ve seen many of these topics previously on the blog: a number of posts have used vector space models, and we’ve also used the dictionary approach to text analysis with TidyText in R.)

Extracting Linguistic Categories With Empath

Import the Library and Test It Out

It’s very straightforward to extract linguistic categories from text with Empath.

We first import the library and initialize a lexicon like so:

from empath import Empath
lexicon = Empath()

The Empath documentation has a nice example test case that gives an intuitive feel for how the coding system works. We simply pass a single sentence to the Empath program, and it returns the coding for all 194 categories. We specify normalize=True in order to get the percentages per document for each category (e.g. number of occurences / total word count):

empath_example = lexicon.analyze("he hit the other person", normalize=True)
empath_example

Which returns a dictionary with the resulting values for each of the 194 categories:

  
{'help': 0.0,
 'office': 0.0,
 'dance': 0.0,
 'money': 0.0,
 'wedding': 0.0,
 'domestic_work': 0.0,
 'sleep': 0.0, 
...
}

Because there are nearly 200 categories, it’s not immediately obvious which categories are picked up by Empath. If we sort the dictionary keys by their values (the Empath scores), we see which categories the Empath program finds in our example text:

# https://stackoverflow.com/questions/613183/how-do-i-sort-a-dictionary-by-value
for w in sorted(empath_example, key=empath_example.get, reverse=True):
    print(w, empath_example[w])

Which returns the categories with scores greater than zero at the top:

  
movement 0.2
violence 0.2
pain 0.2
negative_emotion 0.2
help 0.0
office 0.0
...

According to Empath, the example sentence (“he hit the other person”) contains words that are part of the movement, violence, pain, and negative_emotion categories.

Extract Linguistic Categories from the Song Lyrics Texts

Let’s apply the Empath coding system to our lyrics_clean column in our dataset using a list comprehension:

# apply Empath algorithm to each of the song lyrics texts
empath_analysis = [lexicon.analyze(x, normalize=True) for x in clean_df.lyrics_clean]

This returns a list with 4620 dictionaries of Empath codes, e.g. 1 dictionary per song.

We then turn the dictionaries into a DataFrame, while at the same time multiplying each entry by 100 to have the values expressed in percentages (e.g. .01 becomes 1) - this will make our visualizations easier to read and to understand. Finally, we can look at the resulting dataframe (I’m only showing 5 of the 194 columns for illustrative purposes):

empath_perc = pd.DataFrame(empath_analysis) * 100
empath_perc.iloc[:, 1:6].head()

The first 5 columns and rows of the resulting dataframe, called empath_perc look like this:

Prepare Data for Visualization

Merge Linguistic Categories Into Master Data

We can now merge this dataframe of Empath codes back into our master data, and then aggregate all of the Empath scores by genre in order to make our visualizations.

We first concantenate our original dataframe and the Empath codes like so, checking the shape of the result at the end:

master_empath_df = pd.concat([clean_df, empath_perc], axis = 1)
master_empath_df.shape

The resulting shape is (4620, 199), which is what we should expect - our original 5 columns + the 194 Empath categories results in 199 total columns, while the number of rows has remained the same at 4620.

The head of our resulting dataframe, master_empath_df, looks like this (only first 10 columns shown):

Aggregate the Empath Linguistic Categories by Genre

We now have 1 line per song in our dataframe. We need to summarize the data for each genre in order to make the data visualizations that will help us understand how country and R&B/hip-hop music differ from one another in terms of the most-frequent linguistic categories. Therefore, we will take averages per genre of each of the 194 Empath categories, and use these aggregated data for our plots.

We first identify the names of the columns we will use for our aggregation - which are all the Empath category columns, along with the column indicating genre (which we will use as the grouping variable in our aggregation below):

cols_to_agg = [x for x in master_empath_df.columns if x not in ['song', 'artist', 'lyrics_clean', 'lyrics_scrubbed']]
cols_to_agg[0:10]

The cols_to_agg list contains our “genre” column name, along with all of the Empath categories. The first 10 entries in the list look like this:

  
['genre',
 'help',
 'office',
 'dance',
 'money',
 'wedding',
 'domestic_work',
 'sleep',
 'medical_emergency',
 'cold']

We then take the average of all of the Empath categories per song genre, using a group by command:

agg_df = master_empath_df[cols_to_agg].groupby('genre').mean().reset_index(drop = False)
agg_df.shape

The shape of our aggregated dataframe, called agg_df, is 2 rows and 195 columns. The head of this dataset looks like this (only the first 10 columns shown):

This is exactly what we wanted!

Transform the Aggregated Data From Wide to Long Format

We have one final data transformation step before we can begin to make our visualizations. Specifically, our data is in the wide format, while the seaborn plotting library requires the data to be in the long format. In essence, instead of having one column for each Empath category, we want a longer dataframe with one row per genre/Empath category combination.

We can achieve this using the melt function in Pandas:

long_df = pd.melt(agg_df, id_vars='genre')
long_df.shape

The resulting shape of our dataframe (long_df) is 388 rows by 3 columns, and the head of the dataframe looks like this:

We are now ready to make our data visualizations!

Visualizations

We will make a number of different charts to better understand the linguistic topics that occur in the country and R&B/hip-hop lyrics.

Let’s start by examining the most frequently-occurring Empath topics in each genre.

Top Empath Linguistic Categories for Country Music

In order to make the plots of the top categories for both country and R&B/hip-hop music, I made a function that can be used for this purpose.

This function takes as input the dataframe, the genre to be plotted (country or R&B/hip-hop), the color to use for the plot, the x and y axis labels, and the title. Optional arguments include the directory to save the plot to, and the title of the saved plot.

# function to produce the frequency plots
# function to produce the frequency plots
def plot_top_categories(input_df_f, genre_f, color_f, x_axis_label_f, 
                        y_axis_label_f, title_f, sample_size_f, *args, **kwargs):
    # round the values - so value labels are readable
    input_df_f.value = round(input_df_f.value,2)
    # optional keyword args to save the image to a file - get set up
    plot_out_dir_f = kwargs.get('plot_out_dir_f', None)
    image_file_title_f = kwargs.get('image_file_title_f', None)
    
    # print shapes to make sure subset is working
    print(input_df_f.shape)
    genre_data_f = input_df_f[input_df_f.genre == genre_f]
    print(genre_data_f.shape)
    
    # make a version of dataframe with terms sorted by value
    top_cats = genre_data_f.sort_values(by = 'value',
                                    ascending=False)
    
    # set up the palette - higher numbers have deeper/darker colors
    mypal = sns.light_palette(color_f, n_colors = 15, reverse = True)  
    
    # make the basic barplot object, using above-defined color palette
    ax = sns.barplot(y = 'variable', x="value",
                 data=top_cats[0:15], 
                 palette = mypal)
    # add the value labels for each bar
    ax.bar_label(ax.containers[0])
    # set the x axis label (value needs to be input in the function call)
    ax.set(xlabel=f"""{x_axis_label_f}""")
    # set the y axis label (value needs to be input in the function call)
    ax.set(ylabel=f"""{y_axis_label_f}""")
    # set the font size for the x and y axis labels
    ax.xaxis.get_label().set_fontsize(18)
    ax.yaxis.get_label().set_fontsize(18)
    # set the axis title
    ax.set_title(f"""{title_f}""", fontsize = 25)
    # set the size of the ticks for each axis 
    # (in particular the words on the y axis)
    ax.tick_params(labelsize=15) 
    # add sample size notation in bottom-right side of the plot
    plt.figtext(0.97, -0.01, f'(N = {sample_size_f})', 
            horizontalalignment='right', 
            fontsize = 14) 
    # save out the picture to a file (optional)
    # need to specify plot_out_dir_f and image_file_title_f
    # in function call
    if plot_out_dir_f and image_file_title_f:
        plt.tight_layout()
        fig = ax.get_figure()
        fig.savefig(plot_out_dir_f + f"""{image_file_title_f}""" + '_250.png', 
                    dpi=250, 
                    transparent=False,
                    bbox_inches = "tight") 

We can apply the function to produce the country plot like so:

# apply the function to make the country plot
plot_top_categories(input_df_f = long_df, 
                    genre_f = 'Country',
                    color_f = 'darkred',
                    x_axis_label_f = 'Average % / Category Across All Songs',
                    y_axis_label_f = 'Empath Category',
                    title_f = 'Most Frequent Empath Categories: Country Music',
                    sample_size_f = '2,197')

Which returns the following plot:

The most frequently-occurring topics on average are positive and negative emotion, along with friends and love. Interestingly, optimism is identified as a relatively common category in country music, as are speaking and listening, family, vacation and pain.

Top Empath Linguistic Categories for R&B/Hip-Hop Music

We can apply our function to produce the R&B/hip-hop plot like so:

# apply the function to make the R&B/hip-hop plot
plot_top_categories(input_df_f = long_df, 
                    genre_f = 'R&B/Hip-Hop',
                    color_f = 'darkblue',
                    x_axis_label_f = 'Average % / Category Across All Songs',
                    y_axis_label_f = 'Empath Category',
                    title_f = 'Most Frequent Empath Categories: R&B/Hip-Hop Music',
                    sample_size_f = '2,423')

Which returns the following plot:

The most frequently-occurring topics on average are friends, positive and negative emotion, and love and affection. R&B/hip-hop is definitely a youth-driven musical genre, and there are a number of topics related to this - children and youth, primarily.

Many of the topics overlap with those from the country list, with two exceptions: violence and swearing terms. As we saw in our previous analyses, vulgar language and descriptions of violence are common themes in R&B/hip-hop, particularly when compared to country music.

Largest Differences Between Country and R&B/Hip-Hop Music

The next plot we will make will directly compare each Empath category between the country and R&B/hip-hop genres. We will need to do some additional data preparation in order to make this plot - specifically, we will need to construct a difference score for each Empath category, comparing the percentages for each category between the two musical genres.

Data Preparation

We will start with our long_df object we prepared to make the above two plots.

We can create a difference score for the two genres by grouping the data by variable (which is the Empath category) and then taking a diff between the two values for each category. We then duplicate the difference score for each genre/category combination with bfill.

# create a difference score for rbhh minus country for each theme
long_df['diff_rbhh_country'] =  long_df.groupby(['variable'])['value'].diff().bfill() 
# what does it look like?
long_df.head()

The head of the resulting dataframe looks like this:

I then take the top 15 categories with the largest positive and negative difference scores, and assign them to two separate objects:

# extract the top 15 Empath categories with the largest negative differences (R&B/Hip-Hop - Country)
top_neg_diff = long_df[['variable', 'diff_rbhh_country']].drop_duplicates().sort_values(by = 'diff_rbhh_country')[0:15]
# extract the top 15 Empath categories with the largest positive differences (R&B/Hip-Hop - Country)
top_pos_diff = long_df[['variable', 'diff_rbhh_country']].drop_duplicates().sort_values(by = 'diff_rbhh_country', ascending = False)[-0:15]

I then concatenate the two difference dataframes to make the final data to plot:

# concatenate above dataframes with top positive/negative scores
top_pos_neg_df = pd.concat([top_pos_diff, top_neg_diff], axis = 0).reset_index(drop = True)
top_pos_neg_df.shape

Our final dataframe, called top_pos_neg_df, has 30 rows and 2 columns, and looks like this:

Plotting Between-Genre Differences

We are now ready to make our final plot of the Empath categories with the biggest between-genre differences.

The code to produce this plot uses all of the tricks that we used above, specifying the main and axis titles, font sizes, colors, etc.:

# define the color palette for country 
country_pal = sns.light_palette('darkred', n_colors = 15, reverse = False)  
# define the color palette for R&B/hip-hop  
rbhh_pal = sns.light_palette('darkblue', n_colors = 15, reverse = True)  
# make the basic barplot object, using above-defined color palettes
ax = sns.barplot(y="variable", 
                 x="diff_rbhh_country", 
                 data=top_pos_neg_df.sort_values(by = 'diff_rbhh_country', 
                                                 ascending = False ),
                palette = rbhh_pal + country_pal)
# add the value labels for each bar
ax.bar_label(ax.containers[0], fontsize = 10)
# set the x axis label 
ax.set(xlabel='Difference: R&B/Hip-Hop - Country')  
# set the y axis label 
ax.set(ylabel='Empath Category')
# set the chart title
ax.set_title("Empath Categories With Largest Between-Genre Differences", fontsize = 25)
# set the font size for the x and y axis labels
ax.xaxis.get_label().set_fontsize(18) 
ax.yaxis.get_label().set_fontsize(18)
# set the size of the ticks for each axis 
# (in particular the words on the y axis)
ax.tick_params(labelsize=12)
# set the figure subtitles - give context as to what
# positive & negative scores mean
# https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.figtext.html
plt.figtext(0.90, 0.05, 'Used More in \n R&B/Hip-Hop', 
            horizontalalignment='right', 
            fontstyle = 'italic', 
            fontsize = 11) 
plt.figtext(0.13, 0.05, 'Used More in \n Country', 
            horizontalalignment='left', 
            fontstyle = 'italic', 
            fontsize = 11)

And it returns the following plot:

At the top of the plot are Empath categories that are more frequent in R&B/hip-hop as compared to country. The number one category is swearing_terms, which is no surprise given our previous analyses of these data. Violence is also a category that is more frequent in R&B/hip-hop, as are business, banking and valuable (which are primarily due to mentions of money).

At the bottom of the plot are Empath categories that are more frequent in country music as compared R&B/hip-hop. The number one category difference here is shape_and_size, which is ambiguous - we’ll dive into this difference in more detail below. Interestingly, country songs appear to have more references to night than do R&B/hip-hop. There are a number of categories (driving, car, and vehicle) related to vehicles and driving (as we saw in an earlier post, trucks are a common theme in popular country music). Alcohol and liquid are related to the numerous references to drinking in country (vs. R&B/hip-hop) music, while weather and vacation seem to indicate the “kick back and relax” trope that characterizes some country songs.

Visualizing Empath Category Differences With Scattertext

The plot with the between-genre differences is certainly provocative, but it is not immediately obvious why some Empath categories are more frequent in one genre vs. the other. We will use the Scattertext library (which we used in the first blog post in this series) to visualize these data. The Scattertext visualization has an interactive feature that will allow us to click on an Empath category, and see examples of song lyrics (and the specific words) that are linked to each Empath category.²

We can produce the Scattertext visualization as a standalone html file with the following code:

import scattertext as st
feat_builder = st.FeatsFromOnlyEmpath()
empath_corpus = st.CorpusFromParsedDocuments(clean_df,
                                            category_col='genre',
                                            feats_from_spacy_doc=feat_builder,
                                            parsed_col='lyrics_clean').build()
html = st.produce_scattertext_explorer(empath_corpus,
                                        category='R&B/Hip-Hop',
                                        category_name='R&B/Hip-Hop',
                                        not_category_name='Country',
                                        width_in_pixels=1000,
                                        metadata=clean_df['song'],
                                        use_non_text_features=True,
                                        use_full_doc=True,
                                        topic_model_term_lists=feat_builder.get_top_model_term_lists())
open("rbhh-country-empath.html", 'wb').write(html.encode('utf-8'))

Which produces the following html visualization:

What Do We See?

Using the interactive visualization, it is possible to dive into detail for all of the Empath categories. Clicking on a category in the chart gives examples of songs for each genre that contain the category, and the specific words in the text that are linked to the category are placed in bold. The interactive Scatterplot visualization is a tremendous tool to quickly and easily understand why a pattern revealed by the data analysis exists.

As one concrete example, let’s take the shape_and_size category, which is the top category that appears in country but not R&B/hip-hop music. If we click on this category in the Scatterplot visualization, we get examples of how the term is used in the lyrics texts for both music genres. Country, in particular, seems to have a great many uses of big (e.g. “big ol’ city”, “big plans”), small (the “small town” is a recurrent feature in many country songs), and most especially little (“kiss a little more”, “it’s a little too late”, “the little man”). When looking at the lyrics texts, the use of “little” is very striking - it occurs in many songs, and even in the song titles themselves (e.g. “Make a Little”, “Smoke a Little Smoke”, “A Little More Country Than That”, etc.). It appears that a music genre that has its heart in the wide-open country makes frequent use of the term “little” in its lyrics!

Summary and Conclusion

In this post, we took R&B/hip-hop and country songs from the Billboard Year End 100 song lists from 1990 through 2021 and used the Python Empath library to extract linguistic categories from the song lyrics. Analysis of the top categories per genre revealed more similarities than differences. In particular, positive and negative emotion, as well as friends, love and children are common topics across both music genres.

However, the genres differ from one another in terms of their linguistic categories in important ways. We used two visualization techniques to analyze the differences in Empath topics between country and R&B/hip-hop music. The first was a simple comparison of the topics that differed the most between the genres. R&B/hip-hop lyrics, in comparison to country lyrics, contain more references to vulgar language (e.g. swearing terms), along with references to violence, money (via the categories business, banking, and valuable), and youth and giving. Country music lyrics, in comparison to R&B/hip-hop lyrics, contain more references to shape and size (in larger part due to the surprisingly frequent use of the word “little” in country lyrics), night, references to driving (e.g. driving, car, vehicle; trucks are an evergreen country music subject), references to drinking alcohol (the “drinking song” is a classic country music trope), and references to “kicking back and relaxing” (through the topics weather and vacation).

The second visualization we used in order to understand the differences between the Empath topics between country and R&B/hip-hop music was the Scattertext plot of the Empath topics. This visualization allowed us to examine all of the 194 topics, and to explore the data interactively by choosing a category and viewing the song lyrics and the words linked to the category. This interactive feature is of tremendous use in understanding the results of a quantitative text analysis, and is a critical part of NLP projects where the results must be understood and explained to others. This type of analysis is less in vogue at the moment (the big topic in text analysis is currently LLMs like Chat GPT), but nevertheless has its place in many applied projects where the goal is to make a decision based on the results of a data analysis!

You can find the data and code used for this analysis on Github here.

The Data

The data come from two different sources. The sampling frame is the Billboard year end top 100 song charts for two different genres: country and R&B/hip-hop from the years 1990 until 2021. Note that while the second genre encompasses both R&B and hip-hop, from the period 1990 onward, the bulk of the songs listed lean more towards hip-hop than R&B.

I scraped most of the Billboard data in July 2021 and used the excellent Python package LyricsGenius to extract the song lyric data from the Genius website. Hat tip to Mark MacArdle’s script on Github that made it really straightforward to get these data!¹

In total, the raw dataset contains lyrics for 2754 R&B/Hip-Hop songs and 2444 Country songs that appeared in the Top 100 year-end Billboard song rankings. Some songs are repeated in the raw dataset, because a given song can be popular across multiple years. After removing duplicate songs, we are left with 4620 songs for the current analysis: 2423 R&B/hip-hop and 2197 country songs.

The head of our dataset, called clean_data, looks like this:

The column lyrics_clean contains the song lyrics, from which I’ve removed carriage returns and additional text that is not part of the lyrics (e.g. [Verse 1], etc.). The column lyrics_scrubbed contains the same text as lyrics_clean, but with stopwords removed and all letters set to lower case.

The Tidy Approach to Text Analysis

We have already used the tidy approach to text analysis in a previous post examining the text of Pitchfork music reviews. The basic idea behind the tidytext framework is that we represent our data with 1 line per token (a sub-division of a longer text, typically a single word although here we will use two-word combinations called bigrams), keeping track of important meta-data (e.g. the song title, genre, etc.) in additional columns.

Step 1: Lyrics Texts to Bigrams

In the first step, we will pass our individual song lyrics column in the original data and extract two word combinations called “bigrams”. We will also do two counting exercises in order to select which bigrams to include in our analysis.

The first exercise will be to count how many times each bigram appears in each song. The second will be to count how many times each bigram appears in each genre. The reason that we do this is to identify which bigrams only appear in a single song. Because song lyrics can be more repetitive than other types of written text, we want to exclude bigrams that appear very frequently but only in one song.²

We can accomplish all of this with the dplyr and tidytext packages, and view the head of the resulting dataframe, with the following code:

  
# load packages that we'll need
library(readr)
library(dplyr)
library(tidytext)
library(tidyverse)
library(stringr)

# make a tidy df with the bigrams
song_bigrams <- clean_data %>% select(song, genre, lyrics_clean) %>% 
  # extract bigrams (2-word combinations) from the "lyrics_clean" text column
  unnest_tokens(bigram, lyrics_clean, token = "ngrams", n = 2) %>%
  # count - how many times does each bigram appear in each song? 
  group_by(song, bigram) %>%
  mutate(n_bigram_per_song = n()) %>%
  ungroup() %>%
  # count - how many times does each bigram appear in each genre?
  group_by(genre, bigram) %>%
  mutate(n_bigram_per_genre = n())

head(song_bigrams)

Which returns a dataframe called song_bigrams, with 1,924,106 rows, one for each two-word combination in all of the songs in our corpus. The head of the song_bigrams dataframe looks like this:

We can see from the column n_bigram_per_song how many times each line’s bigram appears in the song, and in the column n_bigram_per_genre, how many times each line’s bigram appears in the song’s genre. When these two columns are equal, it means that the given bigram on a given line appears in a single song in its genre. In the sample of the data shown above, “move slowly” appears once in the song “Nobody’s Home”, and once in the country music genre: meaning that this bigram only appears in a single song. In the analysis that follows, we will exclude rows where this is the case, in order not to consider bigrams that appear frequently, but only in one song.

Step 2: Counting Gender Bigrams Per Genre

Next, we take our song_bigrams dataframe and produce an aggregated dataframe with the counts of each bigram in each genre. Specifically, the following code first removes bigrams that only appear in a single song (e.g. lines where n_bigram_per_song is not equal to n_bigram_per_genre), and aggregates the bigrams to produce a dataset which contains one line per bigram per genre, with the counts of the number of occurrences of each bigram in a column called “total”. Note that we also split the bigrams into two columns, one per word, and only retain those bigrams that begin with she or he.

Our code to produce this aggregated dataset looks like this:

  
bigram_counts <- song_bigrams %>%
  # we *only* want bigrams that appear in more than 1 song
  # we can make this selection by only keeping rows
  # where n_bigram_per_song is not equal to n_bigram_per_genre
  # (if these are equal, it means that the bigram only appears in 1 song
  # in the given genre)
  filter(n_bigram_per_song != n_bigram_per_genre) %>%
  # count the number of occurrences of each bigram in each genre
  count(genre, bigram, sort = TRUE) %>%
  # split up the bigrams into two columns, one per word
  separate(bigram, c("word1", "word2"), sep = " ") %>%
  # only keep bigrams that start with he or she
  filter(word1 %in%  c("he", "she")) %>%
  # removes some repetition - she she
  filter(word1 != word2) %>%
  # rename the bigram count column "total"
  rename(total = n)

head(bigram_counts)

And yields the following dataset, called bigram_counts, with 880 rows:

Step 3: Visualize Common Words Paired with “She” and “He”

The first set of analyses will examine the common words paired with “she” and “he”, separately for each genre. We can take our bigram_counts dataframe, and for each genre, plot the top 10 most frequent words that are paired with “she” and “he” (a huge hat tip to Julia Silge’s post on ordering bar charts per group). The colors used for the gendered descriptions are taken from this very interesting blog post on using color to represent topics related to gender and male/female differences.

The code to produce the country chart is as follows (the R&B/hip-hop code is available in the Github repo):

  
# colors inspired by:
# https://blog.datawrapper.de/gendercolor/
# #8624f5 - for women
# #1fc3aa - for men

hex_pal = c('#8624f5' , '#1fc3aa')

# country
# https://juliasilge.com/blog/reorder-within/
bigram_counts %>%
  # select only country songs
  filter(genre == 'Country') %>%
  # group by first word of bigram
  group_by(word1) %>%
  # and keep only the 10 most-frequently
  # occurring bigrams
  slice_max(total, n = 10) %>%
  ungroup %>%
  # set factor levels so that graph for women appears 
  # on the left
  # and re-order the dataset - by second bigram word,
  # then the total number of occurrences and first bigram word (she/he)
  mutate(word1 = factor(word1, levels = c('she', 'he')),
         word2 = reorder_within(word2, total, word1)) %>%
  # set up our plot with ggplot
  ggplot(aes(word2, total, fill = word1)) +
  # we want a bar chart with no legend
  geom_col(show.legend = FALSE) +
  # separate panels for she & he, panels
  # can have their own scales 
  # (there are more words for women than men)
  facet_wrap(~word1, scales = "free_y") +
  # set the labels for the chart
  labs(x = NULL,
       y = "Number of occurences following 'she' vs. 'he'",
       title = "Most common words paired with 'she' and 'he' in popular country songs",
       subtitle = "Billboard Top 100 Year End Country Songs (1990-2021)",
       caption = "(N = 2,197)") +
  # flip x and y in chart
  coord_flip() +
  # black and white theme (more basic than standard one)
  theme_bw() +
  # set the colors 
  scale_fill_manual(values = hex_pal) +
  # from tidytext - Reorder a column before plotting with faceting, 
  # such that the values are ordered within each facet.
  # (re-order words separately for she/he facet panels)
  scale_x_reordered() +
  # set limits for axis showing the total number of occurrences
  scale_y_continuous(expand = c(0,5))

Common Words Paired with Gender Pronouns in Country Music

The plot for country music looks like this:

The first observation is that there are far more bigrams that begin with “she” than begin with “he.” This is perhaps unsurprising - most of the country songs in the Billboard lists are sung by men, and a common theme (as we saw in the previous post) is love and romantic relationships, which in this traditional music genre, tend to be with women.

The second is that the most frequent words across genders are more-or-less the same. In country music, gender pronouns are followed by the following words: said, was, don’t, ain’t, can’t, says, and had. All in all, country music lyrics concern themselves with what women and men are saying (said, says), and what they were doing (was, had), and what they are not (don’t, ain’t, can’t).

Common Words Paired with Gender Pronouns in R&B/Hip-Hop Music

The plot for R&B/hip-hop music looks like this:

As with country music, there are far more bigrams that begin with “she” than begin with “he”. As with country music, the R&B/hip-hop charts tend to be dominated by male performers, with the narrative focusing on women.

There is also a great deal of overlap among the words following she and he in R&B/hip-hop music (though less so than for country), with the following words common among both genders: got, said, ain’t, was, don’t, and gon. As with country music, in R&B/hip-hop lyrics, the lyrics describe what men and women are saying, doing and what they are not.

Step 4: Visualize Differences Between Words Paired with “She” and “He”

The final analyses will focus on the differences in the bigrams that start with she and he. In other words, how do country and R&B/hip-hop music genres describe women differently from men? This analysis is focused on gender differences within each genre, though we will also comment on differences between the genres.

In order to conduct this analysis, we need to analyze the bigrams in terms of their differences across the genders, which we will do separately per music genre. We use code adapted from here and here to do so. The following code describes the process for country music; for the analagous analysis for Hip-Hop/R&B music, check out the Github repo.

We first filter out any bigrams that appear fewer than 10 times in the country song lyrics. We then create separate columns for each bigram with the counts following “she” and “he”, and then calculate a ratio which is the number of times the bigram appears following each gender pronoun, divided by the total number of bigram occurrences for each gender pronoun. Finally, we take the binary logarithm (log2) of the ratio of the bigram ratios for “she” vs. “he” - this is our “log odds ratio” which we will use as the x-axis in our visualization. The advantage of using the binary logarithm (e.g. log2) instead of the natural logarithm (e.g. log) is that an increase of 1 on the log2 scale corresponds to a doubling of the values on the original scale, making for an intuitive scaling in the x-axis of our graph.

The following code performs these operations and makes the plots, using the colors we defined above for women and men:

  
# country
word_ratios_country <- bigram_counts %>% 
  # select only country music songs
  filter(genre == 'Country') %>% 
  # group by music genre and second bigram word
  group_by(genre, word2) %>%
  # filter any bigrams that appear fewer than 10 times
  filter(sum(total) > 10) %>% 
  ungroup() %>%
  # long to wide transformation, making separate columns
  # for she and he, with the total number of occurrences per gender
  # as data points. Fill empty cells with zero
  spread(word1, total, fill = 0) %>% 
  # https://www.tidytextmining.com/twitter.html#comparing-word-usage
  # ratio, for each bigram: total uses of bigram (+1),
  # divided by the sum of all of the bigram uses (+1)
  mutate_if(is.numeric, list(~(. + 1) / (sum(.) + 1))) %>%
  # divide she ratio by he ratio, and take the log2 of the result
  # we use log2 because the interpretability / logic is better
  # with log2 vs. log (increase of 1 indicates doubling of original metric)
  mutate(logratio = log2(she / he)) %>%
  # sort the dataframe by the log ratio variable
  arrange(desc(logratio)) 

Differences Between How Women and Men Are Described in Country Music

The plot for country music looks like this:

There are clearly differences between how women and men are described in country music. Here are some of the things that jump out to me in the above plot.

Women

• There is a group of words that describe women in terms of desires, needs and pleasures (needs, needed, wants, likes). Women, more so than men, are described in terms of their preferences - not by what they do, but by what they need, want, or enjoy. Examples include - she needs him, what she wants is love, and she likes to feel the sand beneath her feet.

• Giving is an important theme for women in country music. Women are described as giving many different things (She gives me hot and cold fever, she gives me a kiss, she gives you the green light, she gives her love, she gives me hope); the receivers are typically men. By describing them as givers, the country lyrics attribute some agency to women, although one could argue that someone who gives something to someone else is in a comparatively subservient position to the recipient.

• Falling (fell) is another important word that occurs more frequently following “she” than “he.” In country music, this refers (perhaps unsurprisingly) to falling in love (she fell this time and broke her heart in two, she fell in love). As we saw in the previous post, in both country and R&B/hip-hop, love and romance is a prominent theme. In the country music lyrics, it’s more often the women who fall in love (with men, exclusively, in these popular songs).

• There are two variations on coming (comes & came) that are more common following “she” vs. “he” in country lyrics. Upon closer examination of the lyrics, it appears that these word choices often describe male reactions to women who show up at any place where the male narrator happens to be. Examples include she comes in here, here she comes, she came in looking good, and she came across so cool. This makes sense, given that most country songs are written from the perspective of a male narrator, and that women are therefore described from a masculine viewpoint.

• Finally, one interesting term that comes up more following “she” vs. “he” is calls. This term is primarily used to describe talking on the telephone (she calls him), or referring to the male narrator using a term of endearment (e.g. she calls me honey / buttercup / baby, etc.)

Men

• The number one term that is more frequent in country lyrics following “he” vs. “she” is stood. On the one hand, the lyrics sometimes evoke the “strong silent type” who stands in the middle of the action - what’s important is his presence and not necessarily his actions or words (e.g. He stood and he stared a while when their eyes met or He stood there smilin’, holdin’ on). On the other hand, the term is sometimes used to represent the male character’s values or beliefs, as in He stood for Uncle Sam. All in all, the use of the word stood evokes a man who stands steady in his boots, regardless of the situation, and who stands up for his beliefs and values.

• Interestingly, thought is more common in country lyrics following “he” vs. “she.” Examples include he thought that she’d surely run away, happiness is what he thought he’d find and he thought she’d be sitting home crying. To me, this suggests that the inner life of men is more taken into consideration, which is understandable if most of the country songs are told from the perspective of a male narrator. But the comparison to the patterns for women we saw above is striking; in country music, women want, need, and like, but men think.

• There are two different words in the chart that refer to seeing - (saw & looked). Men, more so than women, are described as observers in the situations in which they find themselves. For example, he saw the hit, the run, the slide, there ain’t no bigger fan, well he looked me up and he looked me down, and he looked at me with knowing eyes. Seeing and looking are particular types of action that fit in with the larger pattern of male agency in popular country music lyrics.

• Finally, there are two words that describe men (vs. women) in absolute terms: always and never. Lyrically, it is perhaps easier (arguably too easy) to use these terms to describe a character in a song. Examples of always include he always sang a cowboy’s song, he always drinks for free, he always had a plan, while examples of never include he never said a word, he never had a lot of luck with the ladies, and he never polished his boots. By describing men in these absolute terms, country music lyrics paint male characters with certainty as solid, inflexible, with little room for ambiguity or nuance.

Differences Between How Women and Men Are Described in R&B/Hip-Hop Music

The plot for R&B/hip-hop music looks like this:

It is clear that there are also differences between how men and women are described in R&B/hip-hop music. These are some of the things that I find striking:

Women

• The number one term that distinguishes women vs. men is my (also note that mine appears in the list). In the R&B/hip-hop song lyrics, there are many different examples of what the woman is in regards to the (mostly male) narrators: she my girlfriend, she my fiance, she my better half, along with she my lil’ boo and even she my therapist. Nearly all of these mentions are meant as compliments, but what they all have in common is that the woman is defined in terms of her relationship to the male lyricist.

• The second term that distinguishes women vs. men is twerkin. For the uninitiated, twerking is an act “performed chiefly but not exclusively by women, [in which] performers dance to popular music in a sexually provocative manner involving throwing or thrusting their hips back or shaking their buttocks, often in a low squatting stance.”. In a way, it makes sense that twerkin appears more following she than he, as it is a dance move performed mostly by women, and because as we saw in the previous post, a big theme in hip-hop/R&B music is “the club.” Nevertheless, it is an inherently sexual description that portrays women as objects of desire.

• The third term in R&B/hip-hop lyrics that occurs more frequently following “she” vs. “he” is bad. Some examples include she bad to the bone, she bad as it get, she bad as hell, and she bad as controversy. These lyrics suggest that bad is not necessarily meant in a negative way. It seems to be related to the notion of the bad bitch, who according to Urban Dictionary, is a “badass, solid chick with self respect.” In other words, the R&B/hip-hop lyrics describe women (not men) as bad as a compliment (although using a word that is, in most other cases, used to describe something negative).

• Finally, there is a cluster of words that are more common after “she” vs. “he” that are related to physical appearance: lookin, working, and rockin. In the lyrics texts, lookin often refers to the woman’s appearance (e.g. she lookin’ decent), working often refers to physical attributes (e.g. she working that back, she working them jeans, etc.), and rockin often refers to physical appearance or clothing (e.g. she rockin that thang, she rockin Sean John [the clothing brand], etc.).

Men

• Makes is the number one term that is more prevalent following “he” vs. “she”. There are many different uses of “he makes” in the song lyrics (e.g. he makes his money, he makes a mess, he makes love etc.). What they all seem to have in common is action and agency - men, in contrast to women, are doing things in the hip-hop lyrics, imposing their way upon the world.

• Should is the second term that differs the most between “he” and “she.” The use of should is interesting because it describes the way things ought to be, not the way the they actually are. Examining the R&B/hip-hop lyrics, should is often used when depicting bad or messed-up situations, and suggesting how things might be better if only the male character acted differently (he ain’t gonna love you the way he should, a man would treat a woman like he knew he should, he probably think he could… but I don’t think he should, I’m colder than your man he should be your ex now). This analysis suggests that the R&B/hip-hop lyrics are more likely to take a position on how men (vs. women) ought to act (vs. how they are currently behaving).

• A final theme that emerges from this analysis is that negations (e.g. wasn’t, wouldn’t, can’t, doesn’t) are more commonly used following “he” than “she” in R&B/hip-hop lyrics. These words describe the way things are not, in contrast to the way things are. Examples include he wasn’t frontin’, he wasn’t man enough, he wouldn’t believe us, he can’t seem to keep hisself out of trouble, for his life he can’t tell the truth, he doesn’t know he’s to blame, and [he] acts like he doesn’t even care. This use of negations describes male (vs. female) song characters in opposition to how they actually are - in terms of what he wasn’t, doesn’t, wouldn’t or can’t do or be.

Summary and Conclusion

In this post, we took R&B/hip-hop and country songs from the Billboard Year End 100 song lists from 1990 through 2021 and used the tidytext analytic framework to examine how both music genres describe women and men. Specifically, we looked at two-word combinations called bigrams and examined the words that were most likely to be paired with “she” and “he”.

The first analyses examined the most common words and found that, in both country and R&B/hip-hop, there were more words that followed “she” as opposed to “he.” In other words, the lyrics for both genres were more likely to contain descriptions of women vs. men. This is likely due to the overwhelming gender imbalance in both genres among the performers represented on the music charts; because men are the singers/rappers in these mainstream genres (where LGBTQ+ themes are not common), and because (as we saw in the previous post) love and romance is a frequent subject in both genres, the lyrics more often focus on descriptions of women. Nevertheless, in both country and R&B/hip-hop lyrics, the top words following he/she were largely the same within each genre. Country lyrics are focused on what women and men are saying (said, says), what they were doing (was, had), and what they are not (don’t, ain’t, can’t) while R&B/hip-hop lyrics are focused on what men and women have (got), say (said), are (was), and what they will do in the future (gon).

We next examined, within each genre, the differences between words that followed “she” vs. “he.” In country music, women (vs. men) were more likely to be described in terms of what they desired, needed or liked (often men or love), as givers (of kisses, love, affection, etc.), as falling in love (of course, with a man), and as calling (either using the telephone or calling a male narrator by a nickname such as honey, buttercup, etc.). In country music, men (vs. women) were more likely to be described as standing in the middle of the action (e.g. the strong silent type) or standing up for their values or beliefs, as thinkers whose inner life is worthy of consideration in the song lyrics, as observers who saw and looked at the world around them, and as solid and inflexible individuals who who always or never engaged in certain behaviors.

In R&B/hip-hop music, women (vs. men) were more likely to be described in terms of their relationship to the male narrator (she my girlfriend, she my lil’ boo), as engaging in sexually provocative danse moves (twerkin), as being full of confidence and self-respect (bad), and in terms of their physical appearance (she lookin’ decent, she working them jeans, she rockin that thang). In R&B/hip-hop music, men (vs. women) were more likely to be described as agentic, doing and making things happen, as not behaving properly (with the lyrics taking a position refering to what characters should be doing), and with many negations (wasn’t, wouldn’t, can’t, doesn’t) which serve to define men in opposition to how they actually are (e.g. he wasn’t man enough).

A Mea Culpa From a Music Fan

In sum, these analyses show that, despite the many cultural and musical differences between the two genres, both country and R&B/hip-hop music engage in stereotypical portrayals of men and women (men as strong, agentic “doers” and women as givers, accessories to the males, and sexualized objects of desire).

I’m of course not the first to point out gender imbalances in country music, or the stereotypical portrayal of women in country music (there’s even an entire country song devoted to this). Furthermore, much ink has been spilled in discussing sexist portrayals of women in hip-hop/R&B music (there’s even a whole wikipedia article devoted to this subject). My observations in this blog post are data-driven but they are hardly original.

Despite the gender imbalances and the problematic ways in which men and women are described, I nevertheless remain a fan of both music genres. If you can ignore the lyrics and focus on the music (which I understand not everyone can or will do), both country and hip-hop/R&B music have many unique and compelling qualities. Hip-hop is a celebration of rhythm and rhyme. In the best R&B/hip-hop music, these elements come together in tight, lyrical, energetic package that I personally find very compelling. Country music can be deceptively simple - a single idea told in 3 versus, a couple of choruses and perhaps a bridge. Harmonically, it’s rarely surprising or adventurous (in comparison to jazz or prog rock, for example). However, there is much to appreciate in the musicianship on the records - the bands are composed of terrific musicians playing real instruments, often live, with impeccable skill, feeling, and musicality. The singers are truly excellent vocalists and the production is top-notch. Despite not agreeing with the stereotypical gender roles espoused by certain lyrics, I am still able to appreciate much about both of these musical genres.

Coming Up Next

In our next post, we will return to this dataset once again and use a different approach - dictionary-based thematic coding - to examine themes that country and R&B/hip-hop music lyrics deal with.

Stay tuned!

You can find the data and code used for this analysis on Github here.

The Data

The data come from two different sources. The sampling frame is the Billboard year end top 100 song charts for two different genres: country and R&B/hip-hop from the years 1990 until 2021. Note that while the second genre encompasses both R&B and hip-hop, from the period 1990 onward, the bulk of the songs listed lean more towards hip-hop than R&B.

I scraped most of the Billboard data in July 2021 and used the excellent Python package LyricsGenius to extract the song lyric data from the Genius website. Hat tip to Mark MacArdle’s script on Github that made it really straightforward to get these data!¹

In total, the raw dataset contains lyrics for 2754 R&B/Hip-Hop songs and 2444 Country songs that appeared in the Top 100 year-end Billboard song rankings. Some songs are repeated in the raw dataset, because a given song can be popular across multiple years. After removing duplicate songs, we are left with 4620 songs for the current analysis: 2423 R&B/hip-hop and 2197 country songs.

The head of our dataset, called clean_df, looks like this:

The column lyrics_clean contains the song lyrics, from which I’ve removed carriage returns and additional text that is not part of the lyrics (e.g. [Verse 1], etc.). The column lyrics_scrubbed contains the same text as lyrics_clean, but with stopwords removed and all letters set to lower case.

Visualization with Scattertext

Scattertext is a Python text analysis library that produces visualizations displaying distinguishing terms between different categories of text. Scattertext makes use of a spaCy pipeline to process the text data for plotting. Therefore, we will need first to load a spaCy language model (the first time you run this code, you’ll also need to download a model to your computer). Because we are not using word vectors or any other model output for our Scattertext analysis below, I’ve chosen to use the small English language model here. For more information about spaCy models see here.

Import Libraries and Load spaCy Model

We can import spaCy, load the language model, and import all the Scattertext functions like so:

import spacy
# load the spaCy English language model
# we use the small one here
# for more info see:
# https://spacy.io/models/en
nlp = spacy.load('en_core_web_sm')

# import scattertext
import scattertext as st

Create Scattertext Corpus Object

We next create a Scattertext corpus object directly from our Pandas dataframe using the spaCy pipeline. We need to define the category column, in our case genre, which we’ll use to define the X and Y axes of our scatterplot. We also need to define which column contains the text we would like to analyze - here I select the lyrics_scrubbed column. The spaCy word tokenizers split off tokens like “’s”, which makes sense for some applications, but in our case clutters up the graph with commonly appearing tokens which provide no additional insights. The lyrics_scrubbed column, in contrast, has been cleaned to remove these awkward tokens and other stopwords, and so produces a much cleaner, more interpretable graph.

# lyrics scrubbed
corpus_scrubbed = st.CorpusFromPandas(clean_df,
                             category_col='genre',
                             text_col='lyrics_scrubbed',
                             nlp=nlp).build()

Produce the Plot

In order to produce the Scattertext plot, we simply pass our corpus object to the text plotting function. We must specify the focal category and its name (this category will appear on the y-axis of our plot), as well as the name we would like to give our “other” category (which will appear on the x-axis). I set a minimum term frequency to 50 (otherwise the plot is too full of infrequently-occurring words), set the size of the plot to be 1000 pixels, and specify the song (our “documents”) as the meta-data.

html = st.produce_scattertext_explorer(
    corpus_scrubbed,
    category='R&B/Hip-Hop', category_name='R&B/Hip-Hop', not_category_name='Country',
    minimum_term_frequency=50, pmi_threshold_coefficient=4,
    width_in_pixels=1000, metadata=corpus_scrubbed.get_df()['song'],
    transform=st.Scalers.dense_rank
)
open('lyrics_scrubbed_scatterplot.html', 'w').write(html)

This code produces a standalone html file with our visualization, which I’m embedding below:

What Do We See?

The plot displays each word as a point. The position on the x and y axes is determined by their frequency percentiles for each category. For example, a term at the middle of the x-axis will be mentioned in country songs at the median frequency.

The color of the words is determined by the scaled F score, a value that aggregates both A) the frequency of a word in a given category (e.g. how many times a word appears in a given category, divided by the total word count of that category) and B) the precision of the word (e.g. the number of times a word appears in a given category, divided by the the number of times the word appears across all categories in the entire corpus of documents). Words with higher frequency and precision have higher scaled F scores, which range between -1 and 1. In the plot, words with a higher scaled F score for hip-hop are colored in a darker shade of blue, while words with a higher scaled F score for country are colored in a darker shade of red.²

The top left corner contains words that are characteristic of R&B/hip-hop but not for country. The bottom right corner shows words that are characteristic of country but not R&B/hip-hop. The top right corner displays words that are common in both musical genres, while the bottom left corner displays words that are infrequent in both genres.

The bottom of the plot contains a legend that gives some information about the corpus. As noted above, we have 2423 hip-hop songs and 2197 country songs. The plot also displays the word count for each genre. While the number of songs is more-or-less balanced between the genres, the R&B/hip-hop songs collectively contain more than twice as many words as the country songs! While both genres often share a common song structure of three verses and a chorus (often called the “hook” in hip-hop), hip-hop songs contain more words than the comparatively “lyrically tighter” country songs. (My general sense is that, purely in terms of the vocal performance, rapping allows one to get out more words per measure than singing).

Finally, note that you can hover over each point to get additional information for each word (the scaled F score for the dominant genre and its frequency for both genres per 25000 words). Clicking on a word in the plot opens up a panel underneath which shows selected examples of the word used in the original documents (though because the text was cleaned before passing it to the Scattertext algorithm, the resulting output is difficult to interpret).

Words Unique to R&B/Hip-Hop

The words that are most typical of R&B/hip-hop (vs. country) are displayed in a list to the right of the plot (under the title “Top R&B/Hip-Hop”). We can get a broader sense of these typical words by looking at the upper-left quadrant of the plot. The words that are most characteristic of R&B/hip-hop (regardless of whether or not they also occur in country music) are displayed in a list at the right of the plot under the title “Characteristic”.

Many but not all of these words are vulgarities that I will display in the plot but not repeat here. As we’ve seen elsewhere on this blog, this type of language is very characteristic of rap & hip-hop. The most typical non-vulgar words in the upper-left quadrant include shawty, woo and game (a word often used to describe the rap music industry).

Words Unique to Country

The words that are most typical of country (vs. R&B/hip-hop) are displayed in a list to the right of the plot. We can get a broader sense of the typical words for country music by looking at the lower right quadrant of the plot. It appears that there are comparatively fewer unique words for country music. Not surprisingly whiskey, beer, country, and truck are comparatively more frequent in country music.

Words Common to Both Genres

Despite the evident differences in language use between country and R&B/hip-hop music, it is interesting to see that some words are common in both genres. Among the common terms, we find mentions of love, boy meeting girl, leaving and staying; in sum, many different matters of the heart. Some topics are truly universal!

Infrequently Used Words in Both Genres

There are many words that are rarely used across both genres, clustered together at the bottom left hand corner of the plot. Some of those that lean towards the country side are redneck and dirt road, while some that lean towards the R&B/hip-hop side include gangsta and frontin. These words are used more often in one genre vs. the other, but appear infrequently in the song lyrics corpus.

Summary and Conclusion

In this post, we took R&B/hip-hop and country songs from the Billboard Year End 100 song lists from 1990 through 2021 and used the Python Scattertext library to visualize differences in word use between the genres.

Many but not all of the most unique words in the R&B/hip-hop genre were curse words; use of this type of language is rather characteristic of songs in this genre. Other notable R&B/hip-hop words included club (an important theme in many songs, and also a place where hip-hop music is played), shawty (a term of endearment for one’s female companion), and game (hip-hop shorthand for the music business).

In contrast, there were comparatively fewer unique words in the country music lyrics. Among the top unique terms were country, whiskey, beer, and truck, which fits nicely with the stereotypical conception of what country music is about. Interestingly, both country and R&B/hip-hop music are rather “meta” in that they reference their own music genre and the surrounding industry quite often in their lyrics (see the references in the hip-hop lyrics to the “game”).

Despite their many differences in terms of language use, this analysis revealed certain thematic similarities between R&B/hip-hop and country music. In particular, words such as love, boy and girl, references to hearts and baby suggest love and relationships as central topics in both genres. Country music might describe goings on on a dirt road in a small town in the country, and R&B/hip-hop music might describe what happens in the club (shawties bouncing and shaking?), but songs from both genres clearly draw on timeless subjects like physical attraction, love and the joys and pains of romantic relationships.

Coming Up Next

In our next post, we will return to this dataset and examine the lyrical content in greater detail. In particular, we will look at gender roles in country and R&B/hip-hop lyrics, in order to understand how the different genres portray men and women.

Stay tuned!

In the previous post, the intrepid Jesse Blum and I analyzed metadata from over 6,500 job descriptions for data roles in seven European countries. In this post, we’ll apply text analysis to those job postings to better understand the technologies and skills that employers are looking for in data scientists, data engineers, data analysts, and machine learning engineers.

In this post we present results from text analyses that show that:

• Data analysts are expected to have skills in reporting, dashboarding, data analysis, and office suite software.
• Data scientists are expected to know more about data science, statistics, mathematics, and making predictions.
• Data engineers are expected to industrialize organizrations’ cloud and data architecture and infrastructure.
• Machine learning engineers are expected to use artificial intelligence and deep learning frameworks such as TensorFlow and Pytorch.

The results of this analysis complement and extend the results we presented last time, showing that employers have distinct visions of the (mostly technical & software-related) skillsets that data analysts, data scientists, data engineers, and machine learning engineers should possess.

The Data

The data come from a web scraping program developed by Jesse and myself. Every 2 weeks, we scraped job advertisements from a major job portal website, extracting all jobs posted within the previous 2-week period for the following job titles: Data Engineer, Data Analyst, Data Scientist and Machine Learning Engineer for the following countries: the United Kingdom, Ireland, Germany, France, the Netherlands, Belgium and Luxembourg.

We started data collection mid-August and finished by the end of December, 2021, ending up with 6,590 job descriptions scraped. All the data and code used for this analysis are available on Github. Feedback welcome!

Results

Our dataset includes job descriptions for data roles across four languages (English, French, Dutch and German). We wanted to see if there were any differences in word usage among the different roles (data scientist, data engineer, machine learning engineer and data analyst), and therefore conducted language-specific analyses to contrast and compare the roles according to the words used to describe the job openings.

Word Clouds

Our first set of analyses uses a great R function to create comparison clouds. This type of analysis allows us to compare the frequency of words across groups of documents, and highlight words that appear more in a given group versus the others.

Jesse and I are more comfortable in English, French, and Dutch than German, so we limited our analysis to those three languages. However, there were far fewer Dutch job descriptions than for the other two, so the resulting Dutch comparison cloud was not particularly informative. Below, we focus on the English and French wordclouds and what they reveal about employers’ expectations for the different roles.

The French word cloud looks like this:

The English word cloud looks like this:

Overall, we found that there were clear differences between the roles in the language used in the job advertisements. Furthermore, these differences were largely consistent across the English and French language job ads.

The following table summarizes the comparison:

Some other observations that we found noteworthy:

• There are strikingly few terms that are unique to the data scientist role, suggesting large overlaps with the other profiles. As recently as a couple of years ago, the roles of data engineer and machine learning engineer were much less prevalent and many of the responsibilities currently assigned to these roles fell under the purview of data scientists. With the growth of other data roles and a resulting divvying up of data work, it seems as though organizations are not entirely clear as to what exactly the unique characteristics of data scientists are.

• While the conclusions from the wordclouds were virtually identical across languages, there were some notable differences among the different roles between English and French. For example, the French machine learning engineer ads were more likely to include innovation than the English ones, perhaps suggesting that this work is taking place in R&D or innovation centers of larger companies. The French job descriptions for data engineers were more likely to mention agile methodology, and the French job descriptions for data analysts were more likely to mention SQL (in English, this technology was more prevalent for the data engineer job ads).

• Finally, it was interesting to note that many of the terms used in French job descriptions are actually English words. For example, cloud, reporting, and deep learning could all be translated into French, but they’re usually left in English. Other jargon surrounding data professions, however, has well-established French equivalents. For instance, tableau de bord is the French equivalent of dashboard, intelligence artificielle is the French equivalent of artificial intelligence, and apprentissage automatique is the French equivalent of machine learning. So if you’re trying to understand the tech industry in France, it’s perhaps worth brushing up on your English vocabulary!

Using Skills-ML to Extract Skills from Job Ads

The Skills ML library is a great tool for extracting high-level skills from job descriptions. The Skills ML library uses a dictionary-based word search approach to scan through text and identify skills from the ONET skill ontology, allowing for the extraction of important high-level skills mapped by labor market experts. This approach is more comprehensive than simply counting words (as we did with the comparison clouds above), and it takes into account the fact that some words are synonyms or represent the same skill or technology (e.g.”database”, “data warehouse”, “data lake”, etc. can be grouped under a higher-level term such as “data storage”). Because the ONET skills are only available in English, this analysis was conducted only on the English-language job descriptions.

Most Common Skills

As the following figure shows, Python was the most common skill represented in the English-language job descriptions. Other top skills include R, programming, mathematics, Tableau, visualization, writing, Git, and physics. However, this analysis collapses all the skills across the four data roles. We saw in the wordcloud analysis above and in the previous analysis of job keywords that the desired skillsets can look quite different between the different data profiles.

Clustering Skills and Roles

In order to get a sense of how the extracted skills differed across the data roles, we made a cluster map using the Python Seaborn library. Specifically, we calculated the percentage of job ads per role that contained each skill, filtering on skills that appeared in more than 50 job ads. These percentages were converted to z-scores, such that higher numbers indicate that a given skill is mentioned more often for a given role compared to the others. This final matrix was then passed to the cluster map algorithm, which performs a simultaneous clustering of both the job roles and of the extracted skills.

The results of this analysis showed that there are clear clusters of skillsets required for different types of data-related roles.

In the clustering diagram, shades of red indicate a higher prevalence of a given skill for a given role compared to the others, while shades of blue indicate a lower prevalence of a given skill for a given role compared to the others.

Along the horizontal axis, individual skills are clustered together in logical ways. For instance, at the right side of the chart, Microsoft Office is grouped together with Microsoft Excel and Google Analytics.

On the vertical axis, roles cluster into three separate groups according to their required skills:

• Data analysts are in their own cluster at the top of the graph, with skills that are most different from the other roles. In particular, job ads for data analysts are more likely to mention office-suite software (e.g. Microsoft Office & Excel, Google Analytics), data visualization / dashboarding tools (e.g. Tableau), and sales management & tracking tools (e.g. Salesforce). Job ads for data analysts are less likely to mention programming tools such as Git, Python, programming languages, etc.
• Data engineers are grouped in an overall cluster with data scientists and machine learning engineers, but have a separate branch from the other two. The job ads for data engineers were comparatively more likely to mention data tools (Oracle, noSQL, MySQL, MongoDB, PostgreSQL, and Apache Spark). This suggests that data engineers are expected to play greater roles in the development and maintenance of an organization’s data infrastructure, compared to the other three roles.
• Data scientists and machine learning engineers are placed together in the same cluster at the bottom of the graph. These two roles overlap in terms of computer science skills like programming, Python, and Git, and in domain knowledge in scientific fields such as biology and physics. Furthermore, job ads for these two roles are also less likely to require office suite software capabilities (e.g. Excel), visualization (e.g. Tableau), and business tools (e.g. Salesforce) that are most characteristic of data analysts. However, there are some differences between data scientists and machine learning engineers. Data scientists’ job ads have higher prevalence of mathematics, chemistry and R, while job ads for machine learning engineers have higher prevalence of operating systems (e.g. Unix and Linux), and programming languages (e.g. Javascript and C).

The Added Value of Analyzing Job Description Texts

Overall, the above analysis serves as a useful extension of the Metadata analysis we described in our previous post. Here, we first presented comparison clouds showing the relative frequency of words that were unique to a given role compared to the others. We made separate word clouds for the texts of the English and French job ads, respectively, and found that the main conclusions from these visualizations were the same. Interesting findings from this analysis included:

• Data analysts are expected to work with dashboarding, data analysis and Office tools like Excel. Of all of the profiles, job descriptions for data analysts were more likely to mention contact with the business, interacting with stakeholders and generating and communicating insights.

• Data scientists, in contrast, had relatively few unique words in their job descriptions. Compared to the other roles, they are expected to know about statistics, mathematics and making predictions from models. Our sense was that, given the recent growth of other data roles such as data engineers and machine learning engineers, there is some degree of ambiguity regarding the distinct characteristics that data scientists should have compared to the other roles.

• The job ads for data engineers had a long list of data storage and transfer technologies that were unique to this role. Data engineers are expected to master many different types of databases and cloud platforms in order to move data around and store it in a proper way.

• Finally, job ads for machine learning engineers were more likely to contain mentions of artificial intelligence and deep learning frameworks like Pytorch and TensorFlow, applied to domains such as computer vision.

We also extracted skills from the English language job descriptions using the ONET skill classification. As in our previous analysis of skill keywords, Python was the most frequently-appearing skill. We then made a clustermap to see how the extracted skills differed across the roles.

In this analysis, the data analysts role had least in common with the others. Data analysts in particular were more likely to use office tools (Excel, Google Analytics), visualization tools (e.g. Tableau) and business software (e.g. Salesforce), and less likely to use programming tools and languages (e.g. Git and Python).

Data Engineers also had their own specialties, being particularly likely to work with a wider variety of data storage, big data, and query technologies (e.g. many flavors of SQL, Apache Spark etc.) This analysis shows that data analysts and data engineers have very different skillsets, with data analysts being more focused on office and business software, and data engineers being more focused on programming and databases. This highlights the importance of having both roles on a team in order to have a well-rounded skillset, and the unlikeliness of having one person being equally good at both skillsets (the long-sought after but rarely-found “unicorn” profile).

The End

This is the final post that we’ll make of the analysis of these job description data. All of the data and code for these analyses are available on Github, and we encourage you to explore them further!

This exercise was very meta for us, challenging ourselves across data analysis, data science, data engineering. Both the metadata analysis presented previously and the current text analysis helped us clarify our thinking about the market for data profiles in Europe, and we hope to have expanded your understanding of the data professions and the skills that unite and differentiate them. The job market is evolving quickly, as are the technologies and tools that data professionals are being asked to master. Our analysis of European job descriptions offers a snapshot of the current job market, and we are excited to see what the future brings as European companies’ and institutions’ data efforts mature and as the market continues to evolve!